e-Informatica Software Engineering Journal, Volume 10, Issue 1, 2016, pages: 89–123, DOI 10.5277/e-Inf160105

Software Startups – A Research Agenda

Michael Unterkalmsteinera, Pekka Abrahamssonb, XiaoFeng Wangc, Anh Nguyen-Duca,Syed Shahd, Sohaib Shahid Bajwac, Guido H. Baltese, Kieran Conboyf , Eoin Cullinaf ,

Denis Dennehyf , Henry Edisonc, Carlos Fernandez-Sanchezg, Juan Garbajosag,Tony Gorscheka, Eriks Klotinsa, Laura Hokkanenh, Fabio Koni, Ilaria Lunesuj,

Michele Marchesij, Lorraine Morgank, Markku Oivol, Christoph Seligk, Pertti Seppänenl,Roger Sweetmanf , Pasi Tyrväinenm, Christina Ungererk, Agustin Yagüeg

aBlekinge Institute of Technology, Sweden, bNorwegian University of Science and Technology, Norway,cFree University of Bolzano-Bozen, Italy, dSICS, Sweden, eLake Constance University, Germany,

fNational University of Ireland Galway, Ireland, gTechnical University of Madrid, Spain,hTampere University of Technology, Finland, iUniversity of São Paulo, Brazil, jUniversity of Cagliari, Italy,

kNational University of Ireland Maynooth, Ireland, lUniversity of Oulu, Finland,mHochschule Konstanz, Germany, nUniversity of Jyväskylä, Finland

mun@bth.se, pekkaa@ntnu.no, xiaofeng.wang@unibz.it, anhn@idi.ntnu.no, shah@sics.se,bajwa@inf.unibz.it, guido.baltes@cetim.org, kieran.conboy@nuigalway.ie,

eoin.cullina@outlook.com, denis.dennehy@nuigalway.ie, henry.edison@inf.unibz.it,carlos.fernandez@upm.es, jgs@eui.upm.es, tgo@bth.se, ekx@bth.se, laura.hokkanen@tut.fi,

fabio.kon@ime.usp.br, ilaria.lunesu@diee.unica.it, michele@diee.unica.it,lorraine.morgan@nuim.ie, markku.oivo@oulu.fi, cselig@htwg-konstanz.de,

pertti.seppanen@oulu.fi, roger.sweetman@nuigalway.ie, pasi.tyrvainen@jyu.fi,christina.ungerer@htwg-konstanz.de, ayague@etsisi.upm.es

AbstractSoftware startup companies develop innovative, software-intensive products within limited timeframes and with few resources, searching for sustainable and scalable business models. Softwarestartups are quite distinct from traditional mature software companies, but also from micro-,small-, and medium-sized enterprises, introducing new challenges relevant for software engineeringresearch. This paper’s research agenda focuses on software engineering in startups, identifying,in particular, 70+ research questions in the areas of supporting startup engineering activities,startup evolution models and patterns, ecosystems and innovation hubs, human aspects in softwarestartups, applying startup concepts in non-startup environments, and methodologies and theoriesfor startup research. We connect and motivate this research agenda with past studies in softwarestartup research, while pointing out possible future directions. While all authors of this researchagenda have their main background in Software Engineering or Computer Science, their interestin software startups broadens the perspective to the challenges, but also to the opportunities thatemerge from multi-disciplinary research. Our audience is therefore primarily software engineeringresearchers, even though we aim at stimulating collaborations and research that crosses disciplinaryboundaries. We believe that with this research agenda we cover a wide spectrum of the softwarestartup industry current needs.

Keywords: software startup, research agenda, software-intensive systems

90 Michael Unterkalmsteiner et al.

1. Introduction

Researchers are naturally drawn to complex phe-nomena that challenge their understanding ofthe world. Software startup companies are anintriguing phenomenon, because they develop in-novative software-intensive1 products under timeconstraints and with a lack of resources [2], andconstantly search for sustainable and scalablebusiness models. Over the past few years, soft-ware startups have garnered increased researchinterest in the Software Engineering (SE) com-munity.

While one could argue that software star-tups represent an exceptional case of how soft-ware products are developed and brought tothe market, several factors suggest a broaderimpact. From an economical perspective, star-tups contribute considerably to overall wealthand progress by creating jobs and innovation [3].Digital software startups2 are responsible for anastonishing variety of services and products [5].In the farming sector, venture investment inso-called “AgTech” startups reached $2.06 billionin just the first half of 2015; this figure neared the$2.36 billion raised during the whole of 2014 [6].From an innovation perspective, startups oftenpave the way for the introduction of even morenew and disruptive innovations [7]. Kickstarter ischanging the retail and finance industries, Spotifyis offering a new way to listen to and purchasemusic, and Airbnb is reinventing the hospitalityindustry [8]. From an engineering perspective,startups must inventively apply existing knowl-edge in order to open up unexpected avenues forimprovement [9]; e.g., they must provide educa-tion for full stack engineers, develop techniquesfor continuous lightweight requirements engineer-ing, or develop strategies to control technicaldebt.

Despite these promising conditions, softwarestartups face challenges to survival, even in con-texts where they play a key role in developingnew technology and markets, such as cloud com-puting [10]. These challenges may arise because,while developing a product can be easy, selling itcan be quite difficult [11]. Software startups faceother challenges, such as developing cutting-edgeproducts, acquiring paying customers, and build-ing entrepreneurial teams [12]. Such diverse fac-tors underscore the need to conduct researchon software startups, which will benefit bothscholarly communities and startup leaders.

This paper’s research agenda is driven bypast and current work on software startups. Weoutline the various research tracks to providea snapshot of ongoing work and to preview fu-ture research, creating a platform for identifyingcollaborations with both research and startupenvironments and ecosystems. This effort is nota one-way path. We have therefore founded a re-search network, the Software Startup ResearchNetwork (SSRN)3, which enables interactionsand collaborations among researchers and inter-ested startups. SSRN envisions to: (1) spreadnovel research findings in the context of soft-ware startups; and (2) inform entrepreneurs withnecessary knowledge, tools and methods thatminimize threats and maximize opportunitiesfor success. As part of the network initiatives,an International Workshop of Software Startupswas established in 2015. The first edition of theworkshop was held in Bolzano4 (Italy) in 2015,and the second took place in Trondheim5 (Nor-way) in 2016. This paper provides a researchagenda based on the activities carried out by theresearchers in the network.

The rest of the paper is organized as follows.After we clarify the meaning of software startupand what we know about software startups from

1 ISO 42010:2011 [1] defines software-intensive systems as “any system where software contributes essentialinfluences to the design, construction, deployment, and evolution of the system as a whole” to encompass “individualapplications, systems in the traditional sense, subsystems, systems of systems, product lines, product families, wholeenterprises, and other aggregations of interest”.

2 In our article, digital startups refer specifically to startups in which the business value of the solution is createdby means of software [4].

3 https://softwarestartups.org4 http://ssu2015.inf.unibz.it/5 https://iwssublog.wordpress.com/

Software Startups – A Research Agenda 91

prior research in the Background section, Sec-tion 3 introduces the research topics on softwarestartups, organized under six main tracks thatwe have either investigated or envision investi-gating in the future. Wherever possible, eachtopic is illustrated and motivated by previousstudies. Section 4 highlights the implications ofthese main tracks for future research. The paperconcludes with Section 5, which points out fu-ture actions that can establish and consolidatesoftware startups as a research area.

2. Background

2.1. What is a Software Startup?

To understand software startups, we must firstclarify what a startup is. According to Ries [13],a startup is a human institution designed tocreate a new product/service under conditionsof extreme uncertainty. Similarly, Blank [14] de-scribes a startup as a temporary organizationthat creates high-tech innovative products andhas no prior operating history. These defini-tions distinguish startups from established or-ganizations that have more resources and al-ready command a mature market. In addition,Blank [14,15] defines a startup as a temporary or-ganization that seeks a scalable, repeatable, andprofitable business model, and therefore aims togrow. Blank’s definition highlights the differencebetween a startup and a small business, whichdoes not necessarily intend to grow, and conse-quently lacks a scalable business model.

Even though sharing common characteris-tics with other types of startups, such as re-source scarcity and a lack of operational his-tory, software startups are often caught up inthe wave of technological change frequently hap-pening in software industry, such as new com-puting and network technologies, and an in-creasing variety of computing devices. They alsoneed to use cutting-edge tools and techniquesto develop innovative software products andservices [16]. All these make software startupschallenging endeavours and meanwhile fascinat-ing research phenomena for software engineer-

ing researchers and those from related disci-plines.

In 1994, Carmel first introduced the termsoftware startup, or, to be more precise, softwarepackage startup, in SE literature [17]. Carmel [17]argued that software was increasingly becominga fully realized product. Since then, other re-searchers have offered their own definitions ofsoftware startup. Sutton [16] considers softwarestartups as organizations that are challengedby limited resources, immaturity, multiple influ-ences, vibrant technologies, and turbulent mar-kets. Hilmola et al. [18] claim that most softwarestartups are product-oriented and develop cut-ting edge software products. Coleman and Con-nor [19] describe software startups as unique com-panies that develop software through various pro-cesses and without a prescriptive methodology.

Currently, there is no consensus on the defini-tion of software startup, even though many sharean understanding that software startups dealwith uncertain conditions, grow quickly, developinnovative products, and aim for scalability. Dif-ferent definitions emphasize distinct aspects, andconsequently may have varying implications forhow studies that adopt them should be designed,e.g., who qualifies as study subjects, or which fac-tor is worth exploring. For this reason, despite thelack of a single agreed-upon definition of softwarestartup, it is important and recommended thatresearchers provide an explicit characterizationof the software startups they study in their work.The research track in Section 3.1.1 is dedicatedto develop a software startup context model thatwould allow for such a characterization.

2.2. What are the Major Challenges ofSoftware Startups?

Software startups are challenging endeavours,due to their nature as newly created companiesoperating in uncertain markets and working withcutting edge technology. Giardino et al. [20] high-light software startups’ main challenges as: theirlack of resources, that they are highly reactive,that they are by definition a new company, thatthey are comprised of small teams with littleexperience, their reliance on a single product

92 Michael Unterkalmsteiner et al.

and innovation, and their conditions of uncer-tainty, rapid evolution, time pressure, third-partydependency, high risk, and dependency (they arenot self-sustained). Further, Giardino et al. [12]apply the MacMillan et al. [21] framework in thesoftware startup context, categorizing the keychallenges faced by early stage software startupsinto four holistic dimensions: product, finance,market, and team. The findings of Giardino etal. [12] reveal that thriving in technological un-certainty and acquiring the first paying customerare the top key challenges faced by many startups.In another study, Giardino et al. [22] discoverthat inconsistency between managerial strategiesand execution could lead to startup failure.

Although research exists on the challengessoftware startups face, there is no study dedi-cated to their success factors. Block and Macmil-lan’s [23] study highlights the success factors forany new business, including generating ideas tocomplete product testing, completing a proto-type, and consistently re-designing or makingamendments. Researchers have yet to explorethese general factors’ applicability to the specificsoftware startup context.

2.3. What do We Know about SoftwareEngineering in Software Startups?

Software development comprises a softwarestartup’s core activity. However, some initialresearch studies report a lack of software en-gineering activities in software startups. A sys-tematic mapping study conducted by Paternos-ter et al. [2] allows us to start understandinghow software startups perform software develop-ment. The study reveals that software require-ments are often market driven and are not verywell documented. Software development prac-tices are only partially adopted; instead, pairprogramming and code refactoring sessions sup-ported by ad-hoc code metrics are common prac-tices. Testing is sometimes outsourced or con-ducted through customer acceptance and focusgroups, and team members are empowered andencouraged to adapt to several roles. Similarly,Giardino et al. [20] highlight the most com-mon development practices that have been used

in software startup companies, such as: usingwell-known frameworks to quickly change theproduct according to market needs, evolutionaryprototyping and experimenting via existing com-ponents, ongoing customer acceptance throughearly adopters’ focus groups, continuous valuedelivery, focusing on core functionalities that en-gage paying customers, empowerment of teamsto influence final outcomes, employing metricsto quickly learn from consumers’ feedback anddemand, and engaging easy-to-implement toolsto facilitate product development.

Although a few studies provide snapshots ofsoftware engineering practices in software star-tups [9, 24], the state of the art presented inliterature is not enough to base an understand-ing of how software engineering practices couldhelp software startups. Researchers must builda more comprehensive, empirical knowledge basein order to support forthcoming software star-tups. The research agenda presented in this paperintends to inspire and facilitate researchers inter-ested in software startup related topics to startbuilding such knowledge base.

3. Research Agenda

The Software Startup Research Agenda, initial-ized in June 2015, was developed by a networkof researchers interested in studying the startupphenomenon from different angles and perspec-tives. This variety of research interests not onlyopens up new avenues for collaboration, but alsosheds light on the complexity of the studiedphenomenon. Initially, ten researchers createda mind map of different research areas, aiming toprovide an overview of software startup researchareas and how they connect to each other. Overa period of six months, more researchers joinedthe network, added their research tracks, andcontinuously expanded the map. A working ses-sion with twenty researchers at the 1st workshopon software startup research in December 2015was devoted at discussing the identified areasand finding potential interest overlaps amongthe participants. After this meeting, the authorsof this paper prepared eighteen research track

Software Startups – A Research Agenda 93

Figure 1. Overview of the Software Startup Research Agenda

descriptions according to the following pattern:background of the area, motivation and relevancefor software engineering in startups, researchquestions, potential impact of answering these re-search questions on practice and research, poten-tial research methodologies that can be employedto answer the proposed research questions, andrelated past or ongoing work. Most of the authorsinteracted in the past or are currently active asadvisory board members, mentors, founders orteam members of software startups.

The leading authors of this paper grouped theeighteen research tracks into six major clusters,based on the thematic similarities and differ-ences of the tracks. While this grouping is one ofthe several possible ways to create the clusters,it served the purpose to ease the presentationand discussion of the research agenda, shown inFigure 1. Supporting Startup Engineering Activ-ities (Section 3.1) encompasses research foci thataddress specific software engineering challengesencountered by startup companies. Startup Evo-lution Models and Patterns (Section 3.2) focuseson the progression of startups over time, trying tounderstand the underlying mechanics that drivea company towards success or failure. HumanAspects in Software Startups (Section 3.3) coversresearch tracks that investigate factors related tothe actors involved in startups. The research onApplying Startup Concepts in Non-Startup Envi-

ronments (Section 3.4) seeks to strengthen inno-vation by extracting successful software startuppractices and integrating them in traditional en-vironments. Startup Ecosystems and InnovationHubs (Section 3.5), on the other hand, investi-gates whether and how a thriving environmentfor software startups can be designed. Finally, allof these areas are connected by research tracksthat develop methodologies and theories for soft-ware startup research (Section 3.6).

Figure 1’s illustration of the research agendaincludes reference to research areas outside thispaper’s current scope. Marketing and Businessand Economic Development are directions thatare likely relevant for the performance of softwarestartups. These and other areas may be addedto the research agenda in later editions whenmore evidence exists regarding whether and howthey interact with software startup engineering,i.e. the “use of scientific, engineering, managerialand systematic approaches with the aim of suc-cessfully developing software systems in startupcompanies” [9].

3.1. Supporting Startup EngineeringActivities

The research tracks in this cluster share thetheme of studying, identifying, transferring, andevaluating processes, methods, framework, mod-

94 Michael Unterkalmsteiner et al.

els, and tools aimed at supporting softwarestartup engineering activities.

3.1.1. The Context of Software IntensiveProduct Engineering in Startups

Rapid development technologies have enabledsmall companies to quickly build and launchsoftwareintensive products with few resources.Many of these attempts fail due to market con-ditions, team breakup, depletion of resources, ora bad product idea. However, the role of softwareengineering practices in startups and their impacton product success has not yet been explored indepth. Inadequacies in applying engineering prac-tices could be a significant contributing factor tostartup failure.

Studies show that startups use ad-hoc engi-neering practices or attempt to adopt practicesfrom agile approaches [25, 26]. However, suchpractices often focus on issues present in largercompanies and neglect startup-specific challenges.For example, Yau and Murphy [25] report thattest-driven development and pair programmingprovide increased software quality at an expenseof cost and time. Also keeping to a strict backlogmay hinder innovation. Since neglecting engineer-ing challenges can lead to sub-optimal productquality and generate waste, engineering practicesspecific to the startup context are needed. Theoverarching questions in this research track are:– RQ1: To what degree is the actual engineering

a critical success factor for startups?– RQ2: How can the startup context be defined

such that informed decisions on engineeringchoices can be made?

– RQ3: What engineering practices, processesand methods/models are used today, and dothey work in a startup context?An answer to RQ1 could help practitioners

to decide on what activities to focus on andprioritize allocation of resources. Several studies,e.g Paternoster et al. [2], Giardino et al. [12] andSutton [16], emphasize the differences betweenestablished companies and startups, noting thatstartups are defined by limited resources anddynamic technologies. However, these charac-terizations are not granular enough to support

a comparison of engineering contexts in differ-ent companies, making the transfer of practicesfrom company to company difficult [27]. Thus,understanding the engineering context of star-tups (RQ2) is an important milestone in develop-ing startup context specific engineering practices(RQ3). While there exists work that providessystematic context classifications for the fieldof software engineering in general [27–31], thesemodels are not validated and adapted for usewithin startups. The work in this research trackaims to develop such a software startup contextmodel by analysing data from startup experiencereports [24]. Provided that engineering contextsamong startups and established companies canbe compared at a fine level of detail, the contextmodel can be used to identify candidate practices.Moreover, researchers can develop decision sup-port by mapping specific challenges with usefulpractices, thereby validating the model and help-ing practitioners select a set engineering practicesfor their specific context and set of challenges.

3.1.2. Technical Debt Management

The software market changes rapidly. As dis-cussed by Feng et al. [32], in fast changing envi-ronments, the product management focus evolvesfrom the more traditional cost or quality orien-tation to a time orientation. New product devel-opment speed is increasingly important for orga-nizations, and a commonly shared belief is thattime-to-market of new products can build a com-petitive advantage [32]. In the software startupcontext, it may be vital to be the first to mar-ket in order to obtain customers. Since softwarestartups also lack resources, quality assuranceis often largely absent [2]. However, long-termproblems will only be relevant if the productobtains customers in the short term [33]. Thisshort-term vision may produce software codethat is low-quality and difficult to change, com-pelling the company to invest all of its effortsinto keeping the system running, rather than in-creasing its value by adding new capabilities [33].Scaling-up the system may become an obstacle,which will prevent the company from gaining newcustomers. Finding a viable trade-off between

Software Startups – A Research Agenda 95

time-to-market demands and evolution needs isthus vital for software startups.

One promising approach to performing sucha trade-off is technical debt management. Techni-cal debt management consists of identifying thesources of extra costs in software maintenanceand analysing when it is profitable to invest ef-fort into improving a software system [33]. Hence,technical debt management could assist startupsin making decisions on when and what to fo-cus effort on in product development. Technicaldebt management entails identifying the techni-cal debt sources, the impact estimation of theproblems detected, and the decision process onwhether it is profitable to invest effort in solvingthe detected sources of technical debt [34, 35].Only those sources of technical debt that providereturn on investment should be resolved. Moreimportantly, technical debt should be managedduring project development [36] in order to con-trol the internal quality of the developed software.Several research questions need to be answeredto successfully manage technical debt in this way:– RQ1: What kind of evolution problems are

relevant in the software startup context? Howcan we identify them?

– RQ2: How can we prioritize the possible im-provements/changes in the context of soft-ware startups?

– RQ3. What factors beyond time-to-marketand resource availability must be consideredin trade-offs?

– RQ4: How can we make decisions about whento implement the improvements/changeswithin the software startup roadmap?

– RQ5: How can we provide agility to technicaldebt management, necessary in an environ-ment plenty of uncertainty and changes?

Answering these questions will impact on bothpractitioners and researchers focused on softwarestartups. Practitioners will be able to make bet-ter decisions considering the characteristics ofthe current software product implementation.The current implementation could make it im-possible to reach a deadline (time to market), be-cause of the complexity of the changes to performto implement a new feature, assuming a givenamount (and qualifications) of effort to be de-

ployed. Furthermore, it will be also possible todecide between two alternative implementations,with different costs, but also with different po-tential for the future, assuming that the “future”has been previously outlined. For researchers,answering these questions could help clarify therole of design decisions in software developmentin the context of a software product roadmap,similarly to what happens in other engineeringdisciplines.

Technical debt is context dependent sincequality tradeoffs are context dependent [37].While technical debt is as important to softwarestartups as it is to mature companies, the kind ofdecisions to take and the consequences of makingthe wrong decisions are not the same, justifyingresearch on technical debt specifically in softwarestartups.

In general, there is a lack of specific studies ontechnical debt management in software startups,and current literature reviews on technical debtmanagement do not address this topic [34, 35].Moreover, there are several specific challengesto managing technical debt that are of specialrelevance for software startups. For one, very fewstudies address how to prioritize improvements tosolve technical debt problems, especially for com-mercial software development [35]. In addition,technical debt management literature often refersto time-to-market, but very few studies actuallyaddress it [34], perhaps because it is a topic thatstraddles engineering and economics.

3.1.3. Software Product Innovation Assessment

Startup companies strive to create innovativeproducts. For firms in general, and software star-tups in particular, it is critical to know as soonas possible if a product aligns with the market,or whether they can increase their chances tolead the market and recruit the highest possiblenumber of customers [38].

The need to invest in infrastructures to mea-sure the impact of innovation in software washighlighted by OECD [39], and more recentlyby Edison et al. [40]. These measures will en-able companies to assess the impact of innova-tion factors and achieve the expected business

96 Michael Unterkalmsteiner et al.

goals, as well as to improve the understandingof success yield high returns on investments inthe innovation process [39]. Product innovationassessment is thus very relevant for product de-velopers, and especially for startups, which aremore sensitive to market reactions. Product in-novation assessment is complex, particularly forsoftware products [41].

Product innovation assessment is reportedin literature as the combination of a number ofmulti-dimensional factors impacting the successor failure of a software product [42]. Factor’smeasures intend to engage people in the in-novation process to think more deeply aboutfactors affecting product innovation. Factors suchas time-to-market, perceived value, technologyroute, incremental product, product liability,risk distribution, competitive environment, lifecycle of product, or strength of market could begrouped into dimensions likemarket, organization,environment, or any other terms of impact on themarket and business drivers [43]. These factorscan act as innovation enablers or blockers [44].

Since these factors are not always indepen-dent, it is critical to identify the existing de-pendencies and gain a better understandingof each factor’s impact. It would be necessaryto relate these factors to characteristics spe-cific to software products, such as, but not lim-ited to, software quality attributes proposed byISO/IEC [45].

There is a lack of specific literature on soft-ware product innovation assessment; most of thepast research refers to products in general, andnot specifically to software products [40,46], lead-ing to the following research questions:– RQ1: What should be the components

of a software product innovation assess-ment/estimation model?

– RQ2: What factors can help measure innova-tion from a software product and a marketperspective?

– RQ3: To what extent are factors that canhelp measure innovation dependent on thesoftware product and the market perspective?

– RQ4: What is the relation between softwareproduct innovation factors and quality fac-tors?

– RQ5: What kind of tools for software productinnovation estimation could support softwarestartups in decision making?

While innovation has been widely studied fromthe process perspective, the product perspec-tive, by nature, has been addressed mainly fromthe viewpoint of specific products and indus-tries. However, software products are differentcompared to other kinds of products [47] andinnovations in the software industry happen fast.Hence, answers to RQ1-RQ4 would provide a fun-damental understanding on software product in-novation assessment and be beneficial for bothresearchers and practitioners. Software startupsneed to be fast and spend resources in an efficientway. Therefore, to be able to estimate existingproducts or design new products, consideringthose characteristics that experience shows thatare relevant from an innovation point of view,can be essential for software startups to developsuccessful products (RQ5).

3.1.4. Empirical Prototype Engineering

Startups often start with a prototype, whichserves as a form to validate either a new technol-ogy or knowledge about targeted customers [2].Traditionally, prototyping implies a quick andeconomic approach to determining final prod-ucts [48–50]. Defined as a concrete representationof part or all of an interactive system, proto-types has been intensively researched and usedin Software Engineering, with well-developedtaxonomies, such as horizontal and vertical,low-fidelity and high-fidelity prototypes [50]. Thestrategy of developing a prototype can greatlyvary due to a great variety of prototype types,their development efforts and value they canproduce.

While much about prototyping techniquescan be learnt from the SE body of knowledge,the discussion about prototyping in the contextof business development process is rare. Recentwork on startup methodologies, such as LeanStartup [13] and Design Thinking [51] emphasizesthe adoption of prototypes to increase chancesof success through validated learning. Alterna-tively, startup prototypes need to be developed to

Software Startups – A Research Agenda 97

satisfactorily serve their purposes, i.e. technicalfeasibility test, demonstration to early customers,and fund raising. We argue that the prevalentSoftware Engineering practices used by startupsto develop their first product inefficiently inte-grate into startups’ dynamic contexts. Hence wecall for research in understanding the develop-ment and usage of prototypes in startup con-texts:– RQ1: How can prototyping be used to maxi-

mize learning experience?– RQ2: How can prototyping be used for opti-

mization?– RQ3: How can prototyping be used to support

communication with external stakeholders?– RQ4: How do prototypes evolve under the

multiple influences of startups’ stakeholders?Early stage startups are lacking actionable guide-lines for making effective prototypes that canserve multiple purposes. We believe that manystartups will economically and strategically ben-efit by having proper practices in prototyping,such as technology evaluation (RQ1), strate-gic planning (RQ2) and customer involvement(RQ3).

To understand prototype development andits usage in startups, i.e. answering the first threeresearch questions, exploratory case studies canbe conducted. Cases would be selected to coverdifferent types of startup prototypes at differentphase of startup progress. A large-scale surveycan be used to understand the prototype usagepatterns, i.e. answering RQ4.

Despite an increasing body of knowledge onsoftware startups [2], empirical research on pro-totyping processes and practices are rare. A fewstudies have investigated the adoption of soft-ware prototypes in combination with DesignThinking [52] and proposed prototyping tech-niques [52–54]. However, these studies rely ona very limited number of cases. Moreover, differ-ent constraints on prototyping decisions are oftenneglected. Future work can address antecedencefactors, i.e. the involvement of lead-users, avail-able human resources, and technological push,and how they impact prototyping strategies andusages in different startup contexts [55].

3.1.5. Risk Management Tools forSoftware Startups

The management of risk, namely the risk of fail-ing to meet one’s goals within given constraintsin budget and/or time, is of paramount impor-tance in every human activity. In the contextof software startups, risk management looks un-conventional, because startups naturally involvea much higher risk than traditional businesses.Yet, perhaps even more so than in traditionalcontexts, evaluating and managing risk in thesoftware startup context might be a key factorfor success.

Risk factors can be identified as a check-list ofthe incidents or challenges to face. Each of themcould be categorized and prioritized accordingto its probability and the impact level of itsconsequences. This research track aims to study,model, and quantify various aspects related torisk management in software startups, with thegoal of providing tools, based on process simula-tion, that control risk. Being able to efficientlymodel and simulate the startup process and itsdynamics, would support startups in timely deci-sion making. While numerous other approachesto risk control exist [56], we have found in ourprevious work [57, 58] that process simulationscan be effective in risk management. Therefore,the overarching questions in this research trackare:– RQ1: To what extent do software startups

explicitly manage risk?– RQ2: To what degree is it feasible to model

software development processes in startups?– RQ3: To what extent can these models be

used to quantify the risk of exceeding projectbudget or time?

– RQ4: What systematic ways exist to under-stand when to pivot or persevere [13], andwhat might be the cost of a wrong or untimelydecision?

Following our previous experiences in softwareprocess modelling and simulation, to gain a bet-ter understanding is necessary to identify andanalyse significant activities, not limited tothe software development phase, of a software

98 Michael Unterkalmsteiner et al.

startup (RQ1). This is necessary to be able toidentify the critical aspects of startup develop-ment risks that are suitable for simulation. Inour previous work we studied the application ofEvent-Driven models and/or System Dynamicsto the software development processes. From thiswork we know that it is possible analyse projectvariations in time and budget with a Monte Carloapproach, by performing several simulations ofthe same project, varying the unknown parame-ters according to given distributions, and calcu-lating the resulting distributions of cost and timeof the simulated projects. Such analysis allowsone to compute the Value At Risk (VAR) of thesequantities, at given VAR levels. While Cocco etal. [57] and Concas et al. [58] provide exemplarstudies of the application of these techniques inmature (agile) software development contexts,the question is whether such an approach is suit-able and beneficial for software startups, andunder what conditions (RQ2). By simulating theevolution of a startup as a process, we mightbe able to make predictions on its future devel-opment. Such predictions, or a result that canbe rapidly be drawn from simulations, might becrucial for startups to understand which deci-sions are less costly and/or risky (RQ3). This isparticularly true for decisions related to fieldssuch as market strategies, team management,financial issues or product development (RQ4).

3.1.6. Startup Support Tools

Support tools can help software startups gettheir business off the ground with less painand more guidance. These tools generally em-bed crucial knowledge regarding startup pro-cesses and activities. A plethora of tools (mostlysoftware tools) exist for meeting the differentneeds of entrepreneurs and supporting variousstartup activities. For example, the web-page6by Steve Blank, a renowned entrepreneurshipeducator, author, and researcher from Stan-ford University, contains a list of more than1000 tools. Well-designed portals such as Star-tupstash.com ease access to these supportingtools.

However, due to the lack of time, resources,and/or necessary knowledge, entrepreneurs can-not easily find the tools that best suit their needs,or cannot effectively utilize these tools to theirpotential. Existing studies provide limited in-sights on how entrepreneurial teams could find,use and benefit from support tools. Hence, theoverarching questions in this research track are:– RQ1: What are the needs of software startups

that can be supported by software tools?– RQ2: What are the tools that support differ-

ent startup activities?– RQ3: How can support tools be evaluated

with respect to their efficiency, effectiveness,and return-on-investment?

– RQ4: How can support tools be effectivelyrecommended to entrepreneurs and used bythem?

RQ1 and RQ2 are targeted at identifying a matchbetween the needs of software startups and theavailable tool support. To enable robust recom-mendations, both the individual startups and thesoftware tools need to be objectively character-ized allowing for their evaluation w.r.t. certainquality criteria (RQ3). There are potential syn-ergies with the research track looking at the con-text characterization of software startups (Sec-tion 3.1.1). Answers to these research questionscan be also valuable input for software tool ven-dors to develop the right tools that are needed bystartups. In addition, the findings can be usefulfor future studies that develop proof-of-conceptprototypes to support startup activities.

To investigate the proposed questions, vari-ous research methods can be applied, includingsurvey of software startups regarding their needsand usage of support tools, in-depth case studyof adoption and use of support tools, and de-sign science approach to develop recommendersystems of support tools (RQ4).

Research on tooling aspects in the softwarestartup context is scarce. Edison et al. [59] arguethat, despite the fact that different startup sup-porting tools have been developed and publishedover the Internet, new entrepreneurs might nothave sufficient knowledge of what tools they needwhen compared to experienced entrepreneurs. In

6 http://steveblank.com/tools-and-blogs-for-entrepreneurs/

Software Startups – A Research Agenda 99

addition, not all tools will help entrepreneursin certain tasks or situations. Entrepreneurs’ ex-periences using the tools can serve as the basisfor evaluating and recommending appropriatetools. Besides suggesting a new categorization ofexisting startup support tools, Edison et al. [59]propose a new design of a tool portal that willincorporate new ways to recommend tools toentrepreneurs, especially to those who engage forthe first time in a software startup endeavour.

3.1.7. Supporting Software Testing

Testing software is costly and often compromisedin startups [60], as it is challenging for startupsto fulfil customer needs on time, while simultane-ously delivering a high quality product. In manysoftware startups there is a common slogan thatsays “done is better than perfect”, which indi-cates a general tendency toward a lack of testingand quality assurance activities [61]. However, itis sometimes also observed that startups do notknow how and what to test; they lack expertise totest requirements as they do not have knowledgeabout their customers and users [61]. Thereforeconsidering testing in software startups poses thefollowing research questions:– RQ1: To what extent does software testing

in startup companies differ from traditionalcompanies?

– RQ2: To what extent does testing evolve overtime in software startup companies?

– RQ3: What is an optimal balance betweencost/time spent on testing and developmentactivities?

– RQ4: How can a software startup leveragecustomers/users for testing?

Answering RQ1 would provide insights on theaspects that differentiate the software testingprocess in startups from mature companies. Forexample, integration testing is likely very impor-tant for startups due to the fast paced productdevelopment. At the same time however, startupstend to work with cutting edge technologies, re-quiring a robust and flexible test integration plat-form. Connected to this is the question whethertesting needs change over time, while the soft-ware startup matures. Answers to RQ2 and RQ3

would be particularly valuable for practitionerswho could then better allocate resources. Usersof software could be used for different testingpurposes. On one hand, users provide valuablefeedback in testing assumptions on customersneeds. On the other hand, early adopters thatare more robust towards deficiencies can help toimprove product quality before targeting a largermarket. Answers to RQ4 would provide strategiesto harvest these resources.

In order to answer these research questions,various empirical research methods could be uti-lized. The studies would be devised in a way that“contrasting results but for anticipatable reasons”could be expected [62], i.e. different softwarestartup companies would be taken into accountto acquire a broad view of testing in softwarestartups.

To the best of our knowledge, software testingin software startups has been scarcely researched.Paternoster et al. [2] highlighted the quality as-surance activities in software startups in theirmapping study. They found that it is importantto provide software startups effective and effi-cient testing strategies to develop, execute, andmaintain tests. In addition, they highlighted theimportance of more research to develop practical,commercial testing solutions for startups.

3.1.8. User Experience

User experience (UX) is described as “a person’sperceptions and responses that result from theuse or anticipated use of a product, system orservice” [63]. Good UX can be seen as providingvalue to users, as well as creating a competitiveadvantage. UX is important for software startupsfrom their earliest stages. Firstly, human-centreddesign methods such as user research and usertesting can help startups better understand howthey can provide value to users and customers, aswell as what features and qualities need testingfor users to be satisfied with their product. Com-bined with business strategy, this human-centredapproach helps startups move towards successful,sustainable business creation. Secondly, provid-ing an initially strong UX in the first productversions can create positive word of mouth [64],

100 Michael Unterkalmsteiner et al.

as well as keep users interested in the product fora longer time [65]. Genuine interest from usersfor the product idea while the product is stilla prototype helps gain meaningful feedback [65].Compared to more established businesses, soft-ware startups may pivot resulting in new targetmarkets and user groups. This means efforts putinto designing UX need to be faster and lessresource consuming. Furthermore, failing to de-liver satisfying UX can be fatal to small startupsthat can not cover the costs of redesigning. Theoverarching questions in this research track are:– RQ1: What useful methods and practices ex-

ist for creating UX in startups?– RQ2: What is UX’s role during different

phases of a startup’s life-cycle?– RQ3: To what extent are UX and business

models connected in customer value creation?An answer to RQ1 can provide software startupsmethods for developing strong UX in the firstproduct versions which can keep users interestedin the product for a longer time [65]. Genuineinterest from users for the product idea whilethe product is still a prototype helps to gainmeaningful feedback [65]. For business creation,understanding the value of UX for startups (RQ2)helps assigning enough resources for creation ofUX while not wasting resources where there isno value to be gained (RQ3).

Research on startups and UX has been verylimited. Some case studies report UX’s role inbuilding successful startups [66, 67]. Practicesand methods for UX work in startups have beenreported in [65,68,69]. A framework for creatingstrong early UX was presented by Hokkanen etal. [70]. These provide some results on feasibleand beneficial UX development in startups, butmore generalizable results are needed.

3.2. Startup Evolution Modelsand Patterns

The research tracks in this cluster share thetheme of studying, identifying, and differenti-ating the transformation of startups in differentstages. This also includes studies about differentbusiness and technical decision-making practices.

3.2.1. Pivots in Software Startups

It is very difficult for software startups to un-derstand from start what are the real problemsto solve and what are the right software solu-tions and suitable business models. This is evi-denced by the fact that many successful softwarestartups are different from what they startedwith. For example, Flickr, a popular online photosharing web application, originally was a mul-tiplayer online role playing game [71]. Twitter,a famous microblogging application, was bornfrom a failed attempt to offer personal podcastservice [71].

Due to their dynamic nature, software star-tups must constantly make crucial decisions onwhether to change directions or stay on the cho-sen course. These decisions are known as pivotor persevere in the terms of Lean Startup [13].A pivot is a strategic decision used to test fun-damental hypothesis about a product, market,or the engine of growth [13]. Software startupsdevelop technology intensive products in nature.Due to this, these are more prone to the rapidlychanging technology causing pivots. Similarly,certain types of pivots are more relevant to soft-ware startups e.g. zoom in pivot: a pivot whereone feature of a product become the whole prod-uct as in the case of Flickr. Pivot is closely linkedto validated learning, another key concept fromLean Startup. The process to test a businesshypothesis and measure it to validate its effectis called validated learning [13], whereas pivot isoften the outcome of validated learning. A recentstudy [22] reveals that startups often neglect thevalidated learning process, and neglect pivotingwhen they need to, which leads to failure. Thisshows the importance of pivoting for a startup tosurvive, grow, and eventually attain a sustainablebusiness model. In order to better understandand explore the pivoting process in the softwarestartup context, the following fundamental re-search questions can be formed:– RQ1: To what extent is pivoting crucial for

software startups?– RQ2: How do software startups pivot during

the entrepreneurial/startup process?

Software Startups – A Research Agenda 101

– RQ3: What are the existing pro-cess/strategies/methods to make a pivotingdecision in a startup context?

– RQ4: How do pivots occur during differentproduct development and customer develop-ment life cycles?

Answering RQ1–RQ2 is necessary to understandpivoting in the context of software startups,building a fundamental framework on reasons forpivoting and their types. RQ3–RQ4, on the otherhand, are targeted at understanding pivoting de-cisions and mechanisms. The overall contributionof answering the stated research questions hasimplications for both researchers and practition-ers. The answers would provide an empiricallyvalidated conceptual and theoretical basis forthe researchers to conduct further studies regard-ing the pivot phenomenon. For the practitioners,it would help them to make informed decisionregarding when and how to pivot in order toincrease the chances of success.

Due to the nascent nature of software startupresearch area, exploratory cases studies is a suit-able approach to answer the research questions.Followed by the case studies, quantitative surveyscan also be conducted to further generalize theresults regarding pivoting in software startups.

Recently, there were some studies conductedon pivots in software startups. A study by Vander Van and Bosch [72] compares pivoting deci-sions with software architecture decisions. An-other study by Terho et al. [73] describes howdifferent types of pivots may change business hy-pothesis on lean canvass model. However, thesestudies lack the sufficient detail to understanddifferent types of pivots and the factors triggeringpivots. A study by Bajwa et al. [74], presentsan initial understanding of different types of piv-ots occurred at different software developmentstages, however it lacks the deeper understandingof the pivoting decision that can only be achievedby a longitudinal study.

3.2.2. Determination of Software StartupSurvival Capability through BusinessPlans

Software startups are highly specialized froma technological point of view. Focusing on the

economic exploitation of technological innova-tions [75], they belong to the group of newtechnology-based firms. Literature suggests thatone of their major challenges is the transforma-tion of technological know-how into marketableproducts [76,77]. New technology-based firms of-ten struggle with unlocking the product-marketfit [78] and commercializing their technologicalproducts [76]. Applying a resource-based viewdoes thus not suffice for explaining survival andgrowth of software startups [79,80]: a crucial suc-cess factor is the ability of new technology-basedfirms to understand and interact with the marketenvironment to position their products accord-ingly [81,82].

Particularly in early lifecycle stages, newtechnology-based firms need to build net-work relations with the market. Networktheory literature suggests that with increas-ing network maturity, the chances for sur-vival and growth increase [83–85]. The abil-ity to transform resources in response totriggers resulting from market interactionscan be described as a dynamic capabil-ity [86–89] which helps software startups com-mercialize their products. This transforma-tion process captures the evolution of newtechnology-based firms in their early-stages.Current research is based on the constructof “venture emergence”, which provides a per-spective on the evolutionary change processof new technology-based firms [81, 90]. Ven-ture emergence reflects the interaction processwith agents and their environments [91]. Busi-ness plans of new technology-based firms areused as the artefact for measuring the sta-tus of venture emergence. They contain de-scriptions of transaction relations [92–94] newtechnology-based firms build in four marketdimensions: customer, partner, investor, andhuman resources [95]. This research track in-tends to answer a number of research ques-tions:– RQ1: How reliably can annotated transac-

tion relations from business plan texts de-termine the venture emergence status oftechnology-based startups?

– RQ2: To what extent are the number andstrength (“level”) of identified transaction re-

102 Michael Unterkalmsteiner et al.

lationships useful as an indicator of survivalcapability?

– RQ3: How can patterns of transaction re-lations be used as an indicator for eval-uating strengths and weaknesses of newtechnology-based firms, and thus be used tomore effectively direct support measures?

While it is possible to measure the ven-ture emergence status even in a softwarestartup’s very early stages, the predictivestrength of transaction relations needs to beevaluated (RQ1–RQ2). This use of networktheory to operationalize the venture emer-gence construct is a new approach, whichadds to network theory literature in the con-text of the survival of new technology-basedfirms. It further confirms the business plansof new technology-based firms as a valu-able source of information on startup po-tential. Finally, the resource-based approachto explain venture survival is enriched byapplying a process-oriented perspective: weanalyse resource transformation, rather thanonly looking at the initial resource configu-ration (RQ3). Furthermore, the research cancontribute to the effectiveness of the in-novation system by investigating indicatorsthat reveal strengths and weaknesses of newtechnology-based firms. These can be used to di-rect support measures to software startups moreeffectively.

To answer the stated research questions, onecan use content analysis [96,97], combining hu-man and computer-based coding of businessplans, to determine the number and strengthof transaction relations [98,99].

Initial statistical tests that have been per-formed on a sample of 40 business plans of newtechnology-based firms confirm the relationshipbetween the status of venture emergence of newtechnology-based firms and venture survival [99].Earlier work led to the development of the con-cept for analysing early-stage startup networksand the relevance for survival [95]. Based onthis concept, a coding method for transactionrelations in business plans has been developedand validated with 120 business plans [98].

3.3. Cooperative and Human Aspects inSoftware Startups

The research tracks in this cluster address chal-lenges and practices related to how people coop-erate and work is software startups.

3.3.1. Competencies and Competency Needs inSoftware Startups

Software startups set different competency re-quirements on their personnel than more estab-lished companies. The biggest differences occurin two phases of the evolution of startups whichhave an impact on the nature of software devel-opment and competence needs: (1) in the earlystages of rapid software development when thereis a lack of resources and immature competen-cies in many key areas, and (2) when the rapidbusiness growth of successful startups requiresmanagement of a fast growing personnel andamount of software with limited managementresources and competencies. In the early phasesstrong competition requires the software startupto innovate and react quickly [2], and deploymentof systematic software engineering processes ismany times replaced by light-weight ad-hoc pro-cesses and methods [2,26]. The nature of softwaremakes it possible for successful startups to scalefast [2]. Rapid software-driven growth requiresfast scaling of the software production, distri-bution, and maintenance. The required compe-tences also quickly evolve when software develop-ment moves from rapid greenfield prototyping toprofessional software development and manage-ment. Mastering this demanding situation oftenrequires a broad prior skill basis from the startupteam, including an ability to adjust to changes,and learn quickly.

Research on specific skills and competencyneeds in software startups broadens not onlythe knowledge on software startups themselves,but also broadens the knowledge on softwareengineering conducted under the challenging cir-cumstances of startups. Focusing the researchon the early stages and on the growth pe-riod of the software startups, when the chal-lenges of the software startups are the great-

Software Startups – A Research Agenda 103

est [12,22], brings the most valuable knowledgeto both academia and practitioners. Compe-tency research also brings human factors intofocus [100,101], and reinforces the results of ex-isting software startup research towards a morecomprehensive modelling and understanding.The research questions for studies on competen-cies and competency needs in software startupsinclude:– RQ1: Software startup challenges and com-

petency needs – what software developmentknowledge and skills are needed to overcomethe challenges?

– RQ2: What are the competency needs specificfor software startups compared to the moreestablished software companies?

– RQ3: How do the competency needs changeover the evolution of software startups?

– RQ4: How do the competency needs map ontothe roles and responsibilities of the startupteams in software startups?

– RQ5: How can the growth of software star-tups be managed in terms of competencyneeds for software development practices, pro-cesses and recruitment?

Research on software startups, including researchon competency needs, provides the researchand development of software engineering withnew knowledge and viewpoints on how to di-rect the work in order to best address the spe-cific challenges of the software startups (RQ1).In particular, differences to mature softwarecompanies are interesting to study (RQ2) con-sidering software startups evolve, if they sur-vive, to established companies. Knowing howcompetency needs change might turn out asone key factor for this transition (RQ3). The-oretical models describing the evolution pathsof software startups have been created [13,102], but competency needs and how theymap to roles and responsibilities have beento a large degree ignored (RQ4). Similarly,while software development work [2] and soft-ware engineering practices [26] have also beenstudied, it is unclear how competency needscan be managed in growing software startups(RQ5).

3.3.2. Teamwork in Software Startups

The importance of human aspects in software de-velopment is increasingly recognized by softwareengineering researchers and practitioners. Team-work effectiveness is crucial for the successes ofany product development project [103]. A com-mon definition of a team is ”a small number ofpeople with complementary skills who are com-mitted to a common purpose, set of performancegoals, and approach for which they hold them-selves mutually accountable” [104]. A startupteam is special in the wide range of variety, in-cluding both technicians and entrepreneurs.

While an innovative idea is important for theformation of a startup, startup success or failureultimately rests on the ability of the team to exe-cute. Entrepreneurship research showed that over80 percent of startups that survive longer thantwo years were founded by a group of two or moreindividuals [105]. The dynamic and intertwinedstartups activities require the close collaborationnot only among startup team members, but alsowith external stakeholders, such as mentors andinvestors. Given the diversity in mindsets andskill sets among founders, it is essential that theycan work well together along with the startuplife-cycle. The movement with recent methodol-ogy in Lean startup introduces an opportunityto look at startup teams from various angles, i.e.pivoting, startup culture, team formation, anddecision-making. The overarching questions inthis research track are:– RQ1: Is there a common cultural/organiza-

tional/team characteristic among successfulsoftware startups?

– RQ2: How can a software startup team effec-tively communicate with other stakeholders,i.e. mentors and investors?

– RQ3: How can a software startup manageteam internal relationships?

– RQ4: What are the common patterns ofcompetence growth among software startupteams?

Understanding software startup team behaviourto internal and external environments and relat-ing them to startup success measures would help

104 Michael Unterkalmsteiner et al.

to identify characteristics and teamwork patternsof successful startups. Answering RQ1 would pro-vide practitioners some guidance on how to formstartup teams while answers to RQ2–RQ3 wouldprovide an understanding how internal end ex-ternal team dynamics work and can improved.An answer to RQ4 would also support the workin Section 3.3.1, looking however specifically atcompetence growth patterns that could be valu-able for practitioners when deciding on what tofocus on in competence development. Empiricalstudies, i.e. case studies, surveys and action re-search are all suitable to investigate the statedresearch questions. Among them, comparativecase studies would be the first option to discoverthe difference in startup teamwork patterns.

There exists a large body of literature in busi-ness management, entrepreneurship, and smallventures about entrepreneurial teams’ charac-teristics and their relationship to startup out-comes [105–107]. In Software Engineering, fewempirical studies identified team factors in thefailure of software startups. Giardino et al. foundthat building entrepreneurial teams is one ofthe key challenges for early-stage software star-tups from idea conceptualization to the firstlaunch [12]. Crowne et al. described issues withfounder teamwork, team commitment and skillshortages [108]. Ensley et al. investigated therelative influence of vertical versus shared leader-ship within new venture top management teamson the performance of startups [109]. Other teamdimensions are explored in the business and en-gineering management domain in specific geogra-phies. E.g., Oechslein analysed influencing vari-ables on the relational capital dimension trustwithin IT startup companies in China [105]. Howgeneralizable these influencing variables to othergeographies is yet to be seen.

3.4. Applying Startup Concepts inNon-Startup Contexts

One of the Lean Startup principles claims thatentrepreneurs are everywhere, and that en-trepreneurial spirits and approaches may be ap-plied in any size company, in any sector or indus-try [13]. On the other hand, established organi-

zations face the challenge of innovation dilemmaand inertia caused by the organization’s stabilityand the maturity of markets [110]. Therefore, ap-plying startup concepts in non-startup contextsseems an promising avenue for established orga-nizations to improve their innovation potential.

3.4.1. Internal Software Startups in LargeSoftware Companies

The internal software startup concept has beenpromoted as a way to nurture product innovationin large companies. An internal software startupoperates within the corporation and takes respon-sibility for everything from finding a business ideato developing a new product and introducing it tomarket [111]. Internal software startups can helpestablished companies master the challenge of im-proving existing businesses, while simultaneouslyexploring new future business that sometimescan be very different from existing ones [112].Usually, this involves a conflict of interest interms of learning modes [113] or risk propen-sity [114], which can be prevented by establish-ing dual structures within the organization forimplementing internal software startups [115].Compared to the traditional R&D activities oflarger companies, an internal software startupdevelops products or services faster [2] and withhigher market orientation [116]. This helps estab-lished companies maintain their competitivenessin volatile markets [117].

Besides the fact that the successful imple-mentation of internal software startups facesvarious barriers, such as cultural conflicts [118]or the fear of cannibalization of existing busi-nesses [119], internal software startups can alsobenefit from being part of established compa-nies. Shared resources, such as capital, humanresources [120, 121], and the access to the cor-porates’ internal and external network [122] arejust some benefits.

Earlier research on analysing the results ofstartups’ value creation cycle has taken place inthe context of the evolution of the enterprise [123].However, this occurs over too long of a timeperiod to be useful for guiding software develop-ment. Measuring the cycle time of the software

Software Startups – A Research Agenda 105

engineering process to the completion of a soft-ware feature is also insufficient. The Lean startupapproach [13] has been commonly adopted to newbusiness creation in software intensive ventures.They use the learning loop to discover the cus-tomer value and potential of the new productconcept, as well as to find new means to pro-duce software. Tyrväinen et al. [124] proposethat measuring the cycle time from developmentto analysis of customer acceptance of the featureenables faster learning of market needs. In ad-dition, receiving fast feedback from users makeschanging the software easier for the programmerswho have not yet forgotten the code. Relevantresearch questions regarding internal softwarestartups can be formulated as follows:– RQ1: How can Lean startup be adopted and

adapted for software product innovation inlarge software companies?

– RQ2: What are the challenges and enablersof Lean startup in large software companies?

– RQ3: How should internal software startupsbe managed/lead?

– RQ4: What metrics can be used to evalu-ate software product innovation in internalstartups?

– RQ5: To what extent do internal startupshave a competitive advantage compared toindependent startups (through shared re-sources, etc.)?

Lean startup approach gains more interest fromscholars and academics as a new way to foster in-novation since it helps to avoid building productsthat nobody wants [125]. Some evidence showsthat mature software companies and startupsdiffer in applying Lean startup approach [126];e.g. mature firms start the cycle by collectingdata from existing users and then generatinga hypothesis based on that data, whereas soft-ware startups generate ideas and collect datafrom new users to validate the ideas. However, itseems that, to a large extent, the approach canbe used both in startups and established enter-prises. By answering RQ1–RQ3 we aim at defin-ing structured guidelines on how to introduceLean startup in large software companies, sup-porting practitioners, while answering RQ4-RQ5would provide a motivation for this approach, al-

lowing to compare effectiveness on a quantitativelevel.

Due to the complex nature of the researchphenomenon and the intention to achieve anin-depth understanding of it, we consider mul-tiple case studies [62] as a suitable research ap-proach. The case organizations can be selectedbased on the following criteria: (1) the organiza-tion develops software in-house, (2) a dedicatedteam is responsible from ideation to commercial-ization of a new software, and (3) the softwarefalls out of the current main product line. Theunit of analysis in this study would be a devel-opment team.

Very few studies have investigated how theLean startup [13] can leverage internal startupsin large software companies to improve their com-petency and capabilities of product innovation.Initial steps have been taken and some of theresults have been published to fill this observedgap (e.g. [119, 127]). Marijarvi et al. [128] re-port on Finnish large companies’ experience indeveloping new software through internal star-tups. They also discuss the lifecycle phases ofinnovation work in large companies. The authorsargue that different types of internal organiza-tion may take place in each stage of new productdevelopment. For example, problem/solution fitcan be done in an internal startup or companysubsidiary.

3.4.2. Lean Startup for Project PortfolioManagement and Apen Innovation

Building on the challenges proposed in Sec-tion 3.4.1, we propose that Lean startup couldalso be applied within both (i) project portfo-lio management (PPM), to co-ordinate multiplestartup initiatives within an organization, and(ii) open innovation, wherein internal startupsinvolve multiple organizations, individuals, oreven unknown participants. Both PPM and openinnovation and their main challenges are brieflyintroduced below, followed by research questionsthat require investigation before Lean startupprinciples can be successfully applied in thesenew contexts.

106 Michael Unterkalmsteiner et al.

Software engineering PPM describes the on-going identification, selection, prioritization, andmanagement of the complete set of an organiza-tion’s software engineering projects, which sharecommon resources in order to maximize returnsto the organization and achieve strategic busi-ness objectives [44, 129–131]. Open innovationis defined as the use of “purposive inflows andoutflows of knowledge to accelerate internal in-novation and to expand the markets for externaluse of innovation, respectively” [132]. Popularexamples of open innovation include open sourcesoftware development, crowd-sourcing, and innersource.

Effective PPM is critical to achieving busi-ness value [133, 134], improving cost and timesavings, and eliminating redundancies [135, 136].Unfortunately, existing portfolio managementpractices, which are based on the effective com-pletion of individual projects with only episodicportfolio level reviews [134], fail to manage eitherthe dynamic nature of contemporary projects, orproblems associated with portfolios comprisingtoo many projects [134, 137]. Indeed, many port-folios report an unwillingness to cancel projectsthat no longer contribute to the achievement ofstrategy [134].

Open innovation (OI) presents numerous ad-vantages for organizations, such as access toa requisite variety of experts, a prospectivereduction in overall R&D spending, reducedtime-to-market, improved software developmentprocesses, and the integration of the firm into newand collaborative value networks [132,138,139].Nonetheless, adopting open innovation processescan be significantly challenging. For example,adopters often lack internal commitment, in ad-dition to challenges associated with aligning in-novation strategies to extend beyond the bound-aries of the firm. Moreover, there are concernsregarding intellectual property and managingunknown contributors/contributions, as well asmanaging the higher costs and risks associatedwith managing both internal and external in-novations [140–142]. The role of Lean startupprinciples in addressing these challenges in bothPPM and OI is worthy of further research:

– RQ1: How can Lean startup be implementedwithin a portfolio management or open inno-vation context?

– RQ2: How can Lean startup initiatives driveor accelerate open innovation?

– RQ3: What Lean startup concepts could beadapted to facilitate open innovation pro-cesses in an organization?

– RQ4: How can one ensure Lean startup ini-tiatives conducted across multiple projects ororganizations align with strategy?

– RQ5: How do you reconcile potential conflictsbetween portfolio / open innovation processesand Lean startup processes?

– RQ6: How do you achieve consensus in defin-ing the minimum viable product (MVP) innetworks comprised of multiple autonomous(and sometime anonymous) agents?The successful application of Lean startup

principles (RQ1–RQ3) has the potential to re-duce the costs arising from the poor implemen-tation of PPM and OI practices and increase thevalue achieved from these initiatives. However,because such approaches are often practice led,it is necessary for academic research to developeffective theory to underpin practice and provideempirical data to support, or refute claims of ef-fectiveness (RQ4–RQ6). Rich human interactionsare at the heart of software engineering PPMand open innovation. Accordingly, phenomena inthese domains can be examined using interpre-tive, qualitative methods such as semi-structuredinterviews, case studies and ethnography.

While the principles of lean have been appliedto PPM (e.g. [143, 144], there is little researchlooking at the application of Lean startup princi-ples to PPM. Similarly, while there is interest inthe application of Lean startup principles in openinnovation contexts, to date, such applicationshave predominantly been driven by practice.

3.5. Software Startup Ecosystems andInnovation Hubs

Successful software startups do not live in iso-lation. Normally, they are inserted in a rich en-vironment that includes a number of relevant

Software Startups – A Research Agenda 107

players, such as entrepreneurs, developers, in-vestors, scientists, as well as business and intel-lectual property consultants. To support theseplayers, a number of support programs from theprivate and public sectors are required to providefunding, incubation, acceleration, training, net-working, and consulting. All these elements com-bine into what scholars and practitioners havecalled Startup Ecosystems [145]. In our softwarestartups research agenda, we focus on SoftwareStartup Ecosystems (SSE) and the elements thatare relevant for startups that have software asa key part of their products or services.

By studying how SSEs are created, their maincharacteristics, and how they can evolve, onecan better understand the environments thatfavour, or not, the birth and development ofsuccessful software startups. Research in thisfield can provide, to the relevant stakehold-ers, the concrete actions (e.g., public policies,private activities) that will establish a fruit-ful and vibrant environment for the execu-tion of high-growth innovative projects withinnascent software companies. The main researchquestions that need to be answered are thefollowing:– RQ1: What are the key elements of a fruitful

SSE?– RQ2: Are there different types of SSEs, e.g.

differentiated by size, technology sectors,country economy or other factors?

– RQ3: How do SSEs evolve over time?– RQ4: How can one measure the output and

qualities of an SSE?By answering RQ1, researchers will provide

a better understanding of the way how SSEsand innovation hubs work, instrumenting keystakeholders in taking actions to improve theirecosystems. By identifying what factors promoteor hinder the development of successful startupswithin a certain SSE, policy makers will get sup-port in decision making (RQ2). Entrepreneurswill also be able to better understand what arethe environmental factors and forces that canhelp or hinder the success of their enterprises.

Researchers from Brazil, Israel, and the USAhave developed a methodology to map a specificsoftware startup ecosystem; this methodology

has been applied to Israel [145], São Paulo [146]and New York [147]. Currently, with the help ofdozens of experts worldwide, they are developinga maturity model for SSEs [145, 148], address-ing RQ3 and RQ4. This maturity model needsfurther research and validation before it can beapplied in real scenarios to help practitioners andpolicy makers.

The Global Startup Ecosystem Ranking [149]is crafted by a group of experts that have beenproposing metrics to evaluate regional ecosys-tems around the world and compare them accord-ing to multiple criteria. Frenkel and Maital [150]have developed a methodology to map nationalinnovation ecosystems and use this map to pro-pose policies to promote improvement. Jayshreehas studied the influence of environmental fac-tors on entrepreneurial success [151]. Finally,Sternberg [152] researched the role of regionalgovernment support programs and the regionalenvironment as success factors for startups.

3.6. Theory and Methodologies forSoftware Startup Research

The tracks in this cluster direct their researchtowards identifying means to better study andunderstand software startups.

3.6.1. Overview of the Possible TheoreticalLenses for Studying Software Startups

Theories are important in any scientific field, asthey form the foundation to understand a con-temporary phenomenon better. Theories provideanswers to the “why” questions, and are thereforeuseful for explaining why certain events occurwhile others do not. Software startup researchdoes not operate in a vacuum, but rather canborrow theories from both the software engineer-ing and information systems fields, business andmanagement literature, as well as from the fieldsof organizational and social sciences.

We have identified a few potential theoriesthat can be meaningfully applied in the context ofsoftware startup companies. The proposed theo-ries are the hunter-gatherer model [153], Cynefinmodel [154], Effectuation theory [155] and Bound-

108 Michael Unterkalmsteiner et al.

ary Spanning theory [156]. These theories arebriefly outlined in this section.

Although 90% of human history was occu-pied by hunters and gatherers, who forged forwild plants and killed wild animal to survive,only recently was the hunter-gatherer modelre-discovered by Steinert and Leifer [153] to ex-plain how designers pursue their endeavours insearch of the best design outcome. The modelshows the changes in the design process, as well assubsequently in the design outcome. The modelportrays a distinction between a hunter whoaims to find an innovative idea, and a gath-erer who aims to implement the idea. Bothare needed to achieve concrete results. Whilehunting the idea through ambiguous spaces hasa change-driven, analytical, and qualitative na-ture; gathering the idea across predeterminedpaths has a plan-oriented, manageable, and quan-titative nature. The model has recently beenapplied in software startup research to explainstartups’ evolutionary paths [157].

Complexity theory has been used as a frameof reference, by analysing its implications on soft-ware design and development (e.g. Pelrine [158],Rikkilä et al. [159]). Software projects can becharacterized as endeavours wherein a dynamicnetwork of customers, software designers, devel-opers, 3rd party partners, and external stake-holders interact and can be seen as a Com-plex Adaptive System (CAS). To reason aboutdecision-making in different situations, Snowdenet al. [154] proposed a sense-making frameworkfor such systems. The model has five sub-domainsand divides the world in two parts – ordered andunordered main domains. The ordered domain isthe one in which cause-effect (CE) relationshipsare known (the Known domain), or at least know-able after analysis (the Complicate domain). Incontrast, the unordered domain includes a com-plexity situation, wherein the CE relationshipcan only be perceived in retrospect, but not in ad-vance (the Complex domain), and a chaotic situa-tion, wherein behaviours are completely random,lacking any expected consequence when actedupon. Depending on the problem domain, suit-able approaches include categorizing, analysing,probing or acting [154]. The Cynefin model pro-

vides a framework that can be used to analyse thedecisions made by software startuppers in devel-oping their products. Often they find themselvesin the unordered domain, attempting to makesense out of the current situation and navigateto the ordered domain.

Effectuation theory is a simple model, rootedin entrepreneurship, of decision-making underuncertainty. The effectual thinking is in the op-posite of causal reasoning which starts from de-sired ends to necessary means (top-down). Expe-rienced entrepreneurs reason from means to ends(bottom-up), trying to work out meanings andgoals based on the resources they have at hand.The theory is embodied by five principles: thebird-in-hand principle, the affordable loss princi-ple, the crazy quilt principle, the lemonade prin-ciple, and the pilot-in-the-plane principle [155].The effectuation theory can help to make bettersense of entrepreneurs’ decision-making processin the evolution of software startups, such asproblem validation, value proposition definition,design of MVPs, and pivoting processes. Goodpractices could be discovered using the effectua-tion theory as a theoretical lens.

Startups operate in a dynamic environmentand face expectations and influences from manydirections. In order to survive, they need to ef-fectively collaborate within their team, but alsooutside it. Boundary spanning is a concept thatdeals with the structures of organizations thatare transitioning from a rigid hierarchical struc-ture towards a network-based expert organiza-tion, which gives rise to informal boundariesrather than structural ones [156]. Boundary span-ners are those people and entities who bridgethese boundaries and opportunities. In the soft-ware engineering context, boundary spanning hasbeen studied in the context of global softwaredevelopment [160]. Startuppers can be seen asboundary spanners when they need to bridgebetween various stakeholders. While boundariesare always unavoidable, but also necessary anduseful, knowledge is required on how they canbe crossed, rearranged, or even dissolved whenconsidered harmful [161]. Startuppers should seeboundaries as tools that facilitate and supportmaking sense out of the environment. Boundary

Software Startups – A Research Agenda 109

spanning helps in discovering how to overcomethe challenges of distributed global work, wheremotivations, work styles, and knowledge domainsvary across boundaries. Startuppers can becomeknowledge brokers, transferring and sharing theirknowledge.

There are other theoretical lenses that can beused to study software startups. Startups dealwith innovative services and products, often fornew or emerging markets. Birkinshaw et al. [162]analyse the innovation theories presented andpropose a framework for management innovationprocess. This could be applied to the startup in-novation process context to explore how productdevelopment moves from problem-driven searchthrough trial and error to a finished prototype.The analysis can be complemented with Van deVen and Poole’s [163] four views into organiza-tional changes, in which they present alternateprocesses for organizations to transform.

Theorizing software startups is important,since there is a current lack of understanding ofthe dynamics in startups. Theoretical advance-ments need to be achieved so that researcherscan make better sense out the diverse contexts,situations, and places where startuppers strivefor success.

3.6.2. Defining the Lean Startup Concept andEvaluating Practice

Many positive drivers underpin the Lean Startupmovement. The literature is abound withclaims of reduced risk [13, 125], the benefitsof evidenced-based trials [13, 164], and shortertime-to-market [13]. We certainly know thatthese benefits are needed, given the challenges ex-perienced by early stage software startups [12,22]and the percentage that fail [13]. Indeed, manysoftware startups fail [108, 165] because theywaste too much time and money building thewrong product before realising too late whatthe right product should have been [102, 166].These challenges coupled with high uncertaintymake the Lean Startup Methodology attractiveto software startups as it supposedly offers anintegrated approach to creating products andservices that fit the market [167]. This research

builds on previous research conducted by Den-nehy, Kasraian, O’Raghallaigh, and Conboy [168],which identified a significant absence of frame-works that assisted startups to efficiently andeffectively progress their Minimum Viable Prod-ucts (MVP) to a Product Market Fit (PMV).The theoretical advancement of the lean conceptin contemporary software engineering and soft-ware development literature has been arrested,mainly because the academic research commu-nity has followed “fads and fancies” which char-acterize academic research. The implications forthe arrested theoretical development of lean con-cept, listed next, are the motivation for this re-search.

As is often the case with new and emerg-ing phenomena, Lean Startup practice has ledresearch, with the creation, promotion, and dis-semination of these methods almost completelydue to the efforts of practitioners and consul-tants. Now, Lean Startup research is beginningto gain momentum, as is evident from the in-creasing number of dedicated journal special is-sues, conferences, conference tracks, and work-shops. While there are merits to adopting sucha practice-oriented focus, little if any researcheffort has focused on the conceptual developmentof Lean Startup and its underlying components.As practice has lead research, the definition ofLean Startup has emerged through how it is usedin practice. As a result, Lean Startup adoptionis often defined by how the practices are adheredto, rather than the value gleaned from their use,adaptation, or, in some cases, abandonment. Wesee this in many other methods such as in agile,where many define “being agile” as how manyScrum or XP practices are used, rather than thevalue obtained by their use [169]. As a result,the current body of software startup knowledgesuffers from a number of limitations, including:

1. Lack of clarity: While there is broad agree-ment in principle regarding what constitutes keyconcepts such as MVP, assumptions regardingthe specific definitions, interpretations, use, andevaluations are often unclear in many existingLean Startup studies. This makes critical ap-praisal, evidence-based evaluation, and compari-son across studies extremely difficult.

110 Michael Unterkalmsteiner et al.

2. Lack of cohesion and cumulative tradition:A good concept or theory should cumulativelybuild on existing research. Very little academicresearch has examined Lean Startup using con-cepts that have more mature and substantivebodies of research with theories, frameworks andother lenses that have been thoroughly testedover time. The lean concept has been appliedin manufacturing since WW1, and yet in LeanStartup research we see very myopic and limiteduse of the broad lean frameworks available. Otherconcepts that influence Lean Startup includeagility, flow, and innovation.

3. Limited applicability: Adherence-basedmeasures of Lean Startup inhibit the ability toapply Lean Startup in domains other than thatoriginally intended. Research now attempts to ap-ply Lean Startup in other environments, such aslarge organizations and regulated environments,and so this will become a more prevalent issueas this trend continues. Therefore, questions rel-evant for this research track include:– RQ1: What are the core concepts that under-

pin Lean Startup?– RQ2: What are the components of a higher

abstract Lean Startup that allows the conceptto be applied and evaluated in a value-basedmanner?

– RQ3: What theories, frameworks, metrics,and other instruments from these existingrelated bodies of knowledge can be appliedto Lean Startup?

– RQ4: How can these be effectively appliedto improve the use of Lean Startup in prac-tice, and the study and improvement of LeanStartup in research?

– RQ5: How can Lean Startup then be tailoredto suit environments it was not originallydesigned to support, e.g. large organizations,regulated environments, or peer production?

– RQ6: Does Lean Startup enable or inhibitfundamental leaps in business and softwarebusiness ideas? For example, does MVP placean invisible ceiling, wherein once you reachMVP you subconsciously stop looking for thetruly significant innovation?

As there is reciprocal relationship between prac-tice and academia, where academic research is

informed by practice and practice is informedby academic research, this research would im-pact on research and on practice. By answeringRQ4–RQ6, this research track would providepractice with empirical evidence on the utilityof lean practices in diverse environments, whilealso positioning the lean method at the core ofacademic research (RQ1–RQ3). As case studyresearch is an empirical inquiry that “investigatesa contemporary phenomenon in depth and withinits real-life context” [62], it would be highlysuited to addressing the theoretical limitationsof lean and for answering the questions listedabove. Specifically, the use of a multiple-case de-sign would allow a cross-case pattern to developmore sophisticated descriptions and powerful ex-planations [170] of the lean concept.

The challenges of new product developmentare not confined to software startups. There-fore, software engineering teams working in dis-tributed or regulated environments such as finan-cial services and within multinational companieswould provide rich insights to the advancementof the lean concept.

3.6.3. Research Collaboration Strategies withSoftware Startups

Empirical research in the area of software engi-neering normally requires access to organizationsand artefacts from companies developing soft-ware intensive products and services [171]. In thecase of startups, such access is very limited, dueto several challenges:1. startups have limited resources both in terms

of person hours and calendar time for any-thing but working on their MVP,

2. startups want all investments to yield al-most immediate results, thus investments inlong-term potential are not prioritized, and

3. artefacts and actual products are often verysensitive, as the startup is very vulnerable.

These and other reasons limit empirical research,as reflected in both academic knowledge aboutstartups overall, but also in the superficial na-ture of what is available. For this reason, anyinitiative to seriously collect empirical data aswell as conduct research on core challenges facing

Software Startups – A Research Agenda 111

startups has to originate with a strategy thatovercomes these obstacles. One possible strategyis to pool resources and access to startups, inessence sharing empirical data and coordinatingresearch into startup software engineering. Coor-dination should be seen as equally central, as itenables researchers to limit the impact and costsas each study and project part can be focusedand small, and several larger issues can be tackledthrough coordination. Concrete examples of jointactivities include, but are not limited to:1. joint surveys at the superficial level (pooling

resources to collect many data points),2. complementary surveys and case studies

where each partner does a part only, butthe results can be combined in analysis andsynthesis,

3. formulating a complementary researchagenda with clear interfaces and joint re-search questions, and

4. pooling resources in relation to testing “solu-tions” emerging from the collaboration.While this strategy opens the possibility to

share the resource requirements among the stud-ied startups, there are open questions regardingits implementation:– RQ1: To what extent is data from different

startups and startup ecosystems comparable?In other words, which techniques exist toperform meta-analysis of the gathered het-erogeneous data?

– RQ2: How can we efficiently transfer technol-ogy between researchers and startups, andhow can we measure the impact of transferredsolutions?We conjecture that the software startup con-

text model discussed in Section 3.1.1 would bean enabler for answering RQ1. Confounding vari-ables [172] could then be easier identified, allow-ing for sample stratification and robust statisticalanalyses [173]. In particular, data collected fromdifferent researchers could be aggregated and in-crease the strength of the conclusions drawn fromthe analysis, i.e. enabling meta-analysis [174].

Answering RQ2 would allow us to actuallysupport software startups on a broad basis withthe knowledge gained from the research proposedin this agenda. While different approaches exist

to transfer knowledge from academia to indus-try [175,176], they are mostly targeted at maturecompanies that have the resources to collaboratewith researchers over a longer period of time. Wethink that software startup ecosystems, discussedin Section 3.5, can contribute to technology trans-fer if researchers are active in these structuresand can create a win-win situation where bothstartups and researchers benefit.

4. Discussion

In this section we give a brief overview of the re-search tracks in relation to other work in softwareengineering and their potential impact on thefield. We conclude this section with a discussionon the study’s limitations.

Software startup engineering research centersaround the core knowledge base in Software En-gineering [177]. This is illustrated by the researchtracks proposed in Section 3.1 that encompassproviding support for startup engineering activi-ties. Noticing what is considered “good” softwareengineering practice [177], and the challengesthat software startups encounter [12, 24], we seepotential in directing research towards efficientand effective requirements for engineering prac-tices in startups. Klotins et al. [24] studied 88 ex-perience reports from startups and identified lackof requirements validation, classification (to en-able prioritization), and identification of require-ments sources (to identify a relevant value propo-sition) as causes for engineering uncertainty,which maps to the early-stage startup challengesof technology uncertainty and delivering cus-tomer value, identified by Giardino et al. [12].Unlike large companies, software startups haveunique time and resource constraints and thuscannot afford to develop features and servicesthat will not be used or valued by the customers.We believe that lightweight practices to identify,and, most importantly, analyse requirements fortheir business value can help software startupsin their decision process. Looking at the researchtracks in Section 3.1, several of them touch uponrequirements engineering aspects. Prototypes canbe used to communicate with customers to elicit

112 Michael Unterkalmsteiner et al.

requirements (Section 3.1.4), while product inno-vation assessment (Section 3.1.3) is relevant inthe context of analysing the customers’ perceivedvalue of the offered solutions. Even optimizingthe effort spent in requirements engineering andquality assurance, for example by using test casesas requirements [178], involving product users fortesting (Section 3.1.7), addresses requirementsengineering aspects.

The focus on requirements in software startupengineering research directly relates to the re-search tracks presented in Section 3.2, startupevolution models and patterns, as the cost of piv-oting could be reduced by earlier and less ad-hocanalysis of requirements and value propositionsof the envisioned products. The patterns emerg-ing from the research on survival capabilitiesof software startups, proposed in Section 3.2.2,could provide valuable heuristics leading toa lightweight analysis of product value propo-sitions. The research on pivoting and survivalcapabilities is likely to affect software startuppractitioners on a strategic level by providingthem managerial decision support that drawsfrom models rooted in software engineering prac-tice. An example where such a cross-disciplineapproach has been very successful is value-basedsoftware engineering [179].

The research tracks described in Section 3.3were grouped under the name “cooperative andhuman aspects in software startups”, borrowedfrom the research area in software engineer-ing that is interested in studying the impactof cognitive abilities, team composition, work-load, informal communication, expertise identi-fication and other human aspects on softwareconstruction [180]. We conjecture that study-ing and understanding these aspects better hasa large potential as software startups are drivenby motivated individuals rather than a corporateagenda. Lessons from this research can both bene-fit startup practitioners, in particular in conjunc-tion with the work on software startups ecosys-tems (Section 3.5), and more mature companies,for example by applying models of competencyneeds that could emerge from the work presentedin Section 3.3.1.

The remaining research tracks described inSections 3.4 - 3.5 take a step back from what

happens inside a software startup. The researchtracks in Section 3.4 propose to apply startupconcepts in non-startup contexts. The idea ofextracting a concept from one context andapplying it in another has proven successfulin other areas, such as in systematic litera-ture reviews [181, 182] and open source princi-ples [183–185]. The premise of internal startupsis that the positive traits of “startups in the wild”can be transferred to a corporate environment,fostering innovation and faster product develop-ment. The overall aim of the research tracks de-scribed in Section 3.4 is to evaluate whether thetraits of startups can actually produce thrivingenvironments within mature companies. In com-parison, the research on startup ecosystems andinnovation hubs (Section 3.5) takes a broader andhigher level view of software startup phenomenon.Neither independent startups nor mature com-panies adopting internal startup initiatives livein isolation. A better understanding of startupecosystems and innovation hubs might therebyprovide key insights into the factors that createa fruitful software startup environment.

Finally, the research tracks in Section 3.6 lookat aspects relevant for implementing the researchagenda described in this paper. In particular, the-ories that can be used to better understand thedynamics in and around software startups areof value when attempting to construct a moreholistic understanding of software startups intheir various contexts. For the research on defin-ing the Lean Startup concept, parallels to andlessons from similar endeavours around researchon agile software development [186] should betaken into consideration. In this paper, we fol-lowed a recommendation by Dybå and Dingsøyrto develop a research agenda on the phenomenonof interest [186]. However, in order to implementthis research agenda, we need to also answerthe questions about how to enable efficient andeffective research collaborations with softwarestartups (Section 3.6.3).

4.1. Limitations

The research agenda presented in this paper wasdeveloped “bottom-up”, i.e. the areas of interestwere proposed and described by a sample of soft-

Software Startups – A Research Agenda 113

ware startup researchers without any restrictionon covering certain aspects of the software engi-neering body of knowledge but guided by theirpast, current and future work in the field. Often,these researchers have both a leg in academiaand in the startup community, either as men-tors, founders, or simply as part of the develop-ment team. This approach to develop a researchagenda is not uncommon (see e.g. [187–189]),but is threatened by a potential bias towardsthe preferences of individual researchers. Thisis why we invited a large number of our peersto contribute to the agenda. Even though theresearch tracks cover many software engineeringaspects and beyond, the agenda is only a sam-ple of the potentially relevant future researchon software startups. This means that poten-tially interesting and relevant research topics,such as use of open source software, businessmodel development, legal issues and intellectualproperty rights, are not discussed in this paper.However, we expect that the agenda will growtogether with the research community as soon asthe work on the proposed research tracks bearsfruits, leading to new research questions.

5. Outlook and Conclusions

Software startups are an interesting and stimu-lating phenomenon in the modern economy andare of paramount importance for the societies oftoday. Despite of high failure rates, communities,cities and countries are investing on stimulatingthe creation of software startups. While thesestartups may not solve the unemployment prob-lems of many countries they stimulate a new typeof positive dynamism in societies encouragingpeople to collaborate and develop their personalskills in novel ways. The emergence of the soft-ware startup research area reflects the fact thatwe need to better understand this phenomenonto learn valuable lessons and accumulate validknowledge to benefit future entrepreneurial ini-tiatives. The research agenda described in thispaper is one of the first attempts to establish thesoftware startup as a nascent, yet fast growingresearch area, and to depict its landscape by

highlighting the interesting research topics andquestions to explore.

It is worth emphasizing again that softwareengineering is only one of the multiple disciplinesthat are relevant and can inform software startuppractice. Other disciplines include Economics,Entrepreneurship, Design, Finance, Sociology,and Psychology. Therefore, there is a need to col-laborate with researchers from these disciplinesin order to increase the potential of achieving rel-evant and useful research results that can benefitpractice.

Due to the emerging nature of the field, thereis still much to be done to establish software star-tups as a research area. Relevant concepts needclear definitions, substantive theories need to bedeveloped, and initial research findings need tobe validated by future studies. Software startupsare very diversified in terms of entrepreneurs’varying approaches to their startup endeavours.Without the sound foundation mentioned abovefor this research area, there are risks of askingirrelevant research questions and not being ableto attain rigorous results.

Last but not least, this research agenda isnot meant to be exhaustive, and we are awarethat we may exclude some important SoftwareEngineering topics relevant to software startups.The research agenda is open to additions of newtracks, topics, and research questions by otherresearchers interested in the research area. Withcontributions and commitments from researchersfrom different institutions and backgrounds, col-lectively we can establish software startup asa promising and significant research area thatattracts more exciting discovery and contribu-tion. We welcome those interested in joining theSoftware Startup Research Network in fosteringthe collaboration between researchers and takingthe research agenda further.

References

[1] 42010-2011ISO/IEC/IEEE Systems and soft-ware engineering – Architecture description,ISO Std., 2011.

[2] N. Paternoster, C. Giardino, M. Unterkalm-steiner, T. Gorschek, and P. Abrahamsson,“Software Development in Startup Companies:

114 Michael Unterkalmsteiner et al.

A Systematic Mapping Study,” Informationand Software Technology, Vol. 56, No. 10, 2014,pp. 1200–1218.

[3] “Frequently asked questions about small busi-ness,” U.S. Small Business Administration,Tech. Rep., 2014.

[4] S. Nambisan, K. Lyytinen, A. Majchrzak, andM. Song, “Digital Innovation Management:Reinventing Innovation Management Researchin a Digital World,” MIS Quarterly, 2016, inpress.

[5] “A cambrian moment cheap and ubiquitousbuilding blocks for digital products and serviceshave caused an explosion in startups. specialreport: Tech startups,” The Economist, 18/012014.

[6] WMF, “Intelligent assets unlocking the circulareconomy potential,” World Economic Forum,Tech. Rep., December 2015.

[7] S. Srinivasan, I. Barchas, M. Gorenberg, andE. Simoudis, “Venture Capital: Fueling the In-novation Economy,” Computer, Vol. 47, No. 8,2014, pp. 40–47.

[8] A. Shontell, “The 11 most disrup-tive startups,” Business Insider, 12/072012. [Online]. http://www.businessinsider.com/disruptive-startups-2012-7?op=1&IR=T

[9] C. Giardino, N. Paternoster, M. Unterkalm-steiner, T. Gorschek, and P. Abrahamsson,“Software Development in Startup Companies:The Greenfield Startup Model,” Transactionson Software Engineering, Vol. 42, No. 6, 2016,pp. 585–604.

[10] “Progress without profits. A flock of startupsis making cloud computing faster and moreflexible, but most of them will not survive,”The Economist, 19/09 2015.

[11] “Testing, testing,” The Economist, 18/01 2014.[12] C. Giardino, S.S. Bajwa, X. Wang, and

P. Abrahamsson, “Key Challenges inEarly-Stage Software Startups,” in Proceed-ings 16th International XP Conference (XP).Helsinki, Finland: Springer, 2015, pp. 52–63.[Online]. http://link.springer.com/chapter/10.1007/978-3-319-18612-2_5

[13] E. Ries, The lean startup: How today’s en-trepreneurs use continuous innovation to createradically successful businesses. Crown Books,2011.

[14] S. Blank, The Four Steps to the Epiphany:Successful Strategies for Products thatWin. Cafepress.com, 2005. [Online].http://www.amazon.com/The-Four-Steps-Epiphany-Successful/dp/0976470705

[15] S. Blank and B. Dorf, The Startup Owner’sManual: The Step-By-Step Guide for Buildinga Great Company. K & S Ranch, 2012.

[16] S.M. Sutton, “The Role of Process in a SoftwareStart-up,” IEEE Softw., Vol. 17, No. 4, 2000,pp. 33–39. [Online]. http://dx.doi.org/10.1109/52.854066

[17] E. Carmel, “Time-to-completion in softwarepackage startups,” in Proceedings 27th HawaiiInternational Conference on System Sciences(HICSS). IEEE, 1994, pp. 498–507. [On-line]. http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=323468

[18] O.P. Hilmola, P. Helo, and L. Ojala, “Thevalue of product development lead time insoftware startup,” System Dynamics Review,Vol. 19, No. 1, 2003, pp. 75–82. [Online].http://doi.wiley.com/10.1002/sdr.255

[19] G. Coleman and R.V. O’Connor, “An in-vestigation into software development pro-cess formation in software start-ups,” Jour-nal of Enterprise Information Management,Vol. 21, No. 6, 2008, pp. 633–648. [On-line]. http://www.emeraldinsight.com/10.1108/17410390810911221

[20] C. Giardino, M. Unterkalmsteiner, N. Paternos-ter, T. Gorschek, and P. Abrahamsson, “Whatdo we know about software development instartups?” IEEE Software, Vol. 31, No. 5, 2014,pp. 28–32.

[21] I.C. Macmillan, L. Zemann, and P. Subba-narasimha, “Criteria distinguishing success-ful from unsuccessful ventures in the venturescreening process,” Journal of Business Ven-turing, Vol. 2, No. 2, 1987, pp. 123–137.

[22] C. Giardino, X. Wang, and P. Abrahamsson,“Why early-stage software startups fail: A be-havioral framework,” in Proceedings 5th In-ternational Conference on Software Business(ICSOB). Paphos, Cyprus: Springer, 2014, pp.27–41. [Online]. http://link.springer.com/10.1007/978-3-319-08738-2_3

[23] Z. Block and I.C. MacMillan, “Milestones forsuccessful venture planning,” Harvard BusinessReview, Vol. 63, No. 5, 1985, pp. 184–196.

[24] E. Klotins, M. Unterkalmsteiner, andT. Gorschek, “Software Engineering inStart-up Companies: an Exploratory Study of88 Startups,” Empirical Software Engineering,2016, in Submission.

[25] A. Yau and C. Murphy, “Is a Rigorous Ag-ile Methodology the Best Development Strat-egy for Small Scale Tech Startups?” Tech-

Software Startups – A Research Agenda 115

nical Report MS-CIS-13-01, 2013. [Online].http://repository.upenn.edu/cis_reports/980

[26] E. Klotins, M. Unterkalmsteiner, andT. Gorschek, “Software engineering practices instart-up companies: A mapping study,” in 6thInternational Conference on Software Business.Springer, 2015, pp. 245–257.

[27] K. Petersen and C. Wohlin, “Context in In-dustrial Software Engineering Research,” inProceedings 3rd International Symposium onEmpirical Software Engineering and Measure-ment (ESEM). Orlando, USA: IEEE, 2009, pp.401–404.

[28] P. Clarke and R.V. O’Connor, “The situationalfactors that affect the software developmentprocess: Towards a comprehensive referenceframework,” Information and Software Tech-nology, Vol. 54, No. 5, 2012, pp. 433–447. [On-line]. http://www.sciencedirect.com/science/article/pii/S0950584911002369

[29] T. Dybå, D.I. Sjøberg, and D.S. Cruzes, “WhatWorks for Whom, Where, When, and Why?:On the Role of Context in Empirical Soft-ware Engineering,” in Proceedings Interna-tional Symposium on Empirical Software En-gineering and Measurement (ESEM). Lund,Sweden: ACM, 2012, pp. 19–28. [Online].http://doi.acm.org/10.1145/2372251.2372256

[30] D. Kirk and S.G. MacDonell, “Investigating aConceptual Construct for Software Context,”in Proceedings 18th International Conferenceon Evaluation and Assessment in Software En-gineering (EASE). London, UK: ACM, 2014,pp. 27:1–27:10. [Online]. http://doi.acm.org/10.1145/2601248.2601263

[31] D. Kirk and S. MacDonell, “Categorising Soft-ware Contexts,” in Proceedings 2014 Ameri-cas Conference on Information Systems (AM-CIS). Savannah, USA: AIS Electronic Li-brary, 2014. [Online]. http://aisel.aisnet.org/amcis2014/Posters/ITProjectManagement/8

[32] T. Feng, L. Sun, C. Zhu, and A.S. Sohal, “Cus-tomer orientation for decreasing time-to-marketof new products IT implementation as a com-plementary asset,” Industrial Marketing Man-agement, Vol. 41, No. 6, 2012, pp. 929–939.

[33] E. Tom, A. Aurum, and R. Vidgen, “An ex-ploration of technical debt,” Journal of Sys-tems and Software, Vol. 86, No. 6, 2013, pp.1498–1516. [Online]. http://www.sciencedirect.com/science/article/pii/S0164121213000022

[34] C. Fernández-Sánchez, J. Garbajosa, andA. Yagüe, “A framework to aid in decision mak-ing for technical debt management,” in Pro-

ceedings 7th International Workshop on Manag-ing Technical Debt (MTD). Bremen, Germany:IEEE, 2015, pp. 69–76.

[35] Z. Li, P. Avgeriou, and P. Liang, “A system-atic mapping study on technical debt andits management,” Journal of Systems andSoftware, Vol. 101, 2015, pp. 193–220. [On-line]. http://www.sciencedirect.com/science/article/pii/S0164121214002854

[36] E. Lim, N. Taksande, and C. Seaman, “A bal-ancing act: What software practitioners haveto say about technical debt,” IEEE Software,Vol. 29, No. 6, 2012, pp. 22–27.

[37] F. Shull, D. Falessi, C. Seaman, M. Diep, andL. Layman, “Technical Debt: Showing the Wayfor Better Transfer of Empirical Results,” inPerspectives on the Future of Software Engi-neering. Springer, 2013, pp. 179–190.

[38] F. Johne and P.A. Snelson, “Success factorsin product innovation: A selective review ofthe literature,” Journal of Product InnovationManagement, Vol. 5, No. 2, 1988, pp. 114–128.

[39] D. Lippoldt and P. Stryszowski, “Innovationin the software sector,” Organisation for Eco-nomic Co-operation and Development, Tech.Rep., 2009.

[40] H. Edison, N. bin Ali, and R. Torkar, “To-wards innovation measurement in the softwareindustry,” Journal of Systems and Software,Vol. 86, No. 5, 2013, pp. 1390–1407. [On-line]. http://www.sciencedirect.com/science/article/pii/S0164121213000058

[41] W. Eversheim, Innovation Management forTechnical Products: Systematic and IntegratedProduct Development and Production Planning.Springer Science & Business Media, 2008.

[42] M.M. Crossan and M. Apaydin, “A multi-di-mensional framework of organizational inno-vation: A systematic review of the literature,”Journal of Management Studies, Vol. 47, No. 6,2009, pp. 1154–1191. [Online]. http://doi.wiley.com/10.1111/j.1467-6486.2009.00880.x

[43] R. Balachandra and J. Friar, “Factors for suc-cess in R/D projects and new product inno-vation: a contextual framework,” IEEE Trans-actions on Engineering Management, Vol. 44,No. 3, 1997, pp. 276–287.

[44] R.G. Cooper, “From experience – the invisiblesuccess factors in product innovation,” Journalof Product Innovation Management, Vol. 16,No. 2, 1999, pp. 115–133.

[45] “Software engineering – software product qual-ity requirements and evaluation (SQuaRE) –guide to SQuaRE – ISO/IEC 25000:2005,” In-

116 Michael Unterkalmsteiner et al.

ternational Organization for Standardization,Tech. Rep., 2005.

[46] A. Yagüe, J. Garbajosa, J. Pérez, and J. Díaz,“Analyzing Software Product Innovation Assess-ment by Using a Systematic Literature Review,”in Proceedings 47th Hawaii International Con-ference on System Sciences (HICSS). Waikoloa,USA: IEEE, 2014, pp. 5049–5058.

[47] M. Pikkarainen, W. Codenie, N. Boucart, andJ.A. Heredia Alvaro, Eds., The Art of SoftwareInnovation. Berlin, Germany: Springer, 2011.

[48] H. Lichter, M. Schneider-Hufschmidt, andH. Züllighoven, “Prototyping in Industrial Soft-ware Projects–Bridging the Gap Between The-ory and Practice,” in Proceedings 15th In-ternational Conference on Software Engineer-ing (ICSE). Baltimore, USA: IEEE, 1993, pp.221–229.

[49] M. Beaudouin-Lafon and W.E. Mackay, “Pro-totyping development and tools,” Handbookof Human-Computer Interaction, 2002, pp.1006–1031.

[50] I. Sommerville, Software Engineering, 9th ed.Boston: Pearson, 2010.

[51] T. Brown, Change by Design: How DesignThinking Transforms Organizations and In-spires Innovation. New York: HarperBusiness,2009.

[52] A. Efeoğlu, C. Møller, and M. Sérié, “Solu-tion Prototyping with Design Thinking – SocialMedia for SAP Store: A Case Study,” in Pro-ceedings European Design Science Symposium(EDSS). Dublin, Ireland: Springer, 2013, pp.99–110.

[53] P. Newman, M.A. Ferrario, W. Simm, S. For-shaw, A. Friday, and J. Whittle, “The role ofdesign thinking and physical prototyping insocial software engineering,” 37th IEEE Inter-national Conference on Software Engineering,2015.

[54] C. Grevet and E. Gilbert, “Piggyback proto-typing: Using existing, large-scale social com-puting systems to prototype new ones,” inProceedings 33rd Annual ACM Conference onHuman Factors in Computing Systems. Seoul,Korea: ACM, 2015, pp. 4047–4056. [Online].http://doi.acm.org/10.1145/2702123.2702395

[55] A. Nguyen Duc and P. Abrahamsson, “Mini-mum viable product or multiple facet product?The Role of MVP in software startups,” inProceedings 17th International XP Conference.Edinburgh, UK: Springer, 2016.

[56] T. Raz and E. Michael, “Use and bene-fits of tools for project risk management,”

International Journal of Project Manage-ment, Vol. 19, No. 1, 2001, pp. 9–17. [On-line]. http://www.sciencedirect.com/science/article/pii/S0263786399000368

[57] L. Cocco, K. Mannaro, G. Concas, andM. Marchesi, “Simulating Kanban and Scrumvs. Waterfall with System Dynamics,” in Pro-ceedings 12th Internation XP Conference (XP).Madrid, Spain: Springer, 2011, pp. 117–131.[Online]. http://link.springer.com/10.1007/978-3-642-20677-1_9

[58] G. Concas, M.I. Lunesu, M. Marchesi, andH. Zhang, “Simulation of software maintenanceprocess, with and without a work-in-processlimit,” Journal of Software: Evolution and Pro-cess, Vol. 25, No. 12, 2013, pp. 1225–1248.[Online]. http://onlinelibrary.wiley.com/doi/10.1002/smr.1599/abstract

[59] H. Edison, D. Khanna, S.S. Bajwa, V. Branca-leoni, and L. Bellettati, “Towards a SoftwareTool Portal to Support Startup Process,” inProceedings 1st International Workshop on Soft-ware Startups. Bolzano, Italy: Springer, 2015,pp. 577–583. [Online]. http://link.springer.com/chapter/10.1007/978-3-319-26844-6_43

[60] J. Zettel, F. Maurer, J. Münch, and L. Wong,“LIPE: a lightweight process for e-businessstartup companies based on extreme program-ming,” in Proceedings 3rd International Con-ference on Product-Focused Software ProcessImprovement (PROFES). Kaiserslautern, Ger-many: Springer, 2001, pp. 255–270.

[61] M.D. Kelly, “Lessons Learned from SoftwareTesting at Startups,” in EuroStar-SoftwareTesting Conference, Amsterdam, The Nether-lands, 2012.

[62] R.K. Yin, Case Study Research: Design andMethods, 3rd ed. Sage Publications, 2003.

[63] Ergonomics of Human-system Interaction: Part210: Human-centred Design for Interactive Sys-tems, ISO Std., 2010.

[64] J. Füller, R. Schroll, and E. von Hippel, “Usergenerated brands and their contribution to thediffusion of user innovations,” Research Policy,Vol. 42, No. 6–7, 2013, pp. 1197–1209. [On-line]. http://www.sciencedirect.com/science/article/pii/S004873331300053X

[65] L. Hokkanen and K. Väänänen-Vainio-Mattila,“UX work in startups: current prac-tices and future needs,” in Proceedings16th International XP Conference (XP).Helsinki, Finland: Springer, 2015, pp. 81–92.[Online]. http://link.springer.com/chapter/10.1007/978-3-319-18612-2_7

Software Startups – A Research Agenda 117

[66] B. May, “Applying Lean Startup: An Experi-ence Report – Lean & Lean UX by a UX Vet-eran: Lessons Learned in Creating & Launchinga Complex Consumer App,” in Proceedings Ag-ile Conference (AGILE). Dallas, USA: IEEE,2012, pp. 141–147.

[67] M. Taipale, “Huitale–A Story of a Finnish LeanStartup,” in Proceedings 1st International Con-ference on Lean Enterprise and Software Sys-tems (LESS). Helsinki, Finland: Springer, 2010,pp. 111–114. [Online]. http://www.springerlink.com/index/w48n06l04712621p.pdf

[68] L. Hokkanen, K. Kuusinen, and K. Väänä-nen, “Early Product Design in Startups: To-wards a UX Strategy,” in Proceedings 16thInternational Conference on Product-FocusedSoftware Process Improvement (PROFES).Bolzano-Bozen, Italy: Springer, 2015, pp.217–224. [Online]. http://link.springer.com/chapter/10.1007/978-3-319-26844-6_16

[69] L. Hokkanen and M. Leppänen, “Three Pat-terns for User Involvement in Startups,” in Pro-ceedings 20th European Conference on PatternLanguages of Programs (EuroPLoP). KlosterIrsee, Germany: ACM, 2015, pp. 51:1–51:8.[Online]. http://doi.acm.org/10.1145/2855321.2855373

[70] L. Hokkanen, K. Kuusinen, and K. Väänänen,“Minimum viable user experience: A frameworkfor supporting product design in startups,” inProceedings 17th International XP Conference(XP). Edinburgh, Scotland: Springer, 2016, inpress.

[71] J. Nazar, “14 Famous Business Pivots,” Forbes,2013. [Online]. http://www.forbes.com/sites/jasonnazar/2013/10/08/14-famous-business-pivots/

[72] J. Bosch, V.D. Veen, and J. Salvador,“Pivots and Architectural Decisions: TwoSides of the Same Medal?” in ChalmersPublication Library (CPL), 2013, pp. 310–317.[Online]. http://publications.lib.chalmers.se/publication/189719-pivots-and-architectural-decisions-two-sides-of-the-same-medal

[73] H. Terho, S. Suonsyrjä, A. Karisalo, andT. Mikkonen, Ways to Cross the Rubicon: Piv-oting in Software Startups. Cham: SpringerInternational Publishing, 2015, pp. 555–568.[Online]. http://dx.doi.org/10.1007/978-3-319-26844-6_41

[74] S. Shahid Bajwa, X. Wang, A. Nguven Duc,and P. Abrahamsson, “How do software star-tups pivot? empirical results from a multiplecase study,” in 7th International Conference on

Software Business (ICSOB 2016), Ljubljana,Slovenia, 2016, pp. 169–176.

[75] H. Löfsten and P. Lindelöf, “Science Parksand the growth of new technology-basedfirms–academic-industry links, innovationand markets,” Research Policy, Vol. 31,No. 6, 2002, pp. 859–876. [Online].http://www.sciencedirect.com/science/article/pii/S0048733301001536

[76] J.S. Gans and S. Stern, “The product marketand the market for “ideas”: commercializa-tion strategies for technology entrepreneurs,”Research Policy, Vol. 32, No. 2, 2003, pp.333–350. [Online]. http://www.sciencedirect.com/science/article/pii/S0048733302001038

[77] A. Brem and K.I. Voigt, “Integration of mar-ket pull and technology push in the corpo-rate front end and innovation management– Insights from the German software indus-try,” Technovation, Vol. 29, No. 5, 2009, pp.351–367. [Online]. http://www.sciencedirect.com/science/article/pii/S0166497208000898

[78] A. Maurya, Running Lean: Iterate from PlanA to a Plan That Works. O’Reilly Media, Inc.,2012.

[79] K. Klyver and M.T. Schenkel, “From Re-source Access to Use: Exploring the Impactof Resource Combinations on Nascent En-trepreneurship,” Journal of Small BusinessManagement, Vol. 51, No. 4, 2013, pp. 539–556.[Online]. http://onlinelibrary.wiley.com/doi/10.1111/jsbm.12030/abstract

[80] J. Levie and B.B. Lichtenstein, “A Ter-minal Assessment of Stages Theory: Intro-ducing a Dynamic States Approach to En-trepreneurship,” Entrepreneurship Theory andPractice, Vol. 34, No. 2, 2010, pp. 317–350.[Online]. http://onlinelibrary.wiley.com/doi/10.1111/j.1540-6520.2010.00377.x/abstract

[81] F. Giones and F. Miralles, “Strategic Signalingin Dynamic Technology Markets: Lessons FromThree IT Startups in Spain,” Global Businessand Organizational Excellence, Vol. 34, No. 6,2015, pp. 42–50. [Online]. http://onlinelibrary.wiley.com/doi/10.1002/joe.21634/abstract

[82] B. Clarysse, J. Bruneel, and M. Wright,“Explaining growth paths of youngtechnology-based firms: structuring re-source portfolios in different competitiveenvironments,” Strategic EntrepreneurshipJournal, Vol. 5, No. 2, 2011, pp. 137–157.[Online]. http://onlinelibrary.wiley.com/doi/10.1002/sej.111/abstract

118 Michael Unterkalmsteiner et al.

[83] S.L. Newbert and E.T. Tornikoski, “Supporternetworks and network growth: a contingencymodel of organizational emergence,” SmallBusiness Economics, Vol. 39, No. 1, 2010,pp. 141–159. [Online]. http://link.springer.com/article/10.1007/s11187-010-9300-9

[84] T. Semrau and S. Sigmund, “NetworkingAbility and the Financial Performance ofNew Ventures: A Mediation Analysis amongYounger and More Mature Firms,” Strategic En-trepreneurship Journal, Vol. 6, No. 4, 2012, pp.335–354. [Online]. http://onlinelibrary.wiley.com/doi/10.1002/sej.1146/abstract

[85] P. Witt, “Entrepreneurs’ networks and thesuccess of start-ups,” Entrepreneurship & Re-gional Development, Vol. 16, No. 5, 2004, pp.391–412. [Online]. http://dx.doi.org/10.1080/0898562042000188423

[86] K.M. Eisenhardt and J.A. Martin, “Dy-namic capabilities: what are they?”Strategic Management Journal, Vol. 21,No. 10-11, 2000, pp. 1105–1121. [Online].http://onlinelibrary.wiley.com/doi/10.1002/1097-0266(200010/11)21:10/11<1105::AID-SMJ133>3.0.CO;2-E/abstract

[87] D.J. Teece, G. Pisano, and A. Shuen,“Dynamic capabilities and strategic man-agement,” Strategic Management Journal,Vol. 18, No. 7, 1997, pp. 509–533. [Online].http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1097-0266(199708)18:7<509::AID-SMJ882>3.0.CO;2-Z/abstract

[88] S.L. Newbert, S. Gopalakrishnan, and B.A.Kirchhoff, “Looking beyond resources: Ex-ploring the importance of entrepreneurshipto firm-level competitive advantage in tech-nologically intensive industries,” Technova-tion, Vol. 28, No. 1–2, 2008, pp. 6–19. [On-line]. http://www.sciencedirect.com/science/article/pii/S0166497207000910

[89] B.B. Lichtenstein, K.J. Dooley, and G.T. Lump-kin, “Measuring emergence in the dynamicsof new venture creation,” Journal of Busi-ness Venturing, Vol. 21, No. 2, 2006, pp.153–175. [Online]. http://www.sciencedirect.com/science/article/pii/S0883902605000376

[90] C.G. Brush, T.S. Manolova, and L.F. Edel-man, “Properties of emerging organizations:An empirical test,” Journal of Business Ventur-ing, Vol. 23, No. 5, 2008, pp. 547–566. [On-line]. http://www.sciencedirect.com/science/article/pii/S0883902607000602

[91] J. Katz and W.B. Gartner, “Properties ofEmerging Organizations,” Academy of Manage-

ment Review, Vol. 13, No. 3, 1988, pp. 429–441.[Online]. http://amr.aom.org/content/13/3/429

[92] B. Honig and T. Karlsson, “Institutional forcesand the written business plan,” Journal ofManagement, Vol. 30, No. 1, 2004, pp. 29–48.[Online]. http://jom.sagepub.com/content/30/1/29

[93] D. Kirsh, B. Goldfarb, and A. Gera, “Firm orsubstance: the role of business plans in ven-ture capital decision making process,” Strate-gic Management Journal, No. 30, 2009, pp.487–515.

[94] T. Karlsson and B. Honig, “Judging a businessby its cover: An institutional perspective onnew ventures and the business plan,” Journalof Business Venturing, Vol. 24, No. 1, 2009, pp.27–45. [Online]. http://www.sciencedirect.com/science/article/pii/S0883902607000791

[95] M. König, G. Baltes, and B. Katzy, “On therole of value-network strength as an indica-tor of technology-based venture’s survival andgrowth: Increasing innovation system efficiencyby leveraging transaction relations to prioritizeventure support,” in Proceedings InternationalConference on Engineering, Technology andInnovation/ International Technology Manage-ment Conference (ICE/ITMC). IEEE, 2015, pp.1–9. [Online]. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6904149

[96] K. Dovan, “Reliability in content analysis:Some common misconceptions and recommen-dations,” Human Communication Research,Vol. 30, No. 3, 1998, pp. 411–433.

[97] S. Elo and H. Kyngäs, “The qualitative con-tent analysis process,” Journal of AdvancedNursing, Vol. 62, No. 1, 2008, pp. 107–115.[Online]. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2648.2007.04569.x/abstract

[98] M. König, C. Ungerer, R. Büchele, andG. Baltes, “Agreement on the Venture’s RealityPresented in Business Plans,” in Proceedings22nd International Conference on Engineering,Technology and Innovation (ICE). Trondheim,Norway: IEEE, 2016.

[99] C. Ungerer, M. König, F. Giones, and G. Baltes,“Measuring Venture Emergence and Survival byAnalyzing Transaction Relations in BusinessPlans,” in Proceedings 22nd International Con-ference on Engineering, Technology and Inno-vation (ICE). Trondheim, Norway: IEEE, 2016.

[100] S. Marlow, “Human resource manage-ment in smaller firms: A contradiction interms?” Human Resource Management Re-

Software Startups – A Research Agenda 119

view, Vol. 16, No. 4, 2006, pp. 467–477. [On-line]. http://www.sciencedirect.com/science/article/pii/S1053482206000684

[101] S. Jack, J. Hyman, and F. Osborne,“Small entrepreneurial ventures culture, changeand the impact on HRM: A critical re-view,” Human Resource Management Review,Vol. 16, No. 4, 2006, pp. 456–466. [On-line]. http://www.sciencedirect.com/science/article/pii/S1053482206000672

[102] J. Bosch, H.H. Olsson, J. Björk, and J. Ljung-blad, “The Early Stage Software Startup Devel-opment Model: A Framework for Operational-izing Lean Principles in Software Startups,” inProceedings 4th International Conference onLean Enterprise Software and Systems (LESS).Galway, Ireland: Springer, 2013, pp. 1–15.

[103] T. Dingsøyr and Y. Lindsjørn, “Team perfor-mance in agile development teams: Findingsfrom 18 focus groups,” in Proceedings 14thInternational Conference on Agile Software De-velopment, Vienna, Austria, 2013, pp. 46–60.[Online]. http://link.springer.com/10.1007/978-3-642-38314-4

[104] J.R. Katzenbach and D.K. Smith, “The dis-cipline of teams.” Harvard Business Review,Vol. 71, No. 2, 1993, pp. 111–120.

[105] O. Oechslein and A. Tumasjan, “Examin-ing trust within the team in it startupcompanies–an empirical study in the people’srepublic of china,” in Proceedings 45th HawaiiInternational Conference on System Science(HICSS), Maui, USA, 2012, pp. 5102–5111.[Online]. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6149511

[106] J. Kamm, J. Shuman, J. Seeger, and A. Nurick,“Entrepreneurial teams in new venture creation:A research agenda,” Entrepreneurship Theoryand Practice, Vol. 14, No. 4, 1990, pp. 7–17.

[107] D. Francis and W. Sandberg, “Friendshipwithin entrepreneurial teams and its associ-ation with team and venture performance,”Entrepreneurship: Theory and Practice,Vol. 25, No. 2, 2000, pp. 5–21. [Online].http://go.galegroup.com/ps/i.do?id=GALE%7CA74524630&sid=googleScholar&v=2.1&it=r&linkaccess=fulltext&issn=10422587&p=AONE&sw=w

[108] M. Crowne, “Why software product startupsfail and what to do about it,” in InternationalEngineering Management Conference (IEMC).Cambridge, UK: IEEE, 2002, pp. 338–343.

[109] M.D. Ensley, K.M. Hmieleski, and C.L.Pearce, “The importance of vertical and

shared leadership within new venture top man-agement teams: Implications for the perfor-mance of startups,” The Leadership Quar-terly, Vol. 17, No. 3, 2006, pp. 217–231. [On-line]. http://www.sciencedirect.com/science/article/pii/S1048984306000051

[110] R.G. Cooper, “Perspective: The innovationdilemma: How to innovate when the market ismature,” Journal of Product Innovation Man-agement, Vol. 28, No. SUPPL. 1, 2011, pp. 2–27.

[111] C.K. Bart, “New venture units: use them wiselyto manage innovation,” Sloan Management Re-view, Vol. 29, No. 4, 1988, pp. 35–43.

[112] S.A. Hill and J. Birkinshaw, “Ambidexterityand survival in corporate venture units,” Jour-nal of Management, Vol. 40, No. 7, 2014, pp.1899–1931. [Online]. http://jom.sagepub.com/cgi/doi/10.1177/0149206312445925

[113] J.J.P. Jansen, M.P. Tempelaar, F.A.J. van denBosch, and H.W. Volberda, “Structural differ-entiation and ambidexterity: The mediatingrole of integration mechanisms,” OrganizationScience, Vol. 20, No. 4, 2009, pp. 797–811. [On-line]. http://pubsonline.informs.org/doi/abs/10.1287/orsc.1080.0415

[114] R. Nanda and M. Rhodes-Kropf, “Investmentcycles and startup innovation,” Journal of Fi-nancial Economics, Vol. 110, No. 2, 2013, pp.403–418.

[115] D. Lavie, U. Stettner, and M.L. Tushman, “Ex-ploration and exploitation within and acrossorganizations,” The Academy of ManagementAnnals, Vol. 4, No. 1, 2010, pp. 109–155. [On-line]. http://www.tandfonline.com/doi/abs/10.1080/19416521003691287

[116] J. Lerner, “Corporate venturing,” HarvardBusiness Review, Vol. 91, No. 10, 2013, pp.86–94.

[117] C.A. O’Reilly and M.L. Tushman, “Organiza-tional ambidexterity: Past, present, and future,”Academy of Management Perspectives, Vol. 27,No. 4, 2013, pp. 324–338. [Online]. http://amp.aom.org/cgi/doi/10.5465/amp.2013.0025

[118] D.A. Garvin and L.C. Levesque, “Meeting thechallenge of corporate entrepreneurship.” Har-vard business review, Vol. 84, No. 10, 2006, pp.102–12, 150. [Online]. http://www.ncbi.nlm.nih.gov/pubmed/17040043

[119] H. Edison, X. Wang, and P. Abrahamsson,“Lean startup,” in Scientific Workshop Proceed-ings of the XP conference. Helsinki, Finland:ACM, 2015, pp. 1–7. [Online]. http://dl.acm.org/citation.cfm?doid=2764979.2764981

120 Michael Unterkalmsteiner et al.

[120] C.J. Chen, “Technology commercialization, in-cubator and venture capital, and new ventureperformance,” Journal of Business Research,Vol. 62, No. 1, 2009, pp. 93–103.

[121] S. Coleman, C. Cotei, and J. Farhat, “Aresource-based view of new firm survival: newperspectives on the role of industry and exitroute,” Journal of Developmental Entrepreneur-ship, Vol. 18, No. 01, 2013, pp. 1–25. [On-line]. http://www.worldscientific.com/doi/abs/10.1142/S1084946713500027

[122] G.G. Dess, R.D. Ireland, S.A. Zahra, S.W.Floyd, J.J. Janney, and P.J. Lane, “Emerging is-sues in corporate entrepreneurship,” Journal ofManagement, Vol. 29, No. 3, 2003, pp. 351–378.[Online]. http://jom.sagepub.com/cgi/doi/10.1016/S0149-2063_03_00015-1

[123] A. Croll and B. Yoskovitz, Lean Analytics: UseData to Build a Better Startup Faster, 1st ed.Sebastopol, USA: O’Reilly Media, 2013.

[124] P. Tyrväinen, M. Saarikallio, T. Aho, T. Lehto-nen, and R. Paukeri, “Metrics Frame-work for Cycle-Time Reduction in SoftwareValue Creation,” in Proceedings 10th Inter-national Conference on Software Engineer-ing Advances (ICSEA). Barcelona, Spain:IARIA, 2015. [Online]. https://jyx.jyu.fi/dspace/handle/123456789/47980

[125] T.R. Eisenmann, E. Ries, and S. Dillard,“Hypothesis-driven entrepreneurship: The leanstartup,” Harvard Business School, 2012.

[126] J. Järvinen, T. Huomo, T. Mikkonen, andP. Tyrväinen, “From Agile Software Develop-ment to Mercury Business,” in Proceedings 5thInternational Conference on Software Business(ICSOB). Paphos, Cyprus: Springer, 2014, pp.58–71. [Online]. http://link.springer.com/10.1007/978-3-319-08738-2_5

[127] H. Edison, “A conceptual framework oflean startup enabled internal corporate ven-ture,” in Proceedings 1st International Work-shop on Software Startups. Bolzano-Bozen,Italy: Springer, 2015, pp. 607–613. [On-line]. http://link.springer.com/10.1007/978-3-319-26844-6_46

[128] J. Märijärvi, L. Hokkanen, M. Komssi, H. Kil-jander, Y. Xu, M. Raatikainen, P. Seppänen,J. Heininen, M. Koivulahti-Ojala, M. Helenius,and J. Järvinen, The Cookbook for SuccessfulInternal Startups. DIGILE and N4S, 2016.

[129] S. Meskendahl, “The influence of business strat-egy on project portfolio management and itssuccess – a conceptual framework,” Interna-

tional Journal of Project Management, Vol. 28,No. 8, 2010, pp. 807–817.

[130] B.S. Blichfeldt and P. Eskerod, “Project portfo-lio management – there’s more to it than whatmanagement enacts,” International Journal ofProject Management, Vol. 26, No. 4, 2008, pp.357–365.

[131] J.R. Turner, The handbook of project-basedmanagement. McGraw-Hill, 2014.

[132] H.W. Chesbrough, Open innovation: The newimperative for creating and profiting from tech-nology. Harvard Business Press, 2006.

[133] T. Hatzakis, M. Lycett, and A. Serrano, “Aprogramme management approach for ensur-ing curriculum coherence in is (higher) educa-tion,” European Journal of Information Sys-tems, Vol. 16, No. 5, 2007, pp. 643–657.

[134] B.D. Reyck, Y. Grushka-Cockayne, M. Lockett,S.R. Calderini, M. Moura, and A. Sloper, “Theimpact of project portfolio management on in-formation technology projects,” InternationalJournal of Project Management, Vol. 23, No. 7,2005, pp. 524–537.

[135] B. Kersten and C. Verhoef, “IT portfolio man-agement: A banker’s perspective on IT,” CutterIT Journal, 2003. [Online]. http://www.few.vu.nl/~x/bp/bp.pdf

[136] R. LeFave, B. Branch, C. Brown, andB. Wixom, “How sprint nextel reconfigured itresources for results,”MIS Quarterly, 2008. [On-line]. http://misqe.org/ojs2/index.php/misqe/article/view/213

[137] J. Krebs, Agile portfolio management, 1st ed.Microsoft Press, 2008.

[138] Y.Z. Anbardan and M. Raeyat, “Open innova-tion: Creating value through co-creation,” inProceedings 7th World Conference on Mass Cus-tomization, Personalization, and Co-Creation(MCPC 2014). Aalborg, Denmark: Springer,2014, pp. 437–447.

[139] W. Vanhaverbeke and M. Cloodt, Open Inno-vation: Researching a New Paradigm. OxfordUniversity Press, 2008, ch. Open innovation invalue networks, pp. 258–284.

[140] V. Van de Vrande, J.P. De Jong, W. Vanhaver-beke, and M. De Rochemont, “Open innovationin smes: Trends, motives and management chal-lenges,” Technovation, Vol. 29, No. 6, 2009, pp.423–437.

[141] J. West and S. Gallagher, “Challenges ofopen innovation: the paradox of firm invest-ment in open-source software,” R&D Manage-ment, Vol. 36, No. 3, 2006, pp. 319–331. [On-

Software Startups – A Research Agenda 121

line]. http://dx.doi.org/10.1111/j.1467-9310.2006.00436.x

[142] H. Chesbrough and A. Crowther, “Beyondhigh tech: early adopters of open inno-vation in other industries,” R&d Manage-ment, Vol. 36, No. 3, 2006, pp. 229–236.[Online]. http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9310.2006.00428.x/full

[143] G. Hu, L. Wang, S. Fetch, and B. Bidanda,“A multi-objective model for project port-folio selection to implement lean and SixSigma concepts,” International Journal ofProduction Research, Vol. 46, No. 23, 2008,pp. 6611–6625. [Online]. http://dx.doi.org/10.1080/00207540802230363

[144] M.A. Cusumano and K. Nobeoka, ThinkingBeyond Lean: How Multi-project Managementis Transforming Product Development at Toy-ota and Other Companies. Simon and Schuster,1998.

[145] F. Kon, D. Cukier, C. Melo, O. Hazzan,and H. Yuklea, “A conceptual framework forsoftware startup ecosystems: the case of Israel,”Department of Computer Science, University ofSão Paulo, Tech. Rep. RT-MAC-2015-01„ 2015.[Online]. http://www.ime.usp.br/~kon/papers/SoftwareStartupsConceputalFramework-TR.pdf

[146] M.C. Fonseca, “O ecossistema de startups desoftware da cidade de são paulo,” Master’s the-sis, University of São Paulo, 2016.

[147] D. Cukier, F. Kon, and L.S. Thomas, “Softwarestartup ecosystems evolution: The New YorkCity case study,” in Proceedings 2nd Interna-tional Workshop on Software Startups. Trond-heim, Norway: IEEE, 2016.

[148] D. Cukier, F. Kon, and N. Krueger, “Design-ing a maturity model for software startupecosystems,” in Proceedings 1st InternationalWorkshop on Software Startups. Bolzano, Italy:Springer, 2015, pp. 600–606.

[149] B.L. Herrmann, J.F. Gauthier, D. Holtschke,R. Berman, and M. Marmer, “The globalstartup ecosystem ranking 2015,” Tech. Rep.August, 2015.

[150] A. Frenkel and S. Maital, Mapping NationalInnovation Ecosystems: Foundations for PolicyConsensus. London, UK: Edward Elgar Pub-lishing, 2014.

[151] S. Jayshree and R. Ramraj, “Entrepreneurialecosystem: Case study on the influence of envi-ronmental factors on entrepreneurial success,”European Journal of Business and Management,Vol. 4, No. 16, 2012, pp. 95–102.

[152] R. Sternberg, “Success factors ofuniversity-spin-offs: Regional governmentsupport programs versus regional environment,”Technovation, Vol. 34, No. 3, 2014, pp. 137–148.[Online]. http://www.sciencedirect.com/science/article/pii/S0166497213001399

[153] M. Steinert and L.J. Leifer, “‘Finding One’sWay’: Re-Discovering a Hunter - GathererModel based on Wayfaring,” InternationalJournal of Engineering Education, Vol. 28,No. 2, 2012, pp. 251–252.

[154] D.J. Snowden and M.E. Boone, “A leader’sframework for decision making,” Harvard Busi-ness Review, Vol. 85, No. 11, 2007, pp. 69–76.

[155] S.D. Sarasvathy, “Causation and effectuation:Toward a theoretical shift from economicinevitability to entrepreneurial contingency,”Academy of Management, Vol. 26, No. 2, 2001,pp. 243–263. [Online]. http://www.jstor.org.proxy.lib.ul.ie/stable/info/259121

[156] P. Williams, “The Competent BoundarySpanner,” Public Administration, Vol. 80,No. 1, 2002, pp. 103–124. [Online]. http://onlinelibrary.wiley.com/doi/10.1111/1467-9299.00296/abstract

[157] A. Nguyen Duc, P. Seppänen, and P.K. Abra-hamsson, “Hunter-gatherer cycle: a conceptualmodel of the evolution of software startups,” inProceedings 2015 International Conference onSoftware and System Process. Tallin, Estonia:ACM, 2015, pp. 199–203.

[158] J. Pelrine, “On Understanding Software Agility:A Social Complexity Point Of View,” Emer-gence: Complexity & Organization, Vol. 13, No.1/2, 2011, pp. 26–37.

[159] J. Rikkila, P. Abrahamsson, and X. Wang,“The Implications of a Complexity Perspec-tive for Software Engineering Practice andResearch,” Journal of Computer Engineer-ing & Information Technology, 2012. [On-line]. http://www.scitechnol.com/2324-9307/2324-9307-1-e103.pdf

[160] A. Johri, “Boundary spanning knowledge bro-ker: An emerging role in global engineeringfirms,” in Proceedings 38th Annual Frontiers inEducation Conference. IEEE, 2008, pp. 7–12.

[161] E. Wenger, Communities of Practice: Learning,Meaning, and Identity. Cambridge UniversityPress, 1998.

[162] J. Birkinshaw, G. Hamel, and M.J. Mol, “Man-agement innovation,” Academy of managementReview, Vol. 33, No. 4, 2008, pp. 825–845. [On-line]. http://amr.aom.org/content/33/4/825.short

122 Michael Unterkalmsteiner et al.

[163] A.H.V.D. Ven and M.S. Poole, “ExplainingDevelopment and Change in Organizations,”Academy of Management Review, Vol. 20, No. 3,1995, pp. 510–540. [Online]. http://amr.aom.org/content/20/3/510

[164] S. Blank, “Why the lean start-up changes ev-erything,” Harvard Business Review, Vol. 91,No. 5, 2013.

[165] J.W. Mullins and R. Komisar, Getting to PlanB: Breaking Through to a Better BusinessModel. Harvard Business Press, 2009.

[166] C. Nobel, “Teaching a ‘Lean Startup’ Strat-egy,” HBS Working Knowledge, 2011. [On-line]. http://hbswk.hbs.edu/item/teaching-a-lean-startup-strategy

[167] Y. Harb, C. Noteboom, and S. Sarnikar,“Evaluating Project Characteristics for Select-ing the Best-fit Agile Software DevelopmentMethodology: A Teaching Case,” Journal ofthe Midwest Association for Information Sys-tems (JMWAIS), Vol. 1, No. 1, 2015. [Online].http://aisel.aisnet.org/jmwais/vol1/iss1/4

[168] D. Dennehy, L. Kasraian, O. O’Raghallaigh,and K. Conboy, “Product Market Fit Frame-works for Lean Product Development,” in Pro-ceedings R&D Management Conference 2016“From Science to Society: Innovation and ValueCreation”, Cambridge, UK, 2016.

[169] K. Conboy, “Agility from first principles: Recon-structing the concept of agility in informationsystems development,” Information SystemsResearch, Vol. 20, No. 3, 2009, pp. 329–354. [On-line]. http://pubsonline.informs.org/doi/abs/10.1287/isre.1090.0236

[170] M.B. Miles and A. Huberman, Qualitative dataanalysis: An expanded sourcebook. ThousandOaks, US: Sage Publications, Inc, 1994.

[171] C. Wohlin, A. Aurum, L. Angelis, L. Phillips,Y. Dittrich, T. Gorschek, H. Grahn, K. Hen-ningsson, S. Kagstrom, G. Low et al., “Thesuccess factors powering industry-academia col-laboration,” IEEE software, Vol. 29, No. 2, 2012,p. 67.

[172] M. Unterkalmsteiner, T. Gorschek, A. Islam,C. Cheng, R. Permadi, and R. Feldt, “A concep-tual framework for SPI evaluation,” Journal ofSoftware: Evolution and Process, Vol. 26, No. 2,2014, pp. 251–279.

[173] B. Kitchenham, L. Madeyski, D. Budgen, J. Ke-ung, P. Brereton, S. Charters, S. Gibbs, andA. Pohthong, “Robust Statistical Methods forEmpirical Software Engineering,” EmpiricalSoftware Engineering, 2016, pp. 1–52. [On-line]. http://link.springer.com/article/10.1007/s10664-016-9437-5

[174] W. Hayes, “Research synthesis in software en-gineering: a case for meta-analysis,” in Proceed-ings 6th International Software Metrics Sym-posium. Boca Raton, USA: IEEE, 1999, pp.143–151.

[175] T. Gorschek, C. Wohlin, P. Carre, and S. Lars-son, “A Model for Technology Transfer in Prac-tice,” IEEE Software, Vol. 23, No. 6, 2006, pp.88–95.

[176] R. Wieringa, “Empirical research methods fortechnology validation: Scaling up to practice,”Journal of Systems and Software, Vol. 95, 2014,pp. 19–31.

[177] P. Bourque and R.E. Fairley, Eds., Guide tothe Software Engineering Body of Knowledge,3rd ed. IEEE, 2014.

[178] E. Bjarnason, M. Unterkalmsteiner, E. En-gström, and M. Borg, “A multi-case study ofagile requirements engineering and using testcases as requirements,” Information and Soft-ware Technology, Vol. 77, 2016, pp. 61–79.

[179] B. Boehm, “Value-based Software Engineering,”SIGSOFT Softw. Eng. Notes, Vol. 28, No. 2,2003, pp. 1–12.

[180] C.R.B.d. Souza, H. Sharp, J. Singer, L.T.Cheng, and G. Venolia, “Guest Editors’ Intro-duction: Cooperative and Human Aspects ofSoftware Engineering,” IEEE Software, Vol. 26,No. 6, 2009, pp. 17–19.

[181] C.D. Mulrow, “Rationale for systematicreviews.” BMJ: British Medical Journal,Vol. 309, No. 6954, 1994, pp. 597–599.[Online]. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2541393/

[182] B.A. Kitchenham, T. Dybå, and M. Jørgensen,“Evidence-Based Software Engineering,” in Pro-ceedings 26th International Conference on Soft-ware Engineering (ICSE). Edinburgh, UK:IEEE, 2004, pp. 273–281.

[183] R. Torkar, P. Minoves, and J. Garrigós, “Adopt-ing free/libre/open source software practices,techniques and methods for industrial use,”Journal of the Association for InformationSystems, Vol. 12, No. 1, 2011, p. 88. [On-line]. http://search.proquest.com/openview/beaf4af76e081e0d38c0c1c574218df2/1

[184] E. von Hippel, “Innovation by User Commu-nities: Learning from Open-Source Software,”MIT Sloan Management Review, Vol. 42,No. 4, 2001, pp. 82–82. [Online]. http://adaptknowledge.com/wp-content/uploads/rapidintake/PI_CL/media/InnArticle.pdf

[185] J. West, “How open is open enough?: Meldingproprietary and open source platform strate-gies,” Research Policy, Vol. 32, No. 7, 2003, pp.

Software Startups – A Research Agenda 123

1259–1285. [Online]. http://www.sciencedirect.com/science/article/pii/S0048733303000520

[186] T. Dybå and T. Dingsøyr, “Empirical studiesof agile software development: A systematicreview,” Information and Software Technology,Vol. 50, No. 9–10, 2008, pp. 833–859. [On-line]. http://www.sciencedirect.com/science/article/pii/S0950584908000256

[187] M. Broy, “Challenges in Automotive Soft-ware Engineering,” in Proceedings 28th In-ternational Conference on Software Engineer-ing (ICSE). Shanghai, China: ACM, 2006, pp.33–42. [Online]. http://doi.acm.org/10.1145/1134285.1134292

[188] S. Chandra, V.S. Sinha, S. Sinha, andK. Ratakonda, “Software Services: A ResearchRoadmap,” in Future of Software Engineering(FOSE). Hyderabad, India: ACM, 2014, pp.40–54. [Online]. http://doi.acm.org/10.1145/2593882.2593892

[189] J. Cleland-Huang, O.C.Z. Gotel, J. Huff-man Hayes, P. Mäder, and A. Zisman, “Soft-ware Traceability: Trends and Future Direc-tions,” in Proceedings Future of Software En-gineering. Hyderabad, India: ACM, 2014, pp.55–69. [Online]. http://doi.acm.org/10.1145/2593882.2593891