100204 Designing a Performance Measurement System for the Research Activities a Reference Framework and an Empirical Study

8/4/2019 100204 Designing a Performance Measurement System for the Research Activities a Reference Framework and an …

http://slidepdf.com/reader/full/100204-designing-a-performance-measurement-system-for-the-research-activities 1/14

Designing a performance measurement system for the

research activities: A reference framework and an

empirical study

Vittorio Chiesa a, Federico Frattini a,*, Valentina Lazzarotti b,Raffaella Manzini b

a Politecnico di Milano, Dipartimento di Ingegneria Gestionale, Milan, Italyb Universita Carlo Cattaneo – LIUC, Castellanza, Varese, Italy

1. Introduction

Defining and implementing a performance measurement system (PMS) within the company is

considered a critical activity for supporting decision making, motivating people, stimulating learning,

improving coordination and communication (Shank and Govindarajan, 1993; Schumann et al., 1995;

Kerssens-van Drongelen and Bilderbeek, 1999). In other words, the PMS is nowadays considered

fundamental for achieving the company’s objectives. As a consequence, all the main activities,

processes and/or functions within companies have recently become the object of a PMS: not only the

J. Eng. Technol. Manage. 25 (2008) 213–226

A R T I C L E I N F O

Article history:

Available online 8 August 2008

JEL classification:

O32

Keywords:

Research & development

Performance

measurement

Performance

measurement system

Biotechnology

Pharmaceutical research

A B S T R A C T

Designing a performance measurement system (PMS) for R&D

activities is a very critical but challenging task for supporting

decision making and people motivation. Therefore, the subject iswidely discussed in literature, but the use of a PMS for R&D is still

uncommon among companies.

The paper aims at making a step further in the field, elaborating a

reference framework that describes the logical steps for the

definition of a PMS for R&D. Moreover, the problem of designing

an effective PMS is in-depth studied in a real context, a biotech

company that operates in the field of pharmaceutical research.

ß 2008 Elsevier B.V. All rights reserved.

* Corresponding author at: Politecnico di Milano, Dipartimento di Ingegneria Gestionale, Via Giuseppe Colombo, 40, 20133Milan, Italy. Tel.: +39 02 2399 2796; fax: +39 02 2399 2720.

E-mail address: [email protected] (F. Frattini).

Contents lists available at ScienceDirect

J. Eng. Technol. Manage.

journal homepage: www.elsevier.com/locate/jengtecman

0923-4748/$ – see front matter ß 2008 Elsevier B.V. All rights reserved.

doi:10.1016/j.jengtecman.2008.07.002

mailto:[email protected]

http://www.sciencedirect.com/science/journal/09234748

http://dx.doi.org/10.1016/j.jengtecman.2008.07.002

http://dx.doi.org/10.1016/j.jengtecman.2008.07.002

http://www.sciencedirect.com/science/journal/09234748

mailto:[email protected]



primary ones, such as production or logistics, but also the supporting ones, such as all the adminis-

trative processes.

According to this trend, R&D as well is being considered as a set of activities and processes whose

performance should be monitored and measured, particularly because in many competitive contexts

technological innovation (and, hence, the results of R&D activities) is the main source of sustainable

competitive advantage.

However, defining a PMS for R&D activities is recognized in literature as a very difficult task, since

effort levels may not be observable in quantitative, measurable terms, success is uncertain (and

influenced by uncontrollable factors) and it can be assessed only after long delays (Tipping et al., 1995;

Brown and Svenson, 1988; Kerssens-van Drongelen and Bilderbeek, 1999; Loch and Tapper, 2002). As

a consequence, in the last years many contributions have been written aimed at discussing the subject

and suggesting possible approaches (Pappas and Remer, 1985; Brown and Svenson, 1988; Chiesa and

Masella, 1996; Werner and Souder, 1997; Hauser, 1998; Driva and Pawar, 1999; Driva et al., 2000; Poh

et al., 2001; Godener and Soderquist, 2004). Such contributions have so far concentrated mainly on

performance measurement , i.e. on defining a set of dimensions of performance to be controlled and the

metrics (or indicators) to be used for the measurement of such performance. Far less contributions are

dedicated to the definition of a whole performance measurement system, i.e. an integrated system not

only able to measure a specific set of performance, but also to explain the managerial and

organizational meaning of each measure, to suggest the most appropriate use of each measure and to

analyze R&D performance with respect to the overall company strategy. The main contributions in this

direction come from Kerssens-van Drongelen and Cook (1997) and Kerssens-van Drongelen and

Bilderbeek (1999), who applied the concept of Balanced Scorecard (Kaplan and Norton, 1992) to R&D,

and from other authors (Bremser and Barsky, 2004) who have adopted more recently a similar

approach.

This paper aims at making a further step in this field, i.e. in-depth studying the problem of defining

a system for performance measurement in R&D units, and not only to identify some metrics and

indicators. In particular, the focus is on the R&D units of companies for which technological

innovation, and, hence, R&D activities are critical for competition. These companies, in fact, aregenerally characterized by a very complex and dynamic R&D environment and, hence, represent a

challenging field for this study.

According to this aim, a reference framework is identified in the paper, that represents a

systematization of literature contributions in the field, and describes the logical steps for the

definition of the PMS for R&D. The framework, starting from the corporate strategy and the R&D

strategy, comes to the definition of:

the dimensions of performance to be measured and the related indicators, the structure to be defined for the measurement system, the process aspects to be implemented for the proper operation of the system.

In a few words, the suggested framework gives a ‘‘practical’’ guide that should help managers in the

definition of all the elements of a PMS, in accordance with the overall company’s strategy.

Then, the framework is applied to a specific case study, with the aim of:

1. verifying the actual applicability of the framework in a real context, in which all the ‘‘contingencies’’

are taken into consideration, exploring in detail the problems and difficulties emerging during the

application;2. enriching, if possible, the framework itself and/or modifying it, according to the evidence emerged

during the application;3. giving a very detailed and concrete example of how the huge literature on performance

measurement , which gives many suggestions in terms of dimensions of performance, indicators,process aspects, can be integrated within a system, internally coherent and adequate for a specific

strategic context;4. discussing the possible generalization of the framework, i.e. verifying whether and how it can be

applied to other contexts than the one analyzed in this paper.

V. Chiesa et al. / J. Eng. Technol. Manage. 25 (2008) 213–226 214



The case study, as already pointed into evidence, has been selected in a context where

technological innovation represents the critical source of competitive advantage. The analyzed

company, NiKem Research, is a biotech firm that operates in the field of pharmaceutical research,

mainly offering services with a high technical and scientific content and capable of supporting the new

drug’s R&D process. This is an interesting case, in which a company almost collapses on its R&D

function that represents the 90% of the overall activities.

The paper is structured as follows: Section 2 gives an overview of the literature on R&D PMS and

presents the proposed framework; Section 3 describes the case study and the application of the

proposed framework; finally, Section 4 discusses the results and draws some managerial implications.

2. A literature review and a theoretical framework

There are many definitions of technological innovation. Here, we assume the definition given by

Freeman (1976), as re-adapted by Chiesa (2001). Technological innovation is a process which includes

the technical, design, manufacturing , management and commercial activities involved in the marketing

of a new (or improved) product or the first use of a new (or improved) manufacturing process or

equipment . Different generations of the innovation process are identified in literature beginning

from the 1960s to the present (Chiesa, 2001). In particular, the latter two generations emphasise

that technological innovation is not sequential, is cross-functional by nature and often multi-firm.

Links with suppliers and customers are very strong along with the whole innovation process. Links

with firms take place in a variety of forms (joint ventures, consortia, alliances, contracts, etc.). R&D

activities clearly play an important role in the process of innovation. There are many definitions of

R&D activities as well. A traditional classification concerns the macro-phases which make up the

R&D process (Chiesa, 2001). It is possible to identify: basic research, which is ‘‘an activity aimed to

generate knowledge related to the working principles of natural and social science without direct

relation to industrial applications (products, services, production processes)’’; applied research,

which is ‘‘aimed to the production of knowledge required to define the means to fulfil a specific and

explicit need’’; development , which ‘‘consists of the systematic use of knowledge oriented to thedevelopment of materials, methods, tools, systems’’. Development is composed of a series of

phases: design, prototyping and testing, engineering, installation, maintenance and post-

commercialisation service. In parallel with the changed conception of the innovation process

(the generations of the innovation process), different generations of R&D activities have been

identified. From a concept of R&D with no interaction with the rest of the company and considered

as an overhead cost, the importance of integration in several forms has been progressively pointed

out: e.g. in terms of links between strategy and R&D activities, cross-functional teams and

integration with customers. The last generation emphasises that R&D is ‘‘a part of a total innovation

system including competitors, suppliers, customers and distributors’’.

If we examine the R&D activities in terms of their accountability, that is the most relevant topic for

this work, other significant changes can be pointed out. In fact, R&D was once considered to be aunique, creative and unstructured process that was difficult, if not impossible, to manage and control

(Kerssens-van Drongelen and Cook, 1997). Today, R&D consideration has also changed concerning

with this aspect, although it is generally recognized that it is challenging to establish accountability for

many R&D activities (Tipping et al., 1995; Brown and Svenson, 1988; Kerssens-van Drongelen and

Bilderbeek, 1999). Since the 1990s, relevant changes in the business environment (in terms of

intensified competition, shortened product life cycles, advanced technology and automation, etc.)

have focused managers’ attention on R&D’s contribution to competitive advantage (Kerssens-van

Drongelen and Cook, 1997). These changes have brought companies to pay attention to their R&D

processes in terms of efficiency, internal and external customer focus, time to market (TTM),

innovativeness, etc. (Kumpe and Bolwijn, 1994). In other words, R&D is a key strategic issue that must

be aligned with corporate and business strategies and their various expected performance ( Pearsonet al., 2000; Bremser and Barsky, 2004). In order to encourage such alignment, a performance

measurement system is required, composed of the following elements, linked through reciprocal

relations (Kaplan and Norton, 1992; Kerssens-van Drongelen and Cook, 1997; Kerssens-van Drongelen

and Bilderbeek, 1999; Bremser and Barsky, 2004):




the dimensions of performance to be monitored and the related indicators, qualitative and/or

quantitative; the structure of the system, i.e. the articulation in ‘‘controlled objects’’; the process aspects to be defined for a proper working of the system, i.e. all the norms and rules

governing the PMS, the timing and frequency of measurement for each controlled object and for the

different dimensions of performance, the role and tasks of people involved in the implementation of

the system.

The contributions on the contextual approach give many relevant suggestions for the design of a

PMS for R&D activities, identifying the contextual factors (for a literature review, see Kerssens-van

Drongelen and Cook, 1997) that influence the definition of the PMS’s elements:

the company’s R&D strategy, in terms of long-term objectives (or critical performances) that are

coherent with the business strategy, the competitive context (rules of competition and main

competitive pressures) and the general environmental features (macroeconomic factors, institu-

tional norms, social and cultural characteristics); the entities to be monitored (i.e. R&D division or department, sub-department, project, individual)

organized in a certain structure (i.e. according to scientific disciplines, typology of activities, product

line, project, etc.); the type of activities to be monitored (i.e. basic research; applied research; development, that are

tasks with different degrees of uncertainty); the PMS’s objectives (i.e. the purpose of the measurement, e.g. motivating people, diagnosing

activities, supporting decisions, stimulating learning, improving communication and coordination

between R&D and other company’s organizational units); the resources (time, money, people, competencies) available for the implementation of the PMS.

A tentative systematization of all those suggestions is given in the framework in Fig. 1. Such a

framework describes the logical entities to be considered in the definition of the PMS for R&D and itcan be helpful to practically guide the system design. The framework is made up of two parts:

the contextual factors; the consequent PMS’s elements; and it enlightens the following aspects: within the context, those factors that constitute the R&D ‘‘environment’’ (i.e. R&D strategy, R&D

organization and management, R&D activities) can be distinguished. The R&D strategy drives

Fig. 1. A reference framework for defining a contextual R&D PMS.




Box 1. The links between the contingencies and the PMS elements

Purposes of the measurement and dimensions of performance If, as it is common especially in highly uncertain and dynamic R&D environments, the measure-

ment system serves the purpose of motivating people , particular cautions are required (Kerssens-van Drongelen and Cook, 1997). In the case of motivational purpose, the assumption is that byfeeding back information about their performance, possibly coupled with incentives, people will bemotivated to change their behaviour. Two conditions should be satisfied in order to design aneffective motivational system. First of all, the system has to include only those factors that can becontrolled by the person subjected to the measurement. Furthermore, if an incentive or a rewardprogram is coupled with the measurement (i.e. bonuses, promotions), the system should measureall the important aspects of a work. This is due to the fact that there is a human tendency to directmore effort towards tasks that are being measured and consequently rewarded, giving lessattention to tasks that are not. However, especially in research activities, tasks may not beobjectively measurable (because of their intrinsic high uncertainty and relevant content of crea-tivity), although they are generally critical in order to obtain appreciable results in the long term.This circumstance, linked to the type of activities contingency, makes it particularly challenging toestablish accountability criteria.Purpose of the measurement and process aspects The purpose of the measurement is a contextual factor that is capable of influencing also somePMS process aspects. For example, whether the main PMS’s objective is to diagnose R&Dactivities, it is necessary to choose standards to measure performance against that give thepossibility to objectively judge the value of a specific indicator and to make comparisons overtime, thus enlightening eventual improvement in R&D performance. At the same time, thefrequency and the timing of the measurement must be chosen so that the PMS is capable of gathering and transferring performance data to R&D and top managers timely and coherently withtheir informational needs.Type of activities and dimensions of performance In a research context, literature suggests that the system should include dimensions of perfor-

mance regarding the research process (a dimension of expected performance could be, forexample, ‘‘the achievement of professionalism standards’’, and a possible indicator ‘‘the frequencyof participation to international conferences’’) rather than the research output , i.e. dimension of performance such as ‘‘the work productivity’’, measured by ‘‘the number of patents’’ indicator(Loch and Tapper, 2002; Hauser, 1998; Chiesa et al., 1996). Rewards should not only be financial, butalso emphasise recognition (Loch and Tapper, 2002).Type of activities and process aspects Also the process aspects of the PMS (e.g. standards/norms to measure performance against)should be adapted to the activities features. For example, if the ‘‘number of patents’’ is the indicatorfor individual performance measurement, two opposite norms could be used to measure perfor-mance against: explicitly recorded standards or subjective and implicit norms. The choice shouldtend to vary with the characteristics of the work. Reasonably, for activities with a high uncertainty,

the subjective techniques should be preferred because they take into consideration a wider range of relevant aspects (qualitative aspects and/or environment conditions, that are not under the controlof the researcher).R&D strategy and dimensions of performance This relationship enlightens that the choice about the dimensions of performance to be monitoredis driven by R&D strategy. In effect, the strategic objectives are the critical performances to bepursued to guarantee company’s competitive success. A PMS is useful if it manages to monitorthese critical performances (or, at least, a part of them) that are assumed as dimensions of performance of the system itself.R&D strategy and process aspects The PMS process aspects can be influenced also by the choices the company makes in terms of R&D strategy and objectives. For example, whether the firm means to undertake a time-basedcompetition that requires short time to market (TTM) and technological pioneering behaviours,

it is necessary that coherent decisions are made in terms of timing and frequency of themeasurement, so that the PMS is capable of supporting R&D and top management decisionprocesses.




the choice of the R&D activities to be internally carried out and their organization in a certain

structure; the PMS’s available resources can influence the PMS’s objectives, because they can represent a

constraint that limits the informative completeness of the system and thus the achievable purposes

(and the dimensions of performance to be monitored); the PMS’s objectives are driven also by the R&D management, e.g. because of a leadership style that

stresses more or less the motivational aspects of the system; as far as, the objectives are driven by

the type of activities (i.e. a high level of creativity and complexity can require a particular attention

to researchers’ motivation and coordination); all the contextual factors influence the design of the PMS’s elements (see Box 1 for some examples of

relationships); all the PMS’s elements are interrelated and in close relationship, thus stressing their reciprocal

dependence and the systematical nature of the PMS.

Explanations concerning some relations between the contextual factors and the PMS’s elements

are illustrated in Box 1. Whether it may appear obvious that the structure of the system should be

defined in order to collect information about the performance of all R&D organizational units, the link,

for example, between the purpose of the measurement and the PMS’s dimensions of performance is

less evident. The explanations in Box 1 try to clarify these links and enlighten further complexities

deriving from another contextual factor, i.e. the type of activities that can also influence the definition

of the PMS’s process aspects.

3. The case study and the framework application

The framework has been first applied in a real context with the aim of guiding the definition of a

PMS in a pharmaceutical company, NiKem Research. The problems emerged during the application of

the framework have indicated the need to integrate it with a list of key questions, that replicate its

overall structure and can be used to guide the PMS design coherently with the framework underlyinglogic. This has shown to be an important step towards the practical orientation of the model itself.

Fig. 2 summarizes the results of this empirical investigation and allows for a preliminary

explanation of the reference model underlying logic.

After a brief description of the company’s profile, the contextual factors and the associated PMS’s

elements are illustrated more in detail.

3.1. Company’s profile

NiKem Research is an Italian company founded in 2001 that operates in the field of pharmaceutical

research, mainly offering services with a high technical and scientific content and capable of

supporting the new drug’s R&D process. Its origins could be traced back to the merger between Glaxo-Wellcome and SmithKlineBeecham, two big pharmaceutical companies with Italian operative units,

respectively set in Verona and Milan. The merger took place in 2000 and brought to the birth of GSK

(GlaxoSmithKline); the top management established that all the activities would be aggregated in

Verona, thus implying the partial transferring of SmithKlineBeecham’s personnel from Milan to the

Venetian city. A part of SmithKlineBeecham’s discovery team, that was going to be totally cancelled,

decided not to throw away the excellent competences in medicinal chemistry they have developed

during more than 20 years. They thus presented a spin-off project to GSK aimed at the creation of a

small, highly specialized company, capable of supporting pharmaceutical firms’ R&D processes with

specific services.

3.2. Contextual factors

3.2.1. General environment and competitive context

When NiKem Research was born, the features of the pharmaceutical market were particularly

propitious for the birth and the success of this kind of ventures. In fact, the increasing complexity,




Fig. 2. The framework application.




costs, risks and investments, typical of new drugs’ discovery and development processes, were forcing

big pharmaceutical companies to reorganize their R&D structures and to outsource R&D activities to

smaller and more flexible firms (Muffatto and Giardina, 2003). In order to understand the

characteristics of the service offered by NiKem Research, it could be useful to briefly describe the

structure of the research and development process in the pharmaceutical industry (for more details

see Muffatto and Giardina, 2003; Chiesa, 2003; Katzung, 2000; Paoletti et al., 2001). The identification

of a new drug in the pharmaceutical industry is the result of a complex R&D activity that is basically

composed of two steps: the first one relates to the identification of a new compound (drug discovery),

the second is concerned with the development of the discovered compound (drug development),

followed by approval of the qualified regulatory authority. The drug discovery phase can be split into

two fundamental stages: the biological activities and the chemical activities. The former is composed

of the target identification and the target validation activities, while the latter includes the lead

generation and the lead optimization tasks.

The identification of the target is the starting point of the process, and it consists in the pointing

out of a particular molecular target (e.g. a gene or a protein), that is responsible for a specific

pathology. The next step of the biological activities (target validation) implies a deep investigation

into the characteristics of the identified target in order to discover how it acts and how it generates

the disease. Once the target has been identified and the process through which it works has been

understood, it is necessary to discover, through an intense screening activity, a set of compounds

(named leads) that, interacting with the target, may have positive impact on the pathology. This

stage of the discovery phase is called lead generation, and it ends with the identification of a limited

number of compounds that need to be optimized. The lead optimization phase of the discovery

process is a very critical, risky and innovative one that requires excellent competences in the field of

medicinal and combinatorial chemistry. It basically consists in the analysis of the generated

chemical compounds in order to identify the most balanced chemical entity in terms of potency at

the molecular target, selectivity, efficacy, defects, pharmacokinetic and toxicological properties. To

evaluate the safety and effectiveness of the optimized pharmaceutical compound before it is

destined to the market, it has to be carefully tested, according to very severe and strict procedures.This phase is called drug development, and it is divided into two macro-activities: the pre-clinical

and the clinical tests. The former consists of a set of laboratory tests (called in vitro) and tests on

animals (named in vivo), with the aim of evaluating the drug’s safety and toxicity level. The clinical

tests, on the other hand, are articulated into three sequential steps: phases I, II and III. They are

conducted on human volunteers in order to fix the proper dosage and to assess the drug’s

effectiveness and tolerability. Once the development stage is ended, the results of the tests are

gathered and analyzed, and, eventually, a request for the market approval of the new drug is

forwarded to the qualified authority, that may approve its commercialisation or not. Going back to

the recent evolutions in R&D processes of the pharmaceutical industry, it should be remembered

that, with the ending of the human genome project, all the 25,000 human genes were identified;

about 3000 were recognized to be ‘drugable’ or ‘physiologically relevant’ molecular targets. Theyrepresented, together with the diffusion of high throughput screening (HTS) techniques, an

enormous incentive for pharmaceutical companies to intensify their screening activity; this

brought to the identification of a huge number of lead compounds that needed to be optimized. It

was especially the large scale to be achieved in the tasks of lead generation and optimization in

order to keep competitive, their high level of inherent risk and costs, and the competences they

require, that forced more and more big pharmaceutical companies to outsource these activities

to highly specialized and flexible organizations, thus externally acquiring the missing skills and

critical mass.1

3.2.2. Strategy and objectives

In the propitious context just described, the spin-off project was approved by GSK, that decided totransfer, through a management-by-out operation, personnel, laboratories and other facilities to the

1 Due to the fact that NiKem almost collapses on its R&D function, in Fig. 2, that represents the applied framework there is no

relevant distinction between business strategy and R&D strategy.




new-born NiKem Research, and to support the start-up phase of its activity with a research contract of

2 years, capable of covering the 85% of its start-up costs. In 2001, on August 1st, NiKem Research was

officially established and started working as service provider, highly specialized in the activities of

‘lead generation’ and, especially, ‘lead optimization’, with a staff of 25 researchers. NiKem Research is

thus a typical example of Contract Research Organizations (CRO), where ‘contract research’ is defined

as ‘. . . the activity by which a client hires the services of an external organization to carry out a specific

piece of R&D’ (Haour, 1992).

In 2003, NiKem Research’s top management realized that the company’s business model,

completely based on the provision of specific services, would reveal itself too fragile and not capable of

assuring high value and steady profits on the long term. This was mainly due to the fact that the

pharmaceutical market is traditionally exposed to significant sales fluctuations that negatively reflect

on the performances of small CROs like NiKem Research. Another threat was represented by those

service providers operating in emerging countries like India, Russia or China; it was thought they

would be capable, in 4–5 years, of offering high quality services, comparable to European or American

companies’, but at a significantly lower price (from 1/3 to 1/5). This convinced NiKem Research’s top

management to modify the firm’s strategy and to dedicate part of the available resources to internal

research activities, with the aim of generating novel leads to be internally optimized and developed

until they are ready for the clinical stage of the Drug Development process. In order to carry out the

pre-clinical tests, pharmacological competences are necessary; this has brought NiKem to collaborate

with the ‘‘Istituto Nazionale dei Tumori’’ and with various Italian universities (e.g. Firenze, Ferrara). At

that stage of the R&D process, the leads would be licensed to or partnered with big pharmaceutical

companies that own the financial resources and the complementary assets necessary to complete the

development phase and to introduce the new drugs into the market. The internal research activity

assures NiKem Research another advantage, i.e. the possibility to have a buffer of highly specialized

human resources that can be used to face sudden and transitory increases in the request for discovery

services.

Even if the service provision and the internal research activities require that very similar types

of scientific challenges are faced and that similar technologies and competencies are acquired (e.g.pharmacokinetic, preliminary toxicology, MTS, additional ADME profiling), they significantly differ

when we consider the critical success factors where NiKem Research has to excel in order to gain a

competitive advantage. In the business of discovery services, the objectives to be pursued are the

quality level of the service (at a cost level aligned to the competitors’ one), the respect of the

promised delivery time (punctuality) and the confidentiality in the management of the

information concerning the clients’ compounds. On the other hand, these variables do not

represent competitive pressures in the strategic area of internal research, where it is far more

important to adopt a long-term perspective and to try to gain the leadership in the identification

of novel candidates to clinical development. In this area, even if concrete results are obviously

necessary, time constraints are less severe due to the fact that specific clients’ pressures are not

relevant.After all, a critical objective has been identified at the overall company level, that becomes another

specific objective for each business area. This critical dimension consists in the building of an external

technology reputation. This factor is very important first of all because it provides contacts with new

potential clients in the business of discovery services. In addition, it provides relations with big

pharmas and biotechs in order to improve the internal research pipeline.

3.2.3. R&D activities

According to the traditional classification concerning the macro-phases which compose the R&D

process (see, e.g., Chiesa, 2001), it is possible to classify NiKem Research’s activity as basic and applied

research and therefore strongly creative, uncertain and quite unstructured.

3.2.4. R&D organization and management

It can be stated that NiKem Research is organized on the basis of a matrix structure. The researchers

are grouped in departments, according to the discipline in which they are specialized. The

‘‘departmental dimension’’ of the matrix structure is necessary in order to assure a high level of




specialization in the company’s core scientific competences (the main departments are: medicinal

chemistry, analytical chemistry, computational chemistry, screening). Within each department, a

director is appointed. He is responsible for the management of the activities and the resources in the

department and for its scientific results. Furthermore, each department includes a series of

laboratories. They represent a lower level along the departmental dimension and are defined

according to ‘‘scientific specialization’’ criteria. A ‘‘laboratory head’’ manages the activities of this

organizational unit.

On the other hand, the ‘‘project dimension’’, by means of inter-departmental teams, is important to

put together the transversal skills that are necessary to produce the company outputs. Within each

project, a team leader is appointed as well; he is, in general, the researcher with the major experience,

or charisma, or specialized in the most crucial discipline for the project success. He is responsible for

the results achieved by the team. In this matrix structure, however, a major organizational power is

assigned to the department structure rather than to the project structure. This power can be in

practice appreciated in terms of a wider possibility in the orientation of the researchers’ activities,

through the definition and the valuation of their goals and responsibilities.

After all, NiKem recognizes that human resources are crucial in order to achieve high quality

services. Consequently, the leadership style is surely a participative one with the basic intent of

obtaining people collaboration.

3.2.5. PMS objectives

The leadership approach and the type of activities (i.e. their high level of creativity and complexity

that requires a particular attention to researchers’ motivation and coordination) guide the choice of

the PMS’s objectives towards a clear prevailing of the motivational purpose, even if also the aspect of

diagnosing activities is not ignored. Particular attention has thus been paid to the alignment of the

indicators with the specific goals and responsibilities of the people subjected to the measurement.

This means that the system has to include only those factors that can be controlled by the evaluated

people. This aspect is practically considered when the PMS’s elements are defined, in particular when

the dimensions of performance and the related indicators are assigned to the different objectscomposing the PMS’s structure.

In any case, it has been realized that, in order to enhance the motivational capability of the

performance measurement system, the objectives assigned by a chief to a single researcher must be

collectively defined (‘‘shared objectives’’). Furthermore, they must be measurable, even if they do not

necessarily produce tangible outputs. However, a certain degree of performance ‘‘un-measurability’’ is

expected because of the intrinsic nature of the company’s activity, that is basic and applied research,

and therefore strongly creative, uncertain and quite unstructured. This circumstance influences the

financial incentives (bonuses) that are coupled with the measurement in order to enforce the

motivational purposes of the system. In fact, incentives are depending not only on the specific

individual performances, but they are linked also to the company’s overall earnings. This link is

justified on the basis of the assumption that each researcher (by means of his competences, behaviour,cultural values, etc.), contributes in any case to the company’s success, even if personal, concrete and

measurable results are far from being evident.

The purpose of diagnosing activities can be recognized in the project-progress monitoring. This

type of control is formal and structured, by means of reports and meeting with the clients. The

necessity of a formal approach can be appreciated when we consider in particular the provision of

lead generation and lead optimization services. These stages of the R&D process are, in fact, very

critical for the drug development success and involve customer assets whose intellectual property

must be protected. Besides, the likely client’s criticism is increased by the fact that NiKem has

relationships with companies in competition. Further complexity is caused by the long duration of

the relationships. Theformalization of theapproach is thus considered a mean to limit thecustomer’s

worries.

3.2.6. PMS resources

NiKem Research small dimensions have an inevitable impact on the resources available to design

and then implement the PMS. Above all, human resources availability has been identified as the most




important constraint in the PMS’s design2 as far as few people will undertake the activities necessary

to the proper operation of the system. This circumstance has lead to the definition of a not very

sophisticated system (i.e. including few dimensions of performance, few indicators, a simple

structure), although representing a good starting point for a further deepening.

3.3. PMS elements

3.3.1. Dimensions of performance

The choice of the dimensions of performance to be monitored is driven by R&D strategic objectives.

In Fig. 2 are reported the selected dimensions of performance (one or more) corresponding to each

strategic objective. For example, in the business area of discovery services, the ‘‘service quality level’’

objective has been translated into two dimensions of performance to be monitored in order to assure

its achievement (i.e. service effectiveness and capability of acquiring new technologies and

competencies), as far as the ‘‘service cost level’’ has been turned into a dimension of internal efficiency

(i.e. respect of the planned service costs). On the other hand, in the business area of internal research,

the dimensions of performance, coherent with the objective of ‘‘gaining the leadership in the

identification of novel candidates to clinical development’’, are basically the quality of identified/

optimized leads and the capability of scouting new collaboration opportunities.

3.3.2. Indicators

It is important to choose the correct qualitative and/or quantitative indicators in order to make the

dimensions of performance operatively measurable. It would be important to select those

performance indicators that allow for a simple and direct measurement and that, even if not

questionable, can be collectively discussed. This indicates a clear preference towards quantitative

indicators. Anyway, the characteristics of uncertainty, risk and un-measurability of R&D activities, as

said above, makes it often necessary to integrate quantitative indicators with non-numeric and

qualitative metrics. This tendency can be clearly recognized in the case of NiKem Research, where both

these types of indicators are applied, even to measure the same dimension of performance. A joint useof quantitative and qualitative indicators has been established, for example, to measure the capability

of acquiring new technologies and competencies, where the numeric indicator based on the ‘‘time

needed to acquire new technologies’’ has been integrated with the qualitative appreciation of the

international relevance of the technologies/competencies acquired.

3.3.3. Structure

The structure of the system reflects the organizational R&D structure as described above. This

means that the monitored objects are basically the ‘‘project’’ and the ‘‘department’’. In addition, the

focus on the motivational purpose of the system leads to dedicate a particular attention also to the

performance of individual researchers.

The assignment of the selected indicators to the structure levels is driven by the basic purpose of the performance measurement system, i.e. motivating people. In other words, it is important to make

a specific organizational unit responsible only for those performance indicators it can directly and

completely influence.

In particular, it has been recognized that the controllable factors tend to vary moving across the

organizational levels. For example, the individual researcher cannot be held responsible for the

success of a specific screening task (i.e. the quality of novel identified leads), due to the fact that the

uncertainty of this activity is too high and out of his control (the chances a molecule has of becoming a

lead are very low). On the other hand, it can be considered a responsibility of a research project as a

whole, seen that the project manager can influence its output, e.g. by improving processes and by

allocating resources to a good portfolio of research activities.

The match between the indicators and the structure levels is a very critical process because of itsmotivational implications. Fig. 2 also reports the final output of the assignment choices.

2 Apart from our contribution, a ‘‘design team’’ has been appointed including the Chief Operating Officer, an external

consultant and two researchers, respectively employed in the discovery services and in the internal research activities.




3.3.4. Process aspects

It has been observed that the researchers, because of their high level of education, are very

autonomous in carrying out their work, whose creative nature probably stresses this behaviour (they

often say ‘‘we need to be autonomous in our work’’). The autonomy is typical of both the business

areas. The respect of this working condition requires that the measurement system is not too strict and

constraining. This need has influenced the choice of the norms that NiKem Research is going to

implement. Subjective standards are, in fact, prevalent in order to encourage people initiative also in

the case of results that are different from those agreed. For example, an actual number of patents that

is lower than the standard number, can be anyway considered a good performance through subjective

evaluations by superiors, that can be rightly less severe than a quantitative method objectively

applied.

Considering the timing of the system, it has been decided to assign the individual objectives at the

beginning of the year. The evaluation of the objectives’ progressive achievement occurs about every

four months with a final evaluation at the end of the year.

4. Conclusions and managerial implications

This paper aims at making a step further in the understanding of the problems to be faced when

defining a system for performance measurement in R&D units. According to this aim, a reference

framework is identified, that represents a systematization of literature contributions in the field and

describes the logical steps for a proper definition of the PMS for R&D, thus giving a ‘‘practical’’ guide

that should help managers facing this challenge.

The scheme here suggested explicitly establishes relationships between contextual factors (i.e.

dimensions of the R&D ‘‘environment’’- R&D strategy, R&D organization and management, R&D

activities-, measurement system objectives and resources available for the system design and

implementation) and PMS’s elements that are reciprocally interrelated, coherently with the systemic

nature of the PMS.

Then, the framework has been applied to a company specifically selected in a context wheretechnological innovation represents the critical source of competitive advantage. This is clear when

we consider that NiKem Research, the analyzed firm, almost collapses on its R&D function that

represents the 90% of the overall activities. Above all, the case study has surely provided an enriching

experience regarding the aspects of system definition, thanks to the problems emerged during the

application of the suggested approach. In particular, it was clearly perceivable the need to integrate

the framework with a list of key questions, that replicate its overall structure and can be used to guide

the PMS design coherently with the framework underlying logic. This has shown to be an important

step towards the practical orientation of the model itself, allowing the company to deeply understand

its own strategic objectives and the consequent dimensions of performance to be monitored. The

choice of the specific indicators that allow the measurement of the dimensions of performance

revealed itself to be a very critical task, at least as challenging as their assignment to the structurelevels in order to consider the accountability and motivational implications.

Anyway, the case provides a very detailed and concrete example of how the huge literature on

performance measurement , which gives many suggestions in terms of dimensions of performance,

indicators, process aspects, can be integrated within a system, internally coherent and adequate for

a specific strategic context. In fact, various contextual factors emerge as basic aspects that can

influence the measurement system intents. In particular, it has been recognized that dimensions of

performance (and related indicators) should reflect the activity portfolio derived from the

company’s strategy and its critical long-term objectives. Moreover, the main purpose that has

brought to the introduction of the measurement system (i.e. ‘‘motivating people’’) has decisively

guided its definition. This explains why particular attention has been paid to the alignment of the

indicators with the specific goals and responsibilities of the people subjected to the measurement;the system has to include only those factors that can be controlled by the evaluated people.

Furthermore, it has been recognized that the controllable factors tend to vary moving across

the organizational levels. This means that the organizational structure has a great importance in

the measurement system design. The organizational levels to be considered and matched with the




selected performance indicators, in fact, directly derive from the ‘‘matrix’’ structure applied in

NiKem Research.

In any case, the match between the indicators and the structure levels revels itself to be a very

critical process because of its motivational implications.

The type of activity (basic and applied research) and the peculiarities of the needed human

resources (for example, their autonomy) are the other factors that have had some influence on the

system design. It has been observed that the researchers, because of their high level of education, are

very autonomous in carrying out their work, whose creative nature probably stresses this behaviour.

The respect of this working condition requires that the measurement system is not too strict and

constraining. This need has thus influenced the choice of the norms that NiKem Research is going to

implement. Subjective standards are, in fact, prevalent in order to encourage people initiative also in

the case of results that are different from those agreed.

Finally, NiKem Research small dimensions have had an inevitable impact on the resources available

to design and then implement the PMS. In particular, human resources availability has been identified

as the most important constraint in the PMS’s design and implementation tasks. This circumstance has

lead to the definition of a not very sophisticated system (i.e. including few dimensions of performance,

few indicators and a simple structure), although representing a good starting point for a further

deepening.

Further investigations are surely necessary in order to appreciate the operation of such a system

(‘‘is it accepted by the researchers?’’) and its eventual evolution. From this point of view, the

framework proposed is dynamic in its nature: as far as the contingencies and contextual factors

change, the PMS characteristics (i.e. the dimensions of performance monitored, the structure of the

system, the process aspects) coherently change as well. This is a crucial point, given the dynamic

nature of the competitive context and the rapidly changes in the rules of competition.

The synthetic framework we have elaborated in this article fits the specific characteristics of NiKem

Research, but it represents an important empirical basis for future analysis. First of all, we mean to

further investigate whether it can be usefully applied to other Contract Research Organizations

working in the field of pharmaceutical research, conducting other case studies on the matter. This willgive us the opportunity to improve and correct the proposed scheme and, probably, generalize it, seen

the similarities between CROs in terms of type of activities and strategy. Moreover, it would be

interesting to study, by means of surveys, the possibility to adapt the framework to innovative

companies working in other industries. This will probably require a deep change in the selected

dimensions of performance (and related indicators), due to completely different types of activities and

business models and, therefore, critical long-term objectives.

Another relevant topic for future research concerns the integration of the R&D PMS with the

company’s management control system, i.e. with the control of the overall economic, financial and

non financial performance of the company. This, in fact, poses interesting problems concerning: the

information flows; the style chosen for monitoring performance; the coherence between, on the one

hand, the R&D PMS elements, and, on the other, the company’s PMS elements, particularly in terms of structure and process aspects.

Acknowledgement

We thank Mr. Giuseppe Giardina, Chief Operating Officer in NiKem Research, for his contribution

by means of a deep knowledge of R&D processes and control requirements.

References

Bremser, W.G., Barsky, N.P., 2004. Utilizing the balanced scorecard for R&D performance measurement. R&D Management 34(3), 229–237.

Brown, M.G., Svenson, R., 1988. Measuring R&D productivity. Research-Technology Management 31 (4), 11–15.Chiesa, V., 2003. La Bioindustria. Strategie competitive e organizzazione industriale nel settore delle biotecnologie farm-

aceutiche. ETAS, Milano.Chiesa, V., 2001. R&D Strategy and Organisation. Imperial College Press, London.




Chiesa, V., Coughlan, P., Voss, C.A., 1996. Development of a technical innovation audit. Journal of Production InnovationManagement 13 (2), 105–136.

Chiesa, V., Masella, C., 1996. Searching for an effective measure of R&D performance. Management Decision 34 (7), 49–57.Driva, H., Pawar, K.S., 1999. Performance measurement for product design and development in a manufacturing environment.

International Journal of Production Economics 60, 61–68.Driva, H., Pawar, K.S., Menon, U., 2000. Measuring product development performance in manufacturing organisations.

International Journal of Production Economics 63 (2), 147–159.Freeman, C., 1976. Economics of Industrial Innovation. Pinter Publisher, London.Godener, A., Soderquist, K.E., 2004. Use and impactof performance measurement results in R&Dand NDP: an exploratory study.

R&D Management 34 (2), 191–220.Haour, G., 1992. Stretching theknowledge-base of the enterprise through contract research. R&D Management 22 (2), 177–182.Hauser, J.R., 1998. Research, development and engineering metrics. Management Science 44 (12), 1670–1689.Kaplan, R.S., Norton, D.P., 1992. The balanced scorecard measures that drive performance. HarvardBusiness Review 70 (1),71–79.Katzung, B.G., 2000. Farmacologia generale e clinica. Piccin, Padova.Kerssens-van Drongelen, I.C., Bilderbeek, J., 1999. R&D performance measurement: more than choosing a set of metrics. R&D

Management 29 (1), 35–46.Kerssens-van Drongelen, I.C., Cook, A., 1997. Design principles for the development of measurement systems for research and

development processes. R&D Management 27 (4), 345–357.Kumpe, T., Bolwijn, P.T., 1994. Towards the innovativefirm-challenge for R&D management. Research-Technology Management

37 (1), 38–44.Loch, C.H., Tapper, S., 2002. Implementing a strategy-driven performance measurement system for an applied research group.

The Journal of Product Innovation Management 19 (3), 185–198.Muffatto, M., Giardina, G., 2003. Innovazioni nei Processi di Ricerca in Campo Farmaceutico. Economia & Management 6, 107–

121.Paoletti, R., Nicosia, S., Clementi, F., Fumagalli, G., 2001. Farmacologia generale e molecolare. UTET, Torino.Pappas, R.A., Remer, D.S., 1985. Measuring R&D productivity. Research-Technology Management 28 (3), 15–22.Pearson, A.W., Nixon, W.A., Kerssens-van Drongelen, I.C., 2000. R&D as a business—what are the implications for performance

measurement? R&D Management 30 (4), 355–366.Poh, K.L., Ang, B.W., Bai, F., 2001. A comparative analysis of R&D project evaluation methods. R&D Management 31 (1), 63–76.Shank, J.K., Govindarajan, V., 1993. Strategic Cost Management: The New Tool for Competitive Advantage. The Free Press, New

York.Schumann, P.A., Ransley, D.L., Prestwood, D.C.L., 1995. Measuring R&D performance. Research-Technology Management 38 (3),

45–54.Tipping, J.W., Zeffren, E., Fusfeld, A.R., 1995. Assessing the value of your technology. Research-Technology Management 38 (5),

22–39.

Werner, B.M.,Souder, W.E., 1997. Measuring R&D performance– state of the art.Research-TechnologyManagement 40 (2), 34–42.


Documents

100204 Designing a Performance Measurement System for the Research Activities a Reference Framework and an Empirical Study