Guidelines for assessing the performance of statisticians involved in clinical research in the pharmaceutical industry

Guidelines for Assessing thePerformance of StatisticiansInvolved in Clinical Research inthe Pharmaceutical Industry†

Alan Phillips*

Wyeth Research (UK), Maidenhead, Berkshire, UK

SummaryThis paper seeks to help statisticiansin the pharmaceutical industry to as-sess their own competency and to en-courage continued development. Theapproach adopted is to clarify the roleof statisticians in Phase II/III clinicalresearch. Objective criteria for as-sessing performance in terms of qual-ity, timeliness and productivity areprovided for key job responsibilities.A semi-objective method for obtain-ing an overall assessment of perfor-mance is also proposed. Published in2000 by John Wiley & Sons, Ltd.

Key Words

clinical development plans; protocols;statistical analysis; clinical reports;competency

Introduction

Continuing professional development isessential for the long-term success of

any profession. Encouraging more efficientand effective performance and higher levelsof achievement not only helps companies toachieve their goals but also enables individ-uals to fulfil their personal aspirations andto achieve greater job satisfaction.

In order for statisticians to perform welland continue their development in the phar-maceutical industry, they need to know:

(i) what their job is,(ii) what they are expected to achieve and

the standards they should maintain,and

(iii) how they are performing with regard tothese requirements.

Several authors, such as Moses and Louis[1], Ellenberg [2] and Lesser and Parker [3],have discussed the practice of statisticalconsulting. However, as yet there has beenscant attention paid to clarifying the roleand responsibility of the statistician in clini-cal research in the pharmaceutical industryand to defining criteria for assessing perfor-mance. This paper addresses both issues.The objective criteria outlined in this papercan be used as a checklist to provide statis-ticians with feedback on how their develop-ment is progressing and to help identifyproblems or factors affecting performance.

* Correspondence to: A. Phillips, Wyeth Research (UK),Huntercombe Lane South, Taplow, Maidenhead, Berkshire,SL6 0PH, UK. E-mail: [email protected]† Reproduced from Phillips A. Guidelines for assessing theperformance of statisticians involved in clinical research inthe pharmaceutical industry. Drug Inf J 1999; 33: 987–992with kind permission of the Drug Information Journal. Copy-right © 1999 Drug Information Journal.

Published in 2000 by John Wiley & Sons, Ltd. Qual Assur J 2000; 4, 143–149.

A. Phillips144

Appropriate action can then be taken toenhance performance if necessary. A semi-objective method for pooling information sothat an overall rating of performance can begenerated is also proposed. The method in-creases objectivity in the evaluation of per-formance, encourages open communicationconcerning expected results and helps indi-viduals to assess their own progress.

Types of Statistician inClinical Research

Broadly speaking, within the pharmaceuti-cal industry, there is an accepted careerpath for statisticians in clinical research.Table 1 outlines the qualifications and levelof experience required for the differentlevels.

The senior statistician level is assumed tobe equivalent to the CStat professionalqualification of the Royal Statistical Soci-ety. To qualify for CStat, in addition to 5years practical experience, a person is ex-pected to have the equivalent of a good UKhonours degree (class I or II) and/or anappropriate higher degree either in statisticsor in a subject containing a substantial cov-erage of statistical method and theory. Al-ternatively, a person could have 10 yearspractical experience in applying statistics ata responsible level together with acceptableevidence of knowledge, competence and con-tributions to the subject and its application.

The definition is similar to that beingconsidered by the European Federation ofStatisticians in the Pharmaceutical Industry(EFSPI) for ‘an appropriately qualified and

experienced’ medical statistician for regula-tory purposes. EFSPI’s definition of a medi-cal statistician is someone who has auniversity degree in statistics or equivalent,plus 3 or more years experience in medicalstatistics. An example of an equivalentqualification is a degree in mathematics or arelated subject, involving more than 1year’s (full time equivalent) of courses instatistics.

Within the pharmaceutical industry, astatistician usually has study-level responsi-bilities. This comprises the analysis of clini-cal trial data and co-authorship of finalstudy reports. He/she will also start to be-come involved with the statistical aspects ofclinical trial planning. Individuals wouldtypically assume primary statistical respon-sibility for a few relatively small projects,such as line extensions for existing products.However, in many cases, he/she will con-tribute as part of a team working on dataanalysis and reporting for a larger project.

Senior statisticians are often responsiblefor providing statistical support for clinicalprojects, such as new chemical entities ordrugs. Primary activities include data anal-ysis and co-authorship at the study level, aswell as involvement with the statistical as-pects of planning individual clinical studies.Individuals would also typically assume pri-mary statistical responsibility for integratedefficacy and safety analyses or meta-analyses of the complete clinical project.Senior statisticians start to become involvedwith the statistical aspects of developingclinical development plans and co-ordinat-ing the activities of other members of staffat the study level if appropriate. They also

Table 1. Qualification and experience for different levels of statisticalpersonnel in Phase II/III clinical research

Title Qualification Experience (years) Level of responsibility

Statistician BSc — StudySenior statistician ProjectMSc, PhD ]5

TherapeuticMSc, PhD ]10Principal statistician


145Performance of Statisticians Involved in Clinical Research

start to represent the company at meetingswith regulatory authorities and other out-side professional organizations.

The principal statistician is responsiblefor the overall statistical support for one ormore clinical projects from inception tocompletion. Additional activities to those ofthe senior statistician include statistical re-sponsibility for the clinical developmentplan, interaction with regulatory agenciesand responding to regulatory questions. In-dividuals at this level will typically be re-sponsible for major projects. He/she willfrequently be called on to co-ordinate theactivities of other members of staff and willstart to become involved in mentoringstatisticians.

The remainder of this paper reviews thekey job responsibilities of statisticians inmore detail. Objective criteria in terms ofquality, timeliness and productivity are pro-vided. These can be used to assist in theassessment of performance. The criteriashould be used as guidelines because theyonly constitute part of an individual’s per-formance. Other subjective factors need tobe taken into consideration. For instance,two statisticians who complete the samenumber of analyses in a year may be per-forming at different levels. It may be thatone statistician is working on studies thatare all very similar in nature in a wellworked therapeutic area, whereas the othercould be working in a new complex field onvery different studies.

Clinical Development Plan

As is stated in the International Conferenceon Harmonisation (ICH) E9 [4] guidelines,when developing a drug it is necessary tofind out at which dose(s) and schedule it isboth safe and effective. To achieve thisusually requires a series of clinical studies,each with its own specific objective. Theseare usually described in a clinical develop-ment plan.

Interpretation and assessment of the evi-dence from the total programme of studiesinvolves synthesis of the evidence from theindividual studies, which is often the re-sponsibility of a principal statistician. Con-sequently, common standards should beadopted so that data can be pooled acrossstudies and, thus, should be the focus of theperformance assessment. The followingquestions need to be addressed when assess-ing the quality of statistical input into aclinical development plan.

1. Will the clinical development plan sat-isfy statistical and clinical regulatory re-quirements? Have key decision points inthe drug development process beenclearly identified?

2. Are the objectives of the individual stud-ies clearly defined? Have optimum de-signs been used for each study?

3. Will the total number of patients in theplan be sufficient for a safety evalua-tion? Will the number of patients re-cruited into each study meet theobjectives?

4. Are the key endpoints of each studyclearly defined and will they address thequestions of interest? Have the end-points been defined and collected consis-tently across studies in order to facilitatepooling?

5. What statistical methods will be used toanalyse the key endpoints? Have strate-gies been defined for addressing statisti-cal issues such as multiple treatmentcomparisons, multiple endpoints, identi-fication and handling of treatment bycentre interactions, etc.?

6. Which dictionaries of medical terms willbe used for each study? Can they becombined across studies at the end ofthe programme?

7. Are withdrawals and protocol deviationshandled consistently across the studies?

8. Is the whole programme of studies inte-grated to minimize time and the totalnumber of patients?

Published by 2000 by John Wiley & Sons, Ltd. Qual Assur J 2000; 4, 143–149.

A. Phillips146

The time needed to comprehensively ad-dress and document the statistical aspects ofa clinical development plan can range from1 to 6 man weeks of work, depending on thedisease and the experience of the companyand/or individual. For instance, the clinicaldevelopment plan for a new anti-depressantcan take as little as 1 week if the individualhas recently been involved in the successfulregulatory submission of a similar com-pound and is aware of the latest develop-ments in the field. However, for anindividual with limited experience in thefield of depression, it will take considerablylonger because meetings with external ex-perts, literature reviews etc. will berequired.

Because each clinical development plan isunique in some way or another, it is oftendifficult to assess the performance of thestatistician who provided the input. In caseswhere the issues listed above have not beensatisfactorily addressed, the performance isclearly inadequate. An indication of anabove-average performance, however, isharder to define. Indicative of this would bewhen the individual can demonstrate thatthe project can progress at a faster pacethan expected owing to his creativity.

Protocol

The protocol is a document that providesthe background and rationale for a study. Italso describes the study’s objective(s), de-sign, methodology and statistical consider-ations. When assessing the quality ofstatistical input into a protocol, a similar setof questions to those used for assessing aclinical development plan is needed. Thedifference between a protocol and a clinicaldevelopment plan is that the focus is at thestudy and not the project level; for example:

1. Are the objectives of the study clearlydefined?

2. Will the number of patients recruited besufficient to provide an answer?

3. Have techniques such as randomizationand blinding been used to limit the oc-currence of conscious and unconsciousbias?

4. Are the key endpoints of the study con-sistent with the development plan andwill they answer the question of inter-est? How will they be analysed?

5. Have the analysis populations beenclearly defined?

6. Can the randomization codes be ob-tained in an emergency?

7. Will any interim analyses be carriedout? If so, why? Who will have access tothe results? What impact will the in-terim analysis have on the finalanalysis?

8. Are there plans to review the sample sizeestimation if the original assumptionsare based on clinical judgement?

9. Are the case report forms designed tocollect the data as required by theprotocol?

Provided that adequate documentationexists via the clinical development plan,statistical input into a protocol should takebetween 2–5 man working days. However,this can increase significantly if the docu-mentation is inadequate.

Statistical Analysis

Prior to performing the statistical analysisof any study, a statistical analysis planshould be prepared in order to avoid post hocdecisions that may affect its interpretation.This plan is intended to be a comprehensiveand detailed description of the methods andpresentation of data analyses for the study.As well as issues covered in the clinicaldevelopment plan and protocol, a full anddetailed description of the methods of anal-ysis and presentation should be provided foreach type of data. Consideration should be



given to the following issues when assessingthe quality of a statistical analysis plan.

1. Have methods for handling multicentredata, repeated measurements, multipleendpoints, missing data and outliersbeen defined?

2. Are the rules for calculating derived datadefined?

3. How will baseline values be used? Haveany of the covariates to be used in allstatistical models been defined?

4. Are sub-group analyses required?5. If appropriate, are the rules for stopping

the trial defined and has allowance forthem been made in the analysis?

6. What level of clinical and statistical sig-nificance (one- or two-tailed) will beused?

7. Have the statistical methods for pointand interval estimation been clearlystated?

8. Have methods been proposed for check-ing model assumptions?

The development of a statistical analysisplan should take between 1–3 workingweeks depending on existing available docu-mentation. Once finalized, the plan shouldtake between 4–6 man weeks to implement.Provided sufficient preparation time is al-lowed prior to breaking the blind of anystudy, the statistical results of most studiescould be available within 3–5 working days.An exception would be cases where addi-tional work is needed, for example, if themodel assumptions are not valid.

In terms of assessing integrated efficacyand safety analyses, the timeliness and qual-ity measures for a multicentre trial shouldbe used.

Clinical Report

The clinical report is a document containingan overview of the study and the clinicaland statistical findings. When assessing the

quality of a clinical report, considerationshould be given to the following issues.

1. Are the objectives of the study clearlydefined?

2. Is an explanation for the design of thestudy and the number of patients pro-vided? A comprehensive justificationneeds to be given when techniques suchas randomization and blinding are notused.

3. How was the data processed?4. Have the results been presented in a

clear and precise manner?5. Are all tables and graphs accurate,

clearly labelled and consistent?6. Is the method of statistical analysis de-

fined and appropriate? Have deviationsfrom the analysis plan been clearlydefined?

The time needed to address the statisticalaspects of a clinical report is minimal be-cause most issues should have been coveredearlier in the process. The primary activityis that of quality controlling the contents ofthe clinical report, which can take between2–5 working days.

Professional Currency

For statisticians to continue to provide ef-fective input into the drug developmentprocess, it is essential that they maintaintheir technical competence. A statisticianperforming at a satisfactory level is one whoinvestigates emerging methodologies forpossible advantages and feeds back resultsto colleagues via, for example, technical pre-sentations. An individual who is excellingwith respect to this aspect of the job is onewho takes the initiative to carry out supple-mentary analyses using ‘non-standardmethodology’, where the analyses have thepotential to aid the interpretation of thedata and can be reported in the studyreport.


A. Phillips148

Teamwork

Teamwork is essential for successful clinicalresearch. Statisticians are often part of largemultidisciplinary project teams and theirparticipation on these teams should formpart of any performance review. Areas to beconsidered should include:

1. Responsibility: are commitments madeand followed through?

2. Problem solving: are ideas for solvingproblems creative and openly expressed?Criticisms should be constructive to-wards solving problems.

3. Documents/presentations: are writtendocuments clear, understandable andwell received by other team members?

4. Communication: is information that isimportant to the project communicatedto those who need it? Is scientific/techni-cal information logical, reliable and veri-fiable? Are ideas and messagescommunicated without bias?

5. Initiative: is the status quo challenged?Are problems anticipated and contin-gency plans made? Improvements andinnovation should be owned and readilyimplemented.

6. Working relationships: are working rela-tionships established, co-operative andproductive? Disagreements should be re-solved taking into account achievingteam objectives.

7. Meeting participation: does the team’swork continue without interruption ordelay as a result of regular meetingattendance, preparation and contribu-tion?

All of the above are features of a goodstatistical consultant. Not all areas shouldbe considered to be equally important.More critical areas include problem solvingand initiative. Assessment of teamwork isvery subjective and is best achieved bysoliciting feedback from other team mem-bers.

Overall Assessment ofPerformance

In the pharmaceutical industry and else-where, it is not uncommon for an annualreview of performance to be carried out. Allactivities completed by the individual dur-ing the review period are pooled and oneoverall rating of performance produced. Inthis section, we propose a semi-objectivemethod for generating an overall rating ofperformance.

Table 2 lists the key job responsibilitiesfor statisticians in Phase II/III clinical re-search in the pharmaceutical industry. Toprovide an overall rating, each area of re-sponsibility should be weighted, jointly bythe individual and the manager, at the be-ginning of each review period. The weightsshould be based on the overall importanceof each job responsibility by giving consid-eration to the proportion of time and im-pact of each activity. Example weights forthe different levels are given in the table. Inaddition to the responsibilities listed,specific objectives can be added as isdeemed appropriate.

At the end of the review period, both theindividual and the manager should jointlycompare actual performance against thecriteria listed in the paper. An assessment ofperformance should be derived for each ofthe key job responsibilities using a ratingscale, such as below expectation, meetingexpectations and above expectation. Anoverall rating can then be obtained bycombining the individual ratings. To obtainan above expectation overall performancerating, more than 50% of the individualresponsibilities would need to be rated asabove. A similar approach could be adoptedfor a below expectation performance.

Summary

This paper clarifies the role and responsibili-ties of statisticians in clinical research in the



Table 2. Key job responsibilities for statisticians in Phase II/III clinical research and suggestedweightings for the different levels

ResponsibilitiesWeights

Statistician Senior Principalstatisticianstatistician

10 Produces required quantity ofClinical developmentworkplanProduces quality workMeets deadlines

10 20 20 Protocol developmenta

3040 20 Statistical analysisa

2030 30 Clinical reporta

Keeps current with respect toProfessional currency101010statistical methodologyEmerging methodologies areinvestigated for possibleadvantages

Teamwork2010 Responsibility10Problem solvingDocuments/presentationsCommunicationInitiativeWorking relationshipsMeeting participation

a Assessment needs to be split into three components (quantity, quality and timeliness) as per clinical developmentplan work.

pharmaceutical industry. A checklist of ob-jective criteria for assessing performance interms of quality, timeliness and productiv-ity is provided. The list of criteria is notnecessarily comprehensive nor does it neces-sarily apply to all pharmaceutical compa-nies; however, key job responsibilities arecovered. The criteria should help statisti-cians to better assess their own level ofperformance and to help encourage con-tinued development.

References

1. Moses L, Louis T. Statistical consulting in clinical re-

search: the two-way street. Stat Med 1984; 3: 1–5.

2. Ellenberg JH. Biostatistical collaboration in medical re-

search. Biometrics 1990; 46: 1–32.

3. Lesser ML, Parker RA. The biostatistician in medical

research: allocating time and effort. Stat Med 1995; 14:

1683–1692.

4. International Conference on Harmonisation. Statistical

Consideration in Clinical Trial Design.
