lable at ScienceDirect

Manual Therapy 16 (2011) 614e617

Contents lists avai

Manual Therapy

journal homepage: www.elsevier .com/math

Professional issue

The RCT means nothing to me!

Chris Littlewood*

School of Health & Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, UK

a r t i c l e i n f o

Article history:Received 26 May 2011Received in revised form14 June 2011Accepted 15 June 2011

Keywords:Randomised controlled trialResearch methodologyEvidence based practiceContinuing professional development

* Tel.: þ44 114 222 0888.E-mail address: c.littlewood@sheffield.ac.uk.

1356-689X/$ e see front matter � 2011 Elsevier Ltd.doi:10.1016/j.math.2011.06.006

a b s t r a c t

Research is a cornerstone of evidence based practice with the randomised controlled trial (RCT) regardedas the ‘gold standard’ for evaluating the effectiveness of interventions. However, it is not uncommon forRCT’s to arrive at conflicting conclusions. This conflict might be explained by the quality of the differentstudies and their inherent risk of bias. Despite this, discussion and debate around methodological issuesis limited in physiotherapy specific journals. It is important that clinicians are aware of the inherent riskof bias within studies and what this means for their practice. Hence, this paper presents a clinicallyfocused methodological discussion with the intention of offering a platform upon which readers candevelop their understanding of meaningful critical appraisal and hence gain confidence when readingand appraising published RCT’s.

� 2011 Elsevier Ltd. All rights reserved.

1. Introduction

Research is regarded as a cornerstone of evidence based practice(Greenhalgh, 2010) with the randomised controlled trial (RCT)regarded by many as the ‘gold standard’ for evaluating the effec-tiveness of an intervention (Higgins and Green, 2008). However,the value of this research method with reference to physiotherapyhas recently been questioned (Milanese, 2011). The validity of theseclaims has been challenged but it has been recognised that there isa clear need for more methodological information and debatewithin profession specific journals to facilitate evidence basedpractice (Littlewood, 2011).

With this in mind, this paper offers a comparative criticalappraisal of three published RCT’s of specific interest to the phys-iotherapy profession. The studies are designed to evaluate theeffectiveness of various interventions in the management of rotatorcuff disorders. Shoulder pain is one of the most common muscu-loskeletal symptoms requiring consultation with a physiotherapist(May, 2003). Disorders of the rotator cuff are thought to be thecommonest cause of impaired shoulder function (Lewis, 2009).Long-term outcome is frequently poor despite treatment whichmeans that many patients are exposed to more invasive options,e.g. injections, surgery, and long-term pain and disability(Littlewood and May, 2007).

This is an interesting area because in clinical practice, conflictarises in relation to the management of these disorders (Jonsson

All rights reserved.

et al., 2005) where a range of different interventions are offeredwithout clear knowledge of the ideal treatment (Andres andMurrell, 2008). One reason for this conflict is that apparentlysimilar research offers conflicting conclusions. This conflict mightbe explained by the quality of these different studies and theirinherent risk of bias (van der Windt and Bouter, 2003). Thus toenable useful interpretation and application of research it isnecessary to be able to critically appraise studies to enable thevalidity of their results to be understood (Crombie, 1996).

The intention of this paper is not to provide a systematic reviewof interventions but instead to explore the methodologicalstrengths and limitations of these studies and, in turn, what thismeans for practice and future research. It is anticipated that thisclinically focused methodological discussion will offer a platformuponwhich readers can develop their understanding of meaningfulcritical appraisal and hence gain confidence when reading andappraising future published RCT’s.

2. Critical appraisal

Three studies were included: Bang and Deyle (2000), Brox et al.(1993), Ludewig and Borstad (2003). A brief summary of thesestudies is presented in Appendix 1. The following account isstructured to recognise components that are currently included inmany critical appraisal tools.

2.1. Research aims and research questions

All studies had clearly stated aims including comparisons ofvarious exercise programmes, incorporating loaded exercise, with

C. Littlewood / Manual Therapy 16 (2011) 614e617 615

manual therapy (Bang and Deyle, 2000), surgery (Brox et al., 1993)and no intervention (Ludewig and Borstad, 2003). This enablesa clear judgement about the relevance of the studies to practice tobe made which is a strength of the research. Where a clear state-ment of aim is not made it has also been suggested that this isindicative of poor writing and usually poor quality research (Schulzet al., 2010) which means that the results of the study should betreated with caution.

2.2. Research approach/design

All studies were RCT’s which, when conducted to a high stan-dard, are regarded as the ‘gold standard’ for evaluating the effec-tiveness of an intervention (Higgins and Green, 2008). Thus, theresearch designs were appropriate to answer the research ques-tions. However, although the RCT is an appropriate design, themanner in which the trials are conducted needs to be appraisedbefore accepting the findings (Herbert et al., 2005).

It has been suggested that sub-optimal methods of allocation inRCT’s may lead to an over-exaggeration of treatment effect (Higginsand Green, 2008). Only Ludewig and Borstad (2003) report theirmethod of randomisation which includes measures to conceal theallocation from participants, therapists and researchers. Bang andDeyle (2000) do not mention their method of randomisation orany attempts to conceal allocation whereas Brox et al. (1993) referto allocation by the method of random permuted blocks, which isan accepted method (Bowling, 2002) but do not mention anyattempt to conceal allocation. Due to the influence that the methodof allocation might have on reported treatment effect the results ofBang and Deyle (2000) and Brox et al. (1993) should be treatedwithcaution (Higgins and Green, 2008). In presenting the updatedguidelines for reporting parallel group randomised trials Moheret al. (2010) provide further detail relating to the process of ran-domisation and the consequences of sub-optimal methods inaddition to other important aspects of RCT methodology.

2.3. Sampling

The sampling procedures of Bang and Deyle (2000) and Broxet al. (1993) represented usual practice where participants wererecruited post referral to the treatment centre. This adds credibilityin the form of external validity where the research methods arealigned to real world practice (Greenhalgh, 2010). However,Ludewig and Borstad (2003) recruited local construction jour-neymen through local unions and safety meetings and henceshould be regarded as a non-clinical population. So, even if thisstudy is regarded as credible in terms of internal validity, the use ofa non-clinical population limits the external validity of the findings.

Brox et al. (1993) and Ludewig and Borstad (2003) were the onlystudies to provide evidence of a sample size calculation. The failureto do this has both ethical and scientific implications. A sample sizecalculation is an acceptedmeans of estimating the required numberof participants needed to detect a true treatment effects betweengroups if one does actually exist (Dawson and Trapp, 2001). Bangand Deyle (2000) did not justify their sample size but did identifya statistically significant difference between their groups whichmeans that a Type II error did not occur in this situation (Bowling,2002). However, the failure to include a calculation should beregarded as a potential source of imprecision in this study partic-ularly considering that they only recruited 52 participants in total.

2.4. Data collection

All studies used different measures of outcome and followed uptheir participants at different times. This is a serious shortcoming in

this body of literature which challenges the possibility of synthesis-ing the results of the studies. Only one study (Ludewig and Borstad,2003) clearly stated that they used measures of outcome that hadbeen validated in the population in which they were investigating.Validity relates to how ‘truthful’ a measure is (Bowling, 2001). If anoutcome measure is selected which has not been validated in thechosen population it should be regarded as compromising theinternal validity of the study and again the results of Bang and Deyle(2000) and Brox et al. (1993) should be treated with caution.

In concluding that manual therapy was a useful additionalintervention for participants with subacromial impingementsyndromeBang andDeyle (2000)measuredpain levels at 3e4weeksand function at 2 months. This selective short term outcomemeasurement in favour of one of the treatment arms is a potentialsource of bias andhence the conclusions drawnmight be challenged.

All studies have gone some way to minimising researcher biaswhen collecting the data from the outcome measures. Bang andDeyle (2000) and Brox et al. (1993) incorporate assessor blindingwhilst Ludewig and Borstad (2003) asked participants to inde-pendently complete the measures. Not forgetting the limitations ofthe measures used, methods to minimise assessor bias should beregarded as a strength.

2.5. Data analysis

All studies report using statistical techniques which areaccepted for determining between group differences (Field, 2009).Consideration of whether the results are statistically significant ispresented in all papers but the issue of clinical significancewarrants further thought especially since some of the reportedmeasures are not validated. As mentioned, Ludewig and Borstad(2003) utilise an outcome measure that has been validated, i.e.the Shoulder Rating Questionnaire, but a mean change of 9.9 pointson this measure is not regarded as clinically significant (L’Insalataet al. 1997). This is an important consideration because currentlythe clinicians might ask: ‘What does this actually mean for mypatients and me?’

However, two recognised threats to internal validity includewhether data analysis was undertaken on an intention to treatbasis, i.e. whether the participant’s data was analysed according tothe group they were allocated as opposed to the treatment thatthey actually received (Higgins and Green, 2008). Another impor-tant consideration is how the authors dealt with any missingoutcome data (Higgins and Green, 2008). Brox et al. (1993) reportusing an intention to treat method but do not report how they dealtwith missing values. The impact of this is unclear but might besignificant considering that approximately 30% of participants inthe surgery group were not followed up at 3 months and approx-imately 10% were lost to follow-up at 6 months. Ludewig andBorstad (2003) describe analysing the data of all subjects forwhom post-test data was obtained. This implies that some partic-ipants were lost to follow-up but the authors do not report thisnumber or the reasons for drop-out which casts doubt upon thevalidity of the analysis. Bang and Deyle (2000) report that theycollected complete data sets from all but one of their subjectswhich suggests that bias due to incomplete follow-up is likely to beminimal but intention to treat analysis was not reported and itappears that no consideration was given to the fact that partici-pants might receive other therapy, e.g. the exercise group receivingmanual therapy, during the study period.

2.6. Findings

In the study by Bang and Deyle (2000) both groups experiencedstatistically significant reductions in pain and improvement in

ppendix (continued)

Research question/aimTo compare a supervised

shoulder exerciseprogramme witha supervised shoulderexercise programmecombined with manual

To compare the effect ofarthroscopicsubacromialdecompression,a supervised exerciseregime and placebo

To comparea therapeutic exerciseprogramme with notreatment in patientswith shoulderimpingement

C. Littlewood / Manual Therapy 16 (2011) 614e617616

function but this was greater in the combined group. With regardsto pain, the change could be regarded as clinically significant also. Inthe study by Brox et al. (1993) no statistically or clinically significantdifferences between the surgical and exercise group were detectedbut both improved significantly more than the placebo group. In thestudy by Ludewig and Borstad (2003) there were statisticallysignificant changes in favour of the exercise group.

therapy in patients withshoulder impingementsyndrome.

laser treatment inpatients with rotatorcuff pathology.

syndrome.

Research Approach/DesignRandomised controlled

trial.Randomised controlledtrial.

Randomisedcontrolled trial.

Single blinding only -outcome assessor.

Single blinding only -outcome assessor.

Short-term follow-uponly at 2e3 months.

Short-term follow-up ofpain at 3e4 weeks andshort-term follow-up offunction at 2 months.

Mid-term follow-up at6 months.

Sample52 patients referred by

physician with diagnosisof impingement orrotator cuff tendinitis.58% male, meanage ¼ 43 years.

125 patients referred bytheir GP’s witha diagnosis of rotatorcuff disease. 53% male,mean age ¼ 48 years.

Volunteerconstructionjourneymen recruitedthrough local unionsand safety meetings.100% male, meanage ¼ 49 years.

Inclusion criteria includepositive impingement

Inclusion criteriainclude pain with

Inclusion criteriainclude shoulder pain

2.7. Generalisability

The relatively small sample sizes of the studies limits gen-eralisability but Brox et al. (1993) and Ludewig and Borstad (2003)do justify their sample sizes. All of the studies provide clear detailof the criteria that participants had to meet in order to enter thetrial. This is important because it enables a judgement regardingapplicability, i.e. are the participants in the study sufficientlysimilar to those seen in clinical practice (Moher et al., 2010). Bangand Deyle (2000) and Brox et al. (1993) include participantsrecruited from a clinical environment which could enhance thetransferability of their findings but caution should be exercisedwhen interpreting findings from Ludewig and Borstad (2003)where an all male non-clinical population is recruited whichmight be significantly different from a clinical population dueto possible confounding factors, e.g. motivation, psychologicaldisposition, co-morbidity.

test and pain with activeabduction or resisted test

shoulder abduction,largely maintainedROM and pain withresisted tests.

exacerbated withresisted testing butlargely maintainedROM.

Data collectionVAS was used as a measure

of pain. Function wasmeasured with anunvalidated functionalassessmentquestionnaire developedby the authors from theODI.

The NEER shoulderscore was utilised as theprimary outcomemeasure.

The primary outcomemeasures, theshoulder ratingquestionnaire and theshoulder pain anddisability index whichhave been validatedfor use in populationswith shoulder pain,were self reported bythe patients

Data analysisMultivariate analysis of

variance (MANOVA) wasused to analyse betweengroup differences takinginto account theinteraction ofindependent variables.

The Kruskal WallisANOVA and ManneWhitney U test wereused to assess withingroup and betweengroup differences.

The ANOVA was usedto assess betweengroup differences.

3. Implications & usefulness

In summary, the evidence presented suggests that exercisemight be more beneficial than no intervention, at least as beneficialas surgery with additional benefit conferred when complementedby manual therapy. However, although the study by Ludewig andBorstad (2003) appears to be of a higher methodological quality,all of the studies have a potentially high risk of bias and/orimprecision. So, the value of all of the reviewed studies in terms ofinfluencing clinical decision making is limited.

Despite the limitations of the studies, these findings offera platform upon which to develop future ideas. There is a need forfurther RCT’s which recruit a justified sample size from a specifiedclinical population and utilise and report appropriate methods ofrandom allocation, e.g. distance computer generated random allo-cation, along with validatedmeasures of outcome, e.g. the ShoulderPain & Disability Index, with appropriate length of follow-up, e.g. 3,6 and 12 months, to capture meaningful data.

FindingsBoth groups experienced

statistically significantreductions in pain andimprovement in functionbut this was greater inthe combined group.With regards to pain, thechange could beregarded as clinicallysignificant also.

No statistically orclinically significantdifferences betweenthe surgical andexercise group but bothimproved significantlymore than the placebolaser group.

There werestatistically significantchanges in favour ofthe exercise group.

4. Conclusion

This review has recognised the strengths and limitations of someRCT’s that have evaluated the effectiveness of various interventionsin the treatment of rotator cuff disorders. The implications of thesestrengths and limitations have been discussed. The paper has rec-ognised the limited utility of this research which suggests that highquality research needs to be undertaken to inform clinical practice.

Appendix 1

Author/Date/titleBang and Deyle (2000).

Comparison ofsupervised exercise withand without manualphysical therapy forpatients with shoulderimpingement syndrome

Brox et al. (1993).Arthroscopic surgerycompared withsupervised exercises inpatients with rotatorcuff disease.

Ludewig and Borstad(2003). Effects ofa home exerciseprogramme onshoulder pain andfunctional status inconstruction workers.

A

References

Andres B, Murrell G. Treatment of tendinopathy. Clinical Orthopaedics & RelatedResearch 2008;466(7):1539e54.

Bang M, Deyle G. Comparison of supervised exercise with and without manualphysical therapy for patients with shoulder impingement syndrome. Journal ofOrthopaedic & Sports Physical Therapy 2000;30(3):126e37.

Bowling A. Measuring disease. 2nd ed. Buckingham: Open University Press;2001.

Bowling A. Research methods in health. 3rd ed. Maidenhead: Open UniversityPress; 2002.

C. Littlewood / Manual Therapy 16 (2011) 614e617 617

Brox J, Staff P, Ljungren A, Brevik J. Arthroscopic surgery compared with supervisedexercises in patients with rotator cuff disease. British Medical Journal 1993;307:899e903.

Crombie I. Research in health care. Chichester: John Wiley & Sons Ltd; 1996.Dawson B, Trapp R. Basic & clinical biostatistics. 3rd ed. London: McGraw-Hill; 2001.Field A. Discovering statistics using SPSS. 3rd ed. Sage Publications; 2009.Greenhalgh T. How to read a paper. 4th ed. Chichester: Wiley-Blackwell; 2010.Herbert R, Jamtdedt G, Mead J, Birger Hagen K. Practical evidence based physio-

therapy. Edinburgh: Elsevier Butterworth-Heinemann; 2005.Higgins J, Green S. Cochrane handbook of systematic reviews. Chichester: Wiley-

Blackwell; 2008.Jonsson P, Wahlstrom P, Ohberg L, Alfredson H. Eccentric training in chronic painful

impingement syndrome of the shoulder: results of a pilot study. Knee Surgery,Sports Traumatology, Arthroscopy 2005;14(1):76e81.

L’Insalata J, Warren R, Cohen S, Altchek D, Peterson M. A self-administered ques-tionnaire for assessment of symptoms and function of the shoulder. Journal ofBone and Joint Surgery 1997;79:738e48.

Lewis J. Rotator cuff tendinopathy. British Journal of Sports Medicine 2009;43:36e41.

Littlewood C. Re: the use of RCT’s in manual therapy e are we trying to fit a roundpeg into a square hole? Manual Therapy; 2011. doi:10.1016/j.math.2011.05.013.

Littlewood C, May S. A contractile dysfunction of the shoulder. Manual Therapy2007;12:80e3.

Ludewig P, Borstad J. Effects of a home exercise programme on shoulder pain andfunctional status in construction workers. Occupational & EnvironmentalMedicine 2003;60:841e9.

May S. Implementation of outcome measures for musculoskeletal physiotherapy inprimary care: an audit. Physiotherapy Theory and Practice 2003;19:189e98.

Milanese S. The use of RCT’s in manual therapy e are we trying to fit a round peginto a square hole? Manual Therapy 2011;16:403e5.

Moher D, Hopewell S, Schulz K, Montori V, Gotzsche P, Devereaux P, et al. CONSORT2010 explanation and elaboration: updated guidelines for reporting parallelgroup randomised trials. Journal of Clinical Epidemiology 2010;63:e1e37.

Schulz K, Altman D, Moher D. CONSORT 2010 statement: updated guidelines forreporting parallel group randomised trials. British Medical Journal 2010;340:698e702.

van der Windt D, Bouter L. Physiotherapy or corticosteroid injection for shoulderpain? Annals of Rheumatic Diseases 2003;0:385e7.