RESEARCH

Hek G (1996) Guidelines on conducting a critical research evaluation Nursing Standard. 11,6,40-43.

Guidelines on conducting a

critical research evaluationThis article outlines the reasons why nurses need to be able to read and evaluate research reports critically, and provides a step-by-step approach to conducting a critical

appraisal of a research article.

Date of acceptance: August 1 1996.

Gill Hek MA, RGN, NDN, CertEd (FE), is Senior Lecturer, University of the West of England, Bristol.

KEYWORDS

NURSING RESEARCH

NURSING LITERATURE

QUALITY OF NURSING PRACTICE

These key words are based upon work undertaken by the RCN Library.

This article has been subject to double-blind review.

The ability to evaluate or appraise research in a criti­cal manner is a skill that all nurses must develop. By acquiring and using these skills nurses will be able to understand and appreciate research. For those nurses who have qualified in the last four or five years, criti­cal appraisal or critical evaluation skills may have been learnt during their pre-registration course. However, the vast majority of qualified nurses who undertook their training prior to Project 2000, are unlikely to have had the opportunity to develop such skills. Further­more, many nurses undertake post-registration courses, and most of these courses will require them to gain competency in critical evaluation skills, often assessed through assignments and project work.

WHY IS CRITICAL EVALUATION NECESSARY?Many people assume that research published in a journal or book is of a good quality, and has already been examined and evaluated by an expert before it is accepted for publication. This is true to a certain extent, however, the reader must also take responsi­bility for judging the quality of the research, and con­sidering whether it is worthwhile and appropriate. The review process Before research is pub­lished in a journal, it has usually gone through a process of scrutiny and review. Editors will exam­ine potentially suitable articles for publication which have been submitted to their journal. They will then use members of an expert panel of reviewers or referees to evaluate articles and judge the quality of the research which may include considering its appropriateness and rele­vance for the readers of the journal.

They do this by sending the manuscript to one or two referees who have expertise in the subject. For example, an article about asthma in children may be sent to a clinical nursing practitioner who has experi­ence of nursing children with asthma, and an acade­mic who teaches the management of asthma on a post-registration children's nursing course. Usually, the reviewer will not know any personal details about the author such as name or place of work, and the author will not know who reviewed the article. This is commonly

called 'double-blind' reviewing, and most nursing jour­nals undertake this procedure. The reviewer should provide an objective and unbiased review of the research. The editor of the journal will then consider the comments of the referees, and decide whether to accept the article for publication.

If the reviewing process has been conducted in a conscientious and rigorous manner, the reader of a research article in a journal can have some confidence that the research is ‘sound'. However, if this is not the case, or if the referee has allowed personal biases and beliefs to influence his or her objective judgement, a research article may be published without the neces­sary evaluation and scrutiny.

For this reason among others, there is no certain­ty that published research is reliable and therefore nurses must develop their own skills of critical evalua­tion to allow them to make their own analysis and judgement as to the quality and value of the research. Polgar and Thomas (1995) suggested that the 'proper attitude to published material is hard-nosed scepticism'. Evidence-based practice Nurses need to know that the research they read is credible and reliable for a number of reasons. Probably the most important is to do with providing the highest quality of care to patients and clients. The current healthcare climate places an enormous emphasis on evidence-based' health care. We can under­stand this to mean that the care we provide should be based on appropriate evidence, and this includes research. If we are to give patients the highest quality care possible, we need to know what the best possible care is. This involves searching for the latest evidence and research from journals and books, critically evaluating it, and deciding whether it is appropriate to use in practice.

Nurses may need to do this prior to setting stan­dards or writing protocols, preparing clinical guidelines or auditing a clinical area. Harvey (1996) described the features of audit which include ‘defining the level of quality expected', and this may include the review and critical evaluation of research literature pertinent to the

40 NURSING STANDARD OCTOBER 30/VOLUME 11/NUMBER 6/1996

Fig. 1. Stages of critical evaluation

1. Obtain relevant research article or report, ascertaining that it is research rather than a care study, literature review, discussion or other type of material

2. Read the article thoroughly in order to familiarise yourself with the main elements of the research

3. Break the article down into sections. In some published articles, they are clearly identified by sub-headings, such as:

Introduction and background to the study

Review of the literature

Research design and approach

Sampling strategy and selection of subjects

Data collection techniques and instruments

Data analysis and results

Ethical considerations

Discussion

Presentation

4. Evaluate and analyse each section in detail

5. Consider each section in relation to the entire study

6. Put the whole research back together' and make decisions about the study's value, significance, importance, appropriateness and quality

In many research articles, it is difficult to pick out the sections, and there may be discussion about some of them throughout the article, for example, ethical issues. However, it is important to identify the different sections or ‘themes’ so that they can be fully considered in some depth

area being examined. In setting a standard for mouth care of unconscious patients, for example, a ward team will need to evaluate critically the latest research on the subject, deciding what is the best and highest quality research on which to base the standard against which their own current practice can be compared. To do this, they need critical evaluation skills.

A major report that considered the future of nursing research. Report of the Taskforce on the Strategy for Research in Nursing, Midwifery and Health Visiting (DoH 1993), made some important comments with regard to critical evaluation. It placed responsibility on all nurses to become 'research literate’ to enable them to practise knowledgeably. This included having an 'understanding of the research process (and) the ability to retrieve and critically assess research findings and literature'. These requirements, among others, should be included in training programmes, and in courses designed to help established practitioners to catch up’.

WHAT ARE CRITICAL EVALUATION SKILLS?

Critical evaluation skills enable an individual (or a group of individuals) to assess and evaluate the significance and worth of an individual research article or research report. Through evaluating the research, the various components of a study are examined, analysed and judged. The terms ‘critical appraisal' and 'critical evaluation' are often used interchangeably although ‘evaluation’ is the most commonly used term when referring to an individual research article or report.

The term ‘research critique’ can also be used when referring to the critical evaluation of an article, particu­larly when it is a written account. The process of critical evaluation should be systematic, logical and balanced. The individual undertaking the critical evaluation of a research study attempts to consider all the aspects of the study in an unbiased and non-prejudiced way.

Each section of the study is examined in a bal­anced way, and not in a 'negative' manner as is often thought by the use of the word ‘critical’. When com­bined with evaluation, critical means analytical, bal­anced and rational. Therefore, critical evaluation embraces a systematic, logical, analytical, objective, balanced and decisive assessment and evaluation of a particular research study.

HOW CAN CRITICAL EVALUATION SKILLS RE

DEVELOPED?

Critical evaluation skills can be developed by systemat­ically and methodically working through various stages in an organised and rigorous manner (Fig. 1). When first starting, the whole process can be quite time consum­ing as there are many points to examine and consider. However, practising makes the process of critical eval­uation easier as the various stages become more famil­iar, and the reader becomes more proficient and skilled. Stage one Having obtained a research article that you wish to evaluate critically, the first stage is to examine it closely. It is important to ascertain that it is an original piece of research and not a litera­ture review, case study, discussion paper, or some other type of article or report. All published litera­ture may require some form of critical appraisal, however, not necessarily in the way described in this paper.

A research article or report can usually be deter­mined by finding sections that describe research aims and questions, details about the selection of a sample, and the methods and techniques used for collecting and analysing data. The abstract or summary at the beginning of the research may give some indication as to whether it is an original piece of research.Stage two The next stage is to read the article thoroughly in order to acquaint yourself with the study. It may be helpful to consider the following

questions:Who did what to whom?Why and how and when did they do it?What was the background to the study?

What did they find?Was it morally and ethically sound?

Stage three The third stage is to break up, or separate the article into specific sections. This is sometimes easier if the article has sub-headings to guide you. It is important to understand and com­prehend each section, and it can be helpful to try to imagine what the researcher was doing. If the researcher used techniques or approaches that

OCTOBER 30/VOLUME 11/NUMBER 6/1 996 NURSING STANDARD 41

RESEARCH

REFERENCES

Department of Health (1993)

Report ot the Taskforce on the

Strategy for Research in

Nursing, Midwifery and Health

Visiting. London, Department

of Health.

Harvey G (1996) Relating quality

assessment and audit to the

research process in nursing. Nurse Researcher. 3, 3, 35-46.

Polgar S, Thomas SA (1995)

Introduction to Research in the

Health Sciences. Third edition.

Edinburgh, Churchill

Livingstone.

FURTHER READING

Abbott P (1993) Why do we need

to review literature? Nurse

Researcher. 1,1, 14-22.

Bailie L (1995) Ethnography and

nursing research: a critical appraisal. Nurse Researcher.

3, 2. 5-21.

Behi R (1995) The individual's

right to informed consent.

Nurse Researcher. 3,1, 14-23.

Burns N, Grove SK (1995)

Understanding Nursing

Research. Philadelphia PA,WB Saunders Company.

Cormack DFS (Ed) (1996) The

Research Process in Nursing.

Third edition. London,Blackwell Science.

Couchman W, Dawson J (1995)

Nursing and Health Care

Research: A Practical Guide.

Second edition. London,

Scutari Press.

DePoy E, Gitlin LN (1994)

Introduction to Research:

Multiple Strategies for Health

and Human Services. St. Louis

MO, Mosby.Gibbon B (1995) Validity and relia­

bility of assessment tools.

Nurse Researcher. 2, 4, 48-55.

Hallett C (1995) Understanding

the phenomenological

approach to research. Nurse

Researcher. 3, 2, 55-65.

you did not understand, it will be necessary to find out more about them. Otherwise, you may end up making a judgement about something you do not fully understand.Stage four The fourth stage is where the critical evaluation begins in earnest. Each section identi­fied in stage three is examined in great depth, for example, the selection of the subjects, the tech­niques for collecting the data, and the data analy­sis techniques. The strengths and weaknesses of each section are considered, and any limitations are identified. Each section is analysed and evalu­ated, and judgements made about the manage­ment of the research and the findings of the study. The process involves a balance of positive and negative factors and considerations.Stage five The fifth stage is closely linked to the previous stage and is very important. It involves a consideration of each of the individual sections in relation to the whole study. It can be very easy to have negative opinions about one section, for example, the technique used to select the sample, and this must not be allowed to influence the eval­uation of the whole study. Therefore, the relation­ship of each section to the whole study must be carefully considered in a balanced way.Stage six The final stage is similar to stage two, in that the study is considered in its entirety. However, by this stage in the process, there is much greater understanding about the research, and the study has been thoroughly analysed. The reader is now in a position to make a judgement about the quality and value of the research. The importance and significance of the study can be considered, and whether it is appropriate for the reader to use it in his or her own work.

Using a highlighter pen. or writing notes as you read through an article can be most beneficial. Work­ing with others when critically evaluating can also develop confidence in the process. This could be achieved in a group where members evaluate the arti­cle together, or possibly most useful is when all the individuals have a copy of the research and evaluate on their own, and then join together to discuss their findings.

HOW TO EXAMINE EACH SECTION IN DETAIL Introduction and background to the studyThe research problem should be identified and the rationale and purpose of the research clearly explained. Any limitations to the research should be acknowledged, and the significance, relevance and benefit of the study should be placed in context. Operational definitions and explanations of terms may need to be given.Review of the literature The literature reviewed should be relevant to the study and pro­vide a comprehensive summary of current knowl­

edge about the research problem. Primary sources and recent research and literature should be used whenever possible, and the literature should be critically evaluated, producing balanced and unbiased evidence to support the study. The review should be thorough, logically presented, consistently referenced and include the major studies relevant to the problem being investigated.This section in a research article or report may be quite short, which makes it difficult for the reader to make an objective and balanced judgment.Research design and approach The research approach or strategy should be explained, and any theoretical or conceptual frameworks informing the research should be identified. The research design, for example, experimental design, survey, descriptive design, case study, qualitative field research and so forth, should be clearly linked to the research question(s), research aims and *

objectives, or hypotheses (if appropriate). Research question(s) must be clearly expressed, and any aims and objectives should be consistent with the problem and purpose of the investigation.

Hypotheses should be clear, unambiguous and capable of being tested. Each hypothesis should contain at least two variables and state a predicted relationship between them. The researcher should acknowledge any limitations of the research design. The design and approach of the research should provide the means of examining all the research questions, aims, objectives or hypotheses, and lead on to the sampling strategy, data collection techniques and analysis.Sampling strategy and selection of sub­jects The target’ or accessible population should be described, and should be appropriate for the problem being investigated. The technique used to select a sample should be clearly described, and <f any inclusion/exclusion criteria should be ^

explained. The sampling technique used should be capable of selecting a representative sample from the target population. Potential sample bias­es should be identified, and a description of the key characteristics of the sample selected should be given.

The ways in which the researcher gained 'access’ to the subjects should be apparent, and the subjects should have given ‘informed consent’ prior to taking part in the research. Non-respondents should be accounted for. If subjects are to be assigned to groups, extraneous influences and effects should be controlled.The final size of the sample and number of subjects should be large enough to ensure representativeness, particularly in terms of the power of any statistical analysis, and any generalisations that may be made about the results of the study.Data collection techniques and instru­ments The techniques for collecting data should be adequately described and appropriate to the

42 NURSING STANDARD OCTOBER 30/VOLUME 11/NUMBER 6/1996

Hek G (1995) Sampling techniques

in research- Journal of

Community Nursing. 9. 3, 4-6.

Hicks CM (1990) Research and

Statistics: A Practical

Introduction For Nurses. New

York NY, Prentice Hall.

Holloway I, Wheeler S (1996)

Qualitative Research tor

Nurses. London. Blackwell

Science.

LoBiondo-Wood G, Haber J (Eds)

(1994) Nursing Research:

Methods, Critical Appraisal, and

Utilization. Third edition. S!

Louis MO, Mosby.

Morse JM. Field P (1996) Nursing

Research: The Application ot

Qualitative Approaches.

Second edition. London.

Chapman and Hall.

Oldham J (1994) Experimental and

quasi-experimental research

designs. Nurse Researcher. 1,

4. 26-36.

Parahoo K (1993) Questionnaires:

use, value and limitations.

Nurse Researcher. 1, 2, 4-15.

Polgar S, Thomas SA (1995)

Introduction to Research in the

Health Sciences. Third edition.

Edinburgh, Churchill

Livingstone.Polit DF, Hungler BP (1995)

Nursing Research: Principles

and Methods. Fifth edition.

Philadelphia PA, JB Lippincotl

Company.

Pretzlik U (1994) Observational

methods and strategies. Nurse

Researcher. 2, 2,13-21.Roe B (1993) Undertaking a critical

review of the literature. Nurse

Researcher. 1,1,31-42.Rose K (1994) Unstructured and

semi-structured interviewing.

Nurse Researcher. 1. 3, 23-32.Waterman H (1995) Distinguishing

between 'traditional' and action research. Nurse Researcher. 2,

3, 15-23.

research questions, aims or hypotheses. Any instruments and tools should be sufficiently sensi­tive and able to measure adequately the variables and factors under investigation. The validity and reliability of all measures and instruments should be demonstrated. Instruments designed specifi­cally for the study should have the process of development recounted. A pilot study should have been conducted to test the feasibility of the data collection techniques and instruments, and any other techniques used in the study.

Data analysis and results The procedures for organising and analysing the data should be ade­quately described, and appropriate for the type of data collected. The results should be clearly and wholly presented and relate to the research ques­tions. aims or hypotheses. For quantitative data, appropriate statistical tests should be used and statistical analysis correctly performed. Tables and graphs should be accurately labelled and present summaries of relevant findings. Qualitative data should identify the major ‘themes' that emerged from the analysis. Narrative or accounts from the subjects should support the emergent themes. There should be a description of how the data were coded and analysed and how the findings were validated. The researcher should remain ‘true’ to the data and there should be no evidence of bias in the interpretation of results.

Ethical considerations There are ethical issues in all research, and the researcher should have considered and included them in the report. They may be reported in a specific section of the study, or they may be integrated throughout. The research should be ethically justified, and the researcher should behave in a professional man­ner throughout the study. Access to subjects should be negotiated in an ‘open’ way, and there should be no intimidation or coercion in recruiting subjects. Subjects should be invited to take part in the study, and should give informed and written

consent.The subjects should also be given written infor­

mation about the research. They should be made aware of their right to decline to participate before and during the study. The subjects should not be deceived and should not be submitted to any unnecessary dis­tress or discomfort. If patients or the premises of health authorities are used, permission from a Local Research Ethics Committee must be obtained prior to the research commencing. Research with other types of subjects, for example, students, may need the approval of a Research Board or Committee of some

kind.There should be evidence that the researcher has

stored the data safely, and that it remains confidential to the researcher or research team. The data should be collected in a manner that allows anonymity of subjects,

and protects their privacy during the research and upon publication. The researcher should identify any conflict as a result of being a nurse undertaking research, and the requirements and demands of the research.

Discussion In this section, the researcher draws the study together and makes recommendations. Limitations and weaknesses of the research should be identified before any implications are identified and recommendations made. The find­ings of the study may be discussed in relation to the research questions, aims or hypotheses, and there should be no bias in the interpretation of the findings. The results should be discussed in rela­tion to the literature that was reviewed at the begin­ning of the report, and conclusions should be sup­ported by the findings of the study. Implications and recommendations for practice, education, policy and management should be appropriate, in relation to any limitations of the study. Any recommenda­tions for further research should acknowledge how any weaknesses or problems in the study could be avoided in the future.

Presentation The title of the article or report should be unambiguous and adequately describe the content of the study. The report should be clearly written, free of unnecessary jargon, and be well organised with a logical progression through the research process. Referencing should be con­sistent and accurate, and all citations should be fully referenced at the end of the report. The author/researcher should be appropriate and well qualified to conduct the study. Any sponsorship or support should be clearly acknowledged and the author should write in an objective style with no evidence of bias or prejudice. If an abstract or summary is included, it should incorporate: the research problem; research questions/hypothe­ses; sample size and selection; research design and data collection techniques; and the signifi­

cance of major findings.

CONCLUSIONAll qualified nurses must develop competency in the critical evaluation of research. They must be confi­dent that the nursing care provided is based on high quality research-based evidence wherever it is avail­able. Most significantly, because research is pub­lished in a journal or book does not guarantee that it is reliable and appropriate for clinical practice. Acquir­ing critical evaluation skills takes time and practice, and working in a group with other nurses can make the process more effective. Working systematically and methodically through a research article by using a checklist or guidelines, can assist the beginner in

developing confidence

OCTOBER 30/VOLUME 11/NUMBER 6/1 996 NURSING STANDARD 43