Embed Size (px)
Citation preview
Radiography (2006) 12, 88e95
Assessment of the critical thinking skills ofstudent radiographers
Alan Castle*
University of Portsmouth, Centre for Radiography Education, St George’s Building,Portsmouth PO1 2HY, UK
Received 27 October 2004; accepted 8 March 2005Available online 17 May 2005
KEYWORDSEducation;Critical thinking;Student assessment
Abstract Purpose Enabling students to develop critical thinking skills is one ofthe key aims of higher education and in preparing student radiographers for thefuture, there are increasing demands on educators to teach critical thinking skills tofacilitate reflective, evidence-based practice and inter-professional working. Theaim of the paper is to attempt to compare students’ self-perception of their criticalthinking skills to their actual written assessment performance.Methods Students were asked to self-report how they thought the course haddevelopedtheir critical thinking skills and theoutcomesof thisexercisewerecomparedto the scores of previous assessments that required the demonstration of these skills.Results The results suggest that whilst students report having developed criticalthinking skills during the course, the results of their written assessments requiring thedemonstration of these skills all had a mean score of less than 60% which indicates (interms of the university’s grade criteria guidelines) ‘‘little attempt to use criticaldiscussion in their work.’’Discussion Thirteen components of critical thinking are proposed, together withways in which they could be incorporated into a radiographic curriculum.Conclusions It is suggested that educators may need to review the constructivealignment of their curricula and re-assess their teaching and assessment strategies inorder to effectively develop students’ critical thinking skills.ª 2005 The College of Radiographers. Published by Elsevier Ltd. All rights reserved.
Introduction
Critical thinking is considered a high level activitythat goes beyond simply the acquisition of knowl-
* Tel.: C44 23 9284 5392; fax: C44 23 9284 5388.E-mail address: [email protected]
1078-8174/$ - see front matter ª 2005 The College of Radiograpdoi:10.1016/j.radi.2005.03.004
edge. A taxonomy of educational goals and objec-tives for the cognitive (information and knowledge)aspect of learning, with a hierarchy ranging fromthe lower levels (knowing and understanding) to thehigher levels (analysis, synthesis and evaluation) iswell known.1 To operate at these higher levels it isnecessary to develop the skills of critical thinking,
hers. Published by Elsevier Ltd. All rights reserved.
Assessment of the critical thinking skills 89
which involves the ability to interpret, analyse andevaluate information and ideas, and develop in-dependent opinions and judgements based on soundevidence and reasoning.2 The definition of criticalthinking has been the subject of much debate inrecent years in a variety of educational contexts,but 17 consensus components of critical thinking innursing have been proposed.3 However criticalthinking skills are defined and categorised, it isclear that developing and demonstrating such skillsis important in an age where health professionalsare expected to be reflective practitioners and basetheir practice on sound evidence.4,5
The aim of this paper is to compare students’perception of their critical thinking skills with thatof their actual written assessment performance todemonstrate whether significant changes need tobe made to the curriculum in order to ensurecritical thinking skills are taught and assessedmore effectively. Currently there is no specificcritical thinking skills teaching in the curriculumand any teaching that does take place in a subjectspecific context is left up to individual lecturers.Whilst implicitly there is more emphasis on teach-ing and assessing critical thinking skills as thecourse progresses, this is only explicit in the sensethat assessment questions move from asking stu-dents to describe and explain to asking them todiscuss and evaluate. So whilst it might be reason-able to expect senior year students to have in-creased critical thinking skills compared to juniorstudents, this may only be occurring by accidentrather than by design and it may be that a moreexplicit constructive alignment of the curriculumand transparency in the links between learning andassessment6 is required to improve these skills.
For this study two cohorts of students wereinvolved: third year students at the beginning ofthe fifth semester and second year students at thebeginning of the third semester. Actual studentperformance was measured following marking ofwritten assessments that required demonstrationof critical thinking skills.
Method
In this study no baseline data were available aboutstudents’ perceptions of their critical thinking skillsbefore they commenced the course, so the meth-odology was designed to explore their perceptionsof improvements in critical thinking during thecourse, without making reference to whether theythought their skills were high or low.
The first step was to develop a question-naire with a range of statements that included
appropriate aspects of critical thinking. Bothpositive and negative generic statements werewritten (avoiding neutral or extreme ones whichvirtually everyone would agree with), with the aimof assessing the respondent’s opinion using a five-category agreementedisagreement Likert scale.Subsequently, a self-completion questionnaire,with two sections, was devised. Whilst a shortdefinition of what is meant by critical thinking wasincluded, students had received no formal teach-ing of critical thinking skills. Section one asked forbackground details of gender, age range, highestlevel of qualification obtained and whether pre-vious courses involving learning critical thinkingskills had been undertaken. Section two of thequestionnaire consisted of 12 statements (8 posi-tive and 4 negative) where students were asked toindicate to what extent they agreed or disagreedwith the statement in relation to their currentcourse. The use of negative and positive state-ments was to try to encourage respondents to readthe statements carefully rather simply ticking thesame response for each statement. The question-naires were administered to each of the studentcohorts at the beginning of a session early on ineither their third or fifth semester as appropriate.Section two of the Critical Thinking Questionnaireis shown in Fig. 1
The responses were scored in such a way thatthe endorsement of a positively worded statementand the non-endorsement of a negatively wordedstatement were assigned higher scores (i.e. a scoreof 5 was assigned to a ‘strongly agree’ response tostatement 1 and to a ‘strongly disagree’ responseto statement 2 and so on). In this way a consis-tently high score reflected a superior self-percep-tion of critical thinking skills and vice versa.Therefore, the maximum score that could beattained was 60 and the minimum score was 12.As ordinal data were collected from section two ofthe Critical Thinking Questionnaire, it wasdeemed more appropriate to calculate the medianvalues for the overall and individual statementscores.
To allow for analysis of the spread of opinion, itwas decided to devise four categories that summar-ised the extent to which respondents thought thecourse had promoted their critical thinking skills.Scores over 54 were deemed to be in strongagreement (at least half of the statements scored5 and the other half 4), scores from 42 to 54 weredeemed to be in agreement (at least half of thestatements scored 4 and the other half 3), scoresfrom 30 to 41 were deemed to be in disagreement(at least half of the statements scored 3 and theother half 2) and scores from 18 to 29 were deemed
90 A. Castle
Please indicate the extent to which you agree or disagree with the following statements by placing a in theappropriate circle. You should answer quickly, carefully and truthfully so that the answers you give accuratelyrepresent your views.
Statement Strongly Agree Not Disagree StronglyAgree Sure Disagree
In this course:1. I have learnt more about how to approach
complex issues in a variety of ways.2. I have seldom found myself actively engaged in
thinking about complex issues.3. I have improved my ability to judge the value
of new information or evidence presented to me.4. I have learnt more about how to analyse the key
issues in my subject area.5. I have not improved my ability to give sound
reasons for my beliefs, opinions and ideas.6. I have developed a more open-minded approach
in interpreting, analysing and judging alternativepoints of view.
7. my interest in issues and questions related to mysubject area has increased.
8. I have developed a more focussed and systematicway of thinking.
9. I have learnt more about how to justify whycertain procedures are undertaken in my subjectarea
10. most tutors have encouraged me to explore theideas, theories, assumptions and proceduresrelated to the subject area.
11. most tutors have not demonstrated how to thinkand express myself in a more reasonable,objective and evaluative way.
12. most assessments have not stretched myintellectual abilities.
Figure 1 Critical Thinking Questionnaire (modified version of the teaching for thinking student course evaluationform, California Academic Press18).
to be in strong disagreement (at least half of thestatements scored 2 and the other half 1).
The student scores for written assessments(1500e2000 word essays), completed in semestersone and two (second year students) and three andfour (third year students) of the previous academicyear, asking for the demonstration of criticalthinking skills were recorded as an indication ofperformance. It is acknowledged that, whilst allsubject assessors use standard marking forms, theassessment of the critical thinking skills element ofeach written assignment is superficial (e.g. stu-dents demonstrate a clear ability to evaluatetechniques [high score]:students demonstrate lit-tle/no ability to evaluate techniques [low score]).In terms of weighting, the critical thinking elementof the assessment may account for only about 50%of the overall score (other marks being given forpresentation, referencing, explanation, descrip-tion, conclusions, etc.), although this would beless in the first year and more in the third year.However, despite these limitations, and the factthat there is no clear evidence that what was beingassessed were, in reality, critical thinking skills, or
that they were assessed in a rigorous and reliableway by different assessors, it was possible toprovide some comparison between perceived crit-ical thinking skills and actual performance inwritten assessments requiring the demonstrationof these skills.
Results
Responses to questionnaires
Third year student cohortThe total number of students who completed thequestionnaire was 51 (representing 94% of thetotal cohort). Of these, 82% were female (n Z 42)and 18% male (n Z 9), with 90% (n Z 46) being inthe 18e21 years age group. Over 75% of studentshad attained A/AS level qualifications, with othersobtaining BTEC, GNVQ, HND and Access coursequalifications. Only 2 (4%) students had obtaineda first degree. Most students (76%, n Z 39) re-ported no previous experience of critical thinkingteaching.
Assessment of the critical thinking skills 91
Course Promotes Critical Thinking SkillsStrongly AgreeAgreeDisagreeStrongly Disagree
Total Score54+
42-5430-4118-29
Median = 48SD = 4.2
Total
% (n) Students0
90 (46)10 (5)
0
100 (51)
Figure 2 Agreementedisagreement that course promotes critical thinking skills (3rd year students).
As shown in Fig. 2, the vast majority ofrespondents (90%, n Z 46) agreed that the coursehad promoted critical thinking skills, with theoverall score for the 12 statements having a medianscore Z 48. The median score for all of the 12individual statements Z 4.
Second year student cohortThe total number of year 2 students who completedthe questionnaire was 48 (representing 80% ofthe total cohort). Of these, 77% were female(n Z 37) and 23% male (n Z 11), with 58%(n Z 36) being in the 18e21 years age group. Over75% of students had attained A/AS level qualifica-tions, with others obtaining BTEC and Accesscourse qualifications. Only 5 (10%) students hadobtained a first degree. Most students (65%,n Z 31) reported no previous experience of criticalthinking teaching.
Fig. 3 demonstrates that the majority ofrespondents (75%, n Z 36) agreed that the coursehad promoted critical thinking skills, with theoverall score for the 12 statements having a medianscore Z 44.5. The median score for all of the 12individual statements Z 4.
Assessment performance
Third year student cohortDuring the previous academic year, students com-pleted six written assessments that required themto demonstrate higher-level cognitive skills (dis-cuss, critique, evaluate and analyse). As shown inFig. 4, the mean scores were less than 60% for eachof the six assessments.
Second year student cohortDuring the previous academic year, students com-pleted four written assessments that requiredthem to demonstrate higher-level cognitive skills(discuss, justify, compare and contrast). As shownin Fig. 5, the mean scores were again less than 60%for each of the four assessments.
Discussion
Overall, both cohorts of students reported thatthey had developed critical thinking skills duringthe course, with 3rd year students reportinga higher median score (48) than 2nd year students(44.5). As age range, gender and previous educa-tional qualifications were similar for both cohorts,it maybe that the course has had a positive impacton their critical thinking skills. In general femalesreported higher median scores than males, al-though in both cohorts males only representedapproximately 20% of the respondents. Age, pre-vious educational qualifications and previous crit-ical thinking learning had little overall impact.
It was disappointing to see that all writtenassignments requiring the demonstration of criticalthinking skills resulted in mean scores of less than60%. It also must be borne in mind that each of thewritten assignments only had a weighting ofaround 50% for the critical thinking element ofthe work, which means that most of the markscould have been achieved for lower level skillssuch as knowledge, descriptions and explanations.According to the university grade criteria guide-lines this means that for most students in both
Total Score Course Promotes Critical Thinking Skills (n) Students 54+ Strongly Agree 42-54 Agree30-41 Disagree18-29 Strongly DisagreeMedian = 44.5 SD = 5.5 Total
4 (2) 71 (34) 23 (11) 2 (1)
100 (48)
Figure 3 Agreementedisagreement that course promotes critical thinking skills (2nd year students).
92 A. Castle
Number of Students
Year 2 Assessment Topic Bloom’s Taxonomy
MeanScore
Range SD Number Students scoring <40
Number Students scoring >70
54 Critique Research Papers Evaluation 59.04 40-79 9.42 0 8 48 Discuss safety issues
related to equipment Analysis 48.83 34-66 9.13 3 0
52 Discuss influences of hormones
Analysis 53.13 35-73 11.87 2 3
34* Summarise evidence for inequalities in health
Evaluation 57.54 40-75 9.04 0 3
47 Evaluate renal arteriography
Evaluation 51.85 27-75 13.80 4 4
54 Analyse data set Analysis 54.19 40-67 10.44 0 0 20* Discuss issues related to
health psychology Analysis 58.85 42-79 10.06 0 3
* optional units
Figure 4 Assessment summary for year two assignments 2003/2004.
cohorts there was ‘little attempt to use criticaldiscussion’ in their work. Thus, there appears to bea mismatch between students’ perceptions of theircritical thinking skills and the written evidencethat these skills have been learnt.
Students probably overestimate their criticalthinking skills, as many are often unable to produceappropriate written assignments as supportingevidence. Lecturers’ expectations may be unreal-istic, as students have not been clearly taught theconcept and components of critical thinking and ithas been suggested that the critical thinking abilityof some educators is not significantly higher thantheir students7 as they continue to teach studentswhat to think (teacher-centred/content-orient-ated) rather than how to think (student-centred/learning-orientated).8 If educators are to positivelyinfluence students in developing their criticalthinking skills, then they need to look carefully attheir teaching philosophy.9
If our curriculum is not constructively aligned,with educators not clear about the concept ofcritical thinking, how it should be taught andassessed, then we really will never be clear abouthow successful we are in helping students developthese important skills for their future careers.
Consequently it would appear that a review ofthe current curriculum in terms of teaching andassessing critical thinking skills is required. Aftercareful analysis of the 17 components identified,3
a modified list of 13 skills that should be taught ina radiographic context are proposed (Fig. 6). Inorder to arrive at 13 components the originalinquisitiveness and intellectual dimensions wereincorporated into the information seeking compo-nent, the applying standards dimension was in-corporated into the context component and theintuition dimension was incorporated into theconfidence component.
Thus, having established the component skillsof critical thinking that need to be taught andlearnt, the next stage is effective implementationinto the curriculum. Critical thinking skills aredifficult to teach and assess mainly because ofa lack of clear agreement amongst educators as tohow to teach and assess them. Teaching theseskills can be approached through specific criticalthinking programmes, embedded in specificsubject areas or spread across the curricu-lum.10,11,12,13 They can also be taught in anacademic or clinical context using lectures, jour-nal articles, clinical scenarios, case studies or
Number of Students
Year 1 Assessment Topic Bloom’s Taxonomy
MeanScore
Range SD Number Students scoring <40
Number Students scoring >70
60 Discuss imaging modalities and treatment options
Analysis 56.46 31-68 5.51 3 0
57 Justify poster content and design Evaluation 55.89 40-70 8.14 0 5 60 Discuss results of physics experiment Analysis 58.71 27-75 9.44 3 3 47 Compare and contrast features of
radiographic equipment Analysis 58.62 37-80 10.41 1 7
Figure 5 Assessment summary for year one assignments 2003/2004.
Assessment of the critical thinking skills 93
Component Skills Required Skill Attainment (examples) Information seeking • Inquisitive seeker of knowledge, truth and
understanding • Identify and search relevant sources for
evidence, facts or knowledge and gather data
• Topic related literature search
Analysis • Break a whole into parts to discover components, function and relationships
• Value focussed, systematic and comprehensive approaches to issues
• X-ray tube design and operation • Human skeleton
Evaluation • Read the evidence and clarify issues • Check accuracy and reliability • Make judgements and draw conclusions based
on evidence
• Journal article • Data set of cancer survival rates
Reflection • Contemplate own thinking and assumptions to allow for a deeper understanding
• Reflective diary of clinical practice • Imaging protocols
Creativity • Generate, discover or re-structure ideas and imagine alternatives
• Health promotion opportunities in clinical practice
• SWOT analysis Prediction • Predict/anticipate potential outcomes and
consequences • Clinical case studies • Management styles
Discrimination • Identify inconsistencies • Distinguish relevant from irrelevant • Recognise differences and similarities
• Data set of health statistics for Europe • Compare 2 journal articles on similar
topic Context • Consider the background and influences
relevant to an issue • Health inequalities • NHS Plan
Perseverance • Pursue a course with determination to overcome barriers
• Research project • Oral presentations
Flexibility • Ability to adapt, modify or change thoughts, ideas and behaviours
• Clinical assessments • Physics experiments
Open-mindedness • Tolerant of divergent views • Identify own bias/prejudice
• Factors influencing health • Imaging technologies
Knowledge transfer • Change nature of form or function from one concept to another
• Effects of x-radiation • Cell biology
Confidence • Develop effective communication style • Trust own reasoning skills • Intuitive and insightful understanding
• Communication skills • Image interpretation skills
Figure 6 Components of critical thinking (modified version of the 17 consensus dimensions of critical thinking innursing3).
observation.11 All approaches have advantagesand drawbacks: lectures can cover the conceptsbut are unlikely to have much impact in terms ofapplication in real life situations; evaluating jour-nal articles will only cover a limited number ofcomponents of critical thinking: scenarios andcase studies have to be carefully written if theyare to draw out the components of criticalthinking; observations are time-consuming andlack reliability. In terms of assessment, some formof written assignment is probably necessary toensure that critical thinking skills have beenmastered, but there are problems of subjectivityand inter-assessor reliability in scoring.10 It isdifficult in a 2000 word essay to assess and weightcritical thinking skills in a context where technicaland scientific content is also important. Peerreview of written assignments have been used asa means of promoting critical thinking skills ina wide range of contexts, allowing students toaccess, read, analyse and pass judgement onvarious prescribed aspects of a journal article,
write a short summary based on their deliberationsand then discuss their judgements in a seminarsession with other students who have read, ana-lysed and judged the same article.14,15 However,since only a limited number of components ofcritical thinking skills can be assessed at any onetime, this approach is time consuming and, ifrepeatedly used for each component, may becomelaborious for the student. Using either carefullydesigned and weighted assessments in a range oftopic areas normally assessed in a radiographiccurriculum or short, written responses to reallife clinical scenarios or case studies are thoughtto be the most effective method of teaching andassessment. In this way students can focus on onedimension at a time, the teaching and assessmentof critical thinking can be distributed across thecurriculum, all components can be taught andassessed over time and inter-assessor reliabilityis improved.13 Examples of how each componentof critical thinking skills may be attained areoutlined in Fig. 6.
94 A. Castle
Critical Thinking Skill Component
Demonstration 4 3 2 1
Information seeking Strongly Agree
Agree Disagree Strongly Disagree
A Inquisitive Evidence of extensive search for appropriate literature
B Relevant sources Evidence of accurate and appropriate reference list
C Gathers data Evidence of presentation of appropriate data in text to support work
Max Score = 12 Min Score = 3 Actual Score = Comments
Figure 7 Critical thinking marking grid.
Thus, in order to effectively teach and assesscritical thinking skills, a clear strategy must beadopted in the curriculum rather than simply, as isoften current practice, injecting words such asdiscuss, analyse and evaluate into the stem ofmany assignments in an often vain attempt toencourage students to demonstrate their criticalthinking skills.16,17
Conclusions
It is important that the curriculum is designed toensure that learning activities and assessment tasksare aligned with the learning outcomes, so thatstudents are aware of exactly what learning out-comes we want them to achieve (e.g. demonstratethe components of critical thinking) and then assessthem against assessment criteria that match.6
In order to attempt to improve the quality ofwritten evidence it is suggested that coursesshould:
� review their understanding of the concept andcomponents of critical thinking;� consider how these skills can be taught within
the curriculum;� persuade individual lecturers to focus on one of
the components of critical thinking and teach itwithin their own content area;� encourage students is to think critically;� assess students on one component of critical
thinking at a time using either the context ofwritten assignments or short clinical scenariosor case studies.� ensure individual lecturers learn how to teach
and assess their component of critical thinkingto improve rigor and reliability of assessment
In order to accurately and reliably assess thecritical thinking skill component, it is necessary toensure that all assessors use an appropriatemarking grid. Whilst it is acknowledged that thereare a number of more sophisticated critical think-ing marking grids available,2,19,20,21 the grid inFig. 7 (relating to the critical thinking component:information seeking) could be used and adaptedand would also provide useful feedback tostudents.
The key to teaching critical thinking lies indiscouraging rigid, narrow, biased and emotivethinking and encouraging open-mindedness,a questioning approach and independent thinking.Evidence suggests that students can learn to de-velop and subsequently demonstrate critical think-ing skills in academic scripts over time22 and tofacilitate this educators can help students toclarify issues, look for evidence, develop opinionsand make reasoned judgements. In this wayperhaps the positive perceptions of students canbe more closely matched to the actual outcomesof the learning process.
References
1. Bloom B, Englehart M, Furst E, Hill W, Krathwohl D.Taxonomy of educational objectives: the classification ofeducational goals. Handbook 1: cognitive domain. NewYork: Longmans Green; 1956.
2. Facione PA. Critical thinking: a statement of expertconsensus for purposes of educational assessment andinstruction ‘the Delphi Report’. Millbrae: California Aca-demic Press; 1990.
3. Scheffer BK, Rubenfeld MG. A consensus statement oncritical thinking in nursing. Journal of Nursing Education2000;39:352e9.
4. Department of Health. Meeting the challenge. A strategyfor the allied health professions. London: DOH; 2000.
Assessment of the critical thinking skills 95
5. College of Radiographers. A curriculum framework forradiography. London: COR; 2003.
6. Biggs J. Teaching for quality learning at university.Buckingham: SRHE and Open University Press; 1999.
7. Saarman L, Freitas L, Rapps J, Reigel B. The relationship ofeducation to critical thinking ability and values amongnurses: socialisation into professional nursing. Journal ofProfessional Nursing 1992;8(1):26e34.
8. Kember D. A re-conceptualisation of the research intouniversity academics’ conceptions of teaching. Learningand Instruction 1997;3:255e75.
9. Soden R, Pithers B. A study of teaching, learningand thinking in further education. Paper presented atthe fourth international conference ‘Vocational Educationand Training Research’, University of Wolverhampton;2001.
10. Nisbet J. Teaching thinking: an introduction to the researchliterature. Edinburgh: Spotlight no. 26, SCRE; 1990.
11. McGuiness C. From thinking skills to thinking classrooms:a review and evaluation of approaches for developingpupils’ thinking. Nottingham: DfEE Publications; 1999.
12. Herrington A, Cadman D. Peer review and revising in ananthropology course. College Composition and Communi-cation 1992;42:184e99.
13. Allen GD, Rubenfeld MG, Scheffer BK. Reliability ofassessment of critical thinking. Journal of ProfessionalNursing 2004;20(1):15e22.
14. Newell JA. The use of peer-review in the undergraduatelaboratory. ASEE National Meeting CDROM: Session 2513;1996.
15. Ludlow DK. Using critical evaluation and peer-reviewwriting assignments in a chemical process safety course.Proceedings of the American society for engineeringeducation annual conference and exposition; 2001.
16. Castle A. Overcoming barriers to critical evaluation.Synergy 2003;January:11e4.
17. Adams DS. A cheap and easy method for ‘teaching’ criticalthinking inadevelopmentalbiologycourse.!http://sdb.bio.purdue.edu/SDBEduca/danyadams/criticalthinking.htmlO;2000 [accessed October 2004].
18. California Academic Press (nd). The teaching for thinkingstudentcourseevaluation form. !http://www.calpress.com/evalform2.htmlO [accessed October 2004].
19. Guidance notes for critical essay writing (nd). !http://www.gre.ac.uk/~bj61/talessi/tlr30.htmO [accessed January 2005].
20. The critical thinking community (nd). !http://www.criticalthinking.org/about/internationalCenter.shtml#ICATO[accessed January 2005].
21. California State University. Academic Program AssessmentPlan: Public Policy and Administration Department 2000.!http://www.csus.edu/acaf/Assessment/PPA%20Assessment.pdfO [accessed January 2005].
22. Gopee N. Demonstrating critical analysis in academicassignments. Nursing Standard 2002;16(35):45e52.
