Feedback: critiquing practice, moving forward

Helen Williams

Abstract

Despite attempts by higher education institutions to improve the quality of feedback on assessed work, dissatisfaction expressed by students remains visible. This paper draws upon the preliminary findings of a Higher Education Academy Collaborative Research Project on assessment feedback within two large political science and international relations departments. It offers a critical review of current feedback practices and processes and identifies four key issues – negativity, transferability, intelligibility and consistency – that require attention in order to deliver more effective assessment feedback. The paper also suggests practical ways forward in addressing these issues, highlighting in particular the importance of both structure and timing in helping to produce high quality feedback efficiently.

Introduction

Despite attempts by higher education institutions to improve the quality of feedback on assessed

work, dissatisfaction expressed by students remains visible.1 This paper draws upon a Higher

Education Academy (HEA) Collaborative Research Project on assessment feedback (Project

Grant GEN264) that took place between June 2013 and December 2014 within two large

political science and international relations (IR) departments in the United Kingdom (UK).2 It

offers a critical review of current feedback practices and processes, drawing from the results of

the coding of 400 pieces of feedback at the two institutions, together with data collected from a

series of student-led focus groups and an online survey. This extensive analysis identifies key

issues and themes – negativity, transferability, intelligibility and consistency – that require

attention if departments are to deliver more effective assessment feedback. The paper also

discusses some practical ways forward in addressing these issues, reflecting in particular on the

importance of both structure and timing in helping to produce high quality feedback efficiently.

Bridging the Gap between the Production and Implementation of Feedback

There is now a substantial body of literature on the crucial role that assessment feedback plays in

the development of effective learning (see for instance Irons, 2007; Frankland, 2007; Brookhart,

2008; Joughin, 2008; Carless, 2006; McInerney, Brown, and Liem, 2009; Orsmond and Merry,

2010; McDowell, Sambell, and Montgomery, 2012; Blair and McGinty, 2013; Boud and Molloy,

2013). Indeed, high-quality feedback has been identified as ‘the most powerful single influence

on student achievement’ (Beaumont, O’Doherty, and Shannon, 2011: 671). Yet the literature

also points to high levels of dissatisfaction about feedback practices from students and lecturers

alike and, in particular, to the existence of a significant, and growing, ‘feedback gap’ between its

provision and implementation (Evans, 2013: 73). On the one hand, surveys from across the

world show that students are often unhappy with the quality and/or quantity of the feedback

they receive (Nicol, 2010), with key issues including that feedback comments are difficult to

interpret and therefore to implement, that they are too generalised and vague rather than focused

and personalised, and that they are too negative and are therefore demotivating (Carless, 2006;

Jones and Gorra, 2013). On the other hand, lecturers report spending ‘inordinate’ amounts of

time on the construction of written comments (Bloxham and Campbell, 2010: 292) that are not

always able to read, and still less acted upon, by their students (Nicol, 2010; Orsmond and Merry,

2011; Blair et al., 2013b). They do so, moreover, within the context of larger class sizes, growing

marking loads, heightened workloads, the modularisation of courses, and the end-loading of

assessments – all factors which, in turn, mean that staff have less and less time both to produce

written comments and to provide oral feedback to students in tutorial and other settings

(Hounsell, 2003; Bailey, 2009; Li and De Luca, 2014). And yet the marketization of higher

education, the widening of student access, the need for student retention and completion, and

growing resource constraints also mean that the provision of effective feedback is now more

important than ever before (Price et al., 2010; Evans, 2013; Blair et al., 2013a).

Although there is no general agreement on what precisely constitutes ‘good’ feedback (Evans

and Waring, 2011), there is something of a consensus that if feedback is genuinely to contribute

to effective learning and development, then it must be understood as ‘an active, shared process’

(Bloxham and Campbell, 2010: 291). This reflects a broader shift in which learning has come to

be conceptualised not as the simple transmission of information from teacher to student but

rather as a dialogue between teacher and student so that students are viewed not as passive

receivers but rather as active producers of knowledge (see inter alia Laurillard, 2002; Juwah et al.,

2004; Blair and McGinty, 2013; Crisp, 2007; Higgins, Hartley, and Skelton, 2001; Nicol and

Macfarlane-Dick, 2006; Poulos and Mahony, 2008; Lizzio and Wilson, 2008; Rae and Cochrane,

2008; Johnson et al., 2011). However, and despite the emphasis on student-centred learning both

within the pedagogical literature and from higher education institutions themselves, a variety of

studies show that it is the transmission, not dialogic, model that continues to inform actual

feedback practices in many higher educational contexts (see for instance Juwah et al., 2004;

Carless, 2006; Crisp, 2007; Higgins, Hartley, and Skelton, 2001; Blair and McGinty, 2013; Blair,

Curtis and McGinty, 2013). Simply put, feedback provision all too often takes the form of a

monologue or ‘series of unilateral pronouncements by assessors’ (Crisp, 2007: 578) rather than as

a two-way process through which students can engage with their lecturers on how to improve

(Blair, Curtis and McGinty, 2013). There is therefore a clear need to involve students much

more directly in the development of feedback practices, including in the debates surrounding

those practices, in order to promote open dialogue and discussion and therefore a culture in

which students are encouraged to regard themselves as active learners responsible for the

management of their own learning and development.

Our project aimed to substantially improve the student learning experience by addressing the

frustration that arises when students recognise that they are receiving feedback but do not

understand how to implement it. In the UK, National Student Survey (NSS) scores across

disciplines and universities consistently show that feedback-related questions receive the poorest

scores from students (Ipsos MORI, 2014). Besides poor general feelings about the feedback

they have received, there is a stark disparity between students’ estimations of the level of detail

and promptness of the feedback compared to their ability to implement the feedback: the five-

year averages of the two departments of this bid show a greater than ten-point gap between

responses to ‘I have received detailed comments on my work’ and ‘Feedback on my work has

helped me clarify things I did not understand’. This gap is not inevitable, and initial exploration

demonstrates that summative feedback can be improved and can bridge the communication

divide.

At the same time, and although a number of studies have shown the effectiveness of formative

feedback in particular (Shute, 2008; Juwah et al., 2004), the project also explicitly recognised how

summative assessment is unavoidable for many departments due to constraints of time, staffing

and resources. For example, the two departments involved in this project teach over 1,400

undergraduates a year, processing more than 15,000 individual assessments and providing tens of

thousands of words of feedback. Yet despite the thousands of hours invested in feedback

provision — both written and in-person via tutorials — there are clear disparities between

lecturers’ intentions and students’ interpretations of feedback, between lecturers’ efforts and

students’ perception of such efforts and, more fundamentally, between the desire to promote

student-centred learning and the lack of open dialogue about feedback practices.

The project therefore aimed to explore and address the persistent communication gap between

the provision of feedback by lecturers and the implementation of feedback by students. It did so

by involving students directly in the research and, in so doing, presented the opportunity for

truly dialogic feedback and to make students stakeholders in the learning process. Central to the

project was the use of virtual learning environments (VLEs), which are increasingly being used

by universities and which have the potential to offer different and expanded opportunities for

learning for students whilst also allowing greater flexibility for lecturers (Johnson et al., 2011:

499). In 2011-12, following a scheme piloted by one of the applicants on this project, the

Department of Political Science and International Studies at the University of Birmingham rolled

out electronic marking through GradeMark,3 a component within the widely-used Turnitin

plagiarism detection software. Without changing any other aspects of the feedback process, the

NSS score on utility of feedback jumped by more than 10 points above the previous five-year

average. The project therefore set out to offer a more thorough, detailed investigation in order

to interrogate what constitutes effective students that students can understand and act upon.

Methodology

This project used three sources of empirical data:

A sample (n=400) of coursework feedback drawn from both departments, covering all

three undergraduate levels;

Focus groups with students (n=60) to explore their perceptions of feedback in both

departments;

An online survey of students (n=186) to gain a general understanding of how students

rate the feedback they receive and what they do with it.

The first phase of the project involved analysing a sample of previous years’ written feedback to

determine common themes in feedback emerging from markers’ perspectives. The sample was

drawn from all three years of undergraduate teaching at the two universities.4 All feedback was

related to coursework as opposed to exams, and the vast majority of the coursework was in the

form of essays.5 All pieces were anonymised immediately after collection, retaining only

information about which degree level the feedback was given.6 The purpose of this audit was to

form a snapshot of current feedback practices, focusing particularly on critiques commonly

raised across markers, modules, years, and institutions.

The focus groups were conducted in phases, aligning with the general phases of the project, with

twelve focus groups in total taking place during Phase 1 and Phase 2 (one per year group for

each institution during each phase).7 These were run by students from each year group who had

been trained as facilitators, with a postgraduate research assistant also in attendance for the

purposes of recording the session (but who had no involvement in the discussions). The audio

recordings were subsequently transcribed by the postgraduate research assistants before being

given to the research team in order to retain complete anonymity for the participants in an effort

to encourage honest discussions. The aim of the Phase 1 focus groups was to gain greater insight

into how students understand feedback on summative assessment, pinpointing what they do

with it and how they interpret the ‘language of feedback’. The aim of Phase 2 focus groups was

to test the reception of different formulations of feedback to construct phrases that express

common critiques in language comprehensible to students. Focus group participants also

completed a short questionnaire to get a rough idea of both the participant’s profile and of their

personal view of feedback practices, independent of the discussion.8 Focus group discussions

followed a semi-structured format, with nine core questions addressed in all focus groups. A

general background of the focus group participants’ characteristics can be found in Table 1.

Table 1. Characteristics of focus group participants

Characteristic N

Level

First-year undergraduate 16

Second-year undergraduate 19

Final-year undergraduate 25

Sex

Male 24

Female 36

Achievement to date

First 2

2:1 21

2:2 1

Third 0

No response 36

Secondary school type

Comprehensive 37

Grammar 8

Independent/Private 13

International 2

Other 3

Selectivity of secondary school

Non-selective 28

Partially selective 11

Fully selective 15

No response 6

Total participants 60

Source: compiled by the authors

Transcriptions of the feedback and focus groups were coded qualitatively using the software

programme QSR NVivo, using a largely emergent coding framework. Feedback was coded

according to the topic of the content (e.g. quality of argument, use of sources, referencing);

whether the comment was framed negatively or positively; whether the content was specific to

the essay topic or more general; and whether the comment was focused on explaining the mark

given (feeding back) or provided suggestions for improving future work (feeding forward).

Focus groups were coded according to responses to the semi-structured questionnaire as well as

themes. Alongside, but separate from the focus groups, a short online survey of second-year

students was conducted at both institutions (n=92 at one institution; n=94 at the other). This

was designed to provide contextual information about what students think about the quality and

quantity of the feedback they receive and to get an idea about what students do with the

feedback after reading it.

Findings

In this section we discuss the findings of the feedback audit, the focus groups, and the online

survey. The results from these three different sources of empirical data fall into four broad

categories: negativity, transferability, intelligibility and consistency. The different sources of data

triangulate to give very consistent results on these themes. We will discuss each of these in turn

before offering some suggestions about what we can do to address the issues raised.

Negativity

The feedback audit found that the phrasing of the comments in feedback given to students was

overwhelmingly negative. Although it is to be expected that comments will become more

negative as the quality of the work decreases, it came as a surprise that 56 per cent of the

feedback on submissions awarded a First was negative (Figure 1)9. This was coded and calculated

on a per-word basis, so sentences that had some positive and negative characteristics were split

for coding. This negativity was also highlighted by students in the focus groups, who noted that

they were always told what was wrong with their work but not what was right. One student

explained:

I think it is also important to include good things as well and I think the best sort of feedback

is where they are critical but it’s not harsh like I found a lot of feedback…it comes across like

they are being so mean about it and you just think it’s a bit like unjustified. So, I think, that

needs to come across a bit more approachable.

Although markers may follow a tacit assumption that if they do not comment on something then

it is ‘right’, this is not how students experience those comments. Equally, negative comments

often highlight to the student what is wrong but do not indicate how to achieve a better

outcome. This is linked to the general frustration from students about a lack of transferable

feedback discussed in greater detail below.

Figure 1. Balance of positive to negative comments

n=400 Source: compiled by the authors

Focus group discussions made it equally clear, however, that wholly positive feedback was also

unwelcome. As one focus group participant commented:

Sometimes I don’t think that it’s critical enough, like, it says you did well in this, you did OK

with this but it doesn’t say the things you can improve on.

Students noted that one of the fool-proof ways to achieve student dissatisfaction with feedback

is to give a comment such as ‘This is a very good submission with few faults’, accompanied by a

mark of 65. A mid-Upper Second-Class may be ‘good enough’ for the marker – but not good

enough for students who want to improve. Instead, the focus group discussions across year

groups conveyed the students’ desire to receive a mixture of positive and negative comments.

Students also frequently mentioned the counterproductive effect of having wholly negative

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

First UpperSecondClass

LowerSecondClass

Third Fail

Positive

Negative

and/or excessively harshly phrased comments, the receipt of which caused students to feel

demoralised and to disengage from the learning process. The comment from one focus group

participant that ‘most of the feedback received last year was so bad and harsh, I ignored it’ drew

widespread agreement from other students.

The desire for a mixture of positive and negative feedback was also underscored by the

individual questionnaire results returned by first-year focus group participants. These students

were asked to rate the four characteristics of feedback they perceived as most important from

eleven possible options (Table 2). Eleven of thirteen respondents chose ‘Feedback should help

improve performance’, followed by seven each who chose ‘Feedback should be critical’ and

‘Link between feedback and grade’. This suggests that students do not expect to receive wholly

positive comments, but they do expect to receive constructive criticism that indicates to them

both the negative and the positive. Their expectation that the feedback will explain to them the

mark they received is an expectation held commonly between both lecturers and students and

indicates that, on this at least, both parties agree. The link between feedback and future

performance leads us to the next theme: transferability.

Table 2. Most important characteristics of feedback for studentsa

Rated 1st Rated 2nd Rated 3rd Rated 4th Rated in top 4

Feedback should help improve performance

2 2 6 1 11

Feedback should be critical 3 2 0 2 7

Link between feedback and grade 3 1 0 3 7

Feedback should be from an experienced teacher

1 1 1 3 6

Knowing how to implement advice given in feedback

0 2 2 1 5

Returned in a timely manner 1 1 0 2 4

The quantity of feedback 1 1 1 0 3

Opportunity to discuss feedback 1 2 0 0 3

Consistency between feedback across departmenti

0 0 2 1 3

Subject-specific feedback 1 0 1 0 2

Skill-specific feedback 0 1 0 0 1

Source: compiled by the authors

a This option was only added in the second round of focus groups, so the lower number of selections may not indicate its actual level of importance.

Transferability

Both the focus group discussions and the survey free responses underscored not only the

variability but also the lack of transferability of the feedback they received. Many students

complained that the comments received were so focused on the subject-specific content of the

particular essay as to be inapplicable to improving generally. This student insight proved

consistent with our findings from the feedback audit, which found that seven times as many

words were coded as topic-specific compared to suggestions for improvement. These comments,

some of which could have contained transferable suggestions, were framed in such a way as to

make its transferability obscure to students with lower levels of feedback literacy.

Students’ frustrations were generally focused on the fact that they did not feel that the feedback

gave them enough information to improve future results. As Table 2 shows, ‘Feedback should

help improve performance’ was the most important characteristic of feedback from first-year

students’ perspective. This finding is further underscored by the questionnaire completed by

second- and third-year focus group participants: one-third felt that they received sufficient

feedback on assessments to improve their grade; an identical number disagreed or strongly

disagreed with this. Focus group participants frequently felt that the feedback they received was

more focused on justifying their mark than on telling them how to improve future performance.

Follow-up discussions in Phase 2 focus groups revealed that students especially felt the need for

transferrable feedback in the first two years of their degree but that they felt that topic-specific

feedback played more of a role in their final year. There are some potential underlying

phenomena here: students in their first two years of study are still trying to grapple with the

expectations of university assessment, while many finalists feel that they have generally figured

things out or given up; finalists generally have a wide array of elective modules to choose from

and may use them strategically to learn more about their dissertation topic, making topic-specific

feedback more valuable; and finalists are not looking toward the next year’s assessment when

collecting feedback and are therefore less concerned about its transferability.

Interestingly, students also expressed some dissatisfaction if feedback was too generic and/or

vague. Comments to this effect in the anonymous survey, for instance, included:

[S]eems like the marker hasn’t even read the essay properly but just gives a generic comment

which is pathetic seeing as we pay to go to university and they don’t give good feedback in

order to show how we can improve.

[T]he feedback provided is normally a few general sentences … I want specific comments on

my piece of work.

Needs to be more personal.

Deeper probing of such comments in the focus groups revealed that students are fairly

pragmatic about recycled comments: they do not resent receiving some of the same comments as

their peers, so long as the combination of comments is clearly specific to the individual submission.

The frustration arose not from noticing that lecturers reuse comments but from the feeling that

everyone, regardless of performance, received roughly the same comments. Students then felt that

they have been given some general information that may or may not apply to their submission

and that they are left to sift through it to find something that they can use. This is not, therefore,

an argument against reusing the same critiques, so long as the comments are combined in a

manner that is appropriately specific to the submission.

Intelligibility

The issue of transferability also relates to that of intelligibility, for it is not enough simply to

identify common themes to students that need to be tackled in their work; these themes need to

be articulated in a clear way that students can both understood in principle and implement in

practice. In fact, the feedback audit revealed a number of common critiques being offered by

markers that, moreover, were the same across the two institutions and across all levels of

teaching. The most common themes were grouped around argument, sources, and presentation

standards (Table 3).

Table 3. Common critiques offered by markers

Argument Sources Presentation

Criticality/depth Number General standards Clarity Comprehension Grammar Structure Use of key sources Spelling Originality Referencing Punctuation Use of examples/case studies Use of quotations

Source: compiled by the authors Given that we found that lecturers at all levels consistently critiqued the same errors, it is clear

that we are not conveying adequately to the students what to do to fix many of these problems.

Addressing issues of presentation standards, for instance, might seem to be fairly straightforward

– but students frequently expressed a feeling of inability to transfer the comments received into

concrete improvements. For example, when a lecturer advises that a piece of work needs to ‘be

more critical’, many students can neither identify what this actually means nor understand how to

improve on it in future submissions. This is epitomised by the frustration of one third-year

student:

I don’t really understand what it means. Half of the feedback I get (…) doesn’t really make

sense to me. (…) this is what you did wrong, be more critical but I don’t really understand

how to do that.

This indicates that, though lecturers highlight similar issues with many students’ work, the

students do not understand the language used in the critiques themselves, which prevents

students from improving and can lead them to disengage with future feedback.

Focus group participants frequently expressed the feeling that it would be easier to know what to

do with the feedback they received if common critiques were accompanied by concrete

suggestions. One student suggested:

It’s better when they talk about what’s gone wrong, but they set out what to improve, so you

know what to improve on. Like your structure is poor, this could be helped by signposting at

the start of every paragraph. When they tell you your structure is poor, well obviously you’ve

written the essay so you didn’t see anything wrong with it, so them just telling you it is poor

isn’t helping.

In response, another student felt that, if it were a choice between cryptic feedback and the

marker not commenting on a problem at all, they would rather have the cryptic feedback and try

to figure out what to do with it. What all of this highlights is that students are not able to

interpret potentially transferable suggestions, even when they are provided, because they lack the

conceptual tools to interpret the information provided and to transform that into concrete

improvements on future submissions.

Consistency

This section examines the question of consistency from several angles: perceptions of quantity

and quality of feedback; and views on the consistency of feedback received across modules,

markers, and years. The results of the feedback audit are generally corroborated by the survey

and focus group results: the quantity and quality of feedback varies drastically between markers.

There was large variation in the quantity of feedback given between institutions and between

markers (see Figure 2). The general trend towards more words with decreasing marks is probably

unsurprising, given that there are fewer points to critique in the higher-scoring submissions. The

anomaly, however, is that work receiving a Third received roughly the same amount of feedback

as an Upper Second-Class and quite a bit less than a Lower Second-Class, thereby giving them

less feedback to improve and leaving the impression that the markers have become so

disillusioned as to have given up writing feedback. Furthermore, the means show that even those

receiving the greatest amount of feedback still only received around 150 words, with Firsts

receiving around 90 words. Given the variations between markers, this means that some students

are receiving 200-300 words, while others are receiving one or two sentences. To provide a

measurement for comparison, this paragraph is longer than 150 words.

Figure 2. Words of feedback per essay (mean average)

n=400 Source: compiled by the authors

Clearly, high quantity feedback does equate to high quality feedback but the issue of quantity is

nevertheless relevant to how students perceive the ‘quality’ of the feedback they are receiving.

Students had very mixed responses about the quantity of feedback they were receiving, and the

picture at the two universities was also very different (Figure 3). Student responses and answers

to open-ended questions highlighted the same trend that we noticed when auditing the feedback:

there is huge variability in the feedback students receive from marker to marker, both in terms of

quality and quantity. Only a minority of students (32.9 per cent for one department and 6.4 per

cent for the other) felt that they were receiving enough feedback across-the-board, and a

0

20

40

60

80

100

120

140

160

First Upper SecondClass

Lower SecondClass

Third Fail

minority (37.8 per cent for University 1 and 5.3 per cent for the University 2) felt that the

feedback was consistently of high enough quality to help them improve over time. As one

student noted in the online survey: ‘Some tutors give useful and detailed feedback … It totally

depends on the marker. You can usually tell when they have left the marking to the last minute’.

Another stated: ‘Some tutors give excellent, comprehensive feedback. Others, on the other

hand, give one-line responses … which is useless’. The clear majority (around two-thirds) felt

that both the quantity and the quality varied significantly across assignments.

One perplexing finding in these results is that the department giving fewer words of feedback,

on average, received more positive results about the quantity of feedback being provided. It is

possible that this is partially because the survey was administered at the same time as students

received feedback on a specific piece of coursework and that the quantity of feedback received

on that assessment influenced answers; it is also possible that this is linked to perceptions of

quality versus quantity.

Figure 3. 'Quantity of feedback: are you getting enough of it?'

n=186 Source: compiled by the authors

Views on the quality of feedback also varied between universities, following roughly the same

pattern as for quantity of feedback received (Figure 4). Despite inter- and intra-institutional

variations, the survey results do indicate that very few students find the feedback they receive

universally unhelpful, but students were frustrated by the lack of consistent critiques. Without

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Uni 1 Uni 2 Combined

I'm not getting enoughfeedback on any of mymodules.

Overall, I'm getting anadequate amount offeedback but I'd definitelyfind it helpful to get more.

It varies - sometimes I getloads and other times notenough.

consistent comments, there were no general trends they could identify in the problems identified

by one marker compared to another. This made it very difficult for students to identify what had

gone wrong and therefore what steps they could take generally towards improvement. This was

again paralleled by the findings of the audit: not only was there a paucity of transferable

feedback, there were also no clues in the feedback on modules with multiple coursework

assessment points that could explain why some students’ marks improved or declined drastically

between assessment points.

Figure 4. 'Quality of feedback: how helpful is it?'

n=186 Source: compiled by the authors

Exasperation about inconsistency is tied to feelings about fairness and the general perception

that the whole marking process is very opaque and secretive. The inconsistency of quality

between markers leads to a sense that some students get ‘lucky’ with their marker, while others

are ‘unlucky’:

There is a big disparity depending on who is marking which assignment. Certain students

may have been unlucky and received consistently weak feedback and vice versa.

Consistency also covers students’ need to see the same critiques repeated between markers in

order to identify a pattern of what is going wrong:

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Uni 1 Uni 2 Combined

I never (or rarely) find thefeedback I get helpful, andI'm really struggling toimprove.

The feedback I get is notbad, overall, but it's notalways as helpful as itcould be.

It depends on the module- some is really helpful,some isn't.

The feedback is great - Ireally understand it and it'shelping me to improveover time.

[W]hat I’d do is I take the feedbacks from all my different essays and then kind of see which

points are consistently I’ve failed to do across all of them or I’ve consistently done OK with.

If each marker of each assessment focuses on different aspects to the effect that there is no

overlap between assignments, it is difficult for students to pinpoint overall problems.

Inconsistent feedback, whether from quality, quantity, or critiques, hampers students’

development of feedback literacy and their ability to improve. It also prevents students from

identifying when differences in critiques are because of the different demands of different areas

of their degree — e.g. theory, IR, comparative politics — versus skills they need to develop

across the board. This frustration is captured by a first-year student:

I had an essay back recently and the tutor literally just highlighted random words going

through it and had just written ‘OK’ next to it. I don’t quite know what I’m meant to get out of

that. There were comments and stuff but it wasn’t, it was all a bit minimal. I’m not going to be

able to respond to something if it’s just kind of going through the essay. Another one of my

tutors…went through and wrote really detailed like stuff all the way through the essay with

each line, what’s good, what’s bad, so much annotation and this other tutor I had for another

module literally was just, ‘OK, OK, OK’. I don’t know what that’s meant to be telling me.

What can we do?

Thus far, we have painted a fairly bleak picture of negative, inconsistent, non-transferable

feedback and frustrated students. Although it might be tempting to dismiss our findings are

relevant only to the two departments in question, it should be noted that student satisfaction

with assessment and feedback is poor in other political science departments too (Smith and

Williams, this volume) and that the two departments in question score higher-than-average for

overall satisfaction with assessment and feedback compared to other ‘top’ political science

departments (ibid). This suggests that our findings are relevant for other institutional contexts

and, given that low levels of satisfaction appear to be generalised rather than simply confined to

political science, we believe that they are relevant for other disciplinary contexts too. The most

important thing we want to emphasise, however, is that we are not powerless to improve the

quality and effectiveness of our feedback, and there are some concrete things that we can do to

improve. In this section we argue that the basic, underlying issue of feedback structure underpins

the key problems outlined above, and that such problems can in large part be tackled – if not

entirely overcome – by thinking carefully about the structure (rather than just the specific

content) of feedback offered to students. We also discuss the importance of timing in the

production of feedback, considering the three stages at which we can improve outcomes for

students: before submission, during marking, and after returning the work.

Structure

There are two possible ways we can alter our feedback structurally to address the issues raised in

the previous sections of this paper: through the use of section headings and by harnessing the

benefits of technology.

Section headings

The focus groups discussed students’ feedback experiences in general and some particular

examples of ‘dummy’ feedback modelled on the styles found during the feedback audit. The

format of the example feedback varied, from completely unstructured to fully structured

according to defined themes. The content also varied in length, detail and wording. Participants

were then prompted to discuss the example feedback, giving their opinion of the different

options and indicating which of the examples they would prefer to receive themselves. There

was broad agreement between participants and between institutions about which of the examples

would be their preferred feedback as well as which feedback they thought was the least desirable.

Students expressed a strong preference for the feedback examples that were divided into

sections. Students also preferred examples that contained both positive and negative comments,

even if the mark was undesirable, and which had a paragraph devoted to how students could

improve in the future. This is a clear result that indicates we can signpost our feedback by

dividing it into categories, ensuring one of them is ‘suggestions for improvement’. This would

serve two purposes: to remind us what the students need feedback on and to highlight to

students what information is contained in that section.

Technology

The second structural aspect we can change in our feedback is through the use of technology.

Technology can serve several purposes: to allow us to provide more feedback in less time

through the use of repeated comments; to provide a mixture of in-text annotations and general

comments; and to identify trends across submissions, which can help lecturers to target post-

submission advice to students by highlighting common errors.

As discussed above, whilst students desire ‘personal’ feedback, this does not mean that they

think lecturers should never use repeat comments, just that the comments should be provided in

a combination unique to that student’s submission. Exploiting technology means that lecturers

can develop their comments in greater detail, including examples and further information for

improvement.

The second advantage is through being able to provide both in-text annotations and general

comments if using electronic marking software such as the GradeMark system embedded in

Turnitin. Students expressed a strong desire to receive feedback in both forms. In-text

annotations help the students to pinpoint precisely where they went wrong, for example, where

problems with referencing are and what should be fixed. General comments, on the other hand,

give them an overview of how they did and what they need to do better. Many of the electronic

systems also allow tracking of the number of times different comments are used, which provides

feedback to the marker about common problems that should be addressed with all students.

This leads us to the final sections, which explore the three points at which lecturers have a

chance to provide feedback.

Timing

The importance of timely feedback is well recognised in the pedagogical literature (see for

instance Evans and Waring, 2011; Ferguson, 2011; Higgins, Hartley, and Skelton, 2002; Blair et

al., 2013a; Li and De Luca, 2014) for, as Beaumont et al. (2011: 685) argue, understanding

feedback as a dialogic process rather than as a single event highlights the significance of timing to

learning and how, ‘although principles of good practice are useful’, they must also ‘be

systematically implemented’ at appropriate points within the learning cycle in order to be

effective. Our online survey data also illustrated how timeliness of feedback is critically

important to students, with comments including: ‘The feedback is often given after other

deadlines meaning it’s pretty useless as you cannot use the advice’; ‘On a few occasions the

feedback and marks have come in later than expected which is slightly frustrating because by

then the next assessment has already been handed in so don’t know how to improve’; ‘When I

get feedback it’s good – but it’s no good having feedback a month before your exams – you need

it earlier in the year’; and ‘Often, it doesn’t come back until we’ve already handed the next paper

in too, making the small amount of feedback we do get pretty much useless for immediate

application’. If feedback is indeed understood as a dialogic process rather than as a single event,

then it is crucial to view it within the broader context of the whole assessment cycle. Here we

consider the key stages of assessment and reflect on the implications for the design and

implementation of feedback.

Before submission

Students clamour for clear criteria and examples. Whilst it is not always practical to provide the

latter, we should always make an effort to provide the former, unless we are testing students’

ability to read our minds or follow hidden instructions. Providing clear criteria in advance — and

dedicating time in seminars to discussing the criteria and answering questions about them —

significantly reduces the marking burden. We frequently write very general criteria cloaked in

feedback jargon, figuring that the criteria are more for the markers than for the students. It is

time to change this tradition. While there should be some general elements to the marking

criteria, there should also be elements specific to the learning outcomes, topics, and skill

development expectations of the assignment. There should be a clear connection between the

criteria and the assignment itself. Let us underscore this with two examples from our experience.

The first example is drawn from a workshop presentation about the importance of marking

criteria for outcomes: academic participants were given a series of assignment details and

marking criteria and asked to match the criteria to the assignment. The modules from which the

materials were taken varied widely, from research methods to American politics; and the forms

of assessment varied as well, from empirical data reports to standard essays. Given the diversity,

the differences between the assignments should have been at least vaguely discernible in the

marking criteria. They were not: none of the participants could accurately match the materials.

We have to remember that, especially when we use non-traditional assessment forms, students

are very unsure of what we want and are looking for extra information.

The second example is drawn from a research methods module. Because it was running for the

first time in a very different format, there were no past student submissions to provide as

examples to convey to the students what we were looking for. Instead, students were provided

with a substantial grid with weighted marking criteria and descriptions of the qualities of each of

the different grades for each of the components. Half an hour of one of the seminars was

devoted to going through the criteria and answering questions about them. Students gave very

positive feedback for the criteria and wished that such specific criteria were available for other

modules. Many of the students wrote their assignments with the criteria to hand, and students

were given feedback on their achievement on each individual component when they received

their marks back. The benefits did not stop there, however. This was a large module with nine

people on the teaching team. When the team met in a marking calibration exercise before

embarking on independent marking, each person marked the same submission at the same time.

Referring only to the marking criteria, all markers landed on the same result.

In an era where student satisfaction drives policies and complaints about inconsistency and lack

of transparency are rife, such examples should make us pause to consider the possible impact of

improving the standards of criteria given to the students. Incorporating the criteria into the

marking process and providing feedback on the individual elements also helps students to spot

the smaller areas in which they did well and to identify where improvement is most needed.

The other essential aspect of the pre-submission stage is availability. Again, some academics

seem to feel that students should be able to ‘figure things out’ and that this is somehow part of

the whole assessment process. However, this is nearly guaranteed to make the marking

experience worse for the marker. This can be mitigated by ensuring availability at office hours,

repeatedly encouraging students to use office hours, and answering questions during seminars in

the run-up to the due date. Taking ten minutes or less at the beginning of seminars for the last

weeks before submission has the added benefit of reducing the number of emails received asking

the same questions.

During marking

There are some relatively small changes that we can make to our marking practices to improve

feedback reception and to address the most common student frustrations. The first step is to use

feedback forms with structured sections on different themes. This will ensure basic consistency

of areas of feedback between markers and modules, and the inclusion of a section on

transferable comments and suggestions for improvement can remind markers to provide

forward-thinking advice. We have to remember that, alongside all of the other skills we expect

students to learn, we generally do very little to assist directly in improving their feedback literacy;

providing a basic structure to the feedback gives visual clues to the students about how they

might use the feedback.

The second step is simply to reframe our feedback in two areas: the positive/negative balance

and transferability. As mentioned above, students do not know what they have done correctly if

we do not tell them and therefore learn by trial and error what they should repeat. In addition,

receiving entirely negative comments can be very demoralising and can leave especially struggling

students with the feeling of not knowing where to start. Some of the negative comments can be

framed more positively with slight tweaks to the phrasing, e.g. ‘Your answer to the question is

clear, but the structure (how you’re going to get from here to there) is not’ rather than ‘Structure

is unclear or missing’. This does require writing a bit more, but it should make the feedback

more effective. The same advice applies to transferability: simple rephrasing of the same point

can signpost to the student that this piece of feedback, whilst topic-specific, can also be applied

to other assignments. Compare:

You should have cited some examples when talking about the abortion debate in the USA.

Use of more examples would help to illustrate how well you understand the material. For

example, you talk about the abortion debate in general terms, but applying it to specific

examples would lend greater depth to your argument.

The third step, and linked to the previous point, is to use examples. Just as we ask students to

use examples to apply the ideas they are talking about, we should use examples to show them

what we mean. This can help students to grasp what we mean when we throw around feedback

jargon like ‘critical analysis’, ‘depth’, ‘structure’, ‘argument’, ‘vague’. Some of this can go into

general module feedback and does not need to be repeated on every submission, but students

should have access to examples of the difference between description and critical analysis – one

of the most difficult things for us to explain and for students to understand. Even without

students’ permission to use their examples, we can frequently find examples of what we want

students to emulate in published material.

After feedback is released

The final stage of the feedback process is the follow-up. This can take several different forms

but is critical for closing the feedback loop. It is critical for the assessment to be returned before

the next assessment point so that students have adequate time to implement the feedback they

have received, and one of the most effective ways of closing the loop is to carve out curricular

time to spend on feedback. This could consist of having the students fill out a self-reflection

form, asking them to pinpoint the differences between their expectations and the result, anything

they do not understand, and what they need to ask more questions about. If many students

exhibited the same weaknesses — such as referencing problems, lack of synthesis of sources,

grammar, structuring problems — time could be spent doing a few exercises in these areas as a

group. Even if it is not possible to follow up with class time, it is still important to give students

an idea of the general feedback on the performance in addition to the individual feedback. It is not

helpful to do one without the other, and one of the common ways to frustrate students is only to

give general feedback on an assessment, leaving them to figure out what applies to them and

what does not. However, assembling some critiques of things that commonly went wrong as well

as some of the things that were done really well and an idea of what the average (mean, median,

spread of marks) looked like allows students to compare themselves to their peers and

contextualise their performance.

We also need to do far more to encourage students to follow up on the written feedback they

received. Given that research shows how valuable dialogic feedback and the oral feedback

process are for students’ learning (Bloxham and Campbell, 2010; Evans, 2013), it is problematic

that this is still not a standard part of the feedback process for most students. Nearly a third of

the students surveyed responded that they had never, by mid-way in their second year, followed

up essay feedback by discussing it with their lecturer or seminar tutor. More than 40 per cent had

only done this once or twice; but it was only routine (‘most of the time’ or ‘always’) for five of

the survey respondents. This is despite the seminar tutor and module convenor of the students

surveyed having made repeated invitations to students, having an open-door policy, always

keeping office hours, and frequently meeting with students outside of office hours. The lack of

follow-up is likely to be at least partially a reflection of the perceived lack of transferability of

feedback from one assessment to the next: students do not follow up the feedback because it is

for an assessment that is already done; now they want to move on to the next thing. This means

that addressing the breaks in the feedback loop requires lecturer accessibility, sign-posted

feedback, clear transferability, and a cultural change in how students process and respond to

feedback.

Conclusions

Our findings from the feedback audit, the student survey, and the student focus groups show a

clear gap in the provision and implementation of feedback, in large part due to issues of

negativity, transferability, intelligibility and consistency. These problems are not, however,

impossible to address. Two of the simplest changes we can make is to ensure that feedback

comments are structured into clear categories through the use of section headings, and that

comments do not focus only on specific content (‘what to learn’) but also on future development

(‘how to learn’). Most importantly, we must carve out more time in the curriculum to provide

feedback before, during, and after the assessment submission. Ultimately, for feedback to be

effective, it needs to be embedded into broader practices that treat learning not as a product to

provide to students but rather as a dialogue to share with them. Feedback is perhaps the most

important way that we can actively involve students in this ongoing, dialogic process.

Acknowledgments

We are indebted to the Higher Education Academy (HEA) and University of Nottingham for

jointly funding this research (Project Grant GEN264) and we are also hugely grateful to our

Project Lead, Bettina Renz, and our Participating Investigator and Project Research Assistant,

Hardeep Basra, who organised and undertook the collection of data upon which our analysis is

based. Special thanks must also go to the many students acted as participants in, facilitators of,

and research assistants on, the project, without whom this research would not have been

possible. Finally, we are enormously grateful to the anonymous reviewer for their hugely helpful

and insightful comments on an earlier iteration of this piece.

Notes

1 For example, the Times Higher Education reported in 2014 that although student satisfaction overall had reached a ‘10-year high’ in the National Student Survey (NSS), assessment and feedback was rated the lowest – as it had been in previous years – with just 72 per cent of students reporting that they were satisfied with this element of their learning (Grove, 2014). 2 The School of Politics and International Relations at the University of Nottingham and the Department of Political Science and International Studies at the University of Birmingham. 3 GradeMark comes with a set of pre-programmed ‘quick-marks’ including basic grammatical and argumentation advice, but these were not formulated with extensive input from students and academics. 4 Our aim was to ensure that the sample covered a roughly equal number of modules across each of the years and roughly representative numbers of pieces of feedback across mark boundaries, but fully stratified sampling across institutions was not possible due to informed consent requirements, which causes a self-selection bias in the sample, as lecturers already providing higher quality feedback were more likely to participate. 5 Essays are by far the most common type of assignment to receive detailed feedback comments in the two departments and so this was our focus in the audit, but our findings may be relevant to other forms of assessment such as reflective learning logs. 6 The content of topic-specific comments may make it possible to identify the originating module in some cases, but as far as possible, identifying information has been excluded. 7 Due to space limitations we confine our discussion here to Phases 1 and 2 of the project but we also undertook an additional Phase 3, which included the collation of a ‘bank’ of more highly effective feedback available as OERs through a set of QuickMarks and qualitative rubrics in GradeMark format for institutions using Turnitin-based marking; Adobe fillable forms and Word templates with macros that can be used in paper or electronic form for institutions using paper-based marking. A final round of student-led focus groups were also run in Phase 3, in order to discuss the effectiveness of the feedback examples. 8 Unsurprisingly, the participant profiles indicate a self-selecting sample: of those who chose to provide information regarding their average achievement to date, nearly all participants reported they were achieving at an Upper Second Class level. While the distinct majority of students at both universities graduate with a ‘good’ degree (Upper Second Class or First), it does mean that the focus group results essentially exclude those who are achieving at a lower level, who are frequently harder to reach. 9 See Access to Higher Education (2012) for information on the UK grading system in Higher Education.

References

Access to Higher Education (2012) ‘Grading Scheme Handbook – Section A: Introduction and Summary’, available at https://www.accesstohe.ac.uk/AboutUs/Publications/Documents/Grading-scheme-A.pdf [accessed 6 November 2015].

Bailey, R. (2009) ‘Undergraduate Students’ Perceptions the Role and Utility of Written Assessment Feedback’, Journal of Learning Development in Higher Education 1, 1-14.

Beaumont, C., O’Doherty, M. and Shannon, L. (2011) ‘Reconceptualising Assessment Feedback: A Key to Improving Student Learning?’, Studies in Higher Education 36(6): 671–87.

Blair, A., Curtis, S., Goodwin, M. and Shields, S. (2013a) ‘Learning and Teaching in Politics and International Studies: What Feedback do Students Want?’, Politics 33(1): 66-79.

Blair, A., Curtis, S., Goodwin, M. and Shields, S. (2013b) ‘The Significance of Assignment Feedback: From Consumption to Construction’, European Political Science 12(2): 231-44.

Blair, A., Curtis, S., and McGinty, S. (2013) ‘Is Peer Feedback an Effective Approach for Creating Dialogue in Politics?’, European Political Science 12(1): 102-15.

Blair, A. and McGinty, S. (2013) ‘Feedback-Dialogues: Exploring the Student Perspective’, Assessment & Evaluation in Higher Education 38(4): 466–76.

Bloxham, S. and Campbell, L. (2010) ‘Generating Dialogue in Assessment Feedback: Exploring the Use of Interactive Cover Sheets’, Assessment Evaluation in Higher Education 35(3): 291–300.

Boud, D. and Molloy, E. (2013) Feedback in Higher and Professional Education: Understanding It and Doing It Well, London: Routledge.

Brookhart, S. (2008) How to Give Effective Feedback to Your Students, Alexandria: ASCD. Caffarella, R. and Barnett, B. (2000) ‘Teaching Doctoral Students to Become Scholarly Writers:

The Importance of Giving and Receiving Critiques’, Studies in Higher Education 25(1): 39-52.

Carless, D. (2006) ‘Differing Perceptions in the Feedback Process’, Studies in Higher Education 31(2): 219–33.

Crisp, B. (2007) ‘Is It Worth the Effort? How Feedback Influences Students’ Subsequent Submission of Assessable Work’, Assessment & Evaluation in Higher Education 32(5): 571–81.

Evans, C. (2013) ‘Making Sense of Assessment Feedback in Higher Education’, Review of Educational Research 83(1): 70–120.

Evans, C. and Waring, M. (2011) ‘Student Teacher Assessment Feedback Preferences: The Influence of Cognitive Styles and Gender’, Learning and Individual Differences 21(3): 271–80.

Ferguson, P. (2011) ‘Student Perceptions of Quality Feedback in Teacher Education’, Assessment & Evaluation in Higher Education 36(1): 51–62.

Frankland, S. (2007) Enhancing Teaching and Learning through Assessment: Deriving an Appropriate Model, New York: Springer.

Grove, J. (2014) ‘National Student Survey 2014 Results Show Record Levels of Satisfaction’, Times Higher Education, 12 August 2014.

Hattie, J. and Timperley, H. (2007) ‘The Power of Feedback’, Review of Educational Research 77(1): 81-112.

Higgins, R., Hartley, P. and Skelton, A. (2001) ‘Getting the Message Across: The Problem of Communicating Assessment Feedback’, Teaching in Higher Education 6(2): 269–74.

——— (2002) ‘The Conscientious Consumer: Reconsidering the Role of Assessment Feedback in Student Learning’, Studies in Higher Education 27(1): 53–64.

Hounsell, D. (2003) ‘Student Feedback, Learning and Development’, in M. Slowey and D. Watson (eds.) Higher Education and the Lifecourse, Maidenhead: Open University Press, pp.67-78.

Ipsos MORI (2014) National Student Survey (NSS), available at http://www.ipsos-mori.com/researchspecialisms/socialresearch/specareas/highereducation/nss.aspx, [accessed 31 May 2015].

Irons, A. (2007) Enhancing Learning Through Formative Assessment and Feedback, London: Routledge.

Johnson, E., Cowie, B., De Lange, W., Falloon, G., Hight, C. and Khoo, E. (2011) ‘Adoption of Innovative E-Learning Support for Teaching: A Multiple Case Study at the University of Waikato’, Australasian Journal of Educational Technology 27(3): 499–513.

Jones, O. and Gorra, A. (2013) ‘Assessment Feedback Only on Demand: Supporting the Few Not Supplying the Many’, Active Learning in Higher Education 14(2): 149–61.

Joughin, G. (2008) Assessment, Learning and Judgement in Higher Education, New York: Springer.

Juwah, C., Macfarlane-Dick, D., Matthew, B., Nicol, D., Ross, D. and Smith, B. (2004) Enhancing Student Learning through Effective Formative Feedback, York: Higher Education Academy.

Laurillard, D. (2002) ‘Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies’, New York: Routledge.

Li, J. and De Luca, R. (2014) ‘Review of Assessment Feedback’, Studies in Higher Education 39(2): 378–93.

Lizzio, A. and Wilson, K. (2008) ‘Feedback on Assessment: Students’ Perceptions of Quality and Effectiveness’, Assessment & Evaluation in Higher Education 33(3): 263–75.

McDowell, L., Sambell, K. and Montgomery, C. (2012) Assessment for Learning in Higher Education, London: Routledge.

McInerney, D., Brown, G. and Darmanegara Liem, A. (2009) Student Perspectives on Assessment: What Students Can Tell Us about Assessment for Learning, Charlotte: IAP.

Nicol, D. (2010) ‘From Monologue to Dialogue: Improving Written Feedback Processes in Mass Higher Education’, Assessment & Evaluation in Higher Education 35(5): 501–17.

Nicol, D., and Macfarlane-Dick, D. (2006) ‘Formative Assessment and Self-Regulated Learning: A Model and Seven Principles of Good Feedback Practice’, Studies in Higher Education 31(2): 199–218.

Orsmond, P. and Merry, S. (2011) ‘Feedback Alignment: Effective and Ineffective Links Between Tutors’ and Students’ Understanding of Coursework Feedback’, Assessment & Evaluation in Higher Education, 36(2): 125-36.

Poulos, A. and Mahony, M.J. (2008) ‘Effectiveness of Feedback: The Students’ Perspective’, Assessment & Evaluation in Higher Education 33(2): 143–54.

Price, M., Handley, K., Millar, J. and O’Donovan, B. (2010) ‘Feedback: All That Effort, but What Is the Effect?’, Assessment & Evaluation in Higher Education 35(3): 277–89.

Rae, A. and Cochrane, D. (2008) ‘Listening to Students How to Make Written Assessment Feedback Useful’, Active Learning in Higher Education 9(3): 217–30.

Shute, V. (2008) ‘Focus on Formative Feedback’, Review of Educational Research 78(1): 153-89. Weaver, M. (2006) ‘Do Students Value Feedback? Student Perceptions of Tutors’ Written

Responses’, Assessment & Evaluation in Higher Education 31(3): 379-94. Young, P. (2000) ‘‘I Might As Well Give Up’: Self-Esteem and Mature Students’ Feelings About

Feedback on Assignments’, Journal of Further and Higher Education 24(3): 409-18.

About the Authors

Helen Williams is Assistant Professor of Politics at the University of Nottingham. Between June 2013 and December 2014 she led the Higher Education Academy collaborative project on ‘Closing the loops: bridging the gap between provision and implementation of feedback’. Nicola Smith is Senior Lecturer in Political Science at the University of Birmingham. She was co-investigator on the Higher Education Academy funded project on assessment feedback entitled ‘Closing the loops: bridging the gap between provision and implementation of feedback’ which ran in 2013-14.

Key Quotes

1. ‘Although there is no general agreement on what precisely constitutes ‘good’ feedback, there is something of a consensus that if feedback is genuinely to contribute to effective learning and development, then it must be understood as ‘an active, shared process’. (pp.3-4)

2. ‘Given that we found that lecturers at all levels consistently critiqued the same errors, it is clear that we are not conveying adequately to the students what to do to fix many of these problems’. (p.13)

3. ‘…we are not powerless to improve the quality and effectiveness of our feedback…’ (p.18)

4. ‘Students expressed a strong preference for the feedback examples that were divided into sections’. (p.19)

5. ‘…receiving entirely negative comments can be very demoralising and can leave especially struggling students with the feeling of not knowing where to start’. (p.24)