Critical Review of Research #2

Using Peer Feedback to Enhance the Quality of Student Online Postings: An Exploratory Study

Katie Krieg

EDU 800

Critical Review 2 2

Critical Review

Problem

1. Clarity of the Problem

The problem this article addressed was clearly stated in the introduction of the paper. The

authors explored the use of peer feedback to promote higher levels of thinking during online

discussions. The introduction began by explaining that “student discussion has been identified as

a key component of interactive online learning environments; both instructors and researchers

agree that this is where the ‘real’ learning takes place” (Ertmer, et al., 2007, p. 412). While

feedback has proven effective in face-to-face classes, the authors pointed out how very little

research existed in regards to feedback in online learning environments, with even less focus on

how feedback enhances the quality of discussion responses. The authors also stated “the purpose

of this exploratory study was to fill this gap by examining students’ perceptions of the value of

giving and receiving peer feedback regarding the quality of discussion postings in an online

course” (p. 416). This statement was addressed in the purpose of the study section. From these

sections, the authors made their purpose and the problems they were addressing very clear.

2. Need and Educational Significance

The authors used the introduction to support the idea that good student discussion leads to

valid understanding, but that online discussions often focus around personal stories and do not

include reflections or critical thinking. To help improved reflections and critical thinking during

discussion, the role of feedback in instruction was introduced. Morey (2004) stated “in general,

instructional feedback provides students with information that either confirms what they already

know or changes their existing knowledge and beliefs” (p. 413). As Lynch (2002) and Palloff &

Pratt (2001) further pointed out, “feedback may be even more important in online leanrning

Critical Review 2 3

environments than in traditional classrooms” (p. 414). Students agreed that feedback was an

important factor in online environments and expteced it to be:

1. Prompt, timely, and thorough

2. Ongoing formative and summative

3. Constructive, supportive and substantive

4. Specific, objective, and individual

5. Consistent (p. 414)

Because of the amount of time this level of feedback would require of instructors, peer

feedback was suggested as a possible solution. After addressing the importance of feedback in an

online learning environment, the authors pointed out the lack of research that existed with the use

of feedback in online courses, particularly peer feedback. If discussions are an important element

of online courses, and feedback is essential in reaching higher levels of thinking, but very time

consuming, than it is justifiable to research how peer feedback can be utilized with online

discussion postings.

3. Researchable

This exploratory study focused on the impact of peer feedback on the quality of

discussion responses, and whether the quality could be maintained or increased through peer

feedback. While higher levels of thinking are measurable, they can be difficult to determine. The

authors addressed this challenge by utilizing the levels of Bloom’s taxonomy. “Postings at the

knowledge, comprehension, and application levels received one point; postings demonstrating

analysis, synthesis, or evaluation received two points; non-substantive comments received zero

points” (p. 417).

Critical Review 2 4

Student perceptions of the value of giving and receiving peer feedback were also

examined in this studied. The researchers used entry and exit surveys along with interviews to

obtain both quantitative and qualitative data regarding students’ perceptions of the value of

receiving instructor feedback compared to peer feedback, and giving peer feedback. These

methods would clearly allow the authors to research the problem.

Theoretical Perspective and Literature Review

4. Conceptual Framework

The conceptual framework of this paper focused on the relationships among three

distinct areas: discussions, critical thinking, and feedback. Ideas from these three areas are

consistently intertwined throughout the article. The authors pointed out the importance of student

discussion in the beginning of the paper. Cunningham (1992), noted “that it is the dialog among

community members that promotes learning” (pp. 412-413). As the authors emphasized the

importance of dicussion, they also noted the lack of critical thinking in responses. Deeper

thinking is an important part of the learning process, and therefore an essential part of learner

responses in an online discussion. To help promote deeper levels of thinking in online

discussions, the authors remarked about the importance of feedback and that good feedback

performs these functions:

1. Clarifies what good performance is (goals, criteria, standards)

2. Facilitates the development of self-assessment and reflection

3. Delivers high quality information to students about their learning

4. Encourages teacher and peer dialogue around learning

5. Encourages positive motivational beliefs and self esteem

Critical Review 2 5

6. Provides opportunities to close the gap between current and desired performance

7. Provides information to teachers that can help shape teaching (pp. 413-414)

The authors continued to build upon the three main concepts: discussions, critical thinking,

and feedback. Feedback was addressed from two perspectives: the instructors and peers. By

utilizing Bloom’s taxonomy, the instructors and peers evaluated the critical thinking levels of

discussion postings and provided feedback to participants. Feedback helps learners build upon

what they know, and therefore can help them reach higher levels of thinking, and deep thinking

is a crucial part of effective online discussions.

5. Relevant Theory and Prior Research

The authors began with evidence supporting the value of discussions in the learning

process. When focusing on online environments, the authors noted how discussions often

centered on personal stories and lacked the deep thinking necessary to achieve critical levels of

learning. This lead to information on the benefits of feedback improving learning. The authors

first focused on the general role of feedback in instruction and went further in depth with

information about feedback in online learning environments. Feedback was deemed to possibly

be even more important with online learning than face-to-face settings (p. 414). Specifically,

Ertmer and Stepich (2004) pointed out how “research has shown that the quality of student

discussion responses can be increased through the use of constructive feedback that is prompt,

consistent, and ongoing” (p. 414). Dunlap (2005) noted how instructors would have to be

constantly onlne in order to provide this level of feedback, something that would be rather

impractical (p. 414).

Literature research continued with advantages and challengaes of peer feedback. Previous

research pointed out the benefits of both receiving as well as giving feedback. Challenges

Critical Review 2 6

focused on the general difficulties in giving meaningful feedback. This literature review led to

the purpose of the study where the authors pointed out that little research existed examining the

impact of feedback in online learning environments. “Additionally, very few, if any, studies have

examined the impact of using peer feedback to shape the quality of discourse in an online

course” (p. 416). Due to the limit amount of existing research related to this study, the authors

effectiviely connected their investigation to prior literature.

6. Summary of Literature Review

The flow and clarity of the literature review move effortlessly, building from one idea to

the next. Although, the literature review does not structurally end with a summary of the

literature, the authors did address the brevity of previous literature that referenced online

feedback in the purpose of the study. Their summary of the literature is very brief, containing

only a fraction of a sentence before moving on to the reasons for this study.

While feedback has been demonstrated to be an effective strategy in traditional learning

environments, limited research has been conducted that examines the role or impact of

feedback in online learning environments in which learners construct their own

knowledge, based on prior experiences and peer interactions (p. 416).

This summary only hints at a tiny portion of the literature that was addressed in the beginning of

the article. It does not acknowledge discussion postings, nor does it summarize any perceptions

of feedback received by instructors and peers, or perceptions of giving peer feedback.

7. Clarity of Research Question

The research questions focused on feedback impacting the quality of postings, as well as

the perceptions of giving and receiving peer feedback. The authors very clearly state the research

questions in the purpose of the study (p. 416):

Critical Review 2 7

1. What is the impact of peer feedback on the quality of students’ postings in an online

environment? Can the quality of discourse/earning be maintained and/or increased

through the use of peer feedback?

2. What are the students’ perceptions of the value of receiving peer feedback? How do these

perceptions compare to the perceived value of receiving instructor feedback?

3. What are the students’ perceptions of the value of giving peer feedback?

To address the appropriateness of their research questions, the authors emphasized the gap

that existed in information about feedback in online learning environments. The authors had

previously stressed the importance of discussion as part of the learning process. They also

addressed how critical thinking during discussion is necessary for deep learning to occur (p.

413), supporting the first discussion question. Since time is a crucial factor when providing

feedback, peer feedback was suggested as a solution to instructor feedback (p. 414). The

suggestion of peer feedback formulated the second two research questions, focusing on both

receiving and giving peer feedback.

Research Design and Analysis

8. Study Design

Through this exploratory study, the authors used both descriptive and evaluative

approaches to examine not only the impact of feedback on discussion postings, but the

perceptions of giving and receiving feedback (p. 416). Descriptive studies gather information

without changing or manipulating the environment. They intend to collect data to demonstrate

relationships and describe situations. They do not make predictions or answer how, when or

Critical Review 2 8

why. Evaluation research looks to assess a situation, providing information beyond what one

might gain from a simple observation.

This study focused purely on “what” was happening. The authors wanted to know what

impact feedback had on discussion postings, and what the perceptions were of giving and

receiving feedback. These questions do not require you to use an experimental study and change

the environment. They also do not ask how, when or why. These questions require the researcher

to gather information about a situations, making the descriptive and evaluative methods an

appropriate choice for this study.

9. Sampling Methods

The study included 15 graduate students, 10 female and 5 males. All participants were

enrolled in an online technology integration course during the spring semester of 2005. The

participants include eight administrators, four former or current teachers, two international

students with no previous K-12 experience in the United States, and one student with a teaching

degree, but no teaching experience. Five of the participants were pursuing a master’s degree and

nine were doctoral students (p. 417).

Holton and Burnett (1997) stated that “one of the real advantages of quantitative methods

is their ability to use smaller groups of people to make inferences about larger groups that would

be prohibitively expensive to study” (p. 71). While a small sample size may be acceptable with

qualitative research, it can reduce the reliability. Beyond the sample size, little information is

known about the participants beyond education and gender. The sample does not include much

variety with 13 of the 15 participants having a background in education, which could make

generalizing results difficult. Also, with 10 of the 15 participants being female, this too may have

an impact on the overall ability to generalize results.

Critical Review 2 9

10. Procedures and Materials

Students provided feedback to one another through a numerical score (0-2) based on

Bloom’s taxonomy, and descriptive comments. Bloom’s taxonomy was chosen because of

participants’ familiarity with it, and because of the researchers’ successful implementation of it

in a similar graduate course. The instructors provided feedback for the first six weeks to model

the feedback process. Students provided peer feedback using the same method for the following

six weeks. The instructors received all peer feedback before distributing it to students to ensure

anonymity (p. 417). The scoring method using Bloom’s taxonomy and the scaffolding techniques

used by the instructors helped provide a consistent method for providing peer feedback.

This study collected both quantitative and qualitative data from participant interviews,

scored weekly discussion posts, and collected data from entry and exit surveys. Because postings

were rated by different students throughout the course and not all postings received a peer score,

the feedback scores from the students were inconsistent and incomplete. For data collection

purposes, two researchers scored all postings. All date, time, and discussion question numbers

were removed from postings to ensure this information did not impact results. The researchers

began by scoring a randomly selected discussion question and compared scores after every 10

postings. With 86% agreement between the two raters after processing all posts from the first

question, the two researchers divided and independently rated the remaining 16 discussion

questions. This process provided a uniform procedure for collecting data from students’ posts in

order to identify what, if any, impact peer feedback had on the quality of postings (pp. 418-420).

Students completed a survey with Likert-style items and open-ended questions at the end

of week five, prior to the start of peer feedback, and again in week 16. Participants also

completed an interview with one of the research members, via phone or in person. The survey

Critical Review 2 10

allowed students to rank the importance of various aspects of feedback. Through the open-ended

questions and interviews, researchers were able to gain more information about views on giving

and receiving feedback (p. 420). This combination of data collection procedures offered

consistency as well as insights about feedback perceptions, the exact aspects of the research

questions.

11. Reliability and Validity

The authors addressed both the reliability and validity of their study, although validity

was handled with greater depth than reliability. The use of Bloom’s taxonomy provided a high

degree of face validity, as both the participants and researchers had experience using it to

differentiate levels of thinking (p. 421). Face validity is a subjective judgement, and although it

is necessary throughout the research process, it is often seen as a weak measure of validity

(Drost, 2011, p. 116). Instructors also modeled the scoring process by supplying students with

scores on their discussion postings prior to asking them to provide peer feedback.

Validity was also addressed by triangulating entry and exit surveys with the interviews.

The entry survey indicated how participants felt about instructor and peer feedback. The

interviews provided details and examples of these initial thoughts. The exit survey provided

further support for comments made during interviews (p. 421). This allowed feedback

perceptions to be compared across the open-ended questions and interview responses.

Furthermore, NUD*IST qualitative analysis software was used to help identify recurring patterns

and themes from the interview data.

The use of a standardized interview protocol increased the reliability, while the use of

multiple interview subjects and evaluators reduced interviewer biases. Interrater reliability was

established by having researchers code the discussion responses and compare results. Despite

Critical Review 2 11

these measures, the short duration of the study, along with the small sample size and the fact that

only 12 of the 15 entry surveys were returned effect the reliability of the research.

Interpretation and Implications of Results

13. Conceptual Limitations

The authors included a brief section specific to the limitation of this study. They noted

their limitations as “the small sample size, the relatively short duration of the study, and the

fairly limited scale used to judge the quality of student postings” (p. 428). The authors pointed

out in their discussion sections how using a scoring rubric with “only two meaningful levels of

quality may not have provided enough room for growth, thus causing a ceiling effect to occur”

(pp. 425-426). Student started out with relatively high scores on posts, leaving little room for

growth. Also, many of the discussion questions were not conducive to high-level response, again

making it difficult for students to show improvement in their responses.

Throughout the paper the authors expressed other limitations as they arose. Several times

throughout their study the authors noted how “the ability to give meaningful feedback, which

helps other think about the work they have produced, is not a naturally acquired skill” (Pallof &

Pratt, 1999, pp. 415, 427). Providing more time upfront to train students how to use the scoring

rubric more effectively might have impacted their perceptions of feedback. The lack of training

was evident as students stressed their difficulties using Bloom’s taxonomy to rate the level of

thinking. They felt it was difficult to apply the rubric throughout the course. One student in

particular felt confused when trying to decide which level of Bloom’s taxonomy applied to

responses, and therefore often gave out the top score (pp. 424-425).

Critical Review 2 12

14. Conclusions

The authors included descriptions of their results and offered several quotes from

interviewees. Some of the interview responses supported the data, whereas other comments went

against the quantitative results. While the quantitative data did not demonstrate a significant

improvement in the quality of postings, the authors pointed out how during the interviews, eight

of the 15 participants described how they used information from the feedback process to improve

their posting (p. 422). Six students specifically discussed how providing peer feedback increased

their learning (p. 423). These comments indicate a level of inconsistency with the quantitative

results.

Participants perceived feedback from the instructor as more important than peer feedback

at both the beginning and the end of the course. These scores were supported by student

comments that the instructor was more knowledgeable, or that not everyone was motivated to

provide quality feedback. On the other hand, one student noted how “peers are more often on the

same level and may be able to explain in a manner that makes more sense than the instructor” (p.

423). The authors addressed this inconsistent comment by noting how despite instructor

feedback being the preferred choice, 13 of the participants still valued peer feedback. This was

also validated by comments from students stating peer feedback confirmed that their ideas were

meaningful to others and that they learned from their peers’ perspectives.

Giving feedback was rated at the same level as receiving feedback. The interview

comments were consistent with this data, stating how students “reflected on the feedback they

had given to peers as they formed their own response to discussion questions” (p. 423). Students

also considered how others would view their posts and made additions to their responses.

Critical Review 2 13

15. Relationship of Results to Theory

In the detailed results section, the authors connected their findings back to both the initial

research questions as well as the three main areas of their theoretical base: discussions, critical

thinking, and feedback. Most of the discussion of the results centered on feedback and included

information on its relationship to discussions and critical thinking, as well as the research

questions.

The results showed that students considered the quality of feedback to be more important

than quantity. The importance that feedback be timely also increased by the end of the course (p.

421). These items relate back to the initial discussion on the role of feedback, and the expectation

from students that feedback is prompt, timely, thorough and specific (p. 414). Although the

quality of students posting did not improve with peer feedback, it also did not decrease. This

relates back to the first research question about whether the quality of learning can be maintained

or increased through peer feedback. It also deomonstrates the relationship among discussions,

critical thinking, and feedback.

In regards to the second research question, feedback from the instructor was viewed as

more important that peer feedback on both the entry and exit surveys. This data was supported

with several comments from interviewees. Although instructor feedback was viewed as more

important, 13 of the participants still valued peer feedback (p. 423). Students rated the

importance of giving and receiving feedback at the same level on the exit survey, indicating that,

on average, neither aspect was more important. Interview comments suggested that both giving

and receiving feedback resulted in learning. This ties back to both the second and third research

questions about the perceptions of giving and receiving feedback.

Critical Review 2 14

16. Significance of Study

Considering this article has been cited by 160 other articles since it was published eight

years ago, there is a significance to this study. Noting the importance of feedback in general, and

more specifically the significance of feedback in online learning environments, knowledge about

the impact of feedback is vital. Combine that with the fact that little research existed in regards to

the impact of feedback in online environments, and the authors easily demonstrate a need for

their study.

The main implication in the conclusion of this article was that “if the use of peer

feedback can reduce an instructor’s workload in an online course yet help maintain a high quality

of postings, this may offer an effective strategy for facilitating learning in an online course (p.

429). While this may be true, it pays little attention to the three initial research questions,

focusing on the impact of peer feedback and the perceptions of giving and receiving feedback.

While this study offered information on the perceptions of feedback, it left the door wide

open for future research options. Additional research should look at an alternate scoring rubric,

allowing for greater variation among scores. Studies should also offer greater clarity in the

number and type of postings that will be scored, allowing for students to post social comments

that are not included in the data. Additional research is also necessary to determine the most

effective means for facilitating peer feedback in online learning environments.

Critical Review 2 15

References

Cunningham, R. D. (1992). Beyond educational pschology: Steps toward an educaitonal

semiotic. Educational Psychology Review, 4, 165-194.

Drost, E. A. (2011). Validity and reliability in social science research. Education Research and

Perspectives, 38(1), 105-123.

Dunlap, J. C. (2005). Workload reduction in online courses: Getting some shuteye. Performance

and Improvement, 44(5), 18-25.

Ertmer, P. A., & Stepich, D. A. (2004, July). Examining the relationship between higher-order

learning and students' preceived sense of community in an oline learning environment.

Paper presented at the proceedings of the 10th Australian World Wide Web conference.

Gold Coast, Austrailia.

Ertmer, P. A., Richardson, J. C., Belland, B., Camin, D., Connolly, P., Coulthard, G., . . . Mong,

C. (2007). Using peer feedback to enhance the quality of studnet online postings: An

exporatory study. Journal of Computer-Mediated Communication, 12, 412-433.

Holton, E. H., & Burnett, M. B. (1997). Qualitative research methods. In R. A. Swanson, & E. F.

Holton, Human resource development research handbook: Linking research an practice

(pp. 65-87). San Francisco: Berrett-Koehler Publishers.

Lynch, M. M. (2002). The online educator: A guide to creating the virtual classroom. New

York: RoutledgeFlamer.

Morey, E. H. (2004). Feedback reserch revisited. In D. H. Hansen, Handbook of research on

educational communications and technology (pp. 745-783). Mahway, NJ: Lawrence

Erlbaum.

Critical Review 2 16

Pallof, R. M., & Pratt, K. (1999). Building learning communities in cyberspace. San Francisco,

CA: Jossey-Bass.

Palloff, R. M., & Pratt, K. (2001). Lessong from the cybespace classroom: The realities of online

teaching. San Francisco, CA: Jossey-Bass.