Developing an instrument to characterise peer‐led groups in collaborative learning environments: assessing problem‐solving approach and group interaction

This article was downloaded by: [Universidad de Sevilla]On: 12 November 2014, At: 01:54Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Assessment & Evaluation in HigherEducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/caeh20

Developing an instrument tocharacterise peer‐led groups incollaborative learning environments:assessing problem‐solving approach andgroup interactionPilar Pazos a , Marina Micari b & Gregory Light ba Old Dominion University , Norfolk, VA, USAb Northwestern University , Evanston, IL, USAPublished online: 24 Jul 2009.

To cite this article: Pilar Pazos , Marina Micari & Gregory Light (2010) Developing an instrumentto characterise peer‐led groups in collaborative learning environments: assessing problem‐solvingapproach and group interaction, Assessment & Evaluation in Higher Education, 35:2, 191-208, DOI:10.1080/02602930802691572

To link to this article: http://dx.doi.org/10.1080/02602930802691572

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

http://www.tandfonline.com/loi/caeh20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/02602930802691572

http://dx.doi.org/10.1080/02602930802691572

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

Assessment & Evaluation in Higher EducationVol. 35, No. 2, March 2010, 191–208

ISSN 0260-2938 print/ISSN 1469-297X online© 2010 Taylor & FrancisDOI: 10.1080/02602930802691572http://www.informaworld.com

Developing an instrument to characterise peer-led groups in collaborative learning environments: assessing problem-solving approach and group interaction

Pilar Pazosa*, Marina Micarib and Gregory Lightb

aOld Dominion University, Norfolk, VA, USA; bNorthwestern University, Evanston, IL, USATaylor and FrancisCAEH_A_369327.sgm(Final version received 12 December 2008)10.1080/02602930802691572Assessment & Evaluation in Higher Education0260-2938 (print)/1469-297X (online)Original Article2009Taylor & Francis0000000002009Dr. [email protected]

Collaborative learning is being used extensively by educators at all levels. Peer-led team learning in a version of collaborative learning that has shown consistentsuccess in science, technology, engineering and mathematics disciplines. Using amulti-phase research study we describe the development of an observationinstrument that can be used to assess peer-led group learning. This paper illustratesthe development of a classification system for peer-led learning groups and aninstrument based on this classification system. The instrument evaluates smalllearning groups on two important aspects of group learning: problem solvingapproach and group interaction style. We provide evidence of the factor structureof the two dimensions using both exploratory and confirmatory factor analysis.We also provide information about the reliability of the two scales as measured bythe Cronbach's alpha coefficient. Data from a large peer-led learning programmewas used to conduct the factor analysis. Results from the factor analysis confirmedthat the instrument is actually measuring two key characteristics of small learninggroups: problem solving approach and group interaction style, characteristics thathave been linked to effective functioning of the group and to the student learningoutcomes. This instrument may be particularly appealing to practitioners (facultymembers, those running small-group learning programmes, etc.) because it is easyto use and it does not require extensive time for analysis.

Keywords: collaborative learning; peer-led team learning; instrumentdevelopment; STEM education

Introduction

Today, educators at all levels are using collaborative learning techniques in myriadways, and extensive research has been invested in better understanding how collabo-rative learning works. One version of collaborative learning has shown fairlyconsistent success in science, technology, engineering and mathematics (STEM)contexts: peer-facilitated small-group learning (Born, Revelle, and Pinto 2002; Draneet al. 2005; Lyle and Robinson 2003; Swarat et al. 2004; Tien, Roth, and Kampmeier,2002). In this model, small groups of students meet regularly with a peer – one whohas additional expertise in the subject matter – to work on problems collaboratively.

Peer-led group learning environments are characterised by three key features that,based on educational theory and empirical research evidence, should be expected topromote effective student collaborations and produce meaningful learning outcomesin small-groups. These three features are student participation and interaction,

*Corresponding author. Email: [email protected]

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

192 P. Pazos et al.

facilitation style and student problem-solving. Regarding student participation andinteraction, research suggests that active student participation in the small-grouplearning process is critical to effective learning (Chinn, O’Donnell, and Jinks 2000;Draskovic et al. 2004; Veenman et al. 2005; Webb, Farivar, and Mastergeorge 2002).A number of researchers have examined the role of the facilitator or peer leader, incollaborative learning groups (e.g. Houlden, Collier, and Frid 2001; Johnston andTinning 2001; Maudsley 1999; Neville 1999). There is a general agreement that thefacilitator should maintain a guiding, rather than a teaching, role (Houlden, Collier,and Frid 2001; Neville 1999). Finally, there is a great deal of evidence that studentswho wrestle with problems and expand on their answers learn more effectively thando students who simply seek the correct answer.

Although there is research evidence to support particular practices in peer-led learn-ing groups, there is a need for tools to help assess the quality of collaborative learningexperiences (Dillenbourg et al. 1996; van der Linden et al. 2000). Employing a multi-phase research approach, this study takes a preliminary step towards developing anobservation instrument that can be used to assess peer-led group learning environments.

Methods

Study goals

We used a mixed-methods approach to accomplish the following goals:

● Goal 1: describe the critical aspects differentiating peer-led problem-solvinggroups in the STEM disciplines.

● Goal 2: develop and validate instrument to evaluate small peer-led learninggroups based on the critical aspects identified in Goal 1.

Study context

The study was conducted in an extensive small-group STEM workshop programme ata private research university in the Midwestern USA. Students take part in theprogramme voluntarily. Each year, approximately 90 groups of five to seven first- andsecond-year students, each led by an advanced undergraduate facilitator who hastaken the relevant course one or two years earlier, meet weekly throughout the quarter.Faculty members create conceptually rich problems, which require integration ofinformation rather than mere application of formulas, for these groups to workthrough at each meeting. Problems are typically complex, and some are intentionallyill-structured; that is, there is not an immediately apparent way – and not necessarilya single way – to go about finding the answer.

Phase 1: developing the constructs on which to base the instrument

Our first step was to engage in qualitative observational research to identify the criti-cal aspects differentiating workshop groups. During the 2003–04 academic year, threeresearchers with expertise in education as well as observation and qualitative methodswatched a total of 15 workshop groups (10 on videotape and five live), with thepurpose of identifying key aspects of the group’s functioning that could serve as futureareas of focus in identifying critical group differences. All researchers observed thesame groups. Each group comprised between five and seven students, and the groups

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

Assessment & Evaluation in Higher Education 193

were selected evenly across all five disciplines represented in the programme (biol-ogy, chemistry, mathematics, physics and engineering). The sample was balancedwith respect to the facilitator gender. As these researchers observed the groups, theymade notes about actions, statements or behaviours that appeared central to thegroup’s purpose (i.e. those things related to learning and problem-solving, as opposedto incidental communications or behaviours directed towards some other goal). Theythen collaboratively used a thematic-analysis technique, which is a process used inqualitative research to find patterns in data, to organise all three sets of notes into cate-gories, so that notes referring to a single realm of interaction would be categorisedunder a central heading. The researchers approached this process as described byPatton (2002), first creating an initial coding scheme and then classifying the data intothe coded categories while simultaneously reshaping the coding scheme to best fit thedata. Through this process, the researchers identified five key aspects of group func-tioning: student interaction patterns, style of group’s communication, content ofgroup’s communication, facilitator guidance offered and facilitator management ofgroup processes.

Our next step was to develop a preliminary viewing guide for the observations.In spring 2004, we used the five key characteristics identified in the previous phase,along with a review of the literature, to develop 12 specific questions for considerationin observing the groups. This constituted the preliminary viewing guide (seeAppendix 1 for the guide).

During the summer of 2004, another researcher with a background in STEMeducation used the preliminary viewing guide to observe 10 new videotaped groups,answering all 12 questions for each group. Again, the groups represented all disci-plines in the programme and were balanced in terms of facilitator gender. Based onthese answers, we developed detailed descriptions of each group. Two other research-ers checked the videotapes against these descriptions, made slight revisions andagreed that the descriptions fairly represented the groups. These descriptions were toserve as a basis for developing a model describing key group differences, which wouldin turn serve as the foundation for an observation instrument.

We examined the group descriptions to search for patterns and similarities ordifferences among groups, using the original five key aspects as broad coding tools.What stood out most clearly were differences in the ways in which the group members(including the peer facilitator) interacted and in the ways in which the group addressedthe problems. In terms of interaction, groups tended to vary in the degree to which themembers worked individually or collaboratively; in terms of problem-solving, groupstended to vary in the degree to which they probed beneath the surface versus simplylooking for the correct answer. We therefore developed two dimensions to describethe groups, group-interaction style, with levels labelled ‘individual-oriented’ and‘cooperative’, and problem-solving approach, with levels identified as ‘simple’ and‘elaborated’ (see Appendix 2 for an example of the description of groups at this stage).Two researchers collaboratively developed these dimensions. The dimensions wereshown to two outside learning experts and a group of experienced undergraduate lead-ers trained in group facilitation. The group of experts agreed that the model capturedthe most commonly encountered differences in small-learning-group functioning inthe STEM disciplines. This agreement lent some initial face validity to the theoreticalconstruct and resulted in the development of a 2×2 model (Micari et al. 2007), asshown below in Figure 1.Figure 1. Learning group classification model.Descriptions of each of the four categories in the model are given below.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

194 P. Pazos et al.

‘Simple instruction’: individual-oriented group interaction, simple problem interaction

This type of group resembles a traditional university lecture. There is almost no directstudent-to-student interaction. The facilitator does most of the talking and sets thedirection for the group. Students may ask questions, but they are generally directed tothe facilitator, not to other students. The group works to get the answers to the work-sheet questions, but does not go beyond solving problems in order to discuss conceptsor theories. There is little, if any, discussion of the reasons a particular solution is theright one. The facilitator tends to directly solve the problem without much guidanceon alternative approaches or strategies for solving problems.

‘Elaborated instruction’: individual-oriented group interaction, elaborated problem interaction

In this type of group, there is little student-to-student interaction, with the facilitatordoing most of the talking and setting direction, and questions are generally directed tothe facilitator, not to the other students. However, there is some discussion of theconcepts behind the problem, and the group (students or facilitator) may address thereasons a particular solution works and may discuss some concepts and backgroundideas. The group dedicates a fair amount of time to creating elaborated answers. Thefacilitator may also offer guidance on strategies for approaching problems.

‘Supported discussion’: collaborative group interaction, simple problem interaction

In this type of group, the facilitator plays a ‘content expert’ role who is there to inter-vene when needed. Students may work collaboratively, and they themselves do mostof the explaining and discussing, with the facilitator usually offering input only whenspecifically asked for it. Students focus on getting concise answers to the worksheet

Figure 1. Learning group classification model.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


questions, but do not generally discuss concepts or theories or further expand theiranswers. There is little, if any, discussion of the reasons a particular solution is theright one or of how background information might relate to a problem.

‘Guided discussion’: collaborative group interaction, elaborated problem interaction

The facilitator actively guides this type of group by asking questions to help studentsfind the right approach, but students are also actively engaged in discussion and prob-lem-solving. Students interact frequently, both with each other and with the facilitator.The facilitator provides help or explanation when necessary, but students do most ofthe explaining and discussing. The facilitator encourages all students to participate asnecessary. There is a fair amount of discussion of concepts behind the problems, ofthe reasons a particular solution works, and of background information. There mayalso be discussion of strategies for approaching problems.

Phase 2: quantitative phase and instrument development

Building on the qualitative work described above, we sought to develop an instrumentto evaluate peer-led learning groups. Figure 2 depicts the entire process – bothqualitative and quantitative phases – of developing the instrument.Figure 2. Phases in the instrument development.

Participants

A total of 160 groups of four to seven students were selected to participate during twoconsecutive years. Data from these participants were used to conduct factor analysis,comprising both the initial exploratory factor analysis (EFA) and later confirmatoryfactor analysis (CFA). The sample included 43% males (n = 317) and 57% females(n = 425). The total sample represented approximately 85% of all the groups in theprogramme. Students’ ages ranged from 18 to 21.

Creating and refining the instrument: developing instrument items

We developed an initial 18-item instrument based on the data generated from the qual-itative component of the study, supported by a review of the small-group learningliterature (e.g. Chinn, O’Donnell, and Jinks 2000; Cohen 1994; Draskovic et al. 2004;Springer, Stanne, and Donovan 1999; Veenman et al. 2005; Webb 1991; Webb,Farivar, and Mastergeorge 2002). The items were created to address the two dimen-sions of the 2×2 model: (1) group members’ interaction with one another and(2) group’s approach to problem-solving. One final item asked the observer to assignthe group to one of four descriptions, each of which represented one of the four grouptypes depicted in the model. The purpose of this last item was to assess intra-raterreliability.

To further refine the instrument’s items, two researchers used the instrument toevaluate video recordings of 13 groups (eight of the previously viewed videos as wellas five new ones). They further reviewed the wording of the items to make sure theycaptured group characteristics. They also looked at eliminating redundant items: Incases where they felt an item was not effective in identifying group characteristics orwas creating redundancy in the instrument, they deleted the item. The final instrument

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

196 P. Pazos et al.

consisted of 15 scaled items and the one additional item asking the observer tocategorise the group. The scaled items do not ask the observer to make a positive ornegative evaluation of the group; rather, they ask the observer to assign the group aposition on a continuum between two opposing descriptions of group behaviour (seeAppendix 3 for instrument).

Finally, four trained research assistants used the revised instrument to rate thesame videotaped group session. Observations obtained from the revised instrumentwere used to assess inter-rater reliability. The intra-class correlation coefficient for thefour observers on the resulting instrument was .94, demonstrating high reliability(Portney and Watkins 1993).

The final item in the questionnaire asked the observer to assign each group to oneof the four group types in the classification model. As an additional measure ofconstruct validity we looked at the agreement between the questionnaire items and thefinal classification item. We expected that the observer’s ratings of the groups on theLikert-type items, which are based on the two dimensions, would match his or herfinal group classification. We observed that 75% of the time there was a matchbetween the set of items measuring each dimension and the group classification in thelast item. This provided some evidence for a correspondence between the group

Figure 2. Phases in the instrument development.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


classification system and the scale items. We felt that the percentage would have beenhigher had the descriptions in the final item been more comprehensive.

Piloting the instrument

In fall 2005, we conducted a pilot study to test the instrument. Four previously trainedobservers observed a total of 23 different groups. Observations took between one andtwo hours, and each observation included just one group. The groups in this samplewere selected evenly across five disciplines, and the sample was balanced with respectto facilitator gender.

Our main goal in the pilot study was to identify any problems with the observationprocess or the instrument in a real classroom situation. We asked observers to makenotes about their experiences generally, and about any particular instrument items thatwere problematic. These additional observations led to some further rewording ofitems before the instrument was used to conduct exploratory factor analysis.

Exploratory factor analysis

Our goal in conducting the exploratory factor analysis was to investigate whether theitems had loadings that were consistent with the dimensions in the theoretical model(see Figure 1) derived through the qualitative research and literature review. An initialsample of 70 groups provided data. (An additional sample of 90 additional groups waslater used to apply confirmatory factor analysis, described below.) All the observationswere conducted by trained observers (see inter-rater reliability calculations below).

Before collecting data on the initial 70 groups, we conducted a training session for12 observers. The training session lasted for an hour and a half and included practicesessions using the instrument and further discussion of rating criteria. At the end ofthe training, we assessed inter-rater reliability on a single practice observation byhaving the observers use the instrument to evaluate the same videotaped group. Thecollected data resulted in an intra-class correlation coefficient of 0.95. This suggests ahigh level of consistency in the answers across the group of observers (Portney andWatkins 1993).

The sample of 70 groups was selected to be representative of all five disciplines inthe peer-led learning programme and evenly balanced by the facilitator gender. Thegroups selected represented 64% of all the groups in the programme. Although selec-tion of groups was not perfectly random, the sample was representative of workshopgroups within the programme. Each group was observed for an hour and a half andscored using the instrument described above.

Using exploratory factor analysis, we examined the underlying factor structure ofthe items in the questionnaire. The analysis was conducted using SPSS 14. We usedthe maximum likelihood method and oblique rotation to evaluate the underlyingfactors, and we used the scree plot along with results from the qualitative analysis todetermine the number of factors to include in the model. The scree plot (see Figure 3)suggests that there are two factors above the elbow of the graph. Those two firstfactors map closely onto the two dimensions generated in the qualitative phase of theresearch project: group interaction style and problem-solving approach. Further, theitems in the group interaction factor seemed to address two underlying concepts iden-tified in the literature: the level of student engagement and the balance between facil-itator and student control. The first factor accounted for 35% of the variance and the

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

198 P. Pazos et al.

second factor for an additional 17%. We used the parsimony principle to select thetwo-factor model, since it was the simplest model consistent with the qualitative find-ings as well as prior research. The breakdown of items for each factor can be seen inFigure 4. Five of the items in the initial survey were dropped since they did not loadon any of the factors. The resulting survey included 10 items, five each loading oneach of the two factors.Figure 3. Scree plot from the exploratory factor analysis.Figure 4. Final factors derived from exploratory factor analysis.

Figure 3. Scree plot from the exploratory factor analysis.

Figure 4. Final factors derived from exploratory factor analysis.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


Confirmatory factor analysis: final validation of the instrument

In order to have sufficient data to conduct a confirmatory factor analysis, we trainedsix new observers and conducted an additional 90 group observations. Training wasconducted in the same manner as described above. We again assessed inter-rater reli-ability on a practice videotape using intra-class correlation, which was 0.93. As withprevious observations, the sample was selected to be representative of all disciplinesin the peer-led learning programme and evenly balanced by facilitator gender. Thegroups selected represented 81% of all the groups in the programme. Each group wasobserved for an hour and a half and scored using the instrument described above, andas before, each observation included only one group. Seven of the 90 observations hadmissing data, so that we had 83 usable observations. We used these data in combina-tion with the previous 70 observations for the confirmatory factor analysis, for a totalof 153 observations. Each group was observed only once and the observations wereconducted at different times during the academic quarter to capture variation instudent stress level, workload and so on. Because we had dropped the five items thatdid not load onto either of the factors identified through the exploratory factor analy-sis, the final tested instrument includes 10 items on a five-point scale of ‘1’ to ‘5’ (seeAppendix 3 for final instrument). This final version of the instrument did not includethe final item asking observers to categorise the group.

PROC FACTOR in SAS 9.1 was used to estimate parameters and test the models.Confirmatory factor analysis helped us test the relationship between the observed vari-ables and the underlying latent constructs in the 2×2 model. Before conducting the anal-ysis we identified the items on the instrument that fell under each factor (See Figure 5).Figure 5. Factor structure.

Figure 5. Factor structure.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

200 P. Pazos et al.

The model that we tested hypothesised two latent constructs related to the groups:problem-solving approach and group interaction. The instrument consisted of 10items, five of which were assumed to be indicators of the group interaction styleconstruct, and the other five of which were assumed to be indicators of the problem-solving approach construct. Each item fell under one latent factor only. Figure 5describes the latent structure and Figure 6 shows the model equations. Figure 7 showsthe coefficients for each item.Figure 6. Model equations.Figure 7. Confirmatory factor analysis coefficients.

Figure 6. Model equations.

Figure 7. Confirmatory factor analysis coefficients.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


Next we evaluated the fit of the proposed model. Confirmatory factor analysis withmaximum likelihood provided parameter estimates for the model. Table 1 shows thesummary of fit measures of the optimal model along with their recommended values(Bollen 1989). Most statistics used indicate acceptable model fit, suggesting the factorstructure determined above.

We established the internal consistency of the two scales using Cronbach’s alpha.The group interaction style dimension had a value of α = 0.86, and the problem-solving approach dimension resulted in α = 0.87. We next determined whichparameter estimates were significant.

Relationship of group characteristics to outcome measures

In an attempt to further assess the validity of the group observation instrument, weevaluated the relationship between the two dimensions and some criterion for groupoutcome (DeVellis 1991). Initially, we constructed a regression model with averagecourse grade for the group as response variable, and the two dimensions in the classi-fication model (group interaction style and problem-solving approach) as predictors.We used average GPA as an additional factor to account for pre-existing differencesin student performance. We found no significant relationship between the averagegrade of students in a group and the two dimensions in the model. This result is notsurprising given the unit of analysis of the study. This study has group as the main unitof analysis, making it difficult to identify relationships with grade, which is an indi-vidual variable. As a result of averaging the grades for the students in the groups, welost the variation of the grade data within each group. We also know, there are multi-tude of factors that appear to predict students’ academic performance (Kuncel,Hezlett, and Ones 2004; Ridgell and Lounsbury 2004), so that while group type mayhave an effect on learning, it would not necessarily have a direct impact on students’grade, and less so on average grade of the group.

Because of the limitations of examining grade as an outcome variable, we choseto use an outcome variable that has been linked to academic performance: self-efficacy related to one’s ability to do well in the course (Bandura 1997; Pajares 1996;Pajares and Miller 1994; Wolters and Pintrich 1998). On a short survey that all

Table 1. Summary of fit measures.

Fit measure Recommended value Value

Goodness of fit index (GFI) >0.90 0.9131Bentler’s comparative fit index >0.90 0.9482Bentler and Bonett’s (1980) non-normed index >0.90 0.9315Bentler and Bonett’s (1980) NFI >0.90 0.9077Bollen’s (1986) normed index, rho 1 >0.90 0.8779Bollen’s (1988) non-normed index, delta 2 >0.90 0.9490Chi-square N/A 72.13df N/A 34Chi-square/df ≤3 2.12Root mean square residual (RMR) <0.10 0.0728RMSEA 0.0850

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

202 P. Pazos et al.

observed students were asked to complete, we included a Likert-scaled item stating ‘Ibelieve I will do well in that course’ (‘that’ refers to the course which is linked tothe small-group workshops). From these responses, we created an average groupconfidence score for each observed group, as the average of each of the individualconfidence scores. We used a linear regression model to evaluate whether the twodimension scales (group interaction style and problem-solving approach) wouldpredict average group confidence in course performance. Results suggest thatgroup interaction style was a significant predictor of confidence in course perfor-mance (b = .224, p = 0.045). We thus have some preliminary evidence that the way inwhich students in the group interact and communicate, as measured by this instru-ment, has some relationship with students’ confidence in their ability to succeed in thecourse.

Conclusion

This paper describes the development of a classification system for peer-led learninggroups and an instrument based on this classification system. The instrumentevaluates small learning groups on two important aspects of group learning: problem-solving approach and group interaction style. We provide evidence of the factorstructure of the two dimensions using both exploratory and confirmatory factor anal-ysis. We also provide information about the reliability of the two scales as measuredby the Cronbach’s alpha coefficient.

Based on the factor analyses, it is reasonable to assume that the instrument ismeasuring two discrete characteristics of these small learning groups: problem-solving approach and group interaction style. Further, there is ample evidence in thelearning literature that these two characteristics are vital to the effective functioningof the group and to the student learning outcomes (Boxtel, van der Linden, andKanselaar 2000; Chinn, O’Donnell, and Jinks 2000; Cohen 1994; Draskovic et al.2004; Hake 1998; Haller et al. 2000; Houlden, Collier, and Frid 2001; King 1990;King and Rosenshine 1993; Lambiotte et al. 1987; Neville 1999; Redish, Saul, andSteinberg 1997; Shaw and Webb 1982; Shaw et al. 1979; Slavin 1996; Vedder1985; Veenman et al. 2005; Webb 1989, 1991; Webb, Farivar, and Mastergeorge2002).

The two dimensions identified through the instrument help describe the ways inwhich peer-led group learning is manifested. First, although the students learn in agroup, the ‘groupness’ of the experience can be more or less salient. Secondly, theways in which the group approaches the problems can lead it to simpler or moreprofound engagement with the questions and concepts of the course.

The instrument’s use

This instrument may be particularly appealing to practitioners (faculty members, thoserunning small-group learning programmes, etc.) because it is easy to use and it doesnot require extensive time for analysis. The instrument differs from that of Visschers-Pleijers et al. (2005), another tool for assessing small-group learning, in that it iscompleted by an outside observer rather than the students themselves and that it isdesigned specifically for peer-facilitated groups and includes items related to the peerleader’s interaction with the group.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


Limitations

While the literature provides strong evidence for the importance of these two factorsin student outcomes, we lack direct evidence in the research presented here that thesefactors (group interaction style and problem-solving approach) are linked to academicoutcomes of students participating in the group. Although we did find a relationshipbetween one of the factors (interaction style) and a measure of self-efficacy, thatmeasure was limited to a single survey item. Further research should investigate therelationship of the two factors to outcomes related to academic success, including addi-tional measures of self-efficacy, academic and social integration, intrinsic motivation,approaches to study, and so on. Finally, there are certain practical limitations of theinstrument; for instance, observers need to be trained, and the instrument provides onlya snapshot of the group experience. We recognise that the validity of the instrumentreported in this paper is not guaranteed unless the instrument is used by trained observ-ers. Educators wishing to use the instrument should consider making multiple obser-vations over an academic term to most reliably assess the dynamic nature of groups.

AcknowledgementsWe would like to acknowledge the important contributions to this study of BernhardStreitwieser, Erik Kjeldgaard, Katherine Linsenmeier, Caryn Schnierle, and Le Zhong.

Notes on contributorsPilar Pazos is an assistant professor at the Engineering Management and Systems EngineeringDepartment at Old Dominion University, Norfolk, VA, USA. Previously she was a researchassociate at the Searle Center for Teaching Excellence, Northwestern University, Evanston, IL,USA, working on programme development, research and evaluation in STEM fields. Her mainareas of research interest are collaborative learning, virtual teams and team decision-makingand performance. She has published in the areas of STEM education, team-related research andtraining.

Marina Micari is associate director at the Searle Center for Teaching Excellence, NorthwesternUniversity, Evanston, IL, USA, where she runs teaching and learning programmes for facultyand undergraduates and conducts evaluation research. She has published in the areas of small-group learning, underrepresented students in STEM fields, college student development andeducation for work.

Gregory Light is the director of the Searle Center for Teaching Excellence and an associateprofessor in the School of Education and Social Policy at Northwestern University, Evanston,IL, USA. Prior to arriving at Northwestern University in 2000, he was acting head of the Life-long Learning Group at the Institute of Education, University of London, UK. His research andscholarship focuses on the theory and practice of learning, teaching and educational develop-ment in higher and professional education. He has published over 30 papers and chapters innational and international peer-reviewed publications and given 75 invited talks and conferencepresentations in the area of teaching and learning. He is author of the book Learning andteaching in higher education: The reflective professional.

ReferencesBandura, A. 1997. Self-efficacy: The exercise of control. New York: Freeman.Bollen, K.A. 1989. Structural equations with latent variables. New York: Wiley.Born, W., W. Revelle, and L. Pinto. 2002. Improving biology performance with workshop

groups. Journal of Science Education and Technology 11, no. 4: 347–65.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

204 P. Pazos et al.

Boxtel, C., J. van der Linden, and G. Kanselaar. 2000. Collaborative learning tasks and theelaboration of conceptual knowledge. Learning and Instruction 10, no. 4: 311–30.

Chinn, C., A. O’Donnell, and T. Jinks. 2000. The structure of discourse in collaborativelearning. Journal of Experimental Education 69, no. 1: 77–97.

Cohen, E. 1994. Restructuring the classroom: Conditions for productive small groups. Reviewof Educational Research 64, no. 1: 1–35.

DeVellis, R.F. 1991. Applied social research methods series, 26. Scale development: Theoryand applications. Newbury Park, CA: Sage.

Dillenbourg, P., M. Baker, A. Blaye, and C. O’Malley. 1996. The evolution of research oncollaborative learning. In Learning in humans and machine: Towards an interdisciplinarylearning science, ed. E. Spada and P. Reiman, 189–211. Oxford: Elsevier.

Drane, D., H.D. Smith, G. Light, L. Pinto, and S. Swarat. 2005. The gateway science workshopprogram: Enhancing student performance and retention in the sciences through peer-facilitated discussion. Journal of Science Education and Technology 14, no. 3: 337–52.

Draskovic, I., R. Holdrinet, J. Bulte, S. Bolhuis, and J. Van Leeuwe. 2004. Modeling small-group learning. Instructional Science 32, no. 6: 447–73.

Hake, R. 1998. Interactive-engagement vs. traditional methods: A six-thousand-studentsurvey of mechanics test data for introductory physics courses. American Journal of Phys-ics 66, no. 1: 64–74.

Haller, C.R., V.J. Gallager, T.L. Weldon, and R.M. Felder. 2000. Dynamics of peer educationin cooperative learning workgroups. Journal of Engineering Education 89, no. 3: 285–93.

Houlden, R.L., C.P. Collier, P.J. Frid. 2001. Problems identified by tutors in a hybrid prob-lem-based learning curriculum. Academic Medicine 76, no. 1: 81.

Johnston, A.K., and R.S. Tinning. 2001. Meeting the challenge of problem-based learning:Developing the facilitators. Nurse Education Today 21: 161–69.

King, A. 1990. Enhancing peer interaction and learning in the classroom through reciprocalpeer questioning. American Educational Research Journal 27, no. 4: 664–87.

King, A., and B. Rosenshine. 1993. Effects of guided cooperative-questioning on children’sknowledge construction. Journal of Experimental Education 6, 127–48.

Kuncel, N.R., S.A. Hezlett, and D.S. Ones. 2004. Academic performance, career potential,creativity, and job performance: Can one construct predict them all? Journal of Personalityand Social Psychology 86, 148–61 [In a special section: ‘Cognitive abilities: 100 years afterSpearman (1904)’].

Lambiotte, J.G., D.F. Dansereau, A.M. O’Donnell, and M.D. Young. 1987. Manipulatingcooperative scripts for teaching and learning. Journal of Educational Psychology 79,no. 4: 424–30.

Lyle, K., and W. Robinson. 2003. A statistical evaluation: Peer-led team learning in anorganic chemistry course. Journal of Chemical Education 80, no. 2: 132.

Maudsley, G. 1999. Do we all mean the same thing by ‘problem-based learning’? A review ofthe concepts and a formulation of the ground rules. Academic Medicine 74: 178–85.

Micari, M., P. Pazos, G. Light, B. Streitwieser, and K. Linsenmeier. 2007. Characteristics ofpeer-led workshop groups in STEM disciplines: A validated instrument to categorizegroup functioning. Paper presented at the annual meeting of the American EducationalResearch Association, April, in Chicago, USA.

Neville, A.J. 1999. The problem-based learning tutor: Teacher? Facilitator? Evaluator?Medical Teacher 21, no. 4: 393–404.

Pajares, F. 1996. Self-efficacy beliefs in academic settings. Review of Educational Research66: 543–78.

Pajares, F., and M.D. Miller. 1994. Role of self-efficacy and self-concept beliefs inmathematical problem solving: A path analysis. Journal of Educational Psychology 86:193–203.

Patton, M. 2002. Qualitative research and evaluation methods, vol. 3. Thousand Oaks, CA: Sage.Portney, L.G., and M.P. Watkins. 1993. Foundations of clinical research: Applications to

practice. Norwalk, CT: Appleton & Lange.Redish, E., J. Saul, and R. Steinberg. 1997. On the effectiveness of active-engagement

microcomputer-based laboratories. American Journal of Physics 65, no. 1: 45–54.Ridgell, S.D., and J.W. Lounsbury. 2004. Predicting academic success: General intelligence,

‘Big five’ personality traits, and work drive. College Student Journal 38, no. 4: 607–18.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


Shaw, M., B. Ackerman, N. McCown, A. Worsham, L. Haugh, B. Gebhardt, and P. Small.1979. Interaction patterns and facilitation of peer learning. Small Group Research 10,no. 2: 214–23.

Shaw, M., and J. Webb. 1982. When compatibility interferes with group effectiveness: Facili-tation of learning in small groups. Small Group Behavior 13, no. 4: 555–64.

Slavin, R. 1996. Research on cooperative learning and achievement: What we know, what weneed to know. Contemporary Educational Psychology 21, no. 1: 43–69.

Springer, L., M.E. Stanne, and S. Donovan. 1999. Effects of small-group learning on under-graduates in science, mathematics, engineering, and technology: A meta-analysis. Reviewof Educational Research 69, no. 1: 21–51.

Swarat, S., D. Drane, H.D. Smith, G. Light, and L. Pinto. 2004. Opening the gateway:Increasing student retention in introductory science courses. Journal of College ScienceTeaching 34, no. 1: 2–7.

Tien, L.T., V. Roth, and J.A. Kampmeier. 2002. Implementation of a peer-led learninginstructional approach in an undergraduate organic chemistry course. Journal of Researchin Science Teaching 39: 606–32.

van der Linden, J., G. Erkens, H. Schmidt, and P. Renshaw. 2000. Collaborative learning. InNew learning, ed. R.J. Simons, J. van der Linden, and T. Duffy, 37–54. Dordrecht, TheNetherlands: Kluwer.

Vedder, P. 1985. Cooperative learning: A study on processes and effects of cooperation betweenprimary school children. Westerhaven Groningen, The Netherlands: RijkuniversiteitGroningen.

Veenman, S., E. Denessen, A. van den Akker, and J. van der Rijt. 2005. Effects of a coopera-tive learning program on the elaborations of students during help seeking and help giving.American Educational Research Journal 42, no. 1: 115–51.

Visschers-Pleijers, A., D. Dolmans, I. Wolfhagen, and C. van der Vleuten. 2005. Develop-ment and validation of a questionnaire to identify learning-oriented group interactions inPBL. Medical Teacher 27, no. 4: 375–81.

Webb, N. 1989. Peer interaction and learning in small groups. International Journal ofEducational Research 13, no. 1: 21–39.

Webb, N. 1991. Task related verbal interaction and mathematics learning in small groups.Journal for Research in Mathematics Education 22, no. 5: 366–89.

Webb, N., S. Farivar, and A. Mastergeorge. 2002. Productive helping in cooperative groups.Theory into Practice 41, no. 1: 13–20.

Wolters, C.A., and P.R. Pintrich. 1998. Contextual differences in student motivation and self-regulated learning in mathematics, English, and social studies classrooms. InstructionalScience 26: 27–47.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

206 P. Pazos et al.

Appendix 1. Preliminary viewing guide: initial 12 questions used to develop group descriptionsStudent-to-student communication behaviour

Student interaction1. Does the group work primarily as a unit, or individually?2. Do students ask/explain ideas to one another?3. Are there any individuals who do not participate?

a. Is there any lack of individual participation by gender/ethnicity?Style of communication4. Do students tend to proffer answers that are not fully formed?Content of communication5. Does the group spend time discussing discipline-related concepts beyond what appears

on the worksheet?6. Do students tend to communicate about non-work-related issues?

Facilitator behaviourGuidance offered7. Does the facilitator use questions or probing statements to further students’ analysis of

problems?8. Does the facilitator overtly check to see whether students have understood an

explanation or idea?9. Does the facilitator introduce principles or concepts that students can apply to

problems?10. Does the facilitator offer guidance on how to conceive of problems or study tips?Management11. Does facilitator break the group into subgroups?12. Does facilitator take an active role in managing the group, or let it evolve as it will?

Appendix 2. Condensed descriptions of groupsDescriptions are based on the following areas of focus: student interaction patterns, style ofgroup’s communication, content of group’s communication, facilitator guidance offered andfacilitator management of group processes.

Group #2

Students do quite a bit of asking one another questions and explaining ideas to one another.The group seems to lead itself, with students directing conversation and answering oneanother’s questions. The facilitator asks many questions to help further students’ understand-ing. He also asks students why the answer is what it is and how they got their answers. Thefacilitator offers tips by explaining what professors want students to know for the exam. Thestudents seem to have taken the lead role in the group. The facilitator still provides input thatensures that students 1) come away with an understanding of why things are true, and 2) havea sense of what to expect on their exam.

In terms of the two dimensions, group-interaction approach and problem-solving approach, thisgroup was described as students and facilitator having shared authority for running the sessionand as focusing more on conceptual analysis of the worksheet questions than on simply gettingthe right answer.

Group #3

Students ask and explain to one another, and the facilitator is quiet while this is happening.Facilitator answers questions by asking more questions. He also asks students to work outproblems on the board. Students work mainly independently, asking the facilitator questionsoccasionally. When a question is asked, some stop their own work and pay attention, but othersjust keep working on their own. Discussions of material beyond the worksheet questions occur,

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14


but infrequently and only when specifically introduced by students. Students work togetherwithout explicit guidance from the facilitator. The facilitator is there as a resource, but doeslittle to ensure that all students understand key points.

In terms of the two dimensions, group-interaction approach and problem-solving approach, thisgroup was described as students and facilitator having shared authority for running the sessionand as focusing mostly on getting the right answer, rather than on analysing problems.

Appendix 3. Observation instrument: final version

Workshop-Group Observation Form, rev. 05-07-2008

Facilitator: _________________________ Course: ___________________________

Viewer’s name: _____________________ Date: ___________ Time in: ______________Time out: _____________

In the following items are two statements describing the group along a continuum. Please thinkabout where along the continuum this group lies, and circle the number closest to that point.

1. There is little student-to-student 1 2 3 4 5 There is frequent student-to-student interaction in this group. interaction in this group.

If you had trouble assigning a value to this group, please explain:_______________________________________________________________________

2. The group accepts short answers. 1 2 3 4 5 In this group answers are commonly Answers are rarely expanded expanded upon (by either studentsupon (by either students or or facilitator).facilitator).

If you had trouble assigning a value to this group, please explain:________________________________________________________________________

3. The facilitator does most of the 1 2 3 4 5 Students do most of the talking.talking.


4. When identifying answers, this 1 2 3 4 5 In this group the facilitator or group (facilitator or students) students often emphasise how ordoes not often ask or explain why something is true, not just thathow or why something is true. it is true.


6. The facilitator helps students 1 2 3 4 5 The facilitator points out that solve problems without explicit particular ideas are important forreview of background concepts solving the problem and reviewsor course material. some background concepts or

course material.


7. The facilitator sets the direction 1 2 3 4 5 Students play a role in setting thefor the group. direction for the group.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

208 P. Pazos et al.


9. The facilitator is the focus of 1 2 3 4 5 The students are the focus of thisthe group, and there is a sense group. There is a sense that thethat without the facilitator, the group could function even if thegroup could not function. facilitator were absent.


10. When the group has solved a 1 2 3 4 5 The group sometimes continuesproblem, they move on to discussing a problem evenanother without additional after an answer has beendiscussion. arrived at.


14. The group seems more 1 2 3 4 5 The group (students and/orconcerned with identifying facilitator) seems very concernedanswers to the particular with using worksheet questions thanquestions to discuss and with discussing ideas/conceptsunderstand ideas, concepts and behind the problems.strategies.


15. The facilitator takes 1 2 3 4 5 The group as a whole takesresponsibility for most of the responsibility for most of thecommunication (e.g. questioning, communication (e.g. questioning,answering, explaining). answering, explaining).


Note: This instrument has 10 items but the numbering sequence goes until 15 just to keep theoriginal question numbers from the earlier versions of instrument.

Dow

nloa

ded

by [

Uni

vers

idad

de

Sevi

lla]

at 0

1:54

12

Nov

embe

r 20

14

Documents

Developing an instrument to characterise peer‐led groups in collaborative learning environments: assessing problem‐solving approach and group interaction