Teachers’ Experiences of a Hybrid Professional Development Model of Reading Apprenticeship

Authors and Affiliations: Stephanie Levin, PhD. Impaq International, Mikhail Pyatigorsky, Ph.D., Impaq International, Cheri Fancsali, Ph.D. Impaq International, and Yasuyo Abe, Ph.D., Impaq International

Background

Nationally, two-thirds of high school students are unable to read and comprehend complex academic materials, think critically about texts, and synthesize information from multiple sources, or communicate what they have learned (NAEP, 2013). Without a substantial change in their academic literacy, U.S. high school students face continued academic problems in high school and college because they are unable to handle the quantity and complexity of assigned reading (ACT, 2012). The Reading Apprenticeship instructional framework was developed two decades ago to help teachers provide the literacy support students need to be successful readers in the content areas. It has since reached over 100,000 teachers in schools across the country, at the middle school, high school and college levels.

The complex literacy instruction envisioned by new standards for literacy and college and career readiness remains rare in U.S. secondary schools (Fisher, 2009). National surveys indicate that secondary teachers prioritize content over disciplinary literacy and reasoning (ACT, 2013a, 2013b). These widespread practices detract from students’ opportunity to learn and meet the new standards. Effective PD is needed, at scale, to transform the normative practices of secondary, content-area instruction. Recent years have seen tremendous growth in blended learning as a means to scale PD, yet relatively few studies have examined these formats. This paper presents research on Reading Apprenticeship Writing Connections (RAWC), a hybrid professional development model that incorporates online and face-to-face learning opportunities for educators to support their instruction of literacy in content areas.

To build advanced literacy skills, subject area teachers must help students to draw inferences from academic texts, synthesize information from various sources, follow complex ideas, and write from textual evidence in their disciplines (Heller & Greenleaf, 2007; Lee & Spratley, 2010). However, many middle and high school teachers are unprepared to meet this challenge (Greenleaf & Schoenbach, 2004; Shanahan & Shanahan, 2008). The challenge of providing high-quality inquiry-based professional development at scale has led some to explore online and hybrid-professional development (Perkins & Wiske, 2005; Dede, 2006; Fishman, 2014).

Reading Apprenticeship is a model of professional learning and academic literacy instruction designed to improve literacy skills and academic achievement for all students. Reading Apprenticeship engages

1

teachers in inquiries into their own disciplinary ways of reading, into student and classroom cases with multiple opportunities for professional reading, reflection, connection to classroom practice, and planning. In recent years, Reading Apprenticeship has adopted technological innovations such as online workshops and virtual peer groups to deliver content and support teacher learning. These innovations are expected to efficiently enable more opportunities for community building and learning.

In 2014, the program developers received a Supporting Effective Educator Development (SEED) grant from the Department of Education to scale-up and conduct a randomized controlled trial of the intervention through a project called Reading Apprenticeship: Writing Connections (RAWC). RAWC works with middle school teachers in three subject areas: English, social studies and science. Previous RCTs have tested the efficacy of the Reading Apprenticeship framework and the professional development model in smaller efforts with more closely monitored implementation. These studies have demonstrated strong positive effects on teacher practice resulting from the professional development—most notably, teachers’ increased use of reading comprehension strategy instruction, metacognitive inquiry routines, and collaborative learning structures in their classrooms. They also showed positive effects on students’ literacy and content-area achievement, motivation, and engagement (Greenleaf et al., 2011a, 2011b; Kemple, 2008; Somers, 2010). RAWC differs from other versions of Reading Apprenticeship in that it provides a hybrid of in-person and online PD components.

The first iteration of RAWC’s hybrid course (2014-15) included 80 hours of PD: 7 days face to face and 4 hours/month online during the school year. The online PD activities were primarily synchronous and included large group learning of new Reading Apprenticeship modules (IGNITE sessions) and small group Professional Learning Communities (PLC) which focused on teachers discussing their classroom activities. A-synchronous work included text-based discussion forums. This has been repeated the following year (2015-16) with slight adjustments to the organization of the PD content and greater support for the online components of the initiative.

In addition to presenting preliminary impacts of the RAWC program on teacher practice, this paper addresses the non-traditional delivery of professional development and presents teachers’ perceptions of the hybrid online/in-person content delivery system.

Research Questions

This study explores implementation questions as well as mediating impact on student achievement. Questions included:

1) To what extent do teachers participating in the RAWC professional development change their instructional practices to include use of Reading Apprenticeship strategies and approaches?

2) To what extent do teachers find RAWC professional development to be useful?

2

Logic Model

The logic informing the project design, as shown in Figure 1, is as follows: Professional development in Reading Apprenticeship enables middle and high school teachers of science, history/social studies, and ELA to integrate academic literacy instruction into ongoing content area teaching, thereby increasing the quality of students’ literacy learning opportunities, leading to increased academic engagement and achievement, especially for high-need students. Subject area teachers receive the equivalent of 10 days (60 hours) of subject-specific professional development in a hybrid manner—mix of face-to-face and on-line with on-site follow up over two years, with implementation support between sessions.

Figure 1. Reading Apprenticeship Writing Connections Logic Model

Setting

Professional development was provided both in-person to teachers in a central location within each state and online to teachers through online learning sessions organized in small content area groups (approximately 4 hours a month). Also, teachers were offered on-site support through monthly school meetings led by teacher leaders.

A total of 27 schools from Pennsylvania and Indiana were recruited from the overall sample of schools participating in Literacy Apprenticeship PD to participate in the external evaluation. Thirteen of these schools were randomly assigned to receive the PD during the 2014-15 school year and served as the

3

treatment group. Teachers in these schools attended the Literacy Apprenticeship Summer Institute in 2014 and began implementing Reading Apprenticeship in their classrooms during the 2014-15 school year (Cohort 1). The remaining 14 schools served as the control group. Teachers in these schools participated in the Summer Institute in 2015 and are implementing Reading Apprenticeship in their classrooms throughout the 2015-16 school year (Cohort 2).

Participants

Prior to random assignment, the study team recruited volunteer teachers from each subject area of interest in each school. To be eligible for the study, teachers needed to teach at least two classes in one of the following subject areas: 8th grade ELA, science, or social studies. The random assignment of schools resulted in 13 schools and 43 teachers in the treatment group; 14 schools and 47 teachers in the control group. The student sample consists of students enrolled in target subject area classes instructed by the participating teachers in the 2014-15 school year. Target classes are the one or two classrooms per teacher in which we administered the student survey and the CBAL assessment. Also, teachers were asked to respond to certain survey items based on these target classes. The student sample includes 2,267 students in target classes and approximately 9,198 students in classes taught by study teachers.1 .

Table 1 shows the characteristics of teachers who participated in the study and completed teacher surveys (43 treatment and 47 control), as well as the characteristics of their schools. School characteristics (percent of students proficient on state ELA exams in 8th grade, etc.) represent data from the baseline year (2013-14). Teacher characteristics were collected during the 2014-15 school year. There were some substantive baseline differences between the treatment and control groups. However, none of the differences are statistically significant.2 The impact analysis is estimated controlling for all school and teacher characteristics, thereby mitigating their potential influence.

Table 1. Teacher Sample Equivalence Control group Treatment group Difference

(adjusted) P-valueStandardized diff.

(adjusted)Characteristic Mean SD Mean SDSchool characteristics

% Proficient - 8th grade ELA 0.825 (0.068) 0.775 (0.118) -0.05 [0.123] -0.51

% Asian 0.017 (0.015) 0.021 (0.028) 0.01 [0.337] 0.39

% Black 0.096 (0.129) 0.154 (0.210) 0.06 [0.167] 0.32

% Hispanic 0.109 (0.111) 0.086 (0.067) -0.03 [0.384] -0.33

% Multiple races 0.035 (0.034) 0.035 (0.031) -0.00 [0.419] -0.15

1 These numbers include duplicate students across classes. 2 Equivalence was checked by regressing each characteristic on the block fixed effects and the treatment group indicator variable.

4

Control group Treatment group Difference(adjusted) P-value

Standardized diff. (adjusted)Characteristic Mean SD Mean SD

% White 0.742 (0.187) 0.703 (0.272) -0.03 [0.661] -0.12

% ELL 0.028 (0.039) 0.039 (0.048) 0.01 [0.447] 0.25

% FRPL 0.512 (0.186) 0.493 (0.210) -0.04 [0.432] -0.18

Teacher characteristics3

Female 0.638 (0.486) 0.721 (0.454) 0.11 [0.292] 0.23

Education = more than BA 0.681 (0.471) 0.651 (0.482) 0.02 [0.868] 0.04

Reading specialist 0.085 (0.282) 0.070 (0.258) 0.02 [0.733] 0.07

Years teaching 13.24 (8.38) 13.02 (6.70) -0.27 [0.875] -0.03

Years teaching in subject 10.40 (7.77) 11.37 (6.33) 1.41 [0.369] 0.20

ELA 0.468 (0.504) 0.535 (0.505) 0.10 [0.381] 0.19

Social Studies 0.319 (0.471) 0.302 (0.465) -0.06 [0.586] -0.12

Science 0.213 (0.414) 0.163 (0.374) -0.04 [0.652] -0.10

Intervention

Reading Apprenticeship Writing Connections helps teachers support discipline-specific literacy and learning in their varied content areas by attending to four interacting dimensions of classroom learning culture: Social, Personal, Cognitive, and Knowledge-Building. The social dimension involves building community. The classroom becomes a safe environment where students see other students and their teacher as resources for learning. The personal dimension includes drawing on students’ understandings and experiences as well as developing students’ identities as competent readers, building their awareness of their purposes and goals for reading, and connecting current academic tasks to future career or educational goals. The cognitive dimension involves developing students’ mental processes, including their text-based problem-solving strategies. The knowledge-building dimension includes building students’ knowledge not only of the content of the text but also of language and word construction, genre and text structure, and discipline-specific discourse practices. At the center of Reading Apprenticeship is an ongoing metacognitive conversation carried on both internally through metacognitive reading and reasoning routines and externally, as teacher and students talk about their personal relationships to reading, the social environment and resources of the classroom, their affective responses and cognitive activity, and the knowledge required to make sense of complex texts. This takes place through extensive reading including increased in-class opportunities for students to practice reading complex academic texts in more skillful ways as they collaborate to make meaning of these texts for learning purposes. The framework targets learning

3 All teachers reported their race as white, except for one teacher who did not report race.

5

dispositions as well as literacy skills and knowledge. RAWC’s intensive professional development included 7 days of face-to-face sessions delivered over 12 months, 4 hours a month online learning sessions, with additional in-person, on-site implementation support from teacher leaders.

Research Design

The evaluation uses a mixed-methods approach that combines a randomized controlled trial of teacher practices, student use of literacy strategies and academic behaviors and dispositions, and student achievement at schools with formative feedback on teachers’ training and implementation of Reading Apprenticeship strategies. We estimate a two-level model, with individual students or teachers nested within schools.

Level 1: Student or Teacher

(Eq. 1) Y ij=β0 j+∑q=1

Q

βqj Xqij+εij

where Y ij denotes the outcome for student/teacher i (i = 1, 2, … N) in school j (j = 1, 2, … K), X qij represents individual-level covariates and ε ij is the error term specific to student/teacher i in school j.

Level 2: School

(Eq. 2) β0 j=γ 0+γ 1 (T reatment j )+∑s=1

S

ηsW sj+u j

(Eq. 3) βqj=δ q0

where Treatment is a binary variable indicating whether school j is randomly assigned to receive Reading Apprenticeship (Treatment = 1) or not (Treatment = 0) and its coefficient, γ1, represents the estimated effects of Reading Apprenticeship on the student (or teacher) outcome. W sj represents school-level covariates and u j is the error term specific to the j-th school, representing the random school effects.

In practice, we estimate the reduced-form of the two-level random-intercept model, which can be expressed as follows:

(Eq. 4) Y ij=γ0+γ1 (T reatment j )+∑q=1

Q

δ q Xqij+∑s=1

S

ηsW sj+u j+εij

where variables are as described above. Equation 4 is derived by combining equations 1 through 3. Coefficients δ q and ηs are estimators for marginal effects of individual and school level covariates, respectively. We assume that individual-level effects do not vary across blocks (schools).

6

To evaluate the impact of Reading Apprenticeship training, we tested the null hypothesis that there was no difference between the groups (H0: γ1 = 0). If the null hypothesis was rejected by a two-tailed test at the 5 percent significance level, we concluded that the outcome was different between students (teachers) in schools exposed to Reading Apprenticeship versus students (teachers) in schools that were not exposed to Reading Apprenticeship.

The covariates used in the impact analysis of teacher survey responses include the following:

School levelo Block fixed effectso School characteristics (in baseline year, 2013/14)

• %grade 8 students proficient on state ELA exam, racial composition, %ELL, %FRPL Teacher level

o Gender, race, education, experience, subject taught

Data Collection and Analysis

The primary sources of data collected for this study include student record data collected from the district, teacher surveys (3 per year) collected over two years (6 total), a student survey collected at the end of the implementation year (2014-15), teacher focus groups (2 per year), and a student literacy assessment developed and collected by ETS at the end of the implementation year (see O’Reilly et. al. 2014 for a description of the assessment and reliability information).

Student achievement is measured using two assessments—ETS’ Cognitively Based Assessment of, for, and as Learning (CBAL) assessment of reading and writing and the Indiana and Pennsylvania standardized state reading and writing tests. Students’ academic behaviors and dispositions (e.g., writer/reader identity and perceived self-efficacy) and use of learning strategies, such as metacognitive strategies, is assessed using a student survey. Data on teacher practices and perceptions is collected from teachers using an online teacher survey and focus group interviews. Table 2 presents the evaluation and data collection timeline.

Table 2. Evaluation and Data Collection TimelineSUMMER 2014 SEPT 2014-AUG 2015 SEPT 2015-AUG 2016

COHORT 1 2014 Summer Institute

Year 1 implementation Year 2 implementation

COHORT 2 -- 2015 Summer Institute Year 1 implementation

EVALUATION ACTIVITIES

- 1 teacher focus group in PA, 1 in IN - cohort 1- 3 teacher surveys - CBAL ELA pre- and post-tests (fall,

- 1 teacher focus group in PA, 1 in IN - cohort 2- 2 teacher interviews - cohort 1

7

SUMMER 2014 SEPT 2014-AUG 2015 SEPT 2015-AUG 2016spring) - Student motivation survey (spring) - Collect state reading and writing test data

- 3 teacher surveys - Data analysis and report writing

The primary sources of data collected and presented in this paper are Year 1 teacher surveys (3), a student survey collected at the end of the implementation year (2014-15), and focus group data.

Findings / Results

Key Findings on Teacher Mediating Outcomes Three teacher surveys per year measure the extent to which RAWC had an impact on teacher mediating outcomes, including changes in instructional practice and confidence in literacy instruction. Based on these data, RAWC did not have statistically significant impacts on teachers’ practices and beliefs, as can be seen in Table 3.4

The lack of significant results may stem from the relatively small size of the analysis sample, since we do observe effect sizes well above 0.25, in absolute value. For example, the effect sizes for students practicing metacognitive inquiry and comprehension strategies are 0.45 and 0.29, respectively. On the other hand, the effect sizes for teachers instructing and modeling comprehension strategies are -0.56 and -0.26, respectively. In a larger study of another Reading Apprenticeship project (Reading Apprenticeship Improving Secondary Education) results were similar after only one year of implementation (Fancsali et al., 2015)

For each construct, Table 3 shows predicted values (adjusted means) for treatment and control teachers, as well as the unstandardized (impact estimate) and standardized (effect size) values of the regression coefficient for the Treatment variable, as described in the Research Design section. See appendix A for teacher survey sample and means.

Table 3. Impact Estimates for Teacher Survey Outcomes

Construct

Treatment mean

(adjusted)

Control mean

(adjusted)Impact

estimate S.E. P-valueEffect Size

Variety of Text Types 2.97 3.08 0.10 (0.385) [0.788] 0.08Fostering Student Independence 7.68 8.12 0.44 (1.450) [0.762] 0.10Traditional Reading Strategies5 13.27 12.32 -0.95 (1.908) [0.619] -0.22Teachers Instructing Metacognitive Inquiry 1.50 1.41 -0.10 (0.343) [0.774] -0.10

4 The results presented here are from the model that includes all available school and teacher controls. The results are consistent with more parsimonious models developed via a variable selection process.5 The items making up the third construct, Traditional Reading Strategies, were distractor items, and not expected to be positively impacted by RA.

8

Construct

Treatment mean

(adjusted)

Control mean

(adjusted)Impact

estimate S.E. P-valueEffect Size

Teachers Modeling Metacognitive Inquiry 1.47 1.54 0.07 (0.273) [0.796] 0.07Students Practicing of Metacognitive Inquiry 2.08 2.56 0.48 (0.362) [0.183] 0.45Teachers Instructing Comprehension Strategies 3.26 2.38 -0.89 (0.589) [0.133] -0.56Teachers Modeling Comprehension Strategies 2.97 2.56 -0.41 (0.456) [0.369] -0.26Students Practicing Comprehension Strategies 3.29 3.78 0.49 (0.521) [0.350] 0.29Student Collaboration 10.08 9.36 -0.72 (1.279) [0.574] -0.20Student Engagement 4.23 4.35 0.13 (0.185) [0.498] 0.22Student Differentiation 0.15 0.28 0.13 (0.097) [0.171] 0.43

Key Findings on Student Mediating Outcomes Changes in teacher practices as a result of RAWC are hypothesized to change students’ classroom experiences, attitudes and behaviors. These mediating student outcomes were measured through a year-end student survey.

As we are not yet able to control for student characteristics, we have no impact analysis to report. A preliminary descriptive review of the student responses comparing treatment to the control group shows no significant differences, as can be seen in Table 4.6

Table 4. Student Survey Sample and MeansControl group Treatment Group Standardized

Construct (domain and subdomains) Mean Students Mean Students Difference P-valueIncreased use of comprehension strategies 2.86 589 2.86 918 -0.04 0.74 Use of global reading strategies 2.98 589 2.93 918 -0.07 0.45 Use of problem-solving strategies-mean 3.29 585 3.28 910 -0.03 0.78 Support reading strategies 2.32 583 2.39 907 0.03 0.80 Integration of content and literacy activity 2.68 563 2.71 860 0.02 0.80Increased metacognitive inquiry/use of metacognitive strategies 2.50 573 2.62 883 0.12 0.37 Metacognitive Inquiry 2.53 572 2.66 880 0.13 0.31 Metacognitive strategies-mean 2.45 572 2.55 883 0.10 0.45

Key Findings on Teacher Perceptions of Hybrid Professional Development Model

6 The results presented here do not include school, teacher, or student controls.

9

RAWC professional development included both in-person and online venues. Participating teachers in Indiana and Pennsylvania attended in-person training three times a year, three days in summer; two days in winter, and 2 days the following summer at a central location within their state. Teachers also attended in Ignite Sessions, where specific RAWC content was presented, and Professional Learning Communities, where they discussed classroom experiences and shared successes and challenges of implementing RAWC. Both Ignite sessions and PLC meetings were led by trained Reading Apprenticeship facilitators. Table 5 presents attendance at professional development activities.

Table 5. Professional Development AttendanceSUMMER TRAINING

2014

WINTER TRAINING 2014/15

SUMMER TRAINING

2015

IGNITE SESSIONS 2014/15

PLC MEETINGS

2014/153 days 2 days 2 days 4 sessions 6 meetings

IN mean 2.83 1.58 1.25 1.42 1.67sd 0.64 0.83 0.99 1.69 2.12

PA mean 2.74 1.65 1.43 2.26 3.30sd 0.86 0.78 0.90 1.45 2.24

In surveys and focus groups, teachers were asked about their experiences with RAWC professional development. We first present findings on the two components of online professional development. We then discuss reactions to online activities more generally.

RAWC Professional Development – Ignite Sessions

Ignite sessions are one of two online professional development components of RAWC. During Ignite sessions, facilitators present new content to teachers and provide guidance on implementation of RAWC strategies.

Ignite sessions were held four times over the course of the 2014-2015 school year. Attendance at sessions was relatively low. On average, teachers attended less than half of these sessions, with a great deal of variation among attendance rates (All (n=47): mean =1.83, sd=1.62; Indiana (n=24): mean=1.42, sd=1.69; Pennsylvania (n=23): mean=2.26. sd=1.45). The fall survey response rate is especially low due to the fact that the survey was conducted prior to the first set of Ignite sessions being held for many of the responding teachers.

In teacher surveys conducted three times over the course of the year, teachers provided feedback on the Ignite sessions they attended. Tables 6 through 8 present these questions and responses.

Table 6. Teacher survey responses regarding IgniteHOW EFFECTIVE DID YOU FIND THE IGNITE SESSION(S) AS A LEARNING EXPERIENCE?

Not at all effective

Not so effective

Effective Highly effective

10

FALL (N=11) 19% 47% 28% 9%WINTER (N=27) 7% 60% 22% 11%SPRING (N=29) 10% 35% 52% 3%

In both fall and winter surveys, approximately two-thirds of responding teachers reported that the Ignite sessions were “not at all effective” or “not so effective” (66%, n=11, in fall; 67%, n=27, in winter) as a learning experience. This number decreased in the spring survey to less than half (45%, n=29). It is important to note that many teachers faced difficulties with the online platform at the beginning of the year. This likely explains the high proportion of teachers that found the Ignite sessions to be ineffective. As the year progressed, a higher portion of teachers were more positive about Ignite, possibly due to teachers’ better understanding the RAWC program and their growing comfort with the technology.

Table 7. Teacher survey responses regarding IgniteHOW HELPFUL WAS THE INFORMATION PRESENTED DURING THE IGNITE SESSION FOR IMPROVING YOUR IMPLEMENTATION OF READING APPRENTICESHIP IN YOUR CLASSROOM?

Not at all helpful

Less than moderately

helpful

Moderately helpful

More than moderately

helpful

Very helpful

FALL (N=11) 18% 45% 27% 0% 9%WINTER (N=27) 11% 41% 37% 7% 0%SPRING (N=29) 14% 31% 48% 7% 0%

In the fall survey, just under two-thirds of responding teachers reported that the Ignite sessions were “not at all helpful” or “less than moderately helpful” (63%, n=11) in improving implementation of reading apprenticeship in the classroom. This number steadily decreased over the course of the year, falling to less than half of teachers (45%, n=29) responding to the spring survey. While still only 55% of teachers indicate that Ignite is helpful in any way at the end of the school year, the trend is positive. Again, this may be a result of teachers’ better understanding of RAWC, and their capacity to make better use of the training provided.

Table 8. Teacher survey responses regarding IgniteCOMPARED WITH IN-PERSON TRAININGS, HOW EFFECTIVE WOULD YOU RATE THE IGNITE SESSION AS A LEARNING EXPERIENCE?

Much less effective

Less effective

About the same

More effective Much more effective

FALL (N=11) 27% 54% 9% 9% 1%WINTER (N=27) 37% 44% 15% 0% 4%SPRING (N=29) 34% 41% 24% 0% 0%

While the number of responding teachers reporting more positively about Ignite over the course of the year, when asked how Ignite sessions compare to in-person training, as of the spring, no teachers

11

reported that Ignite was more effective than in-person training and just under a quarter (24%, n=29) reported that Ignite sessions and in-person trainings were “about the same.”

RAWC Professional Development – PLCs

Similar to Ignite sessions, PLC meetings occur online. But they differ in that they are intended to cover materials that are already familiar to RAWC teachers. PLC meetings provide a venue for teachers to share successes and challenges implementing RAWC in their classrooms.

PLC meetings were held six times over the course of the 2014-2015 school year, slightly more often than Ignite sessions. Similar to Ignite sessions, attendance at PLC meetings was low. On average, teachers attended close to half of these sessions, with a great deal of variation among attendance rates (All (n=47): mean=2.47, sd=2.31; Indiana (n=24): mean=1.67, sd=2.12; Pennsylvania (n=23): mean=3.30. sd=2.24).

In teacher surveys conducted three times over the course of the year, teachers provided feedback on the PLCs they attended. Tables 9 through 11 present these questions and responses.

Table 9. Teacher survey responses regarding PLCsOVERALL, HOW EFFECTIVE WAS THE PLC MEETING YOU ATTENDED AS A GROUP LEARNING EXPERIENCE?

Not at all effective

Not so effective

Effective Highly effective

FALL (N=12) 0% 16% 47% 31%WINTER (N=28) 11% 14% 60% 14%SPRING (N=28) 11% 25% 53% 11%

While many responding teachers did not immediately embrace the Ignite sessions, this was not the case for PLCs. As presented in the table above, much higher proportions of responding teachers in the fall (78% [(n=12]), winter (74% [n=28]), and spring (64% [n=28]) indicated that they found the PLCs to be an effective learning experience.

Table 10. Teacher survey responses regarding PLCsHow helpful was the PLC meetings you attended for implementing RAWC strategies in your classroom?

Not at all helpful

Less than moderately

helpful

Moderately helpful

More than moderately

helpful

Very helpful

Fall (n=12) 0% 8% 50% 17% 25%Winter (n=28) 7% 18% 43% 14% 0%Spring (n=28) 7% 25% 18% 36% 0%

12

Similarly, much higher proportions of responding teachers in the fall (92% [(n=12]), winter (57% [n=28]), and spring (54% [n=28]) indicated that they found the PLCs to be helpful in implementing reading apprenticeship strategies in the classroom. In is notable that the percentage of responding teachers reporting positively decreases over the course of the year. This may be due to a drop off in PLC participation over the course of the year. Further investigation is necessary.

Teachers’ reports of their experiences in PLC meetings, presented in Table 11 below, is consistent with the findings above. Almost all responding teachers are extremely positive about their experiences attending PLCs, especially in the fall. According to survey results, this decreases somewhat over the course of the year, but remains high.

Table 11. Teacher survey responses regarding PLC meetingsTo what extent do you agree with the following statements about the PLC meetings you attended so far this year?

% of teachers that agreed or strongly agreed with the following…

Fall(n=12)

Winter (n=28)

Spring (n=28)

The PLC makes me feel supported by my colleagues 100% 86% 79%

I was able to learn by sharing my lessons and student work

92% 71% 79%

I was able to learn by sharing my own challenges to providing instruction

100% 82% 79%

Hearing my colleagues share their approaches and challenges informed my practice

100% 82% 82%

I felt supported by the PLC facilitator 100% 96% 89%

I got the information I need from the PLC facilitator 100% 96% 82%

The PLC caused me to reflect on my own practices 100% 86% 82%

Focus groups with teachers in Indiana and Pennsylvania attending on-line professional development shed more light onto teachers’ views of on-line professional development.

Teachers reported that the most helpful aspects of PLCs include: being part of a professional community; having time to reflect on instructional practices; and receiving support and resources from meeting facilitators. Teachers in both Indiana and Pennsylvania highlighted their appreciation for the opportunity to share their challenges and successes with colleagues in other schools and districts. The PLC meetings generated helpful feedback and suggestions for many.

Teachers also reported frustrating aspects on PLCs. Many discussed the fact that the number of PLC participants was often too low to have meaningful discussions. Teachers also complained about the

13

inconvenient scheduling of the PLC meetings. And a few teachers mentioned that they felt that the content covered was not useful. The most common complaint was about the technology supporting on-line content delivery. At the start of the year, there were many technical issues with both the PLC and Ignite sessions that frustrated teachers and resulted in reduced participation.

Conclusion

Preliminary findings from this study are inconclusive, with no indication that the RAWC hybrid model will achieve the same positive findings as other studies of Reading Apprenticeship. However, given the small sample size and slow start up of the intervention, it is not surprising that we have not seen evidence of changes to teacher practice or student mediating outcomes.

Analysis of teacher survey and focus group data begins to shed light on the potential for online professional development. Many RAWC teachers strongly value being part of a professional community both within and beyond their schools. PLC meetings have supported this, likely explaining the preference for PLC meetings over Ignite sessions for many RAWC teachers.

This study has provided formative information to the Strategic Literacy Initiative, which has been used to adjust program implementation in the current school year. Preliminary results of the 2015 fall survey indicate that a greater portion of this years’ responding teachers respond more favorably to survey questions about online professional development. Further research is needed to determine the extent to which changes to the RAWC program, as well as more time for teachers to incorporate Reading Apprenticeship strategies into their classroom practices, will influence student outcomes.

Given the need for teachers to master instructional practices that can support students meeting rigorous standards, and the high costs of in-person professional development, it stands to reason that hybrid professional development models are worthy of further investigation.

14

References

ACT. (2012). Catching up to college and career readiness. Retrieved from http://www.act.org/research/policymakers/pdf/CatchingUpToCCR.pdf

Fancsali, C., Abe, Y., Pyatigorsky, M., Ortiz, L., Chan, V., Saltares, E., Toby, M., Schellinger, A., Jaciw, A. (2015). The impact of the reading apprenticeship improving secondary education (RAISE) project on academic literacy in high school: A report of a randomized experiment in Pennsylvania and California schools. Final report to Institute for Education Sciences, Investing in Innovation (i3) Validation grant #U39B100255 awarded to WestEd. Retrieved from http://empiricaleducation.com/pdfs/RAISEfr.pdf

Greenleaf, C. L., Litman, C., Hanson, T. L., Rosen, R., Boscardin, C. K., Herman, J., & Jones, B. (2011a). Integrating literacy and science in biology: Teaching and learning impacts of Reading Apprenticeship professional development. American Educational Research Journal, 48(3), 647–717.

Greenleaf, C.L., Hanson, T., Herman, J., Litman, C., Rosen, R., Schneider, S., & Silver, D. (2011b). A Study of the Efficacy of Reading Apprenticeship Professional Development for High School History and Science Teaching and Learning. Final report to Institute for Education Sciences, National Center for Education Research, Teacher Quality/Reading and Writing, Grant # R305M050031.

Kemple, J., Corrin, W., Nelson, E., Salinger, T., Herrmann, S., & Drummond, K. (2008). The Enhanced Reading Opportunities Study: Early Impact and Implementation Findings (NCEE 2008-4015). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

O’Reilly T., Weeks, J., Sabatini, J., Halderman, L., & Steinberg, J. (2014). Designing Reading Comprehension Assessments for Reading Interventions: How a Theoretically Motivated Assessment Can Serve as an Outcome Measure. Educational Psychology Review, 26(3), 403-424.

Somers, M. A., Corrin, W., Sepanik, S., Salinger, T., Levin, J., & Zmach, C. (2010). The Enhanced Reading Opportunities Study Final Report: The Impact of Supplemental Literacy Courses for Struggling Ninth-Grade Readers. NCEE 2010-4021. National Center for Education Evaluation and Regional Assistance.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), various years, 1992– 2013 Mathematics and Reading Assessments. Department of Education, Institute of Education Sciences.

15

Appendix A

Table 1. Teacher Survey Sample and MeansControl group Treatment group

Construct Mean SD Teachers Schools Mean SD Teachers SchoolsVariety of Text Types 3.08 1.29 47 14 2.95 1.14 43 13Fostering Student Independence 7.35 4.05 47 14 8.48 4.39 43 13Traditional Reading Strategies 12.91 4.40 47 14 12.59 4.10 43 13Teachers Instructing Metacognitive Inquiry 1.46 0.94 47 14 1.46 1.04 43 13Teachers Modeling Metacognitive Inquiry 1.45 0.92 47 14 1.57 1.03 43 13Students Practicing of Metacognitive Inquiry 2.02 1.10 47 14 2.63 1.03 43 13Teachers Instructing Comprehension Strategies 3.04 1.70 47 14 2.62 1.42 43 13Teachers Modeling Comprehension Strategies 2.79 1.52 47 14 2.76 1.59 43 13Students Practicing Comprehension Strategies 3.13 1.73 47 14 3.96 1.58 43 13Student Collaboration 9.58 3.62 46 14 9.95 3.60 43 13Student Engagement 4.24 0.59 47 14 4.34 0.52 43 13Student Differentiation 0.18 0.30 47 14 0.24 0.31 43 13

16