Panel organizer: Bryan Cook, University of Virginia bc3qu@virginia.edu

Discussant: Christopher Lemons, Vanderbilt University

Replication in Special Education Research: Definitely Needed, but Rarely Conducted

Panel members: Samantha Gesel, University of North Carolina Charlotte; Erica Lembke, University of Missouri; William Therrien, University of Virginia

Panel Overview: Replication research is a critical part of the scientific endeavor that verifies previous findings and identifies the boundaries of intervention effectiveness. However, systematic replications are rare in special education; and extant replications in the field are predominantly conducted by the same researchers who conducted the study being replicated. In this panel, we (a) examine issues in replication research in special education, (b) present two independent, conceptual replication studies, and (c) discuss an ongoing IES-funded project to develop a crowdsourced platform for conducting large-scale, independent replication studies.

Panel Overview and Introductory Presentation: Replication research is a critical part of the scientific endeavor (Travers, Cook, Therrien, & Coyne, 2016). It is important that researchers conduct replication studies to fully and robustly address important research questions. Additionally, conceptual replication studies are critical for empirically identifying the boundaries of intervention effectiveness (Tincani & Travers, 2019). That is, researchers can systematically vary the elements of an intervention, research samples, settings, and study outcomes across studies to examine which variations and for which populations, outcomes, and settings a practice is and is not effective. The importance of replication research in the field is reflected in the 2018 joint report of The National Science Foundation and The Institute of Education Sciences (IES), as well as IES' current request for applications for Research Grants Focused on Systematic Replication. However, systematic replications are rare in special education; and extant replications in the field are predominantly conducted by the same researchers who conducted the study being replicated (Lemons et al., 2016; Makel et al., 2016), which may pose a threat to the validity. In this panel, we will discuss (a) issues in replication research in special education, (b) two independent, conceptual replication studies, and (c) an IES-funded project to develop a crowdsourced platform for conducting large-scale, independent replication studies.

Conceptual Replication Study #1: The purpose of this study was to replicate the work of Jenkins, Schulze, Marti, and Harbaugh (2017). We administered three oral reading fluency passages per week for 14 weeks with 51 elementary students in special education. We used data from all probes to estimate students' "true growth." We selected specific passages to simulate what the data would have been for different progress monitoring (PM) schedules (e.g., 1 probe/week, 2 probes biweekly) and estimated all applicable weekly slopes for each PM schedule. We compared the "true growth" and weekly slopes to a goal growth rate and determined the adequacy of student growth. We identified whether the determination of response from weekly slope data matched the determination from true-growth data, and calculated the proportion of matched decisions (i.e., decision accuracy) for different PM schedules. Similar to the original results of Jenkins et al., our results indicate that the accuracy and timeliness of intermittent PM schedules and weekly PM schedules are comparable. The accuracy of PM schedules was lower and the timeliness poorer for our sample compared to the results reported by Jenkins et al. We discuss the implications of these results for research and practice.

Conceptual Replication Study #2: This in-progress study also replicates the procedures of Jenkins et al.'s (2017) study of CBM PM schedules, but in the area of mathematics instead of reading. Thus, the two replication studies presented as part of this panel represent a coordinated effort to conduct a systematic series of systematic conceptual replication studies on the same topic, as recommended by Coyne, Cook, and Therrien (2016). We selected the Jenkins et al. study for replication given the impact of this line of research in the field of special education (i.e., wide use of CBM, the need to verify results reported in the original study, and the potential impact of the study's conclusions on schools' use of CBM). In this study, we consider the same two research questions as in the original and the other replication studies: "Is decision-making accuracy from intermittent PM inferior to that from weekly PM, the current standard?," and "How many weeks of PM do these schedules require to reach specific levels of decision accuracy?;" but with a focus on CBM in mathematics computation. We will compare our results to the original results, and to those of the other replication study, to determine whether there is a significant difference between the current, weekly standard for PM and alternative, sparser schedules in mathematics.

Crowdsourcing Replication Studies: Despite the increase in high-quality, experimental research in special education, large-scale, representative, and open replication studies remain scarce. Crowdsourcing is one potential tool we can use to increase the number of high-quality, large-scale replication studies. Crowdsourcing (i.e., combining resources across researchers to conduct studies that could not be accomplished individually) has been used to great effect in other fields. For example in the Many Labs project, crowdsourcing was used to replicate 28 psychology studies multiple times each across 186 research teams (Ebersole et al., 2016). Crowdsourcing in research can take many forms, such as multiple research teams statistically analyzing the same data set (e.g., Silberzahn et al., 2018), multiple research labs each conducting separate replication studies (e.g., Open Science Collaboration, 2015), and many research teams replicating the same study or studies (e.g., Klein et al., 2018). In this portion of the presentation, we will discuss potential pros and cons of utilizing crowdsourcing to conduct replication studies in special education. We will also provide information on the Special Education Research Accelerator (SERA), which is a new platform being developed for conducting crowdsourced replication studies in special education funded by IES.

Discussion Questions:

1. Considering that funding replication studies potentially reduces funding available for new research, what are the best ways to design replication studies and to prioritize where funding should be allocated?

2. What are the best procedures for determining which studies should be replicated (e.g., consider IES's focus on a subset of specific interventions in the most recent RFA)?

3. How should we interpret "failed" replications? Any thoughts on designing replications so that we maximize our learning if initial findings do not replicate?

References: Coyne, M. D., Cook, B. G., & Therrien, W. J. (2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37, 244-253. doi: 10.1177/0741932516648463 Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., ... Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. doi: 10.1016/j.jesp.2015.10.012 Jenkins, J., Zumeta, R., & Dupree, O. (2005).

Measuring gains in reading ability with passage reading fluency. Learning Disabilities Research & Practice, 20, 245-253. doi: 10.1111/j.1540-5826.2005.00140.x Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams Jr, R. B., Alper, S., ... & Batra, R. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443-490. doi: 10.1177/2515245918810225 Lemons, C. J., King, S. A., Davidson, K. A., Berryessa, T. L., Gajjar, S. A., & Sacks, L. H. (2016). An inadvertent concurrent replication: Same roadmap, different journey. Remedial and Special Education, 37, 213- 222. doi:10.1177/0741932516631116 Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A., Simonsen, B., & Coyne, M. (2016). Replication of special education research: Necessary but far too rare. Remedial and Special Education, 37, 205-212. doi:10.1177/0741932516646083 National Science Foundation & Institute for Education Sciences. (2018). Companion guidelines on replication & reproducibility in education research. Author. Retrieved from https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi: 10.1126/science.aac4716 Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., ... & Carlsson, R. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1, 337-356. doi: 10.1177/2515245917747646 Tincani, M., & Travers, J. (2019). Replication research, publication bias, and applied behavior analysis. Perspectives on Behavior Science, 42, 59-75. doi: 10.1007/s40614-019-00191-5 Travers, J. C., Cook, B. G., Therrien, W. J., & Coyne, M. D. (2016). Replication research and special education. Remedial and Special Education, 37, 195-204. doi:10.1177/0741932516648462

Panel organizer: Sarah R. Powell, The University of Texas at Austin srpowell@austin.utexas.edu

Greater Than X: The Latest and Greatest Algebraic Reasoning Research

Panel members: Elizabeth Adams, Southern Methodist University; Leanne Ketterlin-Geller, Southern Methodist University; Jessica Namkung, University of Nebraska-Lincoln; Rajiv Satsangi, George Mason University; Sarah Watt, Miami University

Panel Overview: First, Dr. Elizabeth Adams and Dr. Leanne Ketterlin-Geller present validity evidence for a universal screener designed to assess middle-school students' readiness for algebra. Predictive-related evidence with the state assessment and the following year's high-school Algebra 1 end-of-course scores is examined for students with disabilities. Using hierarchical linear modeling, results show the scores on the universal screener across three timepoints are positively related to test scores (p < .001). Data from 520 Grade 8 students with disabilities were used for these analyses. The results persist after controlling for student and school differences. This data helps set the stage for a focus on intervention research related to algebra.

Second, Dr. Sarah Watt provides a review of effective interventions for teaching algebra to students with learning disabilities. She conducted a comprehensive, systematic review of the literature from 1980-2014. The comprehensive literature and review process yielded 15 studies for inclusion in the review. Meta-analytic calculations were conducted producing overall effect sizes for each included study. Studies using quantitative designs were interpreted using Hedge's g and single-subject designs were interpreted using PND and Tau-U. The 15 studies included 827 participants between 3rd and 12th grade, the largest population of students were sixth graders (34%). Overall, the 10 experimental and 5 single-subject designs produced a moderate overall effect size (g = 0.48). Among the 15 studies 5 interventions were identified as effective, (1) concrete-representational-abstract (CRA), (2) cognitive strategy instruction, (3) enhanced anchored intervention (EAI), (4) tutoring, and (5) graphic organizers.

Third, Dr. Sarah Powell describes a randomized control trial in which 450 3rd-grade students with mathematics difficulty (MD) were randomly assigned to 1 of 2 variants of a word-problem intervention or a business-as-usual (BAU) control. One of the word-problem variants introduced students to two essential components of pre-algebraic reasoning: interpretation of the equal sign as a relational symbol and solving equations with unknowns in various positions of the equation. After receiving the pre-algebraic reasoning component, we used a sequential mediation model to determine that pre-algebraic instruction led to improved equal-sign understanding, which, in turn led to improved equation solving. This equation solving led to improved word-problem performance.

Fourth, Dr. Jessica Namkung describes a study in which she examined the effects of an algebraic equation solving intervention. 48 sixth graders with MD were randomly assigned to the algebraic equation solving intervention, Mystery Math, (n = 24) or a control condition (n = 24) at the individual level. The Mystery Math intervention was designed as a small-group, Tier 2 mathematics program based on the principles of explicit instruction to promote students' algebraic equation solving skills. This multi-component intervention focused on teaching mathematics vocabulary relevant to solving algebraic equations, improving conceptual knowledge of algebraic equation solving via concrete and visual representations, and building fluency with procedural computations and solving for variables. Results indicated that the main effect of intervention was significant for two proximal measures of mathematics

vocabulary and equation solving with large effect sizes. However, the main effect of intervention was not significant for a distal measure of pre-algebra skills.

Fifth, Dr. Rajiv Satsangi discusses the role of assistive and instructional technology in mathematics education for students with MD. Specifically, he discusses whether there is a functional relation between the use of technologies such as virtual manipulatives and strategies, such as video modeling, and the performance of students with MD solving algebraic word problems, multi-step linear equations, and graphing linear equations. Twenty-one students with MD in grades 9-12 participated across 7 single-subject research studies. Findings from each demonstrated a positive relationship between virtual manipulatives and video modeling and students' ability to solve problems. In addition, student performance across measures of problem-solving independence, duration of sessions, and social validity returned favorable outcomes for both strategies, while also highlighting notable limitations of each technology for teachers to consider.

Discussion Questions:

1. How early in a school career should students with MD receive focused interventions on pre-algebraic and algebraic content?

2. What content and pedagogical knowledge do special education teachers need in order to meet the algebraic needs of students with MD?

3. What may be some yet-to-be-developed and tested components of algebraic interventions? Do the components of intervention differ across grade levels?

Panel organizer: Douglas Fuchs, Vanderbilt University doug.fuchs@vanderbilt.edu

Co-organizer: Lynn Fuchs, Vanderbilt University lynn.fuchs@vanderbilt.edu

Discussant: Sharon Vaughn, University of Texas at Austin

Unresolved Methodological Challenges for Advancing Intervention Research

Panel members: Kristen McMaster, University of Minnesota; Lynn Fuchs, Vanderbilt University; Douglas Fuchs, Vanderbilt University

Panel Overview: As the field's attention turns toward children with intensive academic needs, we face unresolved methodological challenges necessary for advancing intervention practices in innovative directions. In this session, we address three such challenges, each with a randomized controlled trial. In the first presentation, Kristen McMaster focuses on the challenge of identifying or developing reliable and valid ways to measure complex constructs like reading comprehension, inferencing, and cognitive processing.

In the second presentation, Lynn Fuchs focuses on the challenge of estimating an innovative intervention component's added value when the dogma of intervention research requires us to hold intervention time constant between the standard intervention condition and the condition that adds the innovative component. In the third presentation, Doug Fuchs focuses on the challenge or wisdom of incorporating instructional activities or components, for which prior studies demonstrate isolated effects, into a multi-component, complex intervention.

McMaster: The purpose of Kristen McMaster's study was to develop and provide initial evidence of the reliability, validity, and sensitivity to growth of the Minnesota Inference Assessment (MIA). MIA is a technology-based language comprehension assessment that does not require decoding skills and thus can be used to assess comprehension of young children who are just beginning to learn to read. It was initially developed as a proximal measure of struggling comprehenders' response to an early language comprehension intervention. In this study, MIA was administered before and after an 8-week inferencing intervention to a diverse sample of n = 272 students in grades K-2. Coefficient alpha was .88, and alternative reliability indices were strong (Guttman's L2 = .88, Feldt-Brennan = .88, Feldt-Gilmer = .88), confirming MIA's internal consistency and reliability. Correlations ranged from r = .322 to .403 between MIA and the Clinical Evaluation of Language Fundamentals: Fifth Edition--Understanding Spoken Paragraphs subtest, and from r = .326 to .339 with the Gates-MacGinitie Reading Test. Pre-post effect sizes ranged from η2 = .08 to .20 depending on which of four forms were used, suggesting sensitivity over an 8-week period and thus promise for MIA to be used to evaluate responsiveness to intervention.

L. Fuchs: The purpose of Lynn Fuchs's study was to assess the added value of different forms of working memory (WM) training beyond effects of a direct skills intervention (schema-based word-problem instruction [SBI] on word-problem (WP) performance. Participants were 2nd graders scoring <25th percentile on WP skill. Children were randomly assigned to 4 conditions: control, validated SBI (a "standard-of-practice" program), validated SBI with domain-specific WM training, and CogMed (domain-general WM training) with 5 min WP practice each session. In the 3 active conditions, intervention time

was held constant: three 30-min sessions per week for 15 weeks. Results indicated that all 3 active conditions ended intervention with significantly stronger WP performance than the control group. Both SBI groups performed significantly stronger than the CogMed condition. However, the two SBI conditions performed comparably. Results will be discussed in terms of the challenge associated with estimating an innovative intervention component's (here, domain-specific WM training's) added value over the standard of care intervention, when the dogma of intervention research requires us to hold intervention time constant between conditions.

D. Fuchs: National reports indicate many students struggle to comprehend written texts, revealing the need for interventions to improve reading comprehension and in particular comprehension of informational texts. We conducted a randomized control trial to further investigate a nonfiction comprehension program previously found to be effective in promoting factual retention and comprehension of informational passages at grades 4-5. We contrasted the standard version of the program, containing a wide variety of comprehension strategies (full), with a slimmed-down version containing a subset of strategies deemed most important (simple). Participants were 149 4th-grade students in 54 classrooms in 13 elementary schools, who were identified on screening measures as weak in reading comprehension. Commercially-available and experimenter-created measures of reading comprehension indexing near-, mid-, and far-transfer were used to assess effects. Results indicated students in both programs retained factual knowledge of informational passages following intervention. Students who received the full program significantly improved their comprehension of unfamiliar, near-transfer informational texts; however, students in the simple program tended to perform better on the mid- and far-transfer measures. Results will be discussed in terms of the challenge of incorporating instructional strategies, for which prior studies demonstrate isolated effects, into a multi-component, complex intervention.

Discussion Questions:

What are your experiences with the types of challenges we've addressed: measuring complex constructs; estimating added value for an innovative intervention component, while holding time constant between standard and innovative conditions; and incorporating activities with prior evidence of isolated effects into multi-component interventions?

How have you addressed these challenges in your research?

What other issues challenge intervention innovation or detection of effects?

Panel organizer and discussant: Nathan Stevenson, Kent State University nsteve15@kent.edu

Defining and redefining engagement: Implications for design, measurement, data quality, and instruction

Panel members: Nathan A. Stevenson, Kent State University; Kelly J. Williams, Indiana University; Leticia Martinez, The University of Texas, Meadows Center for Preventing Educational Risk; Lindsay Foreman-Murray, Western Washington University

Overview: Engagement is a multi-dimensional construct with strong implications for academic learning and behavior management. The degree to which students actively engage in instructional activities is strongly correlated with academic achievement and prevention of many common behavior problems. However, across the research literature, there is considerable variation in the ways engagement is defined. Such variation impacts the quality of data collection and measurement of instructional and behavioral outcomes. The following presentation describes a number of ways in which engagement is defined and measured across study designs, including single-case, surveys, RCTs, meta-analyses, and applied research.

Study 1 (Lindsay Foreman-Murray): The transition from middle to high school is a critical moment in students' academic careers, when school engagement and GPA tend to decline while misbehavior and depression rise. One promising area of research focuses on school efforts to support students' transitions from middle to high school as a means to improve adjustment to high school. Using data from HSLS:09, I investigated associations between types of support for transition and students' ninth-grade school engagement and academic performance. The study sample comprised 808 9th grade students with LD. I used multiple regression to determine the relation between support for transition and student outcomes. None of the supports were consistently predictive of outcomes. Limitations in the ways the variables related to transition support and school engagement were conceptualized and constructed in HSLS:09 suggest a need for greater sophistication in the use of those constructs in nationally representative, large-scale surveys.

Study 2 (Kelly Williams): This presentation reports results from a meta-analysis that examined the impact of the Check & Connect mentoring intervention on measures of academic achievement (e.g., grades, test scores), student engagement (e.g., attendance, behavior referrals, self-report measures), and high school dropout and completion for students in grades 4-12. Studies were included in this meta-analysis if they were published from January of 1990 to October of 2018, implemented Check & Connect with students in grades 4-12, used a quasi-experimental or randomized controlled trial design, and included at least one measure of academic achievement, student engagement, or dropout/school completion. Ten studies met criteria for inclusion as a result of the electronic, hand, and ancestral searches. Data has been extracted from all studies using a code sheet and will be analyzed with the Comprehensive Meta-Analysis software by October 31, 2019. Findings will be shared with respect to each outcome domain (i.e., achievement, engagement, dropout) and implications for future research will be discussed.

Study 3 (Leticia Martinez): The third presentation describes a randomized controlled trial that examined the effects of Check & Connect (Christenson, Stout, & Pohl, 2012) on student engagement outcomes for 358 high school English learners at risk for school dropout. Participants were randomly

assigned to receive either two years of the Check & Connect intervention or to a Business as Usual (BaU) comparison condition during 9th and 10th grades, and pre- and post-intervention data on a self-report measure of engagement and school-based indicators of engagement were collected. Interviews on a subset of 34 students who reported increases or decreases in engagement over the two-year intervention were also conducted. Overall, the students in the intervention did not perform significantly higher than students in the comparison condition on self-reported student engagement, attendance, behavior, or grades. Additionally, students' self-reports of engagement were not corroborated by their actual attendance, grades, and interviews. Students valued the mentors implementing Check & Connect but requested more attention academically and personally from their teachers.

Study 4 (Nate Stevenson): The final presentation examines data from a series of studies that operationalize engagement in reading instruction for students with and without learning disabilities in middle school and high school (Stevenson, 2016, Stevenson & Mussalow, 2019, Stevenson, Swain-Bradway, & LeBeau, 2019). This presentation highlights the shortcomings of typical operational definitions of engagement and unpacks the ways in which such definitions impact the quality of data collection in single-case and group design studies. Specific attention is given to the distinctions (or lack thereof) between academic and behavioral tasks in schools. Attendees will be asked to critique/comment on the implications of definitions for engagement at the individual, class-wide, and school-wide levels.

Audience members are invited to engage with presenters throughout the session. We invite thoughtful comments and feedback on the presentation contents as well as implications for future research and practice in the field.

Discussion Questions:

How can common definitions of engagement be strengthened to improve the consistency and reliability of outcome data?

Under what conditions is improving engagement a meaningful goal as opposed to a vehicle to achieving other outcomes?

How does one balance definitions that yield accurate, reliable, and consistent measurement versus those that maximize social validity and meaning to practitioners?

How does engagement vary as a function of child development and academic expectations in school?

References: Christenson, S. L., Stout, K. E., & Pohl, A. J. (2012). Check & Connect: A comprehensive student engagement intervention: Implementing with fidelity. Minneapolis, MN: University of Minnesota, Institute on Community Integration. Stevenson, N. A., Swain-Bradway, J., LeBeau, B. C. (2019) Examining High School Student Engagement and Critical Factors in Dropout Prevention. Assessment for Effective Intervention. doi: 10.1177/1534508419859655 Stevenson, N. A. & *Mussalow, P. (2019) The Effects of Planning, Goal Setting, and Feedback on Academic Avoidance Behaviors for Students with and without Learning Disabilities. Journal of Positive Behavior Interventions. 21(3), 171-180 doi: 10.1177/1098300718804566 Stevenson, N. (2016). Effects of planning and goal setting on reducing latency to task engagement for struggling readers in middle school. Journal of Behavioral Education. 25(2) 206-222. doi: 10.1007/s10864-015-9238-8

Panel organizer: Elizabeth Swanson, University of Texas at Austin easwanson@austin.utexas.edu

A Variety of Approaches to Address Reading Comprehension in the Middle Grades

Panel members: Alicia A. Stewart, Central Connecticut State University; Jade Wexler, University of Maryland; Amy Barth, Buena Vista University

Panel Overview: Schools face pressure to help students with disabilities meet rigorous achievement standards (ESSA, 2015) and intensive intervention is often necessary to make a meaningful impact on reading comprehension among students in the middle grades (Vaughn et al., 2012). These studies explore different approaches to comprehension instruction including: a) infusing social studies with reading comprehension practices and pairing Tier 1 and Tier 2 instruction, b) reading comprehension instruction for students with inattention, c) knowledge and inference making, and d) professional development to improve practices used by co-teaching pairs.

Infusing Social Studies with Reading Comprehension Practices: An RCT conducted with 80 schools randomized to treatment (i.e. STRIVE) or business as usual (BAU) was conducted to investigate the effects of infusing social studies with vocabulary and reading comprehension practices on 4th graders' reading outcomes. Teachers delivered 36 STRIVE lessons over the course of 18 weeks. Students assigned to the treatment classes outperformed at a statistically significant level peers assigned to the BAU condition on content knowledge, vocabulary, and content reading comprehension (Swanson et al., in prep). Preliminary findings from a smaller study examining the impact of aligning STRIVE Tier 1 and Tier 2 instruction will be discussed as well.

The Effect of STRIVE for Students with Inattention: Although students who struggle to maintain attention tend to embody similar word reading abilities as their typically-developing peers, they perform significantly lower on reading fluency and reading comprehension measures (Ghelani et al., 2004). STRIVE is a set of reading practices that includes evidence-based instructional practices which target vocabulary and reading comprehension using informational text in social studies classrooms. This study investigated the efficacy of STRIVE reading instruction on the reading outcomes of students with inattentive behaviors. Using a quasi-experimental design, we included participants identified with inattentive behaviors based on teacher referral and a brief ADHD measure (Conners 3; Conners, 2008). Students' reading outcomes were compared using an ANCOVA that accounted for pre-test scores. Students in treatment conditions significantly outperformed students in the comparison condition on content knowledge and vocabulary measures as well as a content reading comprehension measure. There were no significant differences between conditions on standardized reading measures.

Role of Knowledge Availability in Inference Making: Availability of knowledge represents a potential source of individual differences in knowledge-to-text inferencing. A total of 145 students in grades 5 and 6, who achieved a score of 92 or greater on the Gates MacGinitie Reading Test, were randomized to Study or Study plus Discussion conditions. Using an experimental procedure to ensure all participants had the same knowledge available to them prior to hearing a story, participants 1) were taught 12 facts about an imaginary planet called Gan, 2) were given 30 seconds to study pictures that depicted the Gan facts (Study condition) or study plus discuss pictures that depicted the Gan facts (Study + Discussion condition) with the experimenter, 3) listened to a six-episode story about the planet Gan, 4) were asked four questions assessing literal and inferential information after each episode, and 5) retention of the

knowledge base was retested. Results showed no significant difference between the two groups. Students across both groups on average performed significantly better at posttest compared to one-week and one-month follow-up tests. There was a significant effect of speed of response on students' overall score; students who took more than average time generally scored higher. The benefit of taking more time to respond to questions was greater for students in the Study plus Discussion group compared to their peers in the Study group.

Improving Literacy Instructional Practices in Co-Taught Content Area Classes: The purpose of this study was to improve the way in which content area and special education co-teaching pairs integrate evidence-based literacy instruction and parity in roles and responsibilities within the middle school classroom. Eleven co-teaching pairs in nine schools were randomly assigned to CALI or a business-as-usual comparison condition. CALI teachers were provided professional development and coaching on effective co-teaching models and content area literacy instruction. CALI teachers demonstrated significantly higher scores than comparison teachers at posttest on a measure of knowledge and skills (0.804, p < .001, Hedges' g = 2.42), perceived personal effectiveness (6.802, p = .078, Hedges' g = 0.89), and perceived co-teachers' effectiveness (12.774, p = .027, Hedges' g = 0.68). Students in the CALI classrooms demonstrated significant gains on a researcher-developed measure of reading comprehension (0.042, p = .080, Hedges' g = 0.39). However, the CALI treatment effect was non-significant for the two standardized measures of reading comprehension (Hedges' g = 0.02; Hedges' g = -0.01).

Discussion Questions:

1. How can we better leverage Multi-Tiered Systems of Supports to improve reading comprehension outcomes for middle grade students?

2. What issues related to measurement of reading comprehension should researchers consider as these lines of research continue?

3. How should researchers address the changing nature of BAU in intervention research?

Panel organizer: Christine Espin, Leiden University espinca@fsw.leidenuniv.nl

Understanding and assessing reading comprehension for struggling readers

Panel members: Christine Espin, Leiden University; Anne Helder, Leiden University; Natalie Foerster, University of Muenster; Elmar Souvignier, University of Muenster

Panel Overview: Reading comprehension is a multidimensional, multicomponent construct. In comprehension, multiple processes come together in combinations that shift constantly during reading of the text as a function of properties of the text and the goals and skills of the text and reader. The complex nature of reading comprehension presents challenges for developing reading comprehension assessments. In this panel session, we bring together a group of international researchers from diverse perspectives (cognitive psychology, educational psychology, and special education). The panel begins with a presentation of research examining the processes underlying reading comprehension. Attention then shifts to research examining the role of such processes in the assessment of comprehension. The focus of each presentation is on students who struggling with reading comprehension.

Presentation 1: Coherence-break detection as a measure of reading comprehension (Anne Helder, Linda van Leijenhorst & Paul van den Broek): When trying to understand a text, readers engage in various cognitive processes. If all goes well, the engagement in these cognitive processes results in the construction of a coherent mental representation of a text, the essence of successful reading comprehension. To construct such a representation, readers need to monitor the coherence of a text. An overview of existing studies on coherence monitoring will be provided with specific attention to coherence monitoring in children. For example, Helder, Van Leijenhorst, & van den Broek (2016) compared online (reading times) and offline (consistency judgments) measures of coherence monitoring in good and poor comprehenders of 8-11-year old children (N = 74), using an inconsistency paradigm. Results showed that both good and poor comprehenders slowed down when they encountered inconsistencies during reading, but poor comprehenders were less accurate in detecting inconsistencies after reading.

Presentation 2: Coherence monitoring during completion of maze tasks (Christine Espin, Elise Swart, Anne Helder, & Arnout Koorneef): Scores from Curriculum-Based Measurement (CBM; Deno, 1985) maze tasks have been shown to be reliable and valid indicators of reading performance and progress (e.g. Wayman, et al., 2007; Conoyer, et al., 2016); however, questions have been raised about the extent to which these scores reflect reading comprehension (see for example, Muijselaar et al. 2017). In this study, we used an inconsistency-paradigm to examine the extent to which maze completion required reading comprehension. Two groups of participants (5th/6th grade students and university students) completed a series of CBM maze tasks while their eye-movements were registered. Maze tasks were created using passages from a previous inconsistency-paradigm study (Helder et al., 2016). For each participant, half of the maze passages contained an inconsistency, and the other half did not. Reading times for the consistent/inconsistent sentences were measured, as were the number of regressions to previous sentences in the text. Preliminary analyses of the children's data reveal that reading times for target sentences were longer in the inconsistent than in the consistent condition, suggesting that children were monitoring comprehension while completing the mazes.

Presentation 3: Assessing progress in reading comprehension: Validity of the slope (Natalie Förster, Mathis Erichsen, & Elmar Souvignier): We developed a series of equivalent web-based tests (quop-L2) to be used for progress monitoring in reading. In each test, reading comprehension is assessed on the word-, sentence-, and text-level. In this study, we (a) evaluate the convergent and divergent validity of the single tests by comparing the quop-L2 assessments with the results of standardized tests in reading, mathematics, and reasoning, and (b) investigate the validity of the slope by comparing growth estimates of quop-L2 with the reading progress in two standardized reading tests using longitudinal data of N = 330 students. The correlation pattern indicated good convergent and discriminant validity of the single tests. Regarding the slope, however, significant positive correlations with the growth in the standardized reading tests were found on the sentence- and text-level but not on the word level. We discuss the results with regard to differences in task characteristics and the sensitivity of reading measures.

Discussion Questions:

1. How can insights into the underlying processes involved in reading comprehension inform the development of reading comprehension assessments?

2. What questions should be asked in establishing the validity of reading comprehension assessments, and how do such questions differ across the various purposes of assessment (e.g., screening, diagnosis, progress monitoring, etc.)?

3. Is reading comprehension the product of multiple component skills (word recognition, fluency, vocabulary, background knowledge), or a separate, unique skill that draws from, but goes beyond, these component skills?

Panel organizer: Steve Graham, Arizona State University steve.graham@asu.edu

Facilitating Writing and Facilitating Learning Through Writing

Panel members: Steve Graham, Arizona State University; Sharlene Kiuhara, University of Utah; Meade Mackay, University of Utah; Stephen Ciullo, Texas State University; Alyson Collins, Texas State University; Young-Suk Kim, University of California; Yaacov Petscher, Florida State University; Alissa Wolters, University of California; Janet Mercado, University of California; Angelique Aitken, University of Nebraska-Lincoln

Panel Overview: Writing is an essential tool for success in society today. Writing is used to support students' learning in school (Bangert-Drowns et al., 1991). It has become a critical tool in both blue and white collar jobs (National Commission on Writing, 2004, 2005). It is frequently used to communicate with friend and others, especially in terms of social media (Graham, 2006). This session focuses on how writing facilitates school learning and how writing is facilitated. The session includes four presentations. The first involves a meta-analysis examining the effects of writing on learning in science, social studies, and mathematics, providing evidence on the effectiveness writing-to-learn with typically developing students and those at-risk. The other three presentations concentrate on factors that facilitate writing performance or development. This includes a study assessing how writing is taught to students with disabilities. Another study examining the relationship between higher-order language skills and writing in Spanish and English for emergent bilingual students. The final presentation studies the impact of choice on students' writing performance.

Accordingly, the first presentation involves a meta-analysis examining if engaging students in writing activities when learning science, social studies, and mathematics facilitated their content-area learning (k = 56 experiments). We reviewed true- or quasi-experiments (with pretests), written in English, and conducted with students in grades 1 to 12 in classes where the writing-to-learn activity was part of instruction. Writing about content statistically enhanced learning (ES = 0.30) and was equally effective at improving students' learning in science, social studies, and mathematics across elementary, middle, and high school students; and with at-risk and non-at-risk learners. Writing-to-learn activities that involved analysis and interpretation, prompted metacognition, and involved informational writing, argumentation, or journal writing statistically improved learning. However, writing activities that focused on recording information (e.g., notetaking) or creating graphical representation of content (e.g., graphic organizers) did not statistically enhance learning.

The second presentation examines the instructional practices of special and general educators who teach writing to students with disabilities (SWDs) in Grade 4 (N = 85). All teachers were each observed four times to examine how writing was taught to these students. The study used an adapted version of the Writing Instruction Observation Protocol to document use of effective practices (e.g., feedback, modeling). Inter observer agreement for 25% of observations exceeded 80%. Participating teachers also completed a survey about their preparation, beliefs, and knowledge of writing instruction for SWDs. Results indicated (a) statistically significant differences between special and general educators for observed items (e.g., student groupings), (b) some items were observed in fewer than 50% of lessons (e.g., modeling), and (c) many participants reported receiving minimal training for teaching writing to students with disabilities. Implications for future research will be discussed.

The third presentation examines (a) cross-language relations (English and Spanish) and their dimensionality of high order cognitive skills such as perspective taking, inference, and monitoring, and (b) their relations to written composition in English and Spanish, using data from Spanish-speaking dual language learners. A total of 318 children in Grades 1 to 3 were assessed on perspective taking (theory of mind), inference, comprehension monitoring, and written composition in English and Spanish. The higher order cognitive skills were all reliable in English and Spanish (alphas ≥ .83). The cognitive skills were moderately related between English and Spanish (.35 ≤ rs ≤ .65). Written composition is currently being coded, which is expected to be completed by October 2019. Confirmatory factor analysis and structural equation modeling will be used to address the two research questions.

The fourth presentation examines the effects of choice in writing through an explanatory sequence mixed methods design. It is commonly assumed that choice has a positive effect on writing, but results have been mixed with most studies finding that choice has no effect. The study involved 242 college students, and applied a randomized control design to assign students to either a choice or no choice condition. Twenty of the participants were interviewed to determine their perceptions of the impact of choice and no choice. The effects of choice were tested as part of a real writing assignment that was included in nine classrooms around authentic activities. Results from hierarchical linear modeling (HLM) analyses found choice had a statistically significant negative effect on holistic writing quality, number of words written, and intrinsic writing motivation. Findings from the semi-structured interviews provided context for understanding the unexpected quantitative results, as those interviewed identified multiple reasons for why no choice had a positive impact.

Discussion Questions:

How and why do writing to learn facilitate learning?

What factors influence writing instruction for students' with disabilities?

What factors facilitate writing performance and development?

References: Bangert-Drowns, R., Hurley, M., & Wilkinson, B. (2004). The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis. Review of Educational Research, 74, 29-58. Graham, S. (2006). Writing. In P. Alexander & P. Winne (Eds.), Handbook of Educational Psychology (pp. 457-478). Mahwah, NJ: Erlbaum. National Commission on Writing in America's Schools and Colleges. (2005). Writing: A powerful message from state government. New York, NY: College Board. National Commission on Writing. (2004). Writing: A ticket to work or a ticket out: A survey of business leaders. Retrieved from www.collegeboard.com

Panel organizer: Karen R. Harris, Arizona State University karen.r.harris@asu.edu

Complexity Science and Methodology in Special Education: Investigating "Wicked Problems" and "Messes"

Panel members: Jonathan C. Hilpert, Learning Analytics, Department of Curriculum, Foundations, & Reading, University of Nevada Las Vegas; Gwen C. Marchand, Director, UNLV Center for Research, Evaluation, & Assessment, University of Nevada, Las Vegas; Joanna K. Garner, Executive Director, The Center for Educational Partnerships, Old Dominion University

Panel Overview: A new approach gaining ground in educational research is acutely important to special education: complexity science. This approach, being used worldwide in the physical and social sciences, has produced new understandings and effective interventions in response to complex challenges such as health and poverty. Complex problems are characterized by non-linearity and are similar to the concept of a 'wicked problem,' as they lack a well-defined structure and straightforward solutions (Harris, 2018). Also referred to as "messes," complex problems are characterized by multiple integrated and difficult to separate dimensions. For example, teacher and student learning and behavior are seen as emergent phenomenon in complex neural, cognitive, situative, social, and cultural systems. Complexity science methodology for addressing teaching, learning, schooling, and policy are discussed by leading educational researchers addressing critical, complex problems in education. Karen Harris will provide an overview of complexity science and applications in special education, and moderate vigorous audience discussion of how these innovative approaches in theory, design, and statistical analyses can and should be applied to the education of students at risk or with disabilities.

Complex systems approaches to research have emerged as a viable way to examine some of the most challenging educational phenomena (Jacobsen, et al., 2019; Hilpert & Marchand, 2018; Koopmans & Stamlovasis, 2016). Jonathan Hilpert has secured NSF funding for complex systems research in learning and has published methodological and research papers using these methods. In this presentation, he will discuss use of research methods and analytic techniques that are outside of the dominant linear paradigm in education research (Kaplan, et al., 2016). He will focus on how these methods involve gathering or simulating intensive data, or data closely spaced in time, which can be used to describe micro genetic changes in target phenomena. Analytic techniques are broken into two categories: equation based techniques and agent based modeling techniques. Agent based modeling techniques allow researchers to simulate the interactions of simplified systems to make inferences about outcome pathways; equation based techniques allow researchers to analyze observations using nonlinear formulas to model emergent mechanisms. Equation based techniques such as social network analysis, dynamic modeling, and time series analysis are quickly becoming promising tools for understanding complex educational systems. Examples and applications will be discussed.

Gwen Marchand has published articles in the Journal of Experimental Education and Educational Psychologist on complexity science and methodology (Hilpert & Marchand, 2018; Marchand & Hilpert, 2018, in press). She has conducted research on classroom engagement, CBM, and reading performance. She recently received NSF funding for research on science teachers' professional learning and student motivation. In this presentation, she will discuss her complex systems research on student mobility, an emergent phenomena shaped by many competing family, school, and community factors and influences that may be in flux during transition periods (Welsh, 2017). To identify and understand factors

influencing student outcomes, the system in which a student is embedded must be defined and multi-level interactions explicitly considered. A nonstructural reactive system of mobility in which children are embedded is characterized by district, school, teacher/classroom, family, and student factors that impact student continuity and adjustment during school transitions and subsequent student outcomes. She will share examples of using social network analytic procedures to characterize patterns of mobility. Analyses represent students and schools as two-mode or bipartite networks (Yang, Keller & Zhang, 2017) and patterns across the schools and student samples are examined.

Joanna Garner is Executive Director of The Center for Educational Partnerships. She focuses on the role of complex dynamic systems in education in identity development, students' learning, and leadership. Complex systems approaches have enabled researches to align theory and method in investigating concepts such as hierarchy, irreducibility, interdependency, emergence, and sensitivity to initial conditions (Harris, 2018; Hilpert & Marchand, 2018). She will argue that this approach is also essential to the work of scholars who design, implement, and evaluate partnership-based initiatives. She will present an instrumental case study (Mills, Durepos & Wiebe, 2010) of a three-year collaborative project between educational researchers, a school district, and the state department of education. She will share one qualitative and one quantitative analytical application of complex systems approaches for individual (serial interviews) and collective (multi-dimensional scaling analysis) units-of-analysis. For the constructs of professional identity development (Garner & Kaplan, 2018) and teachers' beliefs about reform-oriented science instruction (Garner, 2014), she will show how each system displayed elements that were dynamic, interdependent, nonlinear, and contextually sensitive. She will discuss a framework for developing complex systems informed theoretical perspectives and will use researcher-practitioner-policymaker efforts as a worked example.

Discussion Questions: 40 minutes will be dedicated to audience participation and discussion. Facilitating questions include:

1. At the beginning of the session, participants will be asked to be thinking of 1-2 complex problems in their specialty areas as each speaker presents. At the end of the session, we will begin by asking: "Please share one complex problem that matters to you, and some questions or ideas you have on how complexity methods and analyses could be applied." We will ask the audience and the panel to join in further discussing this complex problem and potential research.

2. What concerns do you have about complexity science methods and analyses in special education research?

3. What challenges do you see in conducting complexity science research in special education in schools, communities, policy environments, and so on?

References: Garner, J.K. (2014). The Application of Complex Dynamic Systems Principles to Teacher Professional Development. Presented at the Annual Meeting of the American Psychological Association, Washington DC. Garner, J.K. & Kaplan, A. (2018). A complex dynamic systems perspective on teacher learning and identity formation: A case study. Teachers and Teaching: Theory and Practice 25 7-33. https://doi.org/10.1080/13540602.2018.1533811 Harris, K. (2018). Educational psychology: A future retrospective. Journal of Educational Psychology 110 (2), 163-173. Hilpert, J.C. & Marchand, G. (2018). Complex systems research in educational psychology: Aligning theory and method. Educational Psychologist 53 (3), 185-202. Jacobson, M. J., Levin, J. A., & Kapur, M. (2019). Education as a complex system: Conceptual and methodological implications. Educational Researcher, 48(2), 112-119. Koopmans, M., & Stamovlasis, D. (2016). Complex dynamic systems in education. Switzerland: Springer International Publishing. Marchand, G.C. & Hilpert, J.C. (Guest editors) (in press). Complex Dynamic Systems

Approaches to Educational Research, The Journal of Experimental Education. Marchand, G. C., & Hilpert, J. C. (2018). Design Considerations for Education Scholars Interested in Complex Systems Research. Complicity: An International Journal of Complexity and Education, 15(1), 31-44. Mills, A.J., Durepos, G. & Wiebe, E. (2010). Instrumental case study. Encyclopedia of Case Study Research https://dx.doi.org/10.4135/9781412957397.n175 Welsh, R. O. (2017). School hopscotch: A comprehensive review of K-12 student mobility in the United States. Review of Educational Research, 87(3), 475-511. Yang, S., Keller, F. B., & Zheng, L. (2017). Social network analysis: Methods and examples. SAGE Publications.

Panel organizer and co-discussant: Lindy Crawford, Texas Christian University lindy.crawford@tcu.edu

Co-organizer and co-discussant: Keith Smolkowski, Oregon Research Institute keiths@ori.org

Improving Implementation Fidelity Across Researchers, Teachers, Administrators, and Families

Panel members: Erin A. Chaparro, University of Oregon; Jacqueline Huscroft-D'Angelo, University of Nebraska-Lincoln; Evelyn Johnson, Boise State University; Kathleen Strickland-Cohen, University of Oregon

Panel Overview: The theme of this presentation is implementation fidelity. Discussants will provide an introduction connecting the theme of implementation fidelity across presenters. The first presentation shares data from the development of reliable research protocols across three separate studies (academic; after-care; and families). In the second presentation, researchers briefly describe the development and validation of a teacher-observation protocol (in a research setting), and the steps involved in validating its use in applied settings by school professionals. Following presentations on researcher-validation and teacher- validation of various protocols, the third presenter shares results of statewide-validation efforts as related to a professional development model for addressing academic as well as behavioral outcomes. And, in the fourth presentation, researchers will address the sine quo non aspect of fidelity work, that being stakeholders' perception of the acceptability and the usability of researcher-developed protocols or observation systems (focusing specifically on a tool for improving family-teacher partnerships). Time will be allotted for discussion.

Study One: Supporting Implementation Fidelity Through Development of Reliable Protocols: It is important to develop training protocols and measures for implementation fidelity and reliability among research team members prior to and during intervention delivery. We will discuss the process for developing training protocols and assessments from three projects. First, we will share findings from an intervention study evaluating mathematical reasoning. During training, agreement among research interventionists (N=4) ranged from κ = .54 to κ = .94 and during field delivery average interventionist integrity scores were 94.5% for the control group and 100% for the treatment group. Second, will be an efficacy study of On the Way Home, an educational aftercare program for adolescents departing residential care, Research staff (N=6) were trained (10 hours) to evaluate implementation fidelity of intervention delivery for 196 families via video coding obtaining >90% accuracy. Last, we will share training processes, reliability, and implementation findings among research staff (N=15) from an efficacy study on Parent Connectors.

Study Two: Implementing Evidence-based Instruction with Fidelity: Teacher Observations RESET is a set of observation protocols aligned with evidence-based instructional practices (EBPs) for students with disabilities. As such, RESET offers one approach to facilitate the implementation of EBPs. Our research indicates that the RESET protocols are psychometrically sound and provide consistent measures of special education teachers' instruction. The RESET system however, cannot yet be considered valid when used in schools by school-based personnel. We present our current work with 11 Idaho districts to train 23 coaches to use the RESET protocols with 75 teachers who are part of a statewide project to close the reading achievement gap between fourth grade special and general education students. Through a series of descriptive and quantitative analyses, we seek to understand the differences in how

RESET is implemented in practice that may impact its validity and to determine effective ways to support implementation fidelity when moving from research to practice.

Study Three: Implementing Statewide Academic and Behavioral Initiatives with Fidelity Oregon's brand of multi-tiered systems, called Effective Behavioral and Instructional Support Systems (EBISS) was implemented in 25 school districts through a statewide professional development initiative. The EBISS initiative integrated School-wide Positive Behavior Intervention and Supports (SWPBIS) and the School-wide Reading Model (SWRM) to improve schools for all students and to reduce the number of students at-risk for learning difficulties. Proximal outcomes included gains in the number of building level SWPBIS and SWRM implementation teams and the activities of those teams. Distal outcomes included gains in oral reading fluency (ORF) in first and third grades and decreases in the percentage of students in the intensive category for reading in second and fourth grades. These findings suggest that to optimize improvements in teacher and student outcomes, a rigorous system of professional development and coaching appears necessary. Barriers and facilitators to implementation of statewide initiatives will be discussed.

Study Four: Securing Stakeholder Support: Implementation of a Family-Teacher Partnership Model Effective family-teacher partnership is critical to the successful and sustained implementation of Positive Behavior Interventions and Support (PBIS) in schools. Partnering for Positive Behavior (PPB) is a model designed to help general education teachers and families work together to meet the behavioral needs of students through a shared decision-making process. PPB was piloted in two elementary schools with four parent-teacher dyads (n = 8). After providing a 90-minute training on PPB implementation for teachers, researchers video recorded parent-teacher meetings led by participating teachers. Video data and completed PPB meeting forms were then scored by two independent observers to assess the extent to which teachers were able to implement each component of the PPB teacher-parent meeting strategy with fidelity. Follow-up interviews were also conducted with teachers and family members to assess the perceived acceptability, feasibility, and effectiveness of the PPB model. Results show that teachers were able to successfully implement the PPB meeting strategy with a high degree of fidelity.

Discussion Questions:

1. How might data on social validity, feasibility, implementation fidelity, and the potential for contextual adaptations be collected and used during the intervention development process to help problem solve and improve implementation fidelity and adherence by typical stakeholders?

2. An observation system is a collection of elements that together produce scores representing individual teachers' instructional quality (Hill et al., 2012). These elements include: -The observation protocol, and how the items reflect key constructs of instructional practice that are considered essential for positive student outcomes; -The rating process, and more specifically, who serves as a rater, their relationship to the teacher being served, as well as the rater training and certification process; -The sampling design, including the number and length of observations to be collected per teacher, the number of raters per observation, the time of year the observation is conducted, and the specific students and content areas to be observed (Bell et al., 2015; Hill et al., 2012; Liu et al., 2019). How do these factors vary across research and practice contexts, what is the impact on validity, and what do we do to mitigate their effect on their intended purpose? Can we reliably and validly collect the same or similar information through self-report questionnaires, interviews, or other mechanisms? Might other data collection methods provide different but meaningful information about social validity, feasibility,

implementation fidelity, and potential contextual adaptations? How would those measures inform future program adaptation and research during the efficacy, effectiveness, and scale-up phases?

References: Chaparro, E. A., Smolkowski, K., & Ryan Jackson, K. (2019). Scaling Up and Integrating Effective Behavioral and Instructional Support Systems (EBISS): A Study of One State's Professional Development Efforts. Learning Disability Quarterly, 1-14. https://doi.org/10.1177/0731948719851752 Chaparro, E. A., Smolkowski, K., Baker, S. K., Hanson, N., & Ryan Jackson, K. (2012). A model for system-wide collaboration to support integrated social behavior and literacy evidence-based practices. Psychology in the Schools, 49, 465-483. Crawford, L., Freeman, B., Huscroft-D'Angelo, J., Quebec Fuentes, S., & Higgins, K. N. (2019). Implementation fidelity and the design of a fractions intervention. Learning Disability Quarterly. Online first at https://journals.sagepub.com/toc/ldqa/0/0 Duppong Hurley, K. (2013). Parent Connectors: An Efficacy Study of Peer-Support for Parents of Middle-School Youth with Emotional Disorders. U.S. Department of Education Institute for Educational Sciences. (#R324A130180). Johnson, E. S., Crawford, A. R., Moylan, L. A., & Zheng, Y. (2018) Using Evidence-Centered Design to Create a Special Educator Observation System. Educational Measurement: Issues and Practice, 37(2), 35-44. Kyzar, K., & Strickland-Cohen, M.K. (2017). Family-professional partnerships within SWPBIS: Preliminary examination of the Partnering for Positive Behavior (PPB) intervention. Inclusion, 5, 248-262. Liu, S., Bell, C. A., Jones, N. D., & McCaffrey, D. F. (2019). Classroom observation systems in context: A case for the validation of observation systems. Educational Assessment, Evaluation and Accountability, 31(1), 61-95. doi:10.1007/s11092-018-09291-3 Smolkowski, K., Crawford, L., & Seeley, J. (in-press). Introduction to implementation science for research on learning disabilities. Learning Disability Quarterly. Strickland-Cohen, M. K., & Kyzar, K. B. (2019). Events that help and hinder family-teacher communication within SWPBIS: A qualitative analysis. Journal of Positive Behavior Interventions, 21, 148-158. Trout, A. L., & Epstein, M. E. (2012). On the Way Home: Promoting Transition Outcomes in Youth with EBD or LD an Efficacy and Replication Study. U.S. Department of Education Institute for Educational Sciences. CFDA #R324A120260

Panel organizer: Jessica R. Toste, The University of Texas at Austin jrtoste@austin.utexas.edu

Targeting Motivational Processes to Intensify Reading Interventions: Promise and Pitfalls

Panel members: Stephanie Al Otaiba, Southern Methodist University; Lisa Didion, University of Iowa; Jessica R. Toste, The University of Texas at Austin

Panel Overview: Providing intensive, individualized intervention has been shown to positively improve academic outcomes for students with or at-risk for reading disability (RD). Research-based instruction is intensified by adjusting duration, grouping, and/or the frequency of the intervention by knowledgeable implementers. However, instruction in reading skills alone is often insufficient to improve students' achievement outcomes. The success of intensive intervention may depend on the attention given to students' social-emotional needs (Fuchs, Fuchs, & Malone, 2017). For students who experience repeated failure in reading, their motivation to read often decreases (Aunola et al., 2002). In this panel, we posit that targeting motivation processes and skills within the context of academic intervention may facilitate students' reading performance. The three presentations in the panel address this issue and operationalize motivation within various theory-driven constructs.

First, Toste will begin by presenting an overview of the theoretical and empirical support for this work. She presents findings from a meta-analysis of the relations between motivation and reading (k = 132). Analysis of longitudinal studies suggests that there are robust bidirectional relations between these constructs; however, earlier reading was identified as a stronger predictor of later motivation than motivation was of reading (Toste et al., 2019). That is to say, students initially lose motivation because of repeated failure to master reading skills, but lack of motivation leads to students being less inclined to engage in reading practice overtime. Next, she proposes a theory of change and rationale for targeting motivational processes in students with persistent reading challenges-to guide thinking and discussion as we present findings from intervention research.

Al Otaiba and colleagues explore the potential for targeting growth mindset for struggling readers. Despite growing interest in the study of growth mindset, relatively less work has established whether students' mindset plays a role in reading development, or in their responsiveness to intervention. Only a few studies have examined reading growth on standardized measures. This presentation describes findings from two samples of students. The first sample involves a study examining whether a general factor of growth mindset vs a specific factor of Reading Growth Mindset (RGM) predicted word reading and end of year comprehension scores after controlling for initial skills in fall. The second sample involves an ongoing study examining the role of mindset intervention in a reading intervention. Data analysis is ongoing for the second sample, but findings from the first study indicate that global and RGM were related, but that RGM strongly predicted word reading and reading comprehension. Both global RGM uniquely predicted reading comprehension after controlling for basic word reading skills. Global mindset was found to be more strongly associated with reading comprehension when students had weaker reading comprehension skills; RGM was more strongly associated with reading comprehension for students with stronger reading comprehension skills. Findings suggest the potential importance of assessing general and reading-specific mindset linked to reading.

Next, Didion and colleagues describe the development and efficacy of the Data Mountain program. This program targets self-determined learning (i.e., self-monitoring, goal setting, positive attributions) and

aims to improve students' oral reading fluency (ORF) performance through facilitating these skills. This presentation will highlight findings from a randomized controlled trial with 83 students with and at-risk for RD in 2nd through 5th grades. Students were assigned to one of three conditions: Data Mountain delivered in small groups (DM-G), Data Mountain delivered individually (DM-I), or a reading practice only comparison condition. The two treatment conditions allowed for comparison of program dosage; in a small group, students have additional opportunity for relatedness and peer modeling. HLM was used to examine differences in ORF growth across conditions; and the moderating effects of grade-level, pretest performance, and English learner (EL) status were tested. Results indicate that treatment students, combined DM-G and DM-I, read an average of 31 more words correct per minute than comparison students at the end of the study (p < 0.01). Additionally, DM students' growth rate was twice that of comparison students (p < 0.001). There were no statistically significant differences between delivery formats or moderating effects.

Finally, Toste will conclude with a brief review of evidence from two RCTs investigating the efficacy of a word reading intervention for upper elementary struggling readers, with and without an integrated motivational beliefs training component. Linking these findings to the intervention studies presented by Al Otaiba and Didion, we aim to generate discussion around the opportunities and challenges in developing interventions that target motivational processes and skills. Because motivation does not necessarily promote reading achievement absent existing reading skills, we suggest that integration of direct skills instruction and motivation interventions might produce synergistic effects and optimize gains in reading performance over time. We posit questions related to the conceptualization and operationalization of motivation, validity and stability in construct measurement, and considerations for intervention development.

Discussion Questions:

1. How does the conceptual muddiness of motivation research impact our ability to target specific processes and skills? How do we tease apart potential jingle and jangle fallacies in this work?

2. Theoretical and empirical evidence suggests that motivation is facilitated by self-reinforcing experiences triggered by initial success or failure. As such, should we expect to see change on students' self-report measures of motivation pre-post intervention?

3. What are the research mechanisms through which we can more fully explore the possibility of targeting motivation processes as a method to intensify and individualize intervention?

Panel organizer: Hope K. Gerde, Michigan State University hgerde@msu.edu

Innovations in Writing Assessments across Features of Writing, Ages, and Ability Status

Panel members: Hope Gerde, Michigan State University; Gary A. Troia, Michigan State University; Michael Hebert, University of Nebraska-Lincoln ; Adrea Truckenmiller, Michigan State University; Matthew C. Zajic, University of Virginia

(Additional contributors: Gary Bingham, Georgia State University; Julie Brehmer, Michigan State University; Kaitlin Glause, Michigan State University; Heather Reichmuth, Michigan State University; Frank Lawrence, Michigan State University; Mackenzie Savaiano, University of Nebraska-Lincoln; Marc Goodrich, University of Nebraska-Lincoln; Eunsoo Cho, Michigan State University; Emily Solari, University of Virginia; Nancy McIntyre, University of North Carolina at Chapel Hill; Lindsay Lerro, University of California at Davis)

Panel Overview: Assessment is essential at all levels of writing instruction, particularly for children at-risk or with disabilities. The disparate ways in which writing is measured across ages, ability, and contexts limits our understanding of development and opportunities for intervention. Presentations and discussions examine ways in which elicitation materials, growth trajectories, scoring, and contexts matter for writing assessment from preschool to middle school.

Examination of an Innovative Composing Task for Young Children: Validity and Growth: An innovative story writing measure was examined for evidence of convergent validity and child growth. Participating preschoolers (N=332; M age=50.2 mos, SD=5.49, 52% girls), were individually assessed in September and May. Children wrote their name, five CVC-words, and a story scaffolded with illustrations. Composing was coded for relevance to theme, language structure, verbal/text match. Evidence of construct validity was found through the significant difference between transcription for story and name writing (t=2.452, p=.012), but non-significant difference between story and word writing (t=.132, p=.13). Growth-curve analyses indicated increases in children's transcription (=2.71, p=.012), decreases in verbal composing (=-6.56, p=.001), and increase in verbal-text match reflecting skill specific trajectories.

Direct and Indirect Effects of Literacy Skills and Writing Fluency on Writing Performance across Three Genres: This study examined the contributions to written expression by graphomotor fluency in transcription and text generation. Students from grades 4 (n=60) and 5 (n=115) participated. Regression path analysis identified that text generation (word reading, spelling, and written vocabulary standard scores) transmit their effect to writing quality through typing fluency and text length, while controlling for handwriting fluency, across narrative, opinion, and informative papers. Written text generation consistently has both a significant direct and a serial indirect (through typing fluency and then text length) effect on writing quality, regardless of genre, while controlling for handwriting fluency; antecedent variables explain between 45% and 61% of variance in writing quality. Handwriting fluency has a consistent indirect effect on writing quality via typing fluency and text length.

Sentence Combining Instruction for Students with Visual Impairments: There is extensive heterogeneity in the writing of students with Visual Impairments (VI) (Erin & Wright, 2011; Kreuzer, 2007). Students with VI sometimes require different writing tools or adaptations to writing materials, interventions, and assessments. Sentence combining is an effective instructional approach (Graham &

Perin, 2007) that may have relevance for students with VI. A single-case, adapted alternating treatments design (AATD) was used to compare the effectiveness of presenting sentence combining activities orally or in braille, for two students with VI in grades 3 and 7. Results illustrated differences in the amount of time needed, and the number of writing errors between conditions. Implications for writing assessment are discussed, including how these results can inform the presentation of writing assessment items for students with a variety of disabilities.

Writing Accuracy and Complexity in a Levels-of-Language-Framework in Middle School: The paradox in the development of writing is that accuracy and fluency may decrease as complexity increases because more skilled writers begin to generate complex ideas, choose sophisticated words and adhere to text structure, genre, and conciseness (McCutchen, 2006). Examining students in grades 6-8 (n=289), we empirically evaluated this paradox with variables in a levels-of-language context. At the word level, a significant quadratic relationship exists between spelling errors and descriptive vocabulary. At the sentence level, a significant quadratic relationship exists between sentence-level errors and diversity of sentence types. At the discourse level, a significant quadratic relationship exists between total number of words written and semantic similarity across the students' essays. Accuracy and complexity are differentially related for students with lower and higher rates of transcription fluency with the informational genre.

Lower- and Higher-order Narrative Writing Skills in Children with ASD: This study examined item-level performance on a narrative writing assessment in sixty-two children with ASD (M=12.52,SD=2.10) and FIQs above 73 (M=98.08,SD=15.00). The Test of Written Language, 4th Edition assessed spontaneous narrative writing and was scored for Contextual Conventions (CC; orthographic, grammatic; n=21) and Story Composition (SC; writing quality; n=11). Age and FIQ but not ASD symptom severity showed weak-to-moderate associations with raw CC and SC scores. For CC, partial correlations (FIQ and age) found non-significant-to-weak item-level to overall performance associations for seven items (paragraphs, comma, question mark, fragmentary sentences, run-together-sentences, noun-verb disagreements, and a/an usage) and strong associations for number of correctly spelled words (7+ letters). For SC, partial correlations showed non-significant-to-weak to overall performance associations for story vocabulary and strong associations for five items (sequence, plot, pace, style, and engagement). Results offer insights into lower- and higher-order writing features associated with overall task performance.

Discussion Questions:

1. How do elicitation materials and scoring approaches influence our capacity to accurately assess? 2. Which features of writing could be assessed across a wide range of ages from preschool to

middle school? 3. What considerations need to be made for assessing writing across ages to account for the non-

linear nature of different features at varying times? 4. What features do you think need to be considered when creating writing assessments for

children with disabilities? How do the assessments presented here do this? 5. How might the measures presented here inform classroom practices/instruction in an

individualized way?

References: Erin, J. N., & Wright, T. S. (2011). Learning to Write in Braille: An Analysis of Writing Samples from Participants in the Alphabetic Braille and Contracted (ABC) Braille Study. Journal of Visual Impairment & Blindness, 105, 389-401. Kreuzer, D. T. (2007). An

analysis of writing practices in 4th and 5th grade students with visual impairments. Unpublished Dissertation. Graham, S. G. & Perin, D. (2007). What we know, what we want to know: Teaching adolescents to write. Scientific Studies of Reading, 11, 313-335. McCutchen, D. (2006). From Novice to Expert: Implications of Language Skills and Writing Relevant Knowledge for Memory during the Development of Writing Skill. Journal of Writing Research, 3(1), 51-68.

Panel organizer: Amber B. Ray, University of Hawaii at Manoa amberray@hawaii.edu

Secondary Interventions for the Three Rs

Panel members: Amber B. Ray, University of Hawaii at Manoa; Kaitlin Bundock, Utah State University; Marcia A. Barnes, Vanderbilt University

Panel Overview: Interventions in the core academic areas for secondary students with disabilities are imperative to helping students succeed. This panel will present secondary intervention studies in reading, writing, and mathematics. Implications for instruction and research at the secondary level will be discussed. We will present intervention research conducted with secondary students in the core academic areas of reading, writing, and mathematics.

Reading Intervention. Inference-making has large direct and indirect effects on reading comprehension at the secondary school level (Ahmed et al., 2016). Although few inference-making intervention studies have been conducted, particularly for older students, findings suggest these interventions improve inferential and literal comprehension in struggling readers (Elleman, 2017; Hall, 2016). Because literacy attainments in middle school are highly predictive of post secondary success (ACT, 2014), middle school may be a late developmental window within which to intervene for struggling adolescent readers. The goal of this pilot study was to test a novel inference-making intervention (Connect-IT) for struggling middle school readers.

Connect-IT consists of 26 lessons in text-connecting and knowledge-based inference-making, and incorporates extensive, interleaved practice. The same content and feedback were designed for computer delivery and small group tutoring. 105 sixth to eighth grade students were randomly assigned to Connect-IT-Computer, Connect-IT-Tutor, and Business as Usual.

Connect-IT-Tutor had significantly larger and moderate-sized effects on the Test of Inferential Reading Comprehension (Clemens et al., 2019) compared to Connect-IT-Computer or BaU. Effects on WIAT-III Reading Comprehension were moderated by site, word reading efficiency, and gender. Findings are discussed in terms of: 1) whether inference-making interventions hold promise for improving reading comprehension in struggling adolescent readers; and 2) what can be learned from comparing different intervention delivery systems and potential moderators of intervention effects.

Writing Intervention. Writing is a skill that presents a challenge for many students. While college admission decisions are based on many components, the college entrance exams, including the written assessments, are an important part of the process. Self-regulated strategy development (SRSD) instruction and strategies are powerful tools teachers can use to prepare their students to make meaningful gains on the college entrance writing exam (Ray & Graham, in press; Ray, Graham, & Liu, 2019). This study addressed two main research questions: Does SRSD instruction for the ACT writing assessment enhance the quality of students' advanced plans, overall ACT writing scores, number of argumentative essay elements, and number of transition words? Are the effects maintained over time?

A randomized control trial was conducted with 20 high school students, 9th through 12th grade, with high-incidence disabilities and struggling writers. Treatment students received ACT writing instruction which included learning an argumentative writing strategy, HIT SONGS3, and self-regulation strategies. Strategies were taught using SRSD instructional model which includes six instructional stages. Control

students received ACT math preparation. Fidelity was assessed using an instructor checklist and observation of all lessons. Treatment fidelity was 100%. All essays were scored independently by the first author and a trained rater. After completing SRSD instruction for the ACT writing exam all students made improvements: Planning: Effect Size (ES) 5.54, ACT Writing Score: ES 4.86, Argumentative Elements: ES 4.20, Number of Transition Words: ES 1.78, Genre Knowledge: ES 1.66, Self-Efficacy for Writing: ES 2.18, and Generalization: ES 1.81. Effects of instruction on the ACT writing assessment were maintained over time with maintenance scores statistically higher than pretest scores.

Math intervention. Rate of change, the concept of a unit per unit change in quantities (Bezuidenhout, 1998), is a challenging concept found in high-school mathematics classes as well as in everyday skills such as reading maps and calculating interest. Research indicates that an integrated sequence of concrete, representational, and abstract depictions of mathematics concepts (CRA-I) may effectively improve secondary students’ ability to solve higher-level algebra problems (Strickland & Maccini, 2012), and explicit instructional strategies involving problem-solving heuristics and student verbalizations can help facilitate students’ conceptual understanding of mathematics (Gersten et al., 2009; Hughes, Witzel, Riccomini, Fries, & Kanyongo, 2014; Hwang & Riccomini, 2016). Combining CRA-I and explicit instructional strategies may help improve students’ conceptual understanding and ability to express this understanding through mathematical writing.

The purpose of this study was to examine whether there was a functional relation between an explicit instructional strategy within a CRA-I framework and high-school students’ with disabilities proficiency in solving rate of change problems, as measured by rate of change assessment scores. We used a single-subject multiple-probe design across participants to evaluate the implementation of an intervention consisting of CRA-I and a problem-solving heuristic, the POD check. The intervention consisted of seven core lessons and two review lessons, implemented by a researcher in a one-on-one setting. Participants included three ninth-grade students with disabilities (two with Learning Disabilities and one with Autism) who scored below state proficiency levels in math. Results indicated that all students improved their mathematics scores (combined Tau-U effect size = 0.77, p < 0.001) and maintained improvements 1-7 weeks post-intervention. Specific patterns of students’ results will be discussed, as will implications for research and practice.

Discussion Questions:

1. The presented studies all occurred within secondary settings. What are the unique challenges of conducting intervention research within secondary settings with students with disabilities?

2. How might both researchers and practitioners overcome these challenges to ensure that effective interventions are implemented consistently?

3. The presented studies all involved interventions in core subject areas ( reading, writing, and mathematics). Given the difficulties commonly faced by secondary students with disabilities in other content areas (science, history, etc.), how can we expand research to determine effective interventions and supports in these other content areas?

4. How does this research inform future research and collaboration?future research and collaboration?

References:

ACT. (2014). The Condition of College and Career Readiness 2014. Iowa City, IA: ACT, Inc. http://www.act.org/research/policymakers/cccr14/pdf/CCCR14-NationalReadinessRpt.pdf

Ahmed, Y., Francis, D. J., York, M., Fletcher, J. M., Barnes, M., & Kulesz, P. (2016). Validation of the direct and inferential mediation (DIME) model of reading comprehension in grades 7 through 12. Contemporary Educational Psychology, 44, 68-82.

Bezuienhout, J. (1998). First-year university students’ understanding of rate of change. International Journal of Math Education Science and Technology, 29(3), 389-399. doi.org/10.1080/0020739980290309

Bundock, K., Hawken, L.S., Kiuhara, S.A., O’Keeffe, B., O’Neill, R., & Cummings M. (2019). Teaching rate of change and problem solving to high school students with high incidence disabilities at Tier 3. (Accepted pending minor revisions 7/5/19).

Clemens, N.H., & Barnes, M.A. (2018). Test of inferential reading comprehension. The University of Texas at Austin: Author.

Elleman, A. M. (2017). Examining the impact of inference instruction on the literal and inferential comprehension of skilled and less skilled readers: A meta-analytic review. Journal of Educational Psychology, 109(6), 761-781.

Gersten, R. Chard, D.J., Jayanthi, M., Baker, S. K., Morphy, P., & Flojo, J. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research, 79(3), 1202-1242. doi.org/10.3102/0034654309334431

Hall, C. S. (2016). Inference instruction for struggling readers: A synthesis of intervention research. Educational Psychology Review, 28(1), 1-22.

Hughes, E. M., Witzel, B. S., Riccomini, R. J., Fries, K. M., & Kanyongo, G. Y. (2014). A meta-analysis of algebra interventions for learnings with disabilities and struggling learners. The Journal of the International Association of Special Education, 15(1), 36-47.

Hwang, J., & Riccomini, P. J. (2016). Enhancing mathematical problem solving for secondary students with or at risk of learning disabilities: A literature review. Learning Disabilities Research & Practice, 31(3), 169-181. doi.org/10.1111/ldrp.12105

Ray, A. B. & Graham, S. (in preparation). A College Entrance Essay Exam Intervention for Students with Disabilities and Struggling Writers: A Randomized Control Trial.

Ray, A. B., Graham, S., & Liu, X. (2019). Effects of college entrance essay exam instruction for high school students with disabilities or at-risk for writing difficulties. Reading and Writing: An Interdisciplinary Journal, 32(6), 1507 - 1529. doi: 10.1007/s11145-018-9900-3

Strickland, T. K., & Maccini, P. (2012). The effects of the concrete-representational-abstract integration strategy on the ability of students with learning disabilities to multiply linear expressions within area problems. Remedial and Special Education, 34(3), 142-153. doi.org/10.1177/0741932512441712

Panel organizer: Mike Coyne, University of Connecticut mike.coyne@uconn.edu

Discussant: Russell Gersten, Instructional Research Group

IES MTSS Research Network: Integrating Academic & Behavior Practices Across Tiers

Panel members: Nathan Clemens, University of Texas, Austin; Allison Gandhi, American Institutes for Research; Kathleen Lane, University of Kansas; Amy Sussman, IES National Center for Special Education Research

Panel Overview: IES MTSS Research Network: Integrating Academic & Behavior Practices Across Tiers Abstract: This panel includes researchers from the four research teams that are part of the new IES MTSS Research Network. Teams will share the goals and initial findings of their programs of research that target interventions and systems focused on reading, mathematics, and behavior supports across Tiers 1, 2, and 3.

Introduction: Amy Sussman, IES NCSER Project Officer, will introduce the MTSS Research Network. IES Research Networks Focused on Critical Problems of Policy and Practice in Special Education focus resources and attention on high-priority issues in special education to create both a structure and process for researchers who are working on these issues to share ideas, build new knowledge, and strengthen their research and dissemination capacity. The MTSS Research Network, the first funded by NCSER, includes four Research Teams and a Network Lead examining MTSS at the elementary school level, integrating both academic and behavioral supports and addressing the needs of children with or at risk for disabilities.

Presenter 1: Nathan Clemens, University of Texas at Austin research team, will discuss a program of research evaluating the effects of integrating strategies focused on academic engagement within an intervention intensification process for elementary students with academic difficulties. The goal is to identify and evaluate strategies that support students' academic engagement, a self-regulatory behavior that is directly tied to learning, that can be integrated within reading and mathemathics interventions in efficient and cohesive ways. The research design includes a series of sequential, multiple assignment, randomized trials (SMART designs) and initial findings will be discussed.

Presenter 2: Kathleen Lane, University of Kansas research team, will discuss a program of research, Enhancing Ci3T: Building Professional Capacity for High Fidelity Implementation to Support Students' Educational Outcomes. The Ci3T model was designed to provide a systems approach for schools to address students' academic, behavioral, and social needs in one coordinated model. Specifically, the Ci3T model blends the principles of response to intervention, positive behavioral interventions and supports , along with a validated social skills program. Ci3T offers a comprehensive, integrated, data-driven prevention model with structures for monitoring system- and student-level data to determine effectiveness in meeting system/school goals and to inform instruction for students. The research design includes a randomized control trial and subsequent descriptive analyses.

Presenter 3: Allison Gandhi, AIR and University of Missouri research team, will discuss the development and validation of an Integrated academic and behavior MTSS fidelity rubric that's goal is to reliably and validly measure implementation of integrated MTSS and that can be used by schools and districts to guide implementation and by researchers to advance our understanding of the impacts of MTSS on

student outcomes. The research team will develop the fidelity rubric through an iterative process that builds off existing rubrics, draws input from a broad range of experts in the field and integrates feedback from a beta pilot test. The team will then administer and refine the fidelity tool three times over the course of the study.

Presenter 4: Michael Coyne, University of Connecticut research team, will discuss a program of research that will support schools implement both behavior and reading practices in grades K - 2 by strategically and integrating instruction and intervention practices in reading and behavior supports, rather than implementing separate, parallel practices. The research design includes a randomized control trial evaluating integrated reading and behavior practices in Tier 1 classroom instruction, a regression discontinuity study evaluating integrated supplemental interventions in Tier 2, and a single case study evaluating integrated individual interventions in Tier 3.

Discussant: Russell Gersten, Instructional Research Group, will lead a discussion that will include the following questions.

Discussion Questions:

What are important broader research questions that the Research Network can address across teams, and analytic approaches for answering those questions?

How can the MTSS Research Network support researchers and practitioners interested in MTSS?

What are the opportunities and challenges for researchers conducting studies focused on MTSS practices and systems?

Panel organizer: Jose Blackorby, CAST, Harvard Graduate School of Education jblackorby@cast.org

Co-organizer: Jennifer Yu, SRI International jennifer.yu@sri.com

Discussant: James Basham, University of Kansas

Innovative Technology Based Approaches to Improve Science and Mathematics Learning and Self-Efficacy

Panel members: Tracey Hall, CAST; Jan Bulgren, Kansas University; Kathryn Morrision, SRI; Christian Doabler, University of Texas at Austin; Ben Clarke, University of Oregon

Panel Overview: Achieving the vision of NGSS necessitates new approaches engage active science learning, especially for students with disabilities. (De Jong & Van Joolingen, 1998; Keselman, 2003). Similarly, in mathematics learning, innovative approaches are needed in the instruction of measurement and data analysis, as research suggests that many students experience early and persistent difficulties in these areas (McFarland et al., 2019). Technology based solutions, of informed by Universal Design for Learning (UDL) offers a promising approach for designing learning experiences that engage all students (Meyer, Rose, & Gordon, 2014).

In this panel presentation, we propose to introduce UDL framework, and present design and findings from 3 interventions aimed at improving outcomes in STEM learning. The first study will focus on the Science Notebook in a Universal Design for Learning Environment (SNUDLE). Administered through a tablet, SNUDLE provides students with: (a) space to collect, organize, and display observations and data; (b) space to reflect and make sense of inquiry experiences; and (c) multiple opportunities to demonstrate understanding and receive formative feedback. SNUDLE includes text-to-speech technology with real time highlighting, translation, alt text and long descriptions for images, and a multimedia glossary provided to provide just-in-time support for vocabulary use and development.

In the presentation, we will describe an IES Goal 3 RCT designed to support 4th grade general and special education students acquire and implement inquiry science skills. Specifically, the study evaluated the effects of SNUDLE on students' science content knowledge, broad science knowledge, and motivation for science. The study focused on 4th grade students (n=747) in elementary schools (n=7) in an urban school district. Blocking on schools and teachers, 4th-grade science were randomly assigned to SNUDLE or to traditional paper-based science notebooks.

The second study will focus on an i3 grant development project aimed at improving higher order thinking in secondary science and social studies. The strategy combines Content Enhancement Routines (CER) developed by the University of Kansas, with modern technologies to support both academic content learning and higher order thinking skills (Bulgren & Ellis, 2012). The resulting intervention, referred to as Co-Organize Your Learning (CORGI), includes: science social studies units enhanced with higher order thinking skills (e.g., Cause and Effect) and Google functions including collaborative authoring, commenting, and sharing. In addition, CORGI has embedded videos, models of expert examples, context specific support, text to speech, speech to text, as well as support for vocabulary and translation.

We will share the results of a 2017 pilot study, which included 2 high schools, 11 secondary content teachers, and 649 students. Teachers were randomly assigned to treatment and control conditions. Treatment teachers implemented the 8 enhanced units using CORGI and control teachers implemented them using BAU. Students completed researcher developed end of unit tests. Results suggest a significant difference favoring treatment over control groups on two of the four science unit tests with hedge's g effect sizes of 0.95 and 0.25. Based on these results, a current NSF-funded study will further extend the feature set of CORGI and explore the impact of CORGI on science learning in middle school science settings.

The third study will describe findings from a recent pilot study of Precision Mathematics, funded by the NSF. Precision Mathematics is a first-grade, Tier 2 intervention designed to promote beneficial impact on mathematics outcomes for students at risk for mathematics learning disabilities (MLD). The intervention integrates principles of explicit instruction (Gersten et al., 2009) with domain objectives for Measurement and Data. Lessons include: (a) technology-based activities that offer students opportunities for individualized instruction and practice, and (b) hands-on problem-solving activities that allow students to interact with teachers and peers around critical Measurement and data analysis concepts.

The pilot study was a randomized controlled trial, 96 first-grade students at risk for mathematics difficulties were randomly assigned within classrooms to either treatment (Precision Mathematics) or control (business-as-usual) conditions. A statistically significant positive effect was found on one of five outcome measures, with the other four showing positive but non-significant results. Results also suggested preliminary evidence of differential response based on students' number sense and early literacy risk statuses.

Discussion Questions: Following the presentations, we will engage the audience in a discussion of these interventions within the larger context of innovative approaches to STEM learning. Discussions questions include:

What are the pros and cons of technology based interventions?

What are the challenges to conducting research that can extract and disaggregate specific components?

How should these innovations be made available at greater scale?

References: Bulgren, J., & Ellis J. (2012). Argumentation and evaluation intervention in science classes: Teaching and learning with Toulmin. In Perspectives on scientific argumentation, 135-54. New York: Springer. De Jong, T., & Van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68, 179-201. doi:10.3102/00346543068002179 Gersten, R., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Star, J. R., & Witzel, B. (2009). Assisting students struggling with mathematics: Response to Interven- tion (RtI) for elementary and middle schools (NCEE 2009-4060). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies. ed.gov/ncee/wwc/publications/practiceguides/. Keselman, A. (2003). Supporting inquiry learning by promoting normative understanding of multivariable causality. Journal of Research in Science Teaching, 40, 898-921. doi:10.1002/tea.10115 Meyer, A., Rose, D.H., & Gordon, D. (2014) Universal design for learning: Theory and practice, Wakefield MA: CAST

Panel organizer: Patrick C. Kennedy, University of Oregon ppaine@uoregon.edu

Getting More from Your Literacy Screener Using DIBELS 8th Edition

Panel members: Gina Biancarosa, University of Oregon; Patrick Kennedy, University of Oregon; Beth Harn, University of Oregon

Panel Overview: The 8th Edition of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS 8; University of Oregon, 2018), is a battery of brief fluency measures validated for use in universal screening, benchmark assessment, and progress monitoring in Kindergarten - Grade 8. It measures letter naming fluency (LNF), phonemic segmentation fluency (PSF), nonsense word fluency (NWF), word reading fluency (WRF), oral reading fluency (ORF), and reading comprehension (Maze). Measures take one minute to administer, except Maze, which takes three minutes. Innovations in curriculum-based measurement design and a technology-based data collection approach have helped schools conduct their literacy screening more effectively and efficiently. The proposed panel session describes three studies conducted as part of the DIBELS program of research that give schools confidence in using DIBELS 8 to screen for risk, set ambitious but achievable growth goals for students, and make grouping and instructional decisions to maximize limited teaching time.

Screening for different purposes. DIBELS 8 subtests measure a range of discrete reading-related constructs. Although this makes it possible to screen for many isolated deficits, the cut scores for each subtest were set to optimize their accuracy in predicting overall reading risk, as measured using the Iowa Assessment (University of Iowa, 2015). The purpose of this study is to evaluate DIBELS 8 measures in first grade using multiple criterion measures to explore variations in optimal cut scores for various screening purposes. Specifically, we will establish and compare cut scores for predicting risk on a) the Iowa Assessment; b) the Test of Word Reading Efficiency 2nd Edition (TOWRE-2; Torgesen, J. K., Wagner, J., & Rashotte, 2012), a measure of word reading ability; and c) the Comprehensive Test of Phonological Processing 2nd Edition (CTOPP-2; Wagner, Torgesen, Rashotte, & Pearson, 2013), a norm-referenced test of phonological awareness, phonological memory, and rapid naming ability. The Iowa Assessment was administered to 138 first grade students in five public schools and one private school in the Pacific, East North Central, West North Central, Mountain, and South Atlantic census divisions. The TOWRE-2 and CTOPP-2 were administered to a separate sample of 188 first-grade students in four public schools in the Pacific census division. Implications for DIBELS screening use in determining overall reading achievement risk and construct-specific risk will be discussed.

Setting growth goals. Given the wide range of student skill in many classrooms, many teachers find setting and evaluating learning goals for individual students challenging. Thus, the purpose of this study was to evaluate students' performance on each of the DIBELS 8 measures at the beginning, middle, and end of year to determine normative rates of growth, relative to students' initial skill levels. Across two years (2017-18 and 2018-19), DIBELS 8 data was collected up to three times per year on a total of 8,997 students across grades K-8. For each measure, students were grouped into quartiles by initial skill, and empirical growth percentiles were computed for each quartile. Results show that students' initial skill impacts the amount of growth that can be expected at different percentiles (e.g., average, above average), and that this information can be used to help teachers set ambitious, yet realistic learning goals for students.

Making instructional decisions. Schools need measures that are not only psychometrically sound, but that can inform instructional decision making as well. DIBELS 8 Nonsense Word Fluency was developed specifically to meet this need by better accounting for the frequency with which spelling letter combinations appear (Jones & Mewhort, 2004; Norvig, 2012). Thus, the purpose of this study was to evaluate the extent to which DIBELS 8 NWF replicates the findings of Harn, Stoolmiller, and Chard (2008), who found that for DIBELS 6, students who read nonwords using Ehri's unitization approach (i.e., as a whole unit rather than sound-by-sound) were better readers on a measure of oral reading fluency (DIBELS ORF). Participants in this study were 138 first grade students from five public and one private school in the Pacific, East North Central, West North Central, Mountain, and South Atlantic census divisions. Schools represent towns, large cities, suburbs and rural areas, and the sample was 49.1% male; 40.0% Black, 7.5% Hispanic, and 42.6% White. In addition, 17.5% of students had disabilities, 67.6% were eligible for free or reduced lunch, and 3.8% were English learners. This study includes a more ethnically and geographically diverse sample than Harn et al. (2008) and will examine the relationship between unitization and initial status and growth for both ORF and the Iowa Assessment, a standardized measure of reading comprehension.

Discussion Questions:

What additional evidence would strengthen the argument for DIBELS 8 as a screener for specific literacy constructs, including those invoked in dyslexia screening legislation?

How do these conditional growth goals affect expectations for students? What are the unanticipated consequences of these decisions?

What are the instruction implications of these findings regarding unitization and later reading achievement? How might teachers use this information to adjust instruction?

References: Harn, B. A., Stoolmiller, M., & Chard, D. J. (2008). Measuring the dimensions of alphabetic principle on the reading development of first graders: The role of automaticity and unitization. Journal of Learning Disabilities, 41(2), 143-157. https://doi.org/10.1177/0022219407313585 Jones, M. N. & Mewhort, D. J. K. (2004). Case-sensitive letter and bigram frequency counts from large-scale English corpora. Behavior Research Methods, Instruments, & Computers, 36 (3), 388-396. https://doi.org/10.3758/BF03195586 Norvig, P. (2012). English letter frequency counts: Mayzner revisited. http://norvig.com/mayzner.html Torgesen, J. K., Wagner, J., & Rashotte, C. A. (2012). TOWRE-2 Test of Word Reading Efficiency-Second Edition. Austin, Texas: PRO-ED. University of Iowa. (2015). Iowa assessments. Orlando, FL: Houghton Mifflin Harcourt. University of Oregon. (2018). 8th Edition of Dynamic Indicators of Basic Early Literacy Skills (DIBELS®). Retrieved from University of Oregon website: https://dibels.uoregon.edu/docs/materials/dibels_8_admin_and_scoring_guide_2018.pdf Wagner, R. K., Torgesen, J. K., Rashotte, C. A., & Pearson, N. A. (2013). Comprehensive Test of Phonological Processing 2. Austin, Texas: PRO-ED.