Data Wise Data Presentation University of Massachusetts Boston Aimee D’Avignon, Fabian Torres-Ardila, Janna Kellinger, Mike Gilbert, Kevin Ziomek November,

Data Wise Data Presentation

University of Massachusetts BostonAimee D’Avignon, Fabian Torres-Ardila,

Janna Kellinger, Mike Gilbert, Kevin Ziomek

November, 2014

Overview of the Process

Step 1: Organize for Collaborative Work

Develop well-functioning,

collaborative teams based

on the Data Wise Norms: • Take an inquiry stance• Ground statements in

evidence• Assume positive

intentions• Stick to protocol• Start and end on time• Be here now

Step 2: Build Assessment Literacy

-Understanding what our working conception of assessment literacy is for this project-Understanding limitations and potential uses for assessment and data-Understanding affordances and constraints of presenting data in various ways-Developing a common language around assessment

Step 2: Build Assessment Literacy

Step 3: Choose a Focus Area

A Focus Area is:

-Related to instruction

-Narrows scope of the inquiry while remaining broad enough to be relevant to many/most of the staff member

Possible Focus Areas:

Step 3: Ask a priority question

Priority Questions:

A Priority Question:

-Arises from a collaborative process

-Relates to instruction

-Is actionable

-Further narrows scope of inquiry

-Is genuinely intriguing to faculty and staff

Step 4: Determine a learner-centered problem

Our students struggle with . . .

A Learner Centered Problem:

-Directly relates to the priority question

-Based on digging into multiple data sources

-Is within the school’s control

-Is a statement about student learning (not a question)

-Is specific and small

Step 5: Examine Instruction

Reframe the Learner Centered Problem as a Problem of Practice:

-Directly relates to Learner Centered Problem

-Is based on evidence from examining instruction

-Is within our control

Our faculty struggle with . . .

Step 6: Develop an Action Plan

An Action Plan:

-States specifically what teachers will do to address the problem of practice

-Action steps in this plan should be research-based, evidence -based, high-leverage, assigned to specific people, and time-bound

Step 7: Plan to Assess Progress

The Plan to Assess Progress:

-Helps us determine whether our action plans are, in fact, helping students make progress towards addressing the LCP -Should include short-term, medium-term, and long-term assessments

-Should include specific and measurable student learning

Step 8: Act and Assess

Make it happen

Evaluate success

Our Turn


Develop well-functioning,

collaborative teams based

on the Data Wise Norms: • Take an inquiry stance• Ground statements in

evidence• Assume positive

intentions• Stick to protocol• Start and end on time• Be here now

Two-pronged approach:

1) Datawise Team inquiry projects

2) Dept. wide inquiry project based on our 4 TEAC claims -Each TEAC claim group examine dept. level data to come up with a learner- centered problem

-Then each program examines program level data to examine instruction and develop and carry out an action plan


Strengths Challenges

● We have accreditation as motivation● Infrastructure is in place● We have the support of the Dean and

the Chair of C&I to work on this project

● We are launching a new Undergraduate program which affords us the opportunity to revisit our practices

● The structure of the University and College establishes and reinforces data silos. Therefore obtaining rich and illustrative points of data can be time consuming.

● scheduling around work responsibilities means that all necessary stakeholders are not able to be present at the same time

● Infrastructure (University, College and Department) could be tweaked to improve efficiency

Because we are all stretched for time, the decision was made to use Department meeting time as much as possible to do the department wide Data Wise process work.

Step 2: Build AssessmentLiteracy

One new tool that will be at our disposal is

EDWIN:

- State data that follows our students

from pre-service to in-service- Will include MTELs, PST/PPA data,

Teacher Evaluation Scores, Where our

students teach, etc.

Step 3: Data Overview: Data Inventory

Name of Data Source

Content Area

Dates of Collection

Students Assessed

Who Has Access to these

data?

How are these data

currently used?

How could these data be more effectively used?

MTEL C&I, Curric, Content

Varies, prior to Practicum

All DWT

Exit Survey All Graduation All Christine, faculty exercise at a retreat

Analyze and share with more persons; programmatic change

GPA All Program entry

Demographics All Program Entry

EDWIN All Exit and Employment Data

Aimee Brand new

Analyze and share, prepare graduates, support claims

PPA All Lisa, Christine, GPDs

TEAC, licensure

Identify areas where our students struggle.

Admissions All

Exit Survey (Administered Spring 2013 and Spring 2014)

o Administration is given twice per year (at graduation): the survey assesses students’ perspectives of the program and their preparation (feelings)

o Categories do not add up to 100%

o Some prompts are questions whereas others are statements

o The data displayed corresponds to a subset of 14 questions. This subset was classified as “Student perception of pedagogical competence” by the data team.

o N is small (Spring 2013 = 11; Spring 2014 = 28)

Step 3: Data Overview: Assess Assessments

Step 3: Data Overview: Choose a Focus Area

The focus area for our first inquiry cycle is: TEAC CLAIMS

Our rationale for choosing this focus area is:

We choose these claims because they are important to our College Mission of preparing urban teachers

Our priority question was:

How well are we preparing our graduates to meet our TEAC Claims?

Our graduates will: - demonstrate cultural competence and address social justice issues in urban and other diverse contexts. (QP 1.3, 1.4.2) - demonstrate knowledge of content and pedagogical and assessment practices that promote learning. (QP 1.1, 1.2, 1.4.3) -incorporate family and community resources into their practice. (Q.P. 1.1, 1.2, 1.4.3) -demonstrate the skills necessary to engage in professional and life long learning. (QP 1.41, 1.4.3)

Step 4: Digging into Student Data

1. Real te

aching problems

2. Using fra

meworks to plan

3. Teaching content

4. Inquiry

methods

5. Teach higher o

rder thinking

6. Student m

otivatio

n

7. Edtec

8. Diffe

rentiatio

n

9. Usefulness of te

achr ed classes

10. High-stakes te

sting

11. Teacher R

esearch

12. Using IE

Ps

13. Class-ro

om management

14. Using 504 plans

0

10

20

30

40

50

60

70

80

90

100

Exit Survey - Student Perception Pedagogical Competence Middle/High school Graduates - (Spring 2013 - Spring 2014)

2013 n=11 2014 n=28 Average 2013 Average 2014

% o

f st

uden

ts r

epor

ting

in t

he t

wo

top

cate

gori

es (

out

of 4

cat

egor

ies)

DataWise Team Inquiry Cycle 1

Step 4: Digging intoStudent Data

Student Perception of Pedagogical Competence

Question: My teacher education program prepared me to…

1. UMass-Boston teacher preparation faculty structured their course around real problems of teaching practice.

2. Use the state's curriculum frameworks and standards to plan instruction.

3. Teach content knowledge and skills.

4. Use inquiry methods to create effective learning environments

5. Teach problem solving, conceptual understanding, and other aspects of higher order thinking.

6. Motivate students to participate in academic tasks.

7. Use educational technology as a learning tool.

8. Teach students with different ability levels in the same class

9. Reflect on and improve my teaching performance

10.Teach in a high-stakes testing environment.

11. Use classroom research and inquiry strategies

12. Read and understand Individual Education Plans (IEPs) and provide appropriate accommodations for the individual students in my classroom

13. Use classroom management techniques/procedures.

14. Read and understand 504 plans and provide appropriate accommodations for individual students in my classroom.



Strongest Aspects: “faculty brought a wealth of knowledge to the classroom. The minimum amount of experience the adjunct faculty had was 20+ years, that kept my experience focused on the realities that I will experience.”“Most of the staff actually work in BPS and were great resources to have.”

Suggestions for Program:“More hands on instruction opportunities.”“Application of theory into practice”“Spend more time on practical things like classroom management.”“More emphasis on day-to-day classroom situations: behavior management, how to deliver assessments, how to interpret test results”“More attention to practical aspects of teaching; e.g., classroom management, lesson planning”“More practice less theory.”“Focus more on the practical aspects of teaching like pedagogy.”



Qualitative Data Summary: While instructors have a lot of real-world experience and knowledge of teaching in urban schools, students expressed a need for more practice with real-world situations likely to occur in urban schools. This was reflected in the quantitative data as

well. For example, students rated “structuring classes around real teaching problems” high, but some of the practical applications like using standardized assessment results, classroom management, and using IEP and 504 plans rated low.



Practical ApplicationsClasses structured around

real-world problems



In addition, another quantitative measure on our exit survey supported the aspect of our conclusion relating to instructor strength:

The UMass Boston Teacher Preparation faculty know a lot about the reality of contemporary schools.

Spring 2013-100% Spring 2014-96%



Learner-centered problem:

While our instructors provide many real life examples from their own experiences, our students struggle to apply them to their own teaching.



Problem of practice:

Our instructors impart their own real-life experiences but need to build in more ways for students to practice applying this knowledge

DataWise TeamDataWise Team Inquiry Cycle 1


Potential solutions:

Use instructors real-life experiences to build: in-class role-playing scenarios, case-based teaching, micro-teaching, problem-based learning, analysis and application of real data, simulations, etc.



-Data from future exit surveys

-Examining syllabi

-Examining student work, e.g. core assignments

-Examining PST scores

DataWise TeamDataWise Team Inquiry Cycle 1


We are exploring the use of simulations, case studies, role-playing, etc.


Our Turn: Inquiry Cycle 2

Step 3: Choose a Focus Area

A Focus Area is:

-Related to instruction

-Narrows scope of the inquiry while remaining broad enough to be relevant to many/most of the staff member

Possible Focus Area:

What challenges are our teacher candidates facing that we could better prepare them for?


Step 3: Ask a priority question

Priority Question:

How well prepared are our students to meet the expectations of new state standards and assessment?

A Priority Question:

-Arises from a collaborative process

-Relates to instruction

-Is actionable

-Further narrows scope of inquiry

-Is genuinely intriguing to faculty and staff



1. Real te

aching problems

2. Using fra

meworks to plan

3. Teaching content

4. Inquiry

methods

5. Teach higher o

rder thinking

6. Student m

otivatio

n

7. Edtec

8. Diffe

rentiatio

n

9. Usefulness of te

achr ed cl...

10. High-stakes te

sting

11. Teacher R

esearch

12. Using IE

Ps

13. Class-ro

om management

14. Using 504 plans

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Exit survey - Student perception Pedagogical Competence Graduates - (Spring 2013 and 2014 )

(2013 n=48) 2014 (n=52) Average 2013 Average 2014

% o

f st

uden

ts r

epor

ting

in t

he

"G

ood

or E

xcel

lent

" ca

tego

ries

63%

77%

High Stakes Testing



-20%

-10%

0%

10%

20%

30%

Exit survey - Student perception Pedagogical Competence Graduates - (Spring 2013 and 2014 )

2013 (n=48) 2014 (n=52)

Diff

eren

ce %

of

stud

ents

rep

ortin

g in

the

"Goo

d or

Exc

elle

nt"

cate

gori

es w

ith r

espe

ct t

o av

erag

e

High Stakes Testing



Over the two years of data, the area with the lowest overall rating was “Teach in a High Stakes Testing Environment” The two lowest rating items on the 2014 survey were “teach in a high-stakes testing environment” and “interpret and use standardized test results”


Step 4: Some Qualitative Quotes

“more emphasis on day-to-day classroom situations: behavior management, how to deliver assessments (TRC, Dibels, etc.), how to interpret test results”

“There were a few teachers (2) who were out of date on the new issues facing teachers such as Race to the Top or the new evaluation system.”



Learner-centered problem:

Our students struggle with understanding and implementing effective assessment strategies as well as interpreting and applying external assessment results within the context of PARCC.



Problem of practice:

Given that schools and teachers are increasingly held accountable for student achievement, faculty need to better prepare our students for teaching in this high-stakes testing environment.



Potential solutions:

-Infusing assessment practices across courses

-professional development around Common Core



-Data from future exit surveys

-Examining syllabi

-Examining student work, e.g. core assignments

-Examining PST scores



We have asked someone to talk about how Boston Public Schools analyzes and uses data for improvement at our last department meeting.


Your Turn

Documents

Data Wise Data Presentation University of Massachusetts Boston Aimee D’Avignon, Fabian Torres-Ardila, Janna Kellinger, Mike Gilbert, Kevin Ziomek November,