How a Culture of Inquiry Can Help Catalyze Completion Efforts

www.inquiry2improvement.com

Dr. Rob JohnstoneOhio Board of Regents

Charting the Course for Student CompletionColumbus, OH

December 3rd, 2013

How a Culture of Inquiry Can Help Catalyze Completion Efforts

www.inquiry2improvement.comNational Center for Inquiry & Improvement

Acknowledgements Much of the content in this presentation in conjunction with

national projects such as Completion by Design, the Aspen Prize for CC Excellence, and Bridging Research, Information & Cultures (BRIC)

Content has also been developed by and with a host of national partners, including:

• Community College Research Center (CCRC)• Jobs for the Future• JBL Associates• Public Agenda• The Research & Planning (RP) Group• WestEd

Infographics were primarily designed by Greg Stoup

An Applied Inquiry Framework for Student Completion – April 2013

http://www.inquiry2improvement.com/


The Student Experience

An Applied Inquiry Framework for Student Completion

April 2013



5www.inquiry2improvement.com

A Culture of Inquiry & Action

A RESOURCE for INSTITUTIONAL change

April 2013

www.inquiry2improvement.comNational Center for Inquiry & Improvement 6

What is a Culture of Inquiry?

Institutional capacity for supporting open, honest and collaborative dialog focused on strengthening the institution and the outcomes of its students.



Culture of Inquiry: Features● Widespread access to user-friendly

information on student outcomes● Encouraging more people to ask a wider

collection of questions and use their evidence and conclusions to enhance decision making

● Reflective and dynamic discussions across constituency groups



Culture of Inquiry: More Features

● Continuous feedback so adjustments can be made along the way and processes can be adapted

● A sense of ownership over improving student outcomes – not blaming the student

● Using insight generated from inquiry to act at varying levels of the institution to create the conditions to improve outcomes


9www.inquiry2improvement.com

Applied Inquiry Framework for Student Completion

Developed for Completion by Design

A Resource for Institutional Change


An Applied Inquiry Framework for Student Completion

Stage 1 – Explore how to improve outcomes

Stage 2 – Gather meaningful evidence

Stage 3 – Discuss evidence broadly

Stage 4 – Use evidence to inform change

Stage 5 – Measure the impact of change



Explore how to improve student outcomes

Focus inquiry on designing approaches that improve student outcomes

STAGE 1



Key Questions

When was the last time you sat in a standing committee meeting on your campus that used evidence to explore a key student success outcome for more than 20 minutes?

What types of questions do we spend most of our organizational resources answering?



Gather meaningful evidence

Collect high-quality, meaningful evidence at the student support, classroom, program, and institutional levels

STAGE 2



When gathering evidence, make sure you are focusing on the right data…



Persistence Rate

Success Rate

1989 2008

20 year trend for California CCcourse success & persistence rates

What does that tell us about the usefulness of these metrics in setting institutional strategies?



Three Studies to Energize a Campus Conversation About Student Success

• Grades & Success in Sequenced Courses• Cohort Tracking in Developmental Education• Levels of Preparedness in GE Courses –

success of simultaneously enrolled students in Math / English courses



Study 1:Grades in Sequenced Courses

Note: Data tracked from 2006-07 to 2011-12



Grades in Sequenced Courses Backdrop• Course sequences analyzed in Mathematics,

English, Biology, Chemistry, Physics & Economics• Relationship between grade in 1st course in

sequence and success in 2nd course in sequence was examined.

• Enrollment data tracked from 2006-07 to 2011-12

• Cohort Ns ranged from 1,890 to 126,367



Segmentation of Students Taking Intermediate Algebra

19

A in Math 30810%

B in Math 30813%

C in Math 30812%

Unclear Status in

Math 308*3%

Math 310 Re-peater / Non

Success25%

Math 310 Attempt was First Math

Course29%

Taking Math 308 / 310 in

Same Semester8%

Other1%



Table for Intermediate Algebra

Status in Math 310 Count % of Total N

Success Rate in Math 310

A in Math 308 5,659 10% 81%

B in Math 308 7,726 13% 60%

C in Math 308 7,234 12% 37%

Unclear Status in Math 308* 1,675 3% 16%

Math 310 Repeater / Non Success 14,568 25% 41%Math 310 Attempt was First Math Course 16,812 29% 57%Taking Math 308 / 310 in Same Semester 4,505 8% 68%

Other 487 1% 44%

Total 58,666 100% 53%

20


A

B

C

The grade received by students in Math 306

Intermediate AlgebraPre-Algebra Introductory

Algebra

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Math 308

27%

51%

76%

Developmental Education Math Course Sequence

Math 310Math 306 Math 308

Introductory AlgebraMath 308

B

A

Transfer Level

Those placing directly into

Math 30860%

Math 1300s

A

B

C


Pre-Algebra Introductory Algebra

Intermediate Algebra

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


37%

60%

81%

Developmental Education Math Course Sequence

Math 306 Math 308 Math 310

Intermediate AlgebraMath 310

B

A

Transfer-Level

Those placing directly into

Math 310 57%

Math 1300s

A

B

C



Transfer-Level

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Math 1300s

44%

63%

81%

Intermediate Algebra – Transfer-Level Mathematics

MATH 310 MATH 1300s

Transfer-Level

MATH 1300s

B

AThose placing

directly into Math 1300s

66%

A

B

C


Calculus I Calculus II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


50%

78%

90%

Calculus Course Sequence

MATH 2413 MATH 2414

Calculus II

MATH 2414

B

A Those placing directly into Math

2414

72%

A

B

C

The grade received by students in English 306

Two Levels below

Transfer

One Level below

Transfer

Transfer Level English

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in English 307

49%

66%

77%

Developmental Education Writing Course Sequence

Those placing directly into English 307

ENGL 306 ENGL 307 ENGL 1301

One Level below

Transfer

B

A66%

A

B

C


One Level below

Transfer

One Level below

Transfer

Two Levels below

Transfer

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


46%

61%

74%

Developmental Education Writing Course Sequence

Those placing directly into English 1301

ENGL 307ENGL 306

Transfer Level English

ENGL 1301

B

A71%

A

B

C


Composition I

Composition II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


58%

77%

88%

College English Course Sequence

ENGL 1301 ENGL 1302

Composition II

ENGL 1302

B

A Those placing directly into English 1302

68%

A

B

C

The grade received by students in Biology 1406

Biology I Biology II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Biology 1407

60%

76%

90%

College Biology Course Sequence

BIO 1406 BIO 1407

Biology II

BIO 1407

B

A Those placing directly into Biology 1407

70%

A

B

C

The grade received by students in Biology 2401

Anatomy Physiology

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Biology 2402

50%

72%

88%

College Biology Course Sequence

BIO 2401 BIO 2402

Physiology

BIO 2402

B

AThose placing directly into Biology 2402

67%

A

B

C

The grade received by students in Chemistry 1411

General Chemistry I

General Chemistry II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Chemistry 1412

46%

70%

86%

College Chemistry Course Sequence

CHEM 1411 CHEM 1412

General Chemistry II

CHEM 1412

B

AThose placing directly into

Chemistry 141266%

A

B

C

The grade received by students in Physics 1401

General Physics I

General Physics II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Physics 1402

87%94%

University Physics Course Sequence

PHYS 1401 PHYS 1402

General Physics II

PHYS 1402

B

A Those placing directly into Physics 1402

78%69%

A

B

C

The grade received by students in Economics 2301

Macro Economics

Micro Economics

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Economics 2302

74%81%

89%

Economics Course Sequence

ECON 2301 ECON 2302

Micro Economics

ECON 2302

B

A Those placing directly into

Economics 230271%


A Direct Look at Meaningful Analyses

from Four-Year Schools



Simple and Effective Analyses to Inform Student Progress and Completion Efforts

•Based on analyses from: Florida State University Georgia State University San Diego State University University of North Carolina – Greensboro University of Texas University of Southern California University of Wisconsin Eau Claire Virginia Commonwealth University

34




1. Targeting at-risk populations (UNCG)2. Identifying At-Risk Students Using

Historical Data (GSU, FSU)3. Identifying At-Risk Students Using

Milestones (FSU, GSU)4. Tracking beyond Freshmen to

Sophomore Retention Rates (FSU)5. Tracking Second Year Students with

Sophomore Standing (GSU)

35




6. Identifying Correlates of Success (VCU, UWEC, SDSU)

7. Identifying Correlates of Non-Success (USC)

8. Predictive Model for Retention (VCU, GSU)

9. Impact of Course Withdrawals (VCU)10. Additional Analyses (USC, FSU, GSU,

UWEC)

36



First Year to Second Year Retention, Georgia State University

3/30/10

Fall '00 Fall '01 Fall '02 Fall '03 Fall '04 Fall '05 Fall '06 Fall '07 Fall '080%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

80% 82% 81% 83% 80% 81% 82% 83% 83%

Retention



First Year Retention & Progression Rates - Georgia State University

3/30/10

Fall '00 Fall '01 Fall '02 Fall '03 Fall '04 Fall '05 Fall '06 Fall '07 Fall '080%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

80% 82% 81% 83% 80% 81% 82% 83% 83%

22%27% 28%

33% 35%39%

49%

62%67%

Retention Retained & Sophomore



Discuss evidence broadly

Engage a variety of campus stakeholders in evidence-based discussions about improvements in practice

STAGE 3



Data do not

speak for themselves.



The vital role of exploration

● In order to make data useful, ample time and space are needed to discuss and analyze the information and connect it back to the original research question.

● Answers are not always immediately apparent, so skilled facilitation may be needed to dig out the deeper meaning.

● Multiple perspectives and types of information are often needed to make sense of individual data points.

!



Study 3:•English & Math Preparedness &

Success in GE Courses –


Not in an English course

Engl 826

Engl 836

Engl 100

Engl 110

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

31%

48%

68%

75%Not in an English course

64%

67% 33%

Success Rate of those same students in Psychology 101

English course taking profile for students in Psychology 101

Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade

Success in Psychology 101 for students simultaneously enrolled in an English course

English 826(Two Below)

English 836(One Below)

English 100(Transfer A)

English 110(Transfer B)

Not in a Math course

Funda-mentals

Beginning Algebra


Transfer Level

0%10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

43%

51%

63%

76%Not in an Math course

64%

63% 37%

Success Rate of those same students in Psychology 101

Math course taking profile for students in Psychology 101


Success in Psychology 101 for students simultaneously enrolled in an Math course

Fundamentals Beginning Algebra


Transfer Level Math

0%10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

31%

48%

68%

75%


Success in five highly enrolled GE courses by English enrollment level

Psychology 101 Speech 101 Economics 101 History 101 HSCI 101

54% 57%

73%

82%

27%

56%

73%79%

48%43%

51%

64%60%

74%

82%88%





0%10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

48%

58%

68%

76%


Success in five highly enrolled GE courses by English enrollment level

Fitness 334 Accounting 101 Music 202 Biology 250 Sociology 101

37%

50%

66%

73%

31%

64%

74%80%

40%

60%65%

50%

59%

74%





40%

N/A


Use evidence to inform change

Implement changes in practice and policy based on analyses and discussion of college evidence

STAGE 4



Use evidence to guide innovation

● In this context, research and applied inquiry are fundamentally interventionist in nature.

● We are not seeking absolute truths; rather we are looking for patterns of evidence that inform action-oriented decisions.

● Failure can be seen as an opportunity for learning, especially when outcomes are shared and used to inform further improvements in practice.


The process of inquiry is not a search for an absolute truth

Domain of possible solutions

We answer the questions that eliminate dead end solutions

And what do we do when the evidence is ambiguous?

Gregory M Stoup, Cañada College

What to do when you reach the limits of your research and yet still face multiple choices?

Trust your expertise & choose !


Measure the impact of change

Evaluate the impact of practice changes on student performance

STAGE 5



Exploring the Preventing Loss, Creating Momentum Framework


Loss & Momentum Framework

CONNECTION

Interest to Application

ENTRYEnrollment to Completion of

Gatekeeper Courses

PROGRESSEntry into Course of Study to 75% Requirements

Completed

COMPLETIONComplete Course

of Study to Credential with Labor Market

Value

POLICIES

PRACTICES

PROGRAMS

PROCESSES



PROGRESSENTRY COMPLETIONCONNECTION

Students never apply to college

Students delay entry into college

College counseling patterns that lead to:- under enrollment- little program-specific guidance- missed financial aid opportunities

Unstructured programs / too many choices

Extended onramps delay entry to programs of study

Students fail to enroll/pass Gatekeeper courses

Poor work-school balance

Part-time enrollment forcing long completion times

Progress not monitored / feedback givenLife events / “Stop out or drop out”

Transfer without credentialStudents accumulate credits (& debt) not aligned with completion

Never complete college level math

Credential doesn’t support needed wage & aren’t stackable

Completion by Design Framing ModelSome Known Loss Points

Poor academic preparation



PROGRESSENTRY COMPLETIONCONNECTION

foster college-going norms in High School

expand awareness of college programs and requirements

dual enrollment & AP credit

Completion by Design Framing ModelMomentum Strategies

take placement test in high school

educational planning in high school

aggressive financial aid support

accelerate entry to POS

shorter, faster, cheaper course design

effective academic catch-up programs

mandatory intrusive advising focused on programs of study

programs to incentivize optimal attendance

student progress to completion monitored & feedback provided

accelerated competency-based programs

emergency aid for students

remove barriers to graduation

Learn & Earn and Career Pathway programs

incentives to transfer with credentials

mandatory intrusive advising toward certificates degrees & transfer

First Time Student

Successful Completion



Final Thoughts

An Applied Inquiry Framework for Student Completion – September 2013


Culture of Inquiry: Why All the Fuss?● Because this ongoing work is challenging but

necessary!● Work needs to marry insight & evidence● Problems are large and recurring● No silver bullet / evident answers● Multiple solutions likely needed● Progress is not linear● Requires cross constituency interaction



Next Steps• What are some “quick win” opportunities

for demonstrating the value of a culture of inquiry to practitioners and stakeholders?

• What are some of the challenges to developing or further evolving a culture of inquiry?

Completion by Design? Completion by Accident?



Find Out More

• The National Center for Inquiry & Improvement websitewww.inquiry2improvement.com

• Dr. Rob Johnstone, Founder & [email protected]

• CBD Inquiry Guides: http://www.inquiry2improvement.com/publications-resources


http://www.inquiry2improvement.com/publications-resources

http://www.inquiry2improvement.com/publications-resources

Documents

How a Culture of Inquiry Can Help Catalyze Completion Efforts