ST. PHILLIPS COLLEGE...In 2016, the National Science Foundation awarded St. Philip's College (SPC) in San Antonio, Texas, a Historically Black Colleges and Universities Undergraduate

ST. PHILLIPS COLLEGE

HBCU- UP TIP Grant: STEMed PROJECT YEAR 2 External Evaluation Report

Completed by

KAVITA MITTAPALI, PH.D. DIA ADAMS, M.A.

This material is based upon work supported by the National Science under HBCU UP (award no. 1623270). This report was completed by MN Associates, Inc. 2

MN ASSOCIATES, INC.


TABLE OF CONTENTS

LIST OF TABLES 4

LIST OF FIGURES 4

INTRODUCTION 7

OVERVIEW OF STEMED PROJECT 7

EVALUATION 9

SUMMARY OF FINDINGS 10

SUMMARY OF FINDINGS 17

APPENDIX A: STUDENT SURVEY 19

APPENDIX B: STUDENT RECRUITMENT FLYER 21

APPENDIX C: AMP AND ALL STEM STUDENT GPA PERFORMANCE BY SUBJECT 22


List of Tables Table 1: Overview of Project Activities in Year 2 9

List of Figures Figure 1: FTIC or Veteran STEMed Students by Race/Ethnicity and Gender (N=18) 10 Figure 2: Percentage of Students Who Received a Laptop (N=18) 10 Figure 3: Percent of Students Who Received a C or Higher in AMP Courses (N=18) 12 Figure 4: Pre/Post Increase in ALEKS scores (N=18) 12 Figure 5: Percent of Students Who Saved 1 Year of Degree Completion Time 13 Figure 6: Student Rating of Instructors 14


Executive Summary

In 2016, the National Science Foundation awarded St. Philip's College (SPC) in San Antonio,

Texas, a Historically Black Colleges and Universities Undergraduate Program (HBCU-UP) grant.

SPC used these funds to develop the college’s Targeted Infusion Project: Science, Technology,

Engineering, and Mathematics Enrichment by Design (STEMed) Project, currently at the end of its

second year. STEMed is focused on decreasing the time first-time college (FTIC) or veteran

students take to complete their STEM degrees and graduate. The project offers students support via

developmental math courses and mentoring. The primary goals of the grant are:

GOAL 1 – Recruit & Retain FTIC or Veteran STEM students

1. Recruit 20 FTIC or veteran STEM students from San Antonio Independent School District

(SAISD) and the local community each semester for a total of 120 students over two-year

period. SPC defines “each semester” as fall August – December, 16 weeks; spring January –

May, 16 weeks; and summer May – August, 10 weeks.

2. Award student incentive laptops with Maple software at the end of each semester based on

performance and success in the AMPP.

3. Mentor all 20 current AMP Program participants weekly and all former AMP Program

participants two times per semester (before enrollment and after midterms).

GOAL 2 – Decrease the STEM degree completion time of FTIC or Veteran STEM students

1. Implement the Accelerated Math Preparation (AMP) Program to decrease the number of

semesters needed for developmental math by one year (2 courses) by the end of the program

(16 weeks for fall and spring, 10 weeks for summer).

MNA’s Year 2 evaluation finds the STEMed Program has achieved many successes, including:

● Delivering four developmental math courses as a part of the AMP program.

● Decreasing degree completion time for 72% of AMP participants, the majority of

whom are FTIC students and some of whom are veteran students.

● Distributing compensation to students who successfully decreased their degree

completion time by 1 year, in the form of a laptop and software.


● Offering strong math instruction, as evidenced by student feedback on AMP

courses.

● Regularly providing all AMP student participants with weekly STEM mentoring, as

well as twice per semester mentoring to post-AMP students.

● Increasing the number of program participants from Year 1.

Looking ahead into Year 3, the following recommendations are provided to further enhance the

positive results STEMed is experiencing:

● Further expand efforts to enroll students in STEMed programming. The program has a goal

of reaching 120 students in three years, averaging 20 students per semester. In Year 2, 18

students were enrolled, ten fall 2017 and eight in spring 2018. Year 2 enrollment numbers

were an increase from Year 1, but still short of program targets. It will be important to ramp

up outreach efforts, to mitigate low enrollment, and stay on track toward enrollment goals.

Expanding outreach to more SAISD schools and using existing students as ambassadors for

the program could be of benefit in the future.

● Offer additional academic resources for Pre-Calculus. Pre-calculus had a Year 2 failure rate

of 39%, similar to levels in Year 1 (40%), while the other three AMP courses had very low

failure rates. Program staff should consider whether supplemental resources for Pre-Calculus

are warranted to help increase passage rates in the future. This could perhaps take the form

of outside of class tutoring or online math assistance.


Introduction

With funding awarded by the National Science Foundation’s (NSF) Historically Black

Colleges and Universities Undergraduate Program (HBCU-UP) Targeted Infusion Grant, St. Philip’s

College (SPC), a two-year Historically Black College and a Hispanic Service Institution in San

Antonio, Texas, launched the Targeted Infusion Project: Science, Technology, Engineering, and Mathematics

Enrichment by Design (STEMed) Project. The purpose of NSF’s HBCU-UP Grant is to promote and

implement educational innovations through the use of modularized, emporium model instruction,

and the revolutionary learning program using knowledge space theory: Assessment and Learning in

Knowledge Spaces (ALEKS). Beginning in 2016, SPC’s STEMed project spans a three-year period,

and is currently in Year 2 of the grant. STEMed aims to recruit, retain, and provide math

acceleration for 120 first time in college (FTIC) or veteran students as they matriculate through their

STEM degree program. The project hopes to increase the number of students graduating from SPC,

transferring to a four-year institution to continue their STEM education, and/or entering the

workforce upon completion of the project.

The STEMed Project will strengthen undergraduate STEM education at SPC, by focusing on

increasing the success rates of first time in college (FTIC) African-Americans, Veterans and other

traditionally underrepresented students including students from the Eastside Promise Zone,1 and

lowering their degree completion time by enhancing their skills in developmental math.

Overview of STEMed Project The STEMed project is housed under SPC’s Center of Excellence for Mathematics and the

Mathematics Department leadership. Implementation began during the 2016 – 2017 academic year

(Year 1) and will continue for three years until the 2018 - 2019 academic year (Year 3).

1 The Eastside Promise Zone encompasses 19 contiguous census tracts with 64,125 residents classified as predominantly minority. This area is characterized by high poverty rates, inadequate access to high quality early learning programs, struggling schools, over-age students, low graduation rates, poor health conditions, insufficient or ineffective supportive services, inadequate employment opportunities, a shortage of quality affordable housing, high crime rates, and the lowest college enrollment rates in the city.


The project will feature researched-based best practices, interventions, and strategies to

decrease semesters needed for developmental math. The project objectives are aligned with SPC’s

Strategic Plan objectives2, as well as President Barack Obama’s National Education 2020 goal of

increasing college graduates with degrees in the United States by the year 2020. The primary purpose

of the Targeted Infusion Project: STEM Enrichment By Design (STEMed) Project is to decrease the

time FTIC or veteran students take to complete their STEM degree and graduate.

A linchpin activity of STEMed is the Accelerated Math Preparation (AMP) Program, which

will be offered in each fall, spring, and summer semesters to help FTIC or veteran STEM students

advance into college level math. AMP uses the interactive, artificial intelligence platform Assessment

and Learning in Knowledge Spaces (ALEKS), in addition to modular curriculum and emporium

model instruction, to facilitate math skill development.

The project’s overarching goals and objectives that are tied to these activities are listed

below:

GOAL 1 – Recruit & Retain FTIC or veteran STEM students, with a focus on African American and other traditionally underrepresented students

Objectives

1. Recruit 20 FTIC or veteran STEM students from San Antonio Independent School District

(SAISD) and the local community each semester for a total of 120 students over a threeyear

period.

2. Award student incentive laptops with Maple software at the end of each semester based on

performance and success in the AMPP.

2 This project is directly aligned with SPC’s Strategic Plan Objective 1, Student Success: provide academic and student support and align labor market-based pathways to achieve student completion.


3. Mentor all 20 current AMP Program participants weekly and all former AMP Program

participants two times per semester (before enrollment and after midterms).3

GOAL 2 – Decrease the STEM degree completion time of FTIC or veteran STEM students

Objectives

1. Implement AMP to decrease the number of semesters needed for developmental math by

one year (2 courses) by the end of the program (16 weeks for fall and spring, 10 weeks for

summer).4

Evaluation

SPC contracted with MN Associates, Inc. (MNA), a minority, woman-owned small business

in Fairfax, VA, to serve as STEMed’s external evaluator. A mixed methods approach was adopted to

complete the evaluation. MNA engaged in periodic online correspondence with project staff in

order to receive access to meeting materials and data/records. MNA was selected to conduct a

retroactive evaluation using extant data, as the project’s original evaluator is no longer working on

the project. Data yielded will inform future decisions regarding the implementation of this

educational project in meeting its objectives and outcomes. Data are in an aggregate and no personal

identifying information is used in the report.

Program documentation consists of student and faculty files from a database set up

exclusively for the STEMed Project, with information on all program participants in year 1. The

database houses demographic information, high school designation regarding economically

disadvantaged status, and ethnic group per existing public data source of the Academic Indicator

Excellence System reports. Data on STEM course completion, productive grade rates, graduation

3 To further enhance the quality of the program, Goal 1 objective 3 was strengthened to ask students to meet with mentors weekly, as opposed to monthly as originally proposed. 4 The course sequence was changed from Math 0305 Basic Mathematics (a 4.33-hour course) and two Math 0100 Special Topics (two 1-hour courses) to four co-enrollment courses: Math 0410 Pre-Algebra (a 4.33-hour course) + Math 0320 Intermediate Algebra (a 3.66-hour course) + Math 1414 College Algebra (a 4-hour course) + Math 2412 Precalculus (a 4-hour course). This change did not affect the goals or objectives only the number of contact hours funded by the college.


and transfer rates were gathered from the Institutional Planning, Research and Effectiveness Office

(IPRE), the STEMed Project database and the National Student Clearinghouse. Student surveys

were also used to measure qualitative and quantitative outcomes. All students who completed AMP

coursework were given surveys to share feedback on their course experience (see instrument in

appendix). Lastly, AMP students completed pre and post ALEKS assessments, to gauge changes in

their math skills over time.

Summary of Findings

Year 2 project activities for STEMed are illustrated below in Table 1. Project activities and

data are discussed further below and tied to the project’s explicit goals and objectives to determine

to what extent the program is making progress.

Table 1: Overview of Project Activities in Year 2

Activity Participants Completed

Recruit FTIC and Veteran STEM students NA

Yes

Implement AMP courses 18 Yes

Mentor participants regularly 18 Yes

Award student incentive laptops 10 Yes

Administer student survey 18 Yes

GOAL 1: Recruit & Retain FTIC or veteran STEM students, with a focus on African American and other traditionally underrepresented students

Objective 1: Recruit 20 FTIC or veteran STEM students from San Antonio Independent

School District (SAISD) and the local community each semester for a total of 120 students

over the three-year period of grant.

To recruit students for the STEMed Project grant Co-PI, Maria Rodriguez, reached out to

potential participants via a variety of avenues. A flyer developed by the PI and co-PI to advertise the

program’s offerings was disseminated heavily. Information about AMP was given to SPC’s veteran’s

office, multiple SAISD schools, as well as college advisors. In addition, new students were reached

out to during the college’s New Student Orientation. STEMed also provided extensive information

on the project on their website.


Over the course of Year 2 of the grant, a total of 18 students participated in the program, ten

during the fall 2017 semester and eight during the spring 2018 semester. Sixteen students were FTIC

students and two were veteran students. Figure 1 provides a breakdown of students by

race/ethnicity and gender. Thirty-nine percent of students were female, up from 0% in Year 1, while

61% were male. The majority of students were Hispanic (83%) and the remaining students were

African-American (17%).

Figure 1: FTIC or Veteran STEMed Students by Race/Ethnicity and Gender (N=18)

Objective 2: Award students who successfully completed the AMP a laptop with Maple

software, to attract applicants and retain selected participants to optimize results.

As a reward for completion of all four AMP courses in one-semester (which decreases

degree completion time by one year), students were awarded laptops with Maple software. The

technology is meant to support the students as they matriculate through their STEM degree

program. A total of students ten out of 18 (56%) met the requirements to receive a laptop in Year 2

(see Figure 2). Two students qualified for laptops during fall 2017 and have received their machines.

Eight students have been ordered laptops for spring 2018, and they will be distributed to students as

soon as the computers are received and configured.


Figure 2: Percentage of Students Who Received a Laptop (N=18)

Objective 3: Mentor all 20 current AMP Program participants weekly and all former AMP

Program participants two times per semester (before enrollment and after midterms).

During Year 2 for both the fall 2017 and spring 2018 semesters each STEMed participant

met with PI Jessica Lopez weekly for mentoring. The meetings covered a variety of topics including

tutoring, course selection, student support services, degree planning, registration, and study skills.

Students were also offered post-AMP mentoring twice a semester as well. Appointments for post-

AMP student sessions are made in advance to encourage participation.

GOAL 2: Decrease the STEM degree completion time of FTIC or veteran STEM students

Objective 1: Implement AMP to decrease the number of semesters needed for

developmental math by one year (2 courses) by the end of the program (16 weeks for fall and

spring, 10 weeks for summer.

AMP was developed by project PI, Jessica Lopez. The program encompassed four courses:

Math 0410 Elementary Algebra, Math 0320 Intermediate Algebra, Math 1414 College Algebra (for

Pre- Calculus) and Math 2412 PreCalculus, each of which were four weeks long. Courses used the

Assessment, Learning, in Knowledge Spaces (ALEKS) system, a web-based learning tool that uses

adaptive questioning to determine a student’s content knowledge, and give direction for areas that


require more learning. The emporium model was also employed in classes, which is a computer-

based learning system designed to help students be actively engaged in content mastery. Modularized

curriculum was a third strategy used in AMP coursework. Through the modularized curriculum

model students receive instruction in small discrete modules that are nonsequential. In addition, to

the four math classes mentioned above students were also offered one informal ALEKS course that

spanned the entire 16-week semester which included material from all four classes. Students were

given course grades at the end of the spring semester. Grades were based on completion of each

courses’ respective ALEKS objectives and performance on the course’s final exam.

A total of ten students took courses as a part of AMP during fall 2017 and eight during

spring 2018, for a total of 18 students in Year 2. Figure 3 breaks down how students academically

performed in the courses taken grade wise. Students performed best in Intermediate Algebra, both

of which had a 94% pass rate. The lowest performance was in Pre-Calculus, where only 61% of

students passed, followed by College Algebra and Elementary Algebra both at 89%.

Figure 3: Percent of Students Who Received a C or Higher in AMP Courses (N=18)

Across the board, student performance increased significantly during the program, as

evidenced by student’s pre and post assessment ALEKS scores, which rose sharply. The majority of

student scores (84%) increased by at least 50 percentage points (figure 4), a higher I’m not sure if


this is the correct word to use here but I could be wrong. than the year before (72%)5. Students

increased their ALEKS scores by an average of 62 percentage points in Year 2, similar to the average

of 68 percentage points in Year 1.

Figure 4: Pre/Post Increase in ALEKS scores (N=18)

In order to achieve the goal of decreasing the need for developmental math by one year,

students had to pass Math 0410 Elementary Algebra, Math 0320 Intermediate Algebra, and Math

1414 College Algebra. In Year 2, 72% of AMP students met this goal, and were able to sufficiently

reduce their degree completion time (see Figure 5).

Figure 5: Percent of Students Who Saved 1 Year of Degree Completion Time6

5 Percentages are based on the percent of topics mastered from all four courses (Elementary algebra, Intermediate Algebra, College Algebra, and Precalculus) 6 N-Year 1=7, N-Year 2=18


Through surveys AMP students provided feedback on their course instructors. By and large

students appeared to be satisfied with the quality of AMP instruction offered. Figure 6 highlights

student feedback by course, where they were asked to provide input on the extent they agreed with

the following statements:

● The exams were fair

● The course was well organized

● I clearly understood what was expected of me in this course

● The instructor’s use of examples helped to get points across in class

Across all four courses 100% of respondents agreed or strongly agreed with the above statements

(Figure 6), demonstrating that AMP instructors are doing a solid job of teaching courses from the

perspective of students.

Figure 6: Student Rating of Instructors 7

The course was well organized The instructor's use of examples helped to get points across in class

7 N-College Algebra=3, N-Elementary Algebra=2, N-Intermediate Algebra=4, N-Pre-Calculus=4.


The exams were fair I clearly understood what was expected of me in this course


In addition to the feedback above, students also shared qualitative feedback about their

instructor’s. When asked to share information on specific things their instructor did well, fall 2017

College Algebra students had this to say8:

“He was always very willing and quick to come right over and help us one on one with anything we were struggling

with.”

“He answered questions in a way I could understand.”

“He was always there to assist me. Always seemed very joyful to help me learn something no matter how long it took

me to learn it. Instructor answers any questions I have.”

With regard to specific things they did not like about the course, they had this to say:

“We can't use notes on the final exam. I know I can't remember everything at once so I might fail.”

“You are a great professor, and are always open to any question, there is nothing I personally believe you could or

better stated should change. Thank you for being such great help through this course.”

“I have no problem with the instructor.”

Summary of Findings

At the conclusion on Year 2, the STEMed Project is continuing to show substantive

progress towards meeting its goals and objectives. In particular, the project is doing a strong job of

reducing degree completion time for enrolled AMP students. Our analysis also finds that STEMed

has been successful in offering high-quality math instruction, and that students are by-and-large

satisfied with the instruction they received. Outreach to students has improved in Year 2, as

compared to Year 1, and female students are a part of the program now in near proportionate

numbers. However, there is still room for further improvement if the program is to meet its goal of

serving 120 students in three years.

Looking ahead into Year 3, we recommend the following steps be taken to further

strengthen the program’s positive impact:

8 Qualitative feedback was not available for each class, as some students did not provide feedback on this portion of the survey.


● Further expand efforts to enroll students in STEMed programming. The program has a

goal of reaching 120 students in three years, averaging 20 students per semester. In Year

2, 18 students were enrolled, ten fall 2017 and eight in spring 2018. Year 2 enrollment

numbers were an increase from Year 1, but still short of program targets. It will be

important to ramp up outreach efforts, to mitigate low enrollment, and stay on track

toward enrollment goals. Expanding outreach to more SAISD schools and using existing

students as ambassadors for the program could be of benefit in the future.

● Offer additional academic resources for Pre-Calculus. Pre-calculus had a Year 2 failure

rate of 39%, similar to levels in Year 1 (40%), while the other three AMP courses had

very low failure rates. Program staff should consider whether supplemental resources for

Pre-Calculus are warranted to help increase passage rates in the future. This could

perhaps take the form of outside of class tutoring or online math assistance.


Appendix A: Student Survey 1. The instructor's class presentations were designed for easy note taking. 2. The course was well organized. 3. The course material appeared to be presented in logical content units. 4. There was continuity from one class to the next. 5. Course concepts were related in a systematic manner. 6. The course assignments were clearly specified 7. The objectives of the course were well explained. 8. The course objectives were clear. 9. The instructor's use of examples helped to get points across in class. 10. The instructor's use of technology, when appropriate, was effective. 11. The instructor was enthusiastic about the course material. 12. The instructor seemed to enjoy teaching. 13. The instructor's use of personal experiences helped to get points across in class. 14. The instructor puts material across in an interesting way. 15. The instructor stressed important points in lectures. 16. The instructor treated students with respect. 17. Exams were fair. 18. Course objectives were reflected in the exams. 19. The exams concentrated on important aspects of the course. 20. The exams were graded fairly. 21. The exams stressed the important points presented in class. 22. The types of test questions used in the exams were good. 23. The exams focused on issues discussed in class. 24. The exams helped me to track my progress in the course. 25. The graded assignments were returned quickly enough to benefit me.

26. The graded exams were returned promptly. 27. The syllabus for the course clearly explained the flow of the course.


28. I clearly understood what was expected of me in this course. 29. Classes were conducted in accordance with a prescribed calendar or meeting schedule 30. The instructor was available during posted office hours. 31. The instructor adequately explained the grading system. 32. The instructor appeared to have the equipment or other materials necessary to conduct each class.


Appendix B: Student Recruitment Flyer


Appendix C: AMP and STEM Student GPA Performance by Subject9

On average, AMP students performed better academically when compared to other STEM

students, as measured by GPA in Year 2. This finding is consistent across STEM subject areas, with

the exception of engineering and math, where grant students performed slightly lower than other

STEM students. Overall, grant students had a GPA slightly higher than non-grant STEM student,

2.89 compared to 3.02.

9 N-STEM students= 24, N-Grant Students=13

Documents

ST. PHILLIPS COLLEGE...In 2016, the National Science Foundation awarded St. Philip's College (SPC) in San Antonio, Texas, a Historically Black Colleges and Universities Undergraduate