Student Learning Assessment Report October 2012 · MPH faculty revised their MPH SLAP to delete the first, second, and sixth areas of evaluation and agreed with the SLAC that these

1

Student Learning Assessment Report October 2012

Based on the

Updated Student Learning Assessment Plan Program Name: Master of Public Health

Submitted by Program: May, 2010 Accepted by SLAC: June 30, 2010

Revised: 10-12

2

Table of Contents

Page

3……………………………………………………………...Introduction

4....…………………………………………………….……..Data Collection and Analysis Process

6………………………………………………………………Measures of Student Achievement

20……………………………………………………………. Curriculum Mapping

20…………………………………………………………….. Strengths and Weaknesses in Student Learning and Recommendations to Improve

21..………………..…………………………………………..Recommendations to Improve Evaluation of Student Learning Assessment

22….…………………………………..………………………Appendices

3

Introduction Feedback provided by the Student Learning and Assessment Committee (SLAC) was reviewed and, based on this feedback, the MPH faculty revised their MPH SLAP to delete the first, second, and sixth areas of evaluation and agreed with the SLAC that these were programmatic evaluation criteria and will be addressed in program review. (See Appendix 3) Based on feedback from SLAP and evaluation focused meetings by the MPH faculty the following were instituted by the program:

• The Curriculum Committee has surveyed faculty and mapped program competencies to specific courses. • Course evaluations were revised to include student evaluation of their attainment of program competencies the course

instructor indicated were being addressed in each course • The Internship Coordinator has revised current forms to enhance data collection as identified in the MPH SLAP and CEPH

accreditation criteria. • The university-wide competencies have been mapped to the program competencies and the evaluation of MPH competencies

will also evaluate achievement of the University Competency it is mapped to. (See Appendix 1) • The SLAP report was revised to delete Goals 1, 2 and 6 from Table 3 reporting of results

4

Data Collection and Analysis Process

The following section describes the tools the program used to evaluate student learning during this academic year. . Competency Assessment (Indirect).

• Pre/post-program self assessment (student completes when beginning the MPH program, and after completion of internship) was analyzed using paired t test and affect size was reported. The data was analyzed using students that had completed both assessments. The program did not begin having students do an online entry assessment until 2009 therefore only 11 students had completed their internship portfolio

• Post internship students’ self assessment of competency achievement was analyzed using averaging to assess each competency for percentage of students who rated themselves as either exceptional or proficient. 32 students had completed the online assessment during this last academic year and were used in the analysis. Not all students provided response to all assessment questions.

Graduate Survey (Indirect). Members of the graduating class were emailed a link to the graduate survey (using Feedback Server) asking them to provide their summative views of the MPH program support, curriculum, faculty, faculty advisor, and staff. Program faculty are most interested in the graduate’s view of the overall value of their academic experience in relation to perceived competency in fulfilling the role of public health professional. Student Course Evaluations (Indirect): Aggregation of student course evaluations were analyzed by competency to ascertain what percentage evaluated themselves as exemplary or proficient in the competency mapped to a specific course. A total of 915 responses given through 22 core courses that were delivered over the course of the academic year are used to compile the assessment results. The total number of respondents to each competency question can be gleaned from Table 3 below. Alumni Survey (Indirect) This is currently being conducted for the first time in 3 years Employer Survey (Direct) This is currently being conducted for the first time in 3 years

5

In addition, The MPH program is planning to develop the following data sources: Course Evaluation Data (Direct) The faculty will map competencies to each course objective and select course evaluation tools (Scores on papers, presentations, discussion boards, quizzes, exams) to measure achievement of the competencies Eportfolio (Direct) The eportfolio will be used in the spring term following piloting by the MHA program to begin collecting online evaluation of student attainment data from the internship preceptor and coordinator Eportfolio (Indirect) The eportfolio will be used in the spring term following piloting by the MHA program to begin collecting online self- evaluation of competency attainment data from each student

6

Measures of Student Achievement The following table summarizes the assessment of the MPH program’s student learning goals

Student Measurements/Data/Assessment

Objective

Measure Tool

Direct/Indirect

Benchmark Actual Data and Date

Collected Who uses the

data? Narrative Summary

Action Planned/Taken

Students' self assessment scores on the MPH program competencies will be significantly higher than their scores on the pre-program self-assessment.

Pre program level to be compared to post program participation survey results (Indirect)

Effect Size greater than 0.40

Pre-program self-assessment scores: Average = 3.06

Post-program self-assessment scores: Average = 3.43

Change (Effect size) = 0.48

N = 11

p = 0.06

CQI/Curriculum committees MPH Faculty

Advisory committee

Goal Met

During the 2011/2012 academic year, there were only 11 students who took both pre and post assessments. The assessment for this goal is therefore based on these 11 students.

The n for this data will increase longitudinally as more students who took the pre-competency survey complete the program .

Students will pass the CPH exam.

CPH Exam (Direct)

90% of students who take the CPH will pass.

No students have been reported as taking the CPH Exam


N/A

Faculty will discuss the usefulness of this measure due to the low numbers

7

Advisory committee

of graduates who take this national competency examination

Students are learning relevant skills for public health professionals.

Graduate Survey (Indirect)

Alumni Survey (indirect)

Employer Survey (Direct)

80% of responses are favorable.

Over 8080% % of the graduates responded strongly agree or agree that the program prepared them in relevant public health professional skills with the exception of Integrate populations-based and community-based medicine in clinical practice n= 23 (79% response rate) The Alumni and Employer Surveys were sent out for the first time in 2 years 1 month ago and data is being summarized.


Advisory committee

□ Goal Not Met

The college person responsible for sending out these surveys has met with the Chair and strategies to enhance the return rate have been instituted for the current mailing as well as revising the tool to ask them about current program competencies that were not reflected in this data. That would not include the competency rated as below 80% since this competency is not appropriately

8

worded for public health with the words medicine and clinical practice

Students will demonstrate achievement of the competency: *

Use information technology to acquire data to address priority public health issues

e-portfolio competency self-evaluation forms (Indirect)

Course Evaluation Surveys (Indirect)

90% of students will attain at least 80% on selected course measurements

80% of students will assess that they learned more about course competencies on the final course evaluation.

e-portfolio evaluations:

97%

Course evaluations:

93%


Advisory committee

Goal Met

In the e-portfolio evaluation 32 students responded to the assessment questions for this competency.

The course evaluation result is based on the responses of 168 students who took various courses that incorporated this competency.

A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-

9

assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct measure

Students will demonstrate achievement of the competency:

Develop strategies for continuous quality improvement of programs including policy analysis




80% of students will assess that they learned more about course competencies on the final course

e-portfolio evaluation forms

97%

Course evaluations

84%


Advisory committee

Goal Met


The course evaluation result is based on the responses of 131

A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure

10

evaluation.

students who took various courses that incorporated this competency.

The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct measure

Students will demonstrate achievement of the

e-portfolio

90% of students will attain at least 80% on selected


CQI/Curriculum committees

Goal Met

In the e-portfolio

A Preceptor Internship Evaluation form

11

competency:

Applies communication theory and technology to promote public health outcomes

competency self-evaluation forms (Indirect)


course measurements


97%

Course evaluations

82%

MPH Faculty

Advisory committee

evaluation 31 students responded to the assessment questions for this competency.


for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2)

12

The program will analyze course average grades by program competencies mapped to that course to develop a direct measure


Considers the role of social, cultural and behavioral factors in the accessibility, acceptability, and deliver of public health services




80% of students will assess that they learned more about course competencies on the final course evaluation


97%

Course evaluations

94%


Advisory committee

Goal Met



A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected

13

and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct measure


Utilizes community based research theory and strategies to develop population programs and policies


Course Evaluation Surveys


80% of students will assess that they learned more about


97%

Course evaluations


Advisory committee

Goal Met


A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of

14

(Indirect) course competencies on the final course evaluation.

81%


deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct

15

measure


Translate scientific principles and research into public health practice






100%

Course evaluations

92%


Advisory committee

Goal Met



A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of

16

the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct measure


Applies financial management theory and skills in public health program management






88%

Course evaluations

64%


Advisory committee

□ Goal Not Met

A Preceptor Internship Evaluation form for evaluating student achievement of program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to

17

improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades by program competencies mapped to that course to develop a direct measure


Modifies organizational practices based on social,

e-portfolio competency self-evaluation forms



97%


Goal Met

In the e-portfolio evaluation 31 students responded to the

A Preceptor Internship Evaluation form for evaluating student achievement of

18

political, and economic factors and trends

(Indirect)



Course evaluations

96%

Advisory committee

assessment questions for this competency.


program competencies has been developed and we are in the process of deploying it online as a direct measure The 8 program competencies have been explicated with examples to improve student and preceptor understanding of what is expected and improve the validity and reliability of student self-assessment and preceptor evaluation of student achievement of the competencies (See appendix 2) The program will analyze course average grades

19

by program competencies mapped to that course to develop a direct measure

* See Appendix 4 for data on competencies

20

Where is your program on curriculum mapping?

• Have you mapped your curriculum? When was it last completed? We have done some initial curricular mapping 1 year ago

• At what level is your curriculum mapped: Competencies, Objectives, Assessment, other We have mapped program competencies to course learning objectives. We are now in the process of beginning to map assessment activities to the course objectives as well.

• What tool did you use? We did this manually in an Excel spreadsheet.

Strengths and Weaknesses in Student Learning and Recommendations to Improve

. The major weakness continues to be the dearth of data that we have to make decisions regarding the strengths and weaknesses of our program. The focus of next year is to put in place web based robust data collection tools and to continue to develop the expertise in student evaluation of our CQI and Curriculum Committee. The competency related to financial competency was rated lower than the benchmark. The curriculum committee continues to map course objectives and competencies to assessment activities and will use this data as well as the SLAP data to make suggestions regarding enhancing financial content and assignments into appropriate core courses The MPH Curriculum Committee will review this report and bring suggestions to address identified areas for improvement to the program faculty for discussion during its monthly reports to the Faculty Committee. A summary of this report will be posted on the DMU webpage and Pulse for program stakeholder’s once reviewed by the dean, SLAC, and DMU provost.

.

21

Recommendations to Improve Evaluation of Student Learning Assessment • Purchase and use an ePortfolio tool • Increase faculty expertise in evaluation through faculty development opportunities on campus and at national meetings • MPH program CQI committee develops an ongoing system to better collect and analyze data. • Faculty meetings have a report from CQI committee monthly related to student and program evaluation and faculty use data to

make changes in the program as needed

22

Appendix 1: MPH Competencies Mapped to University Competencies

23

MPH Competency University Competency Use information technology to acquire data to address priority public health issues.

Demonstrate an understanding of research methodology and its relationship to critical thinking


Manifest dedication to the highest standards of professionalism and exhibit a commitment to service


Display an ability to work collaboratively and communicate effectively with others


Value the human experience with compassion and a sensitivity to individual and cultural differences




Demonstrate a knowledge of the foundational sciences

Applies financial management theory and skills in public health program management.


Modifies organizational practices based on social, political, and economic factors and trends


24

Appendix 2: MPH Competencies Examples

25

Fully competent Developing skills, but not fully

competent Not competent

Uses information technology to acquire data to address priority public health issues (Analytical/Assessment Skills)

I can generate variables that measure public health conditions

I can use variables that measure public health conditions

I can neither use nor generate variables that measure public health conditions

I employ ethical principles in the collection, maintenance, use and dissemination of data and information

I adhere to ethical principles in the collection, maintenance, use and dissemination of data and information

I neither adhere to nor employ ethical principles in the collection, maintenance, use and dissemination of data and information

I can interpret quantitative and qualitative data

I can describe the public health applications of quantitative and qualitative data

I can neither describe nor interpret quantitative and qualitative data

Develops strategies for continuous quality improvement of programs including policy analysis (Policy Development/Program Planning Skills)

I can develop plans to implement policies and programs

I can participate in program planning processes

I can neither participate in the program planning process nor develop plans to implement policies and programs

I can develop policies for organizational plans, structures, and programs

I can incorporate policies and procedures into program plans and structures

I can neither incorporate policies and procedures into program plans and structures nor develop policies for organizational plans, structures, and programs

I can develop mechanisms to monitor and evaluate programs for their effectiveness and quality

I can identify mechanisms to monitor and evaluate programs for their effectiveness and quality

I can neither identify nor develop mechanisms to monitor and evaluate programs for their effectiveness and quality

Applies communication strategies and technology to promote public health outcomes (Communication Skills)

26

I have presented demographic, statistical, programmatic, and scientific information for use by professional and lay audiences

I have participated in the development of demographic, statistical, programmatic and scientific presentations

I have neither participated in the development of demographic, statistical, programmatic and scientific presentations nor have presented demographic, statistical, programmatic, and scientific information for use by professional and lay audiences

I can apply communication and group dynamics strategies in interactions with individuals and groups outside of an organization

I can apply communication and group dynamic strategies in interactions with individuals and groups within an organization

I cannot apply communication and group dynamic strategies in interactions with individuals and groups

Considers the role of social, cultural and behavioral factors in the accessibility, acceptability, and delivery of public health services (Cultural Competency Skills)

I have applied the role of cultural, social, and behavioral factors in the accessibility, availability, acceptability and delivery of public health services

I can recognize the role of cultural, social, and behavioral factors in the accessibility, availability, acceptability and delivery of public health services

I cannot recognize the role nor have I considered the role of cultural, social, and behavioral factors in the accessibility, availability, acceptability and delivery of public health services

I can explain the dynamic forces that contribute to cultural diversity in both verbal and written form

I can describe the dynamic forces that contribute to cultural diversity

I can neither describe nor explain the dynamic forces that contribute to cultural diversity

I can assesses public health programs for their cultural competence

I have participated in the assessment of the cultural competence of a public health organization

I have neither participated in an assessment of cultural competence nor can I assess programs for cultural competence

Utilizes community based research theory and strategies to develop population programs and policies (Community Dimensions of Practice Skills)

I can demonstrate the capacity to work in community-based participatory research

I understand community-based participatory research principles

I can neither demonstrate nor have I collaborated in community-based participatory

27

efforts research efforts I can facilitate collaboration and partnerships to ensure participation of key stakeholders

I have collaborated with community partners to promote the health of the population

I have neither collaborated with community partners to promote the health of the population nor can I facilitate collaboration and partnerships to ensure participation of key stakeholders

I have used community input when developing public health policies and programs

I have gathered input from the community to inform the development of public health policy and programs

I have neither gathered input from the community to inform the development of public health policy and programs nor used community input when developing public health policies and programs

Translates scientific principles and research into public health practice (Public Health Science Skills)

I have applied the basic public health sciences (including, but not limited to biostatistics, epidemiology, environmental health sciences, health services administration, and social and behavioral theories) to public health policies and programs

I can identify the basic public health sciences (including, but not limited to biostatistics, epidemiology, environmental health sciences, health services administration, and social and behavioral theories)

I have neither identified nor applied the basic public health sciences

I have conducted a comprehensive review of the scientific evidence related to a public health issue, concern, or, intervention

I can describe the scientific evidence related to a public health issue, concern, or, intervention

I can neither describe the scientific evidence related to a public health issue, concern, or, intervention nor have I conducted a comprehensive review of the scientific evidence related to a public health issue, concern, or, intervention

I can determine the limitations of research I can discuss the limitations of research I can neither discuss nor determine the

28

findings findings limitations of research findings

Applies financial management theory and skills in public health program management (Financial Planning and Management Skills)

I can interpret the organizational structures, functions, and authorities of local, state, and federal public health agencies for public health program management

I can describe the organizational structures, functions, and authorities of local, state, and federal public health agencies

I can neither describe nor interpret the organizational structures, functions, and authorities of local, state, and federal public health agencies

I can prepare a programmatic budget I have participated in developing a programmatic budget

I can neither contribute to the preparation of proposals for funding from external sources nor prepare proposals for funding from external sources

I can evaluate program performance I have participated in program evaluation activities

I can neither report nor evaluate program performance

Modifies organizational practices based on social, political, and economic factors and trends (Leadership and Systems Thinking Skills)

I can incorporate systems thinking into public health practice

I can describe how public health operates within a larger system

I can neither describe how public health operates within a larger system nor incorporate systems thinking into public health practice

I can promote individual, team and organizational learning opportunities

I can use individual, team and organizational learning opportunities for personal and professional development

I can neither use individual, team and organizational learning opportunities for personal and professional development nor promote individual, team and organizational learning opportunities

29

I can contribute to the measuring, reporting and continuous improvement of organizational performance

I can participate in the measuring, reporting and continuous improvement of organizational performance

I can neither participate in nor contribute to the measuring, reporting and continuous improvement of organizational performance

* DMU MPH competencies are based on The Council on Linkages Between Academia and Public Health Practice’s Core Competencies for Public Health

Professionals (adopted May 3, 2010). http://www.phf.org/resourcestools/Pages/Core_Public_Health_Competencies.aspx

30

Appendix 3: Revised MPH SLAP (Changes highlighted in Yellow)

31

Student Learning Assessment Plan Program Name: Master of Public Health Submitted by Program: May 31, 2010

Accepted by SLAC: June 6, 2010 Revised 10-12

Table of Contents

32…………………………………………………………...……………………..Program Context 38……………………….Program Structure for Evaluating Student Learning Assessment Process 43……………………………………....Goals, Measures, Benchmark and Planned Data Collection 48…………………………………………………………………………... MPH Program Terms 49………………………………. Des Moines University Common Glossary of Assessment Terms 51……………………………………………… Appendix 4: Competency Self Assessment Scores

Reference: Walvoord, B.E. (2004) Assessment Clear and Simple. Jossey-Bass: San Francisco

32

Program Context Table 1: Program Profile

Number of faculty 5 Full-time faculty

6 Part-time, 7 Guest Lecturers

Number of students 262 active students Credit hours 45 credits (39 core + 6 electives) Curriculum structure Fall, Spring and

Summer 12 week terms

All core courses are taught twice a year.

MPH Core Curriculum and Course Sequences for Students Admitted 2010 and after (sequencing effective Fall 2011)

• Students may take tier 1 and 2 concurrently and tier 2 and 3 concurrently, but must be towards the end of a tier to take concurrent classes in next tier

• Student must have taken all of tier 1 and 2 and the Community Research Methods course in order to start the Capstone Course Credit Hours

Tier 1

MPH 620 Introduction to Graduate School (take within 6 credits) 1

MPH 621 Overview of the U.S. Health Care System 3

MPH 650 Basic Statistics & Research 3

MPH 657 Survey of Human Health and Disease** 3

Tier 2

MPH 645 Health Services Program Evaluation 3

33

MPH 652 Public Health Law, Ethics and Policy 3

MPH 654 Social & Behavioral Sciences 3

MPH 655 Epidemiology 3

Tier 3

MPH 625 Health Care Financial Management I 3

MPH 651 Environmental and Occupational Health 3

MPH 653 Public Health Administration & Management 3

MPH 671 Community Research Methods 3

Tier 4

MPH 658 Public Health Internship 3

MPH 660 Public Health Capstone 2

**Required for students who do not have clinical experience. If students have clinical experience, they are encouraged to speak with their academic advisor regarding a replacement course.

Total Semester Hours of Core 39 Total Semester Hours of Electives 6 Total Semester Hours for MPH 45

The DMU Master of Public Health (MPH) Program was launched in 1999 with its first graduates completing their degrees in 2000. This program achieved accreditation through the Council on Education for Public Health (CEPH) in 2002. Director and faculty turnover from 2002 - 2004 resulted in CEPH placing the program on probationary status. The program regained full accreditation status in 2008 and will have its next CEPH self-study and site visit in 2015.

34

The program's target student population is the working professional who desires to improve skills and knowledge for improved work performance, increased public health understanding, and career advancement. The program initially offered courses during the weekends and evenings and now offers them online as well. Courses used to be available via the ICN statewide network. Increasingly, courses were offered using the internet through DMU's learning management system. The program could be completed wholly online by the end of 2006. Core courses are offered at least once online and at least once in the classroom annually. The DMU MPH Program's mission, vision, and values are reviewed and revised annually. The MPH Advisory Committee provides input regarding applicability of the mission for practicing public health professionals (our graduates). Faculty conduct their review during the MPH Student Handbook annual revision process. The mission, vision and values for the MPH Program have stayed essentially the same since 2005, with minor changes in wording annually. Likewise, the goals and objectives have remained the same since 2005. It is believed that the mission, values, and goals accurately depict the DMU MPH program. In addition, the stability of this focus has allowed the program to focus on systems improvement. Mission The DMU-MPH program serves humanity by advancing and disseminating core public health knowledge through teaching, research, and practice in an active partnership with our students and the public health community. Vision The DMU-MPH program will build a community of educated, ethical, and inquisitive professionals who are highly capable and driven to optimize public health practice. MPH Values

1. Education: We value a high quality educational experience where practiced professionals share their experiences, knowledge, and process of development, guiding students through curriculum and public health understanding.

2. Students: We value the diverse personal and professional life experiences of each student; their willingness and desire to engage in the program with fellow students and peer professionals; and how their involvement shapes the delivery of the curriculum for a unique and evolving classroom experience.

3. Evidence-based practice: We value enhancing the curriculum through an active partnership between students, faculty, and the community in identifying authentic public health issues where students are involved in assessing and solving problems for the promotion of public health practice.

4. Community: We value a connectedness with communities to promote student learning and professional growth.

35

MPH Educational Goal To deliver core public health competencies through a stimulating educational experience using practiced professionals and faculty to meet the needs of a highly motivated student body where feedback is sought and incorporated in a continuous evaluation of the program. MPH Educational Objectives

1. DMU-MPH will offer a core curriculum that includes epidemiology, biostatistics, public health administration and management, social & behavioral science, financial understanding, occupational and environment health, program evaluation, ethics, and a culminating capstone experience.

2. DMU-MPH will use technology to enhance teaching strategies. 3. DMU-MPH will provide opportunities for public health research. 4. DMU-MPH will provide opportunities for professional experiences. 5. DMU-MPH will utilize community professionals in the delivery and governance of the curriculum.

MPH Service Goal/Objectives To provide leadership to the public health communities through active service on boards and committees, provide public health content expertise to the larger community of health care, and serve as mentors and examples to public health students in community service. MPH Service Objectives MPH Program students, faculty, and practitioner scholars will provide service and leadership through:

1. Membership on committees and boards; 2. Consulting activities relating to promotion of public health; 3. Public presentations and seminars for the benefit of students, alumni, faculty, healthcare executives and the local community;

and 4. Student internships and capstone projects.

MPH Research Goal To advance public health knowledge from an evidence-based perspective, translate evidence-based knowledge into the public health curriculum, and facilitate the transfer of new knowledge into public health practice. MPH Research Objectives MPH Program faculty and practitioner scholars will promote the discovery of knowledge in public health through:

36

1. Developing and maintaining a public health research agenda; 2. Collaborative relationships with other academic, public health, and community partners to further the advancement of public

health and related research; 3. Promotion of evidence-based research opportunities for faculty and student involvement; 4. Partnerships with community stakeholders to address current community needs. 5. Incorporation of research and evidence throughout the curriculum to support public health practice.

MPH Administrative Goal To administer, operate and sustain a CEPH-accredited MPH program through:

1. Competent and collaborative leadership with advocacy and representation of the program to the administration of the College and the University;

2. Strategic planning processes that are aligned with the College and University vision, mission, and values; 3. Operational practices that support faculty, student, and program administration; 4. Resource allocation and budgeting processes that support the delivery of a high-quality academic program; 5. Inclusion of faculty, student, and public health community perspectives into the program mission, vision, values, goals, and

objectives.

37

Program Curriculum Approval Process

Program forwards the approved syllabus to Registrar

Syllabus submitted to MPH Curriculum Committee and then to CHS Curriculum Committee

Course syllabus developed or revised in collaboration between course instructor, field experts, & director

Program determines need for a new course or for substantial revision of an existing

38

Program Structure for Evaluating Student Learning Assessment Process The MPH program has a new Continuous Quality Improvement Committee that is charged with coordinating the SLAP and CEPH self-study data collection and analysis. They will bring suggestions for program improvement based on evaluation data and a scan of current PH practice, research and education literature to faculty. The following is a depiction of our process and tools for evaluation Course-level

• Introduction to Graduate School and the MPH Program • Overview of US Health Care System • Basic Statistics & research • Survey of Human Health and Disease • Community Health Program Planning & Evaluation • Public Health Law, Ethics & Policy • Social & Behavioral Sciences • Health Care Financial Management I • Environmental & Occupational Health • Public Health Administration & Management • Community Research Methods

Assignments for courses include papers, presentations, discussions boards, quizzes and exams (D) Course pass rates (D) Student Course evaluation ratings of attainment of course competencies (I)

39

Internship-level Comprehensive portfolio submission (D) Self-assessment on public health program competencies (I) Evaluation by preceptor (D) Internship coordinator evaluation (D) Final grade (D)

Capstone-level Project Paper (D) Project Presentation (D) Evaluation by preceptor (D) Final grade (D)

40

Program-level Increase in scores from pre-program assessment using public health competencies to post-program assessment on public health competencies (D) Pass rates for CPH Exam (D) Student surveys (I) Alumni surveys (I) Employer surveys (I) Preceptor surveys (I) Employment rates (I)

University-level Student learning assessment report Program Evaluation

Accreditation Annual Report Self-Study

41

Each course has course competencies. Each course competency should support at least one programmatic competency. Upon completing the prescribed courses, students should be at a certain level of competency in all of the programmatic competencies. Additionally, the assignments in the courses will be able to be mapped as evidence to the course and the program competencies Feedback on the MPH SLAP was provided by the Student Learning and Assessment Committee (SLAC). As a result of the review of this feedback, the MPH faculty revised their competencies from 73 to 8. The revised competencies are based on Tier 2 Core Competencies for Public Health Professionals (May 3, 2010) as published by the Council on Linkages between Academia and Public Health Practice (COL). Tier 2 Core Competencies apply to individuals with management and/or supervisory responsibilities. According to the Council, the competencies “reflect the characteristics that staff of public health organizations (collectively) may want to possess as they work to protect and promote health in the community. The Core Competencies are designed to serve as a starting point for academic and practice organizations to understand, assess, and meet education, training and workforce needs. Over 50% of state and local health departments and more than 90% of public health academic institutions are using the Core Competencies to identify and meet workforce development needs.” The following are the competencies written by the MPH Faculty based on the Council on Linkages (COL) competencies and as organized within the domains developed by the Council: Analytical/Assessment Skills Use information technology to acquire data to address priority public health issues. Policy Development/Program Planning Skill Develop strategies for continuous quality improvement of programs including policy analysis Communication Applies communication theory and technology to promote public health outcomes Cultural Competency Considers the role of social, cultural and behavioral factors in the accessibility, acceptability, and deliver of public health services Community Dimensions of Practice Skills Utilizes community based research theory and strategies to develop population programs and policies Public Health Science Skills Translate scientific principles and research into public health practice

42

Financial Planning and Management Skills Applies financial management theory and skills in public health program management. Leadership and System Thinking Skills Modifies organizational practices based on social, political, and economic factors and trends The faculty have also mapped MPH Competencies to DMU Competencies as follows: Table 2- MPH Competencies Mapped to University Competencies

MPH Competency University Competency Use information technology to acquire data to address priority public health issues.





Display an ability to work collaboratively and communicate effectively with others


Value the human experience with compassion and a sensitivity to individual and cultural differences




Demonstrate a knowledge of the foundational sciences

Applies financial management theory and skills in public health program management.


Modifies organizational practices based on social, political, and economic factors and trends


43

Goals, Measures, Benchmark and Planned Data Collection The MPH program will use the following evaluation tools to directly measure goal achievement: Table 3: EVALUATION GOALS AND PROCESSES

Goals Measures Benchmark Planned Data Collection Students' self assessment scores on the MPH program competencies will be significantly higher than their scores on the pre-program self-assessment.

Entry self-assessment scores Post-program self-assessment scores

A paired sample t test effect size of greater than 0.40 (Cohen’s d)

Students complete a survey assessing their pre and post program self-assessment of program competency achievement during the orientation course and when submitting the portfolio

Students will pass the CPH exam.

CPH participation numbers CPH pass rates

90% of students who take the CPH will pass.

NBPHE contacts the program director to verify participants' qualifications to take the exam. NBPHE also reports whether student passes or fails to the program director.

Students are learning relevant skills for public health professionals.

Graduate surveys Alumni surveys Employer surveys

80% of responses are favorable. Results will guide course and program changes.

44

Students will demonstrate achievement of the competency: Use information technology to acquire data to address priority public health issues

Competency self-evaluation Preceptor evaluation of competency attainment Course competencies mapped to course student assessment data

Course evaluations

95% of graduating students will rate themselves as outstanding or highly effective 100% of graduating students will be rated as outstanding or highly effective

90% of students will attain at least 80% on selected course measurements mapped to program competencies 80% of students will assess that they learned more about course competencies on the final course evaluation.

Competency self-evaluation Instruments Preceptor evaluation of student achievement of program competencies Mapped course objectives and student attainment data Students' self-assessment on course evaluation

Students will demonstrate achievement of the competency: Develop strategies for continuous quality

45

improvement of programs including policy analysis Students will demonstrate achievement of the competency: Applies communication theory and technology to promote public health outcomes

Students will demonstrate achievement of the competency: Considers the role of social, cultural and behavioral factors in the accessibility, acceptability, and deliver of public health services

46

Students will demonstrate achievement of the competency: Utilizes community based research theory and strategies to develop population programs and policies

Students will demonstrate achievement of the competency: Translate scientific principles and research into public health practice

Students will demonstrate achievement of the competency: Applies financial management theory and skills in public health program management

47

Students will demonstrate achievement of the competency: Modifies organizational practices based on social, political, and economic factors and trends

48

MPH Program Terms Capstone The final culminating project completed by the student CEPH Council on Education for Public Health, the national accreditation body

for public health programs. CHS College of Health Sciences COL Council on Linkages between Academia and Public Health Practice who

authored the competencies for the MPH program. CPH Certified in Public Health Internship A field placement experience consisting of 180 hours that results in a

student submitting a comprehensive portfolio that includes an evidence-based self-assessment on all of the MPH program competencies.

MHA Master of Health Care Administration program MPH Master of Public Health

49

Des Moines University Common Glossary of Assessment Terms

Assessment: The systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development. (Palomba & Banta, 1999). Authentic Assessment: Authentic assessment examines something a student has produced as evidence of attaining key program objectives. Authentic assessment is flexible and faculty-driven (New Mexico State University). Evaluation: The use of assessment findings (evidence/data) to judge program effectiveness; used as a basis for making decisions about program changes or improvement. (Allen, Noel, Rienzi & McMillin, 2002). Direct Measure of Learning: Measurable student behaviors, knowledge, and skills that demonstrate the intended competency has occurred. Indirect Measure of Learning: Student/faculty beliefs and satisfaction regarding what students have learned as a result of their education experience. Formative Assessment: Activities, tasks, and conversations that provide on-going feedback about a learners progress while developing competency during the course or rotation. Summative Assessment: Demonstration of cumulative learning used to verify a student's exit competence. Benchmarking: A benchmark is a point of reference against which something may be measured. Rubric: Rubrics specify criteria that clearly define for both student and teacher what a range of acceptable and unacceptable performance looks like. Criteria define descriptors of ability at each level of performance and assign values to each level. Levels referred to are proficiency

50

levels which describe a continuum from excellent to unacceptable product (System for Adult Basic Education Support). Learning Outcomes/Competencies: Statements describing intended student behaviors, knowledge, or skills acquisition based on the completion of an academic curriculum. Objectives: Statements which indicate what the student is expected to learn as a result of the educational experience. Although sometimes synonymous with outcomes, objectives are more specific. Typically lecture and lab learning outcomes are termed objectives. Curricular Mapping: A process for reviewing the operational curriculum of an academic program in any education setting. It includes organizing and recording information about the curriculum to permit a visual display of the relationships between and among curricular components (Matveev, 2008).

51

Appendix 4: Competency Self Assessment Scores

52

FIGURE 1SUMMARY OF STUDENT SELF EVALUATION ON THE EIGHT CORE COMPETENCIES ON TWO OCCASIONS - COURSE EVALUATION AND INTERNSHIP PORTFOLIO SELF EVOLUTION.

Total number of responses to course evaluation questions = 915

Total number of responses to post-internship self evaluations = 32

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Use information technology to

acquire data to address priority

public health issues

Develop strategies for continuous

quality improvement of

programs including policy

analysis

Applies communication

theory and technology to

promote public health

outcomes

Considers the role of social, cultural, and behavioral

factors in the accessibility, acceptability,

and delivery of public health

services

Utilizes community

based research theory and

strategies to develop

population programs and

policies

Translate scientific

principles and research into public health

practice

Applies financial management

theory and skills in public health

program management

Modifies organizational

practices based on social,

political, and economic

factors

92.9%

84.0% 81.7%

94.3%

81.5%

91.5%

64.1%

95.7%97.0% 97.0% 97.0% 97.0% 97.0% 100.0%

88.0%

97.0%

Post course self evaluation

Post Internship Portfolio self evaluation

Rubric for Evaluating Program’s Student Learning Assessment Reports – MPH 2012 The highlighted column indicates the average score received from members of the SLAC Committee. Comments follow on the next page.

Report Elements Excellent Acceptable Weak Student learning outcomes – faculty expectations of their graduates

Three to five outcomes, consistently stated in terms of measurable knowledge, skills, or behaviors

Three to five outcomes, most stated in terms of measurable knowledge, skills, or behaviors

Fewer than three outcomes; not stated in terms of measurable knowledge, skills or behaviors

Assessment methods –how faculty will collect evidence to determine how well students meet their expectations

Two or more appropriate measures were described and implemented for each learning outcome

At least one direct measure was described and implemented for each learning outcome

Measures were inadequate or not implemented

Criteria for success – the level of performance that meets faculty standards

Specified the desired level of achievement using indicators other than grades

Desired level of achievement was not clearly described for all outcomes

Criteria for success were not included or inappropriate (used grades)

Findings – the degree to which students met the faculty standards

Reported and analyzed findings to indicate areas where students excelled, met standards, and fell short

Provided evidence of some analysis of students’ learning beyond overall findings

Reported only overall findings or omitted the findings

Use of results – the changes made to address issues identified in the findings and the efficacy of the changes

Gave specific and logical actions taken based on the findings for each of the assessed outcomes and reported efficacy of the actions

Gave specific and logical actions taken for most of the assessed outcomes

Use of results was missing, future-oriented, or indicated that no changes were needed

This rubric has been adapted for use at Des Moines University although originally drafted by Baylor University.

Comments Appears the majority of measures are self-assessments? Program is working on direct measures for programmatic competencies; would encourage focus on1-2 significant competencies for next year (perhaps financial?). Then, use ANGEL to track measures throughout the program for selected competencies rather than tackling full complement. I am not sure some of the survey measures really measure what they want to. “Why do you want them to learn more about course competencies?” The national exam may not be necessary. The first few measures seem program performance directed vs. student learning. Take a careful look at the value of the CPH exam if no one is taking it and there is no link to Graduation. The lack of direct measures is a concern.

Documents

Student Learning Assessment Report October 2012 · MPH faculty revised their MPH SLAP to delete the first, second, and sixth areas of evaluation and agreed with the SLAC that these