Assessment Matrix Workshop

Katherine PerezJacqueline Peña

Office of Planning and Institutional EffectivenessJanuary 2009

SLO matrix SLOs and Corresponding Assessment Methods

Workshop: Revising SLOs and Assessment Methods

Reporting Results and Use of Results Workshop: Writing the Results and Use of Results

PO matrix POs and Corresponding Assessment Methods Reporting Results and Use of Results

Workshop: Revising POs and Assessment Methods + Writing Results and Use of Results

Additional individualized assistance

SACS Accreditation Key SACS Deadlines:

September 10, 2009—Report due March 8-12, 2010—Onsite Review

Key IE Deadlines: February 1, 2009—Fall 2008 SLO matrices due June 1, 2009—Spring 2009 SLO and PO matrices due

Continuous improvement Institutional Effectiveness Sound, research-based and assessment-

based decisions

Student Learning Outcome(Stated in Measurable Terms)

Assessment Method Results (Data Summary and Analysis)

Use of Results for Improving Student Learning

Link to the Mission:

Criteria: Can be observed and measured Relates to student learning towards the end of

the program Reflects an important concept

Formula: Who + Action Verb + What Students will identify and apply the key

components of a learning system. Students will identify and apply the five

key components of a learning system.

Principle 3.4.12 The institution’s use of technology enhances

student learning and is appropriate for meeting the objectives of its programs. Students have access to and training in the use of technology. (Technology use)

Required for all undergraduate programs Optional for 2007-2008 SLO matrices Required for 2008-2009 SLO matrices

General (not appropriate for SACS): Students will use information technology to gather and disseminate

information. More Specific:

Graduates of the program will demonstrate the ability to use appropriate computer technology, software, and the Internet to complete their capstone research papers.

Much more Specific (appropriate for SACS): Students will be able to effectively demonstrate information

technology skills by locating and retrieving information on economic topics and issues, published research in Economics and related fields, and/or by finding information about the generation, construction, and meaning of economic data for their final research projects.

Students will write and present a capstone project that requires the use of Word, Excel, PowerPoint, and information technology.

Action Verbs [Who + Action Verb + What]

Knowledge

Comprehension

Application

Analysis Synthesis Evaluation

List Summarize Apply Analyze Combine AssessDefine Describe Demonstra

teSeparate Integrate Rank

Describe Interpret Illustrate Explain Modify TestIdentify Predict Solve Connect Create MeasureShow Distinguish Examine Classify Design JudgeExamine Estimate Modify Arrange Compose CritiqueState Paraphrase Classify Compare Prepare Discrimina

teFind Compare Calculate Contrast Write SupportLabel Contrast Compute Infer Formulate Hypothesi

zeInfer Construct Order Construct

Implement

Note: This table reflects the original Bloom’s Taxonomy Table. The modified version places evaluation as the fifth column and synthesis (creating) as the sixth column.

1. Artifact2. Collection of the artifact

Where, when, and how Census versus sampling

3. Criteria Minimum standards on a rubric/scale or the

percentage of correct items4. Evaluation of the artifact

Faculty panel or external evaluators (reliability)

Rubrics, embedded questions, etc.

Common Direct Measures

Common Indirect Measures

•Standardized exams•Exit examinations•Portfolios•Pre-tests and post-tests•Locally developed exams•Papers•Oral presentations•Behavioral observations•Thesis/dissertation

•Surveys or questionnaires•Student perception•Alumni perception•Employer perception•Focus groups•Interviews•Student records



Doctoral students will identify, define, and apply key literary theories (including formalism, historicism, mimesis, post-colonialism, structuralism, post- structuralism, psychoanalysis, modernism, and postmodernism) to their specialization areas in the field of English literature.

After completing all their courses, English literature doctoral students will take the theory comprehensive exam, which will be divided into five key literary theories (one theory per exam question) that the students learn throughout their coursework.

All comprehensive exams will be evaluated by a faculty panel of 3 members using a department rubric.

For each of the five literary theories, students will achieve a 3 or better on a 4-point rubric:1 = poor2 = acceptable3 = good4 = excellent


Summary of Results Direct measures

Test items Performance as determined by rubrics

Indirect measures Surveys and questionnaires Interview and focus group data

Format Narrative Tables or charts

Analysis of results If pertinent, explain results in a narrative form by

interpreting results or using qualitative analysis of the data. Every student learning outcome must have at least:

One set of results One student learning improvement strategy (use of results)

All the students passed the assessment. 75% of the students met the criteria for

success. Our students passed the dissertation

defense on the first attempt. 75% of the students (n=15) achieved a 3

or better on all the rubric categories for the capstone course research paper.

Frequency of Student Results for all Four Categories of the Research Paper (n=20)

1 NOVICE

2APPRENT

ICE

3PRACTITIO

NER

4EXPERT

PERCENTAGE MEETING CRITERIA

Grammar 2 (10%) 0 (10%) 8 (40%) 8 (40%) 80% (n=16) met the criteria.

Essay Structure

4 (20%) 0 (15%) 11 (55%) 2 (10%) 65% (n=13) met the criteria.

Coherence of Argument


Application of MLA Rules


Criteria: Students will achieve a 3 or better on a 4-point rubric on all five sections of the master’s level thesis.







20 students took the theory comprehensive exam during the 2008-2009 academic year.

70% of the students achieved the minimum criterion (i.e. scoring a 3 or better on all five theory components).

20% of the students scored less than 3 on at least 1 theory component.

10% of the students (n=2) scored less than a 3 on all five theory components and had to be rescheduled to retake the test during the summer during the summer B session.


DO DON’T•DO focus on making specific improvements based on faculty consensus.

•DON’T focus on simply planning for improvements or making improvements without faculty feedback.

•DO address specific program improvements that will impact student learning.

•DON’T address assessment improvement plans such as revising the rubric.

•DO use concrete ideas. •DON’T write vague ideas.

•DO state strategies that are sustainable and feasible.

•DON’T use strategies that are impossible to complete in one year considering your resources.

•DO use strategies that can improve the curriculum and help students learn outside of courses.

•DON’T focus on only improving the curriculum.

Target met. Will continue to monitor. The faculty will meet and revise the three

introductory courses. A larger sample will be obtained. We will revise the rubric and have a calibration

session with the faculty prior to evaluating the student papers each semester.

A capstone course will be created that emphasizes research and thesis writing methods.

A student resource center will be created with computer equipment, writing tutors, and statisticians who can assist our students with course- and thesis-related writing and research.







20 students took the theory comprehensive exam during the 2008-2009 academic year.

70% of the students achieved the minimum criterion (i.e. scoring a 3 or better on all five theory components).

20% of the students scored less than 3 on at least 1 theory component.

10% of the students (n=2) scored less than a 3 on all five theory components and had to be rescheduled to retake the test during the summer during the summer B session.

Use of Results for Improving Student Learning1. The department will assist faculty in organizing student-led comprehensive exam study groups a year prior to the expected

examination deadline. 2. A theory unit will be added to each literature course so that students are constantly exposed to multiple literary theories throughout

their coursework.3. In addition, the department will create and distribute a study list that includes major topics covered on the comprehensive exams,

useful websites for tutorials and reviews, and a bibliography of texts that are critical for success with the five theoretical concepts.


Assessment Methods Results (Data Summary and Analysis)

Undergraduate English students will communicate effectively in written format.

During their final semester, in their senior seminar course (PPP 4677), students will write a 10-page research paper on a critical topic as applied to an English, Caribbean, or American novel.

20% of the students’ papers will be randomly selected and evaluated by a faculty panel of 3 members using a rubric that measures:(1) Grammar, (2) Essay Structure, (3) Coherence of Argument, (4) Application of MLA rules.

Students will achieve a 3 or better on a 4-point rubric:1 = novice2 = apprentice3 = practitioner4 = expert

100 students took the seminar in 07-08.20 papers were sampled.

100% met the minimum criterion for success (i.e. scoring a 3 or better on all four components).

Use of Results for Improving Student Learning1. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring

for at least one 60-minute session.2. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any

point during their academic program will be referred to a writing tutor at the Center for Academic Excellence.3. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles.

1 2 3 4

G 0 0 12 8

E 0 0 10 10

C 0 0 15 5

M 0 0 20 0

71% 29%

Overview and Comparisons: Give an overview or summary of all the outcomes

together Discuss trends that you have seen over the years

Explanations Provide qualitative explanations for poor results or

exceptionally high results Notes and documentation affecting results

Response rate (e.g. Only 50% of the students completed the project.)

Inter-rater reliability (e.g. 2 faculty members reviewed the artifacts and the inter-rater reliability was only 60%.)

Assessment Commentary Revise or create instrument (e.g. artifact, rubric) Modify assessment methods (e.g. data collection,

sampling, criteria, evaluation process)

Write or revise your Results and Use of Results.

Results: Clearly state the results of each PO assessment (table or

narrative). Use of Results:

Explicitly state how you will use the results to improve the program during the following assessment cycle/year.

Summary Page:1. Overview of results and use of results (looking at the big

picture)2. Trends over the last year or several years3. Comparison to national or state trends or data4. Explanations for poor or unexpected results5. Notes or documentation that could explain the results6. Assessment improvement plans

END OF THE SLO PORTION

What Is Next? 10-minute overview of Program

Outcomes PO writing assistance Continued SLO writing assistance

Program-level outcomes Focus on student success (not student learning) Formula = Who + Action Verb + What Examples:Graduates seeking employment in the field will

find such employment within 6 months of graduation.

Graduates will be satisfied with advising and mentoring services.

Faculty will publish at least one journal article or book chapter every two years.

Program Outcome(Stated in Measurable Terms)


Full-time students will graduate from the English doctoral program within 7 years of program admission.

The program administrative assistant will keep track of students’ progress in the program from date of admission, making updates for each student at the end of each academic semester.

10 students graduated from the English doctoral program during the 2008-2009 academic year.

The program assistant analyzed the completion timeframe for each individual student and discovered that all of the graduating students spent over half of their graduate time working on the dissertation (3-6 years).

Use of Results for Improving Student Learning1. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students’

progress, and the students future goals and timelines. 2. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program.

The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that the group members find helpful for successful completion of the program.

3. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program admission.

Time of Completion for the 10 Students

5 students 6 years2 students 7 years2 students 8 years1 student 9 years

Take 15 minutes to review or write at least one program outcome with corresponding assessment methods, results, and use of results.

PO Formula:Who + Action Verb + What

Assessment Methods Formula:1. State the instrument/method/tool (e.g. survey, national exam

results).2. Explain the data collection steps (e.g. when, where, how).3. State the success criteria (minimum standards for success).

Results: Clearly state the results of each PO assessment (table or narrative).

Use of Results: Explicitly state how you will use the results to improve the program

during the following assessment cycle/year.

Marta Perez Director [email protected] 305-348-2733

Maria Corrales Coordinator [email protected] 305-348-0459

Katherine Perez Coordinator [email protected] 305-348-1418

Jacqueline Peña Coordinator [email protected]. 305-348-1367

Mayelin Felipe Computer Specialist [email protected] 305-348-0115

Karla Felipe Computer Specialist [email protected] 305-348-0115

Amanda Berhaupt Graduate Assistant [email protected] 305-348-2731

Randhir Kaur Graduate Assistant [email protected] 305-348-2731

Documents

Assessment Matrix Workshop