Upload
landon-franklin
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Assurance of Learning in Business Schools: Observations, Implementation Issues & Guidance
Thomas G. CalderonChair & Professor of Accounting/
Director, CBA Quality AssessmentCollege of Business AdministrationThe University of AkronAkron, OH 44325-4802Email: [email protected] Tel: 330 972-6228
Material for this presentation is based on my experience in leading assessment projects since 1996, my various research projects, the AAA’s Teaching & Curriculum Section’s Best Practices in Accounting Program Assessment (Calderon, Green & Harkness) and Kathryn Martell & Thomas G. Calderon, Assessment of Student Learning in Business Schools: Best Practices, Each Step of the Way. Vol. 1&2. AIR/AACSB. 2005. http://aacsb.edu/resource_centers/assessment/default.asp
APLG/FSA 2006 Annual SeminarFebruary 12–14, 2006http://aaahq.org/aplg/seminars/2006/program.htm
2
About this Presentation
This presentation covers:– Assurance of Learning Fundamentals– Six Observations on Assurance of Learning– Lessons learned from each observation
The presentation offers practical guidance for implementing assurance of learning projects and identifies key issues that need attention
3
Congruency, Alignment & Change
Learning goals for each program determined by the faculty and based on the program’s mission
Curriculum alignment Evidence showing that students are achieving
program’s learning goals Use evidence to initiate change and continuous
improvement
5
Curriculum Alignment
What are the program’s learning goals? Where in the curriculum are these goals
addressed? What is taught and how are students
assessed?
Program Mission
Program Learning
Goals
Courses/Co-Curr.
6
Curriculum Alignment – From AIS CourseLearning Goal (Students Will….) Covered? Briefly describe what you do (how covered and how
assessed)
1. Demonstrate knowledge and understanding of core accounting fundamentals (financial reporting, cost and management accounting, auditing, tax, and systems).
Yes This course (i) covers the basic AIS concepts and principles, business processes, data modeling methods, systems documentation techniques, data normalization, database approaches, internal control frameworks, fraud, auditing computer based systems, and systems analysis and design issues (ii) applies various concepts and principles to the five transaction cycles (i.e., revenue, expenditure, payroll, manufacturing and financial cycles). Special emphasis is placed on COSO, SOX and CoBIT frameworks.
3. Demonstrate the ability to think creatively and apply their knowledge of accounting fundamentals in innovative ways.
Yes Students work both individually and in groups to examine internal control issues in various accounting cycles. Further, students work in a semester long project on an emerging issue in AIS. As part of the requirements, students write a detailed report and make a formal presentation before the class.
6. Demonstrate ability to research an issue, analyze qualitative and quantitative data, and integrate information from multiple sources.
Yes The semester long project requires students to research an emerging issue, analyze various issues and prepare a formal report. Students leverage Ohiolink sources, Web based sources and library to conduct research and analysis. Another assignment requires students to analyze SOX section 404 material weaknesses in selected companies’ 10K and 10Q reports accessed from the SEC’s website.
7
Evidence
Not Course Grades Direct based on actual samples of students’ work
linked to program learning objectives Indirect attitudes, opinions and assumptions about
learning linked to program learning objectives Must be “public” shared with faculty in the program Must be based on individual student performance
unless assessing teamwork May be shared with students
8
Closing the loop
Analyze and report evidence Share evidence with faculty and stakeholders Discuss evidence and implications Use evidence in decision making processes Act on evidence, propose improvements Implement proposals and continuously
improve
9
Plan Before You Leap!
Before starting, develop an assessment plan that includes:– Learning objectives– Evidence to be used– Method to collect evidence– Desired expectations– Who is primarily responsible for collecting evidence– When and how will evidence be reported, discussed and used
Follow the plan
10
Observations and Practical Lessons
Six Observations on Assurance of Learning Lessons learned from each observation Examples and implementation ideas
11
Assessment and the apparent dominance of indirect evidence
Observation 1: – Business schools and accounting programs
use multiple types of evidence to assess learning
– Indirect evidence appears to dominate
12
Assessment and the apparent dominance of indirect evidence
Type of Evidence FrequencyPercent
(N=160) Type
Student Survey 114 71 Indirect
Alumni Survey 106 66 Indirect
Required Capstone or Senior Project 92 58 Direct
Business Advisory Board input 88 55 Indirect
Employer Survey 83 52 Indirect
Syllabi Review 77 48 Indirect
ETS Major Field Test 66 41 Direct
EBI surveys 60 38 Indirect
Course embedded assessment with program standards (rubric) 59 37 Direct
Student Interviews 55 34 Indirect
Student GPAs 45 28 Indirect
Focus Groups 41 26 Indirect
13
Assessment and the apparent dominance of indirect evidence
Lesson 1: Recognize important characteristics of assessment evidence– Must be appropriate within the context of the
school’s mission– Focus is on degree program– Must relate to specific learning objectives – Place emphasis on direct evidence of student
learning– Use indirect evidence to supplement direct evidence
14
Critical pragmatism vs. scientific orthodoxy in evidence collection
Observation 2:– Measurement is a concern for everyone– There is an inherent quest to “measure” learning– Measurement comes with certain connotations
Precise Objective and Unbiased Representationally faithful Valid and reliable
– Assessment data are sometimes criticized as unreliable and unscientific, having high inter-rater variability, and difficult to develop and use
15
Critical pragmatism vs. scientific orthodoxy in evidence collection
Lesson 2: Recognize vital evidence collection and analysis issues– Root evidence collection and measurement in
common sense and use evidence that facilitate dialogue and understanding of student learning.
– Avoid fixation on traditional validity concepts; rather employ the contemporary holistic approach to validity – validity in use.
– Be aware of possible scoring variability among different raters. How much are you willing to accept?
16
Critical pragmatism vs. scientific orthodoxy in evidence collection
Lesson 2 Continued: Recognize vital evidence collection and analysis issues– Build consensus at alignment stage and in
planning for assessment (e.g., standardized rubric development)
– Have dialogue and invest time to develop a shared understanding of evidence and traits
– Employ observable behavioral checklists rather than traditional Likert scales with inherently wide variability.
17
TRAIT Good Satisfactory Not Acceptable
Recognizes own personal biases that can influence decision making outcomes.
States assumptions and identifies and clarifies personal beliefs that may affect decision outcomes
States assumptions and identifies but does not clarify personal beliefs that may affect decision outcomes
Does not state assumptions or does not identify personal beliefs that may affect decision outcomes
Learns from history by including a discussion from the past where managers faced similar ethical issues.
Demonstrates a good appreciation for prior history where managers faced similar ethical issues
Demonstrates a fair appreciation for prior history where managers faced similar ethical issues
Does not include a discussion of prior history where managers faced similar ethical issues
Limits the expression of self-interest and other outcomes of marketplace logic.
Demonstrates substantive constraint in the expression of self interest and other outcomes of market-place logic
Demonstrates some constraint in the expression of self interest and other outcomes of market-place logic
Demonstrates no constraint in the expression of self interest and other outcomes of market-place logic
Abstains from the tendency to justify the means by virtue of the end.
Clearly abstains from tendency to justify the means by virtue ends. Issues are clearly considered and decision-making is mindful of this tendency.
Shows awareness of the tendency to justify the means by virtue of the end but actions to avoid the tendency are not very deliberate
Does not abstain from the tendency to justify the means by virtue of the end
Adapted from Fogarty 2005
18
A Hypothetical Summary of Evidence Based on a Standardized Rubric
TRAIT N Good Satisfactory Not Acceptable
Recognizes own personal biases that can influence decision making outcomes.
200 30(15%)
160(80%)
10(5%)
Learns from history by including a discussion from the past where managers faced similar ethical issues.
200 22(11%)
88(44%)
90(45%)
Limits the expression of self-interest and other outcomes of marketplace logic.
200 40(20%)
100(50%)
60(30%)
Abstains from the tendency to justify the means by virtue of the end.
200 36(18%)
160(80%)
4(2%)
Average 16% 64% 21%
19
The assessor’s dilemma and course-embedded assessment
Observation 3: – Assessment is useful and necessary – Assessment requires resources, commitment, and attention
At many schools, students complete a battery of assessment center activities as part of a required course (Bommer, Rubin & Bartles 2005).
Rockhurst University invested significant sums to create an infrastructure to support course embedded assessment (Bassett, Daley, Haefele 2005).
Many schools administer assessment activities in the capstone course (e.g., Business Strategy & Policy courses at The University of Akron)
Surveys used by many schools are costly and generate reams of data (e.g., EBI)
Assessment evidence gathering, analysis and reporting require significant time commitments
Accounting departments spent less than $5,000 in 2000 and 60% said they had no resources for assessment in 2004 (Hindi & Miller 2000; Calderon et al. 2004)
Assessors (faculty) need time for other valued activities
20
The assessor’s dilemma and course embedded assessment
Specialized assessment activities (e.g., assessment center, assessment day, special tests) …Generally more overhead
UA uses the following in-house tests:• The CCT (core curriculum test)• Major field tests in finance• Major field test in accounting • Major field test in marketing
Attitudes & Opinions - Surveys, interviews, focus groups ………Generally more overhead
UA uses the following:• EBI surveys• Graduating students exit surveys• Employer surveys• Alumni surveys• Advisory board reviews
Course-embedded assessment (done as part of regular, on-going activities in a course) …………. Generally less overhead
UA uses the following:• “Management Projects”• Cases, simulations and projects• Writing samples and writing
portfolios
DIRECT INDIRECT
21
The assessor’s dilemma and course embedded assessment
Lesson 3: Course-embedded assessment (CEA) transforms the evidence collection process from a potentially burdensome activity to one that is part of normal classroom activity
– Create a support infrastructure for CEA Consensus, Classroom Activities, Training, Instruments, & Medium for Reporting and Dialogue (CATIM)
– Focus on program learning goals rather than learning goals for a specific course
– Make achievement of program learning goals public– Use incentives to generate faculty interest in CEA – e.g.,
Rockhurst
22
Data reporting and active engagement vs. data hoarding and passive compliance
Observation 4– Close to 75 percent of business school deans report that
their faculty discuss assessment data regularly at faculty meetings.
– Most do so at least once each academic year.– USAF Academy reports data to faculty regularly and have
dialogue on how data compare with expectations– Seton Hall and Ohio Northern prepare an annual
assessment report for faculty – Kings College sets aside an Assessment Day for faculty to
make presentations on assessment results.
23
Data reporting and active engagement vs. data hoarding and passive compliance
How programs report results?Percent (N=105)
Faculty retreat 26
Regular faculty meetings 69
Departmental meetings 44
Committee meetings 44
Oral reports 25
Written reports 50
Special faculty meeting 14
N=160 vs. N=105? Implications
24
Data reporting and active engagement vs. data hoarding and passive compliance
Lesson 4: Assessment reports are the basis for dialogue, reflection, and proposals for change– Offer relatively seamless opportunities for viewing
and reviewing reports (e.g., WebCT)– Discuss reports regularly– Identify opportunities for improvement– Create a set of action items after discussions
25
A Hypothetical Summary of Evidence Based on a Standardized Rubric
TRAIT N Good Satisfactory Not Acceptable
Recognizes own personal biases that can influence decision making outcomes.
200 30(15%)
160(80%)
10(5%)
Learns from history by including a discussion from the past where managers faced similar ethical issues.
200 22(11%)
88(44%)
90(45%)
Limits the expression of self-interest and other outcomes of marketplace logic.
200 40(20%)
100(50%)
60(30%)
Abstains from the tendency to justify the means by virtue of the end.
200 36(18%)
160(80%)
4(2%)
Average 16% 64% 21%
26
Visualization and improvement opportunities
Radar Chart of Item Analysis
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
12
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
1920
2122
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
3940
Observation 5: Some schools use innovative charts and graphs to focus faculty attention (e.g., Redle & Calderon 2005)
27
Trends in U of Akron’s Major Test in Finance
Trend in Mean and Median Student Scores
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
Test date
28
Domain Objectives
59.76%
56.19%
74.09%
54.39%
69.86%
61.77%
76.12%
78.70%
51.87%
50%
55%
60%
65%
70%
75%
80%
Through Fall 2001 During 2002 Through Fall 2002 Through Spring 2004 Through Fall 2004 Through Spring 2005
29
Lesson 5: Attention directing graphs and charts lead to more focused assessment dialogue – Oriented toward problem-solving– Focus may include:
Assessment method Faculty development Student Classroom teaching and learning methods Teaching resources and the physical infrastructure for learning Learning goals and curriculum alignment
Visualization and improvement opportunities
30
Closing the Loop
Percent (N=143)
No 42
Yes 58
Total 100
Observation 6:
Has assessment led to significant curriculum changes?
31
Closing the loop
36% of business schools report that they have a formal written assessment plan
Schools with a formal plan are more likely than those without to report success in using assessment to improve the curriculum (Chi-square=4.77, DF=1, p-value=.02)
32
Closing the loop
Lesson 6: Plan before you leap!!!– Powerful tool for effective management and communication of
assessment activities. – Guides schools through each major assessment activity – A basic plan could be designed as a grid that includes columns
for: program learning objectives assessment methods to be used in collecting evidence about student
learning for each objective identification of the measurement metrics and expectations for each
learning goal a procedure for administering the assessment activity a time line for implementation List of persons responsible for coordinating data collection
33
Assessment Plan – A Basic ExampleStudents will: Method used/
EvidenceExpectations Assessment Procedure Primary
Responsibility of
Demonstrate knowledge and understanding of core accounting fundamentals (financial, cost, systems, auditing, taxation)
Method: Major Field Test in Accounting (MFTA) ---(75 MC questions)
Evidence: Raw percentage scores on the MFTA
- Satisfactory value-added (≥45%) - Satisfactory raw scores on the post-test (≥70%)- Satisfactory achievement on each cognitive learning goals
- Computerized Pre/Post Test. - Pre-test administered in first accounting course-Post-test administered in accounting capstone - Consistent incentives and awards (e.g., 5% of grade in capstone)
Department Chair
Demonstrate ability to apply core accounting fundamentals through case analyses and projects.
Method: Course-embedded case/project to demonstrate ability to apply accounting fundamentals
Evidence: Ratings for individual students based on standardized rubric developed for the program by the faculty
- At least satisfactory rating on all traits in the standardized rubric- Overall, 70% perform at least at the satisfactory level
- Relatively short case/project that span multiple areas in accounting- Students analyze cases or complete project as part of requirements capstone.- Case analyzes/project are rated by instructor and a panel of professionals
Faculty who teach the accounting capstone course
34
Conclusion
Assessment is effective if it leads to congruency, alignment and change
Assessment must be planned, supported, and executed in a systematic manner
Assessment is a systematic and deliberate extension of what faculty normally do
Keep what you do simple Avoid fixations!