Evaluation Juga

7/29/2019 Evaluation Juga

1/25

Session 1

Accountability Policies


2/25

Who is Responsible for your

Childs Education?

The Ministry of Education lists a number of stakeholders

responsible for publicly funded elementary and secondary

education in Ontario:

Ontario Government and the Education Act

Minister of Education School Boards

Principals

Teachers

Students

Parents School Councils

Ontario College of Teachers

Education Quality and Accountability Office

http://www.edu.gov.on.ca/eng/document/brochure/whosresp.html
http://www.edu.gov.on.ca/eng/document/brochure/whosresp.htmlhttp://www.edu.gov.on.ca/eng/document/brochure/whosresp.html


3/25

Accountability

Educational accountability is primarily a relationship

between three key stakeholders:

1. Taxpayers

2. Elected officials

3. Teachers


4/25

Standards-Based Reform

Defined as that which measures the performance of alearner against set standards of achievement or

competence; unit standards are descriptive statements of

what an individual should know and be able to do.

Early 1990s response to concerns that students weren't

learning enough to compete in a global economy and that a

significant gap exists between minority and non-minority

student achievement.


5/25

Standards-Based Reform in the

Western World

Each of the following countries has developed and

implemented a form of standards-based reform:

United States

England

Canada

Australia New Zealand


6/25

Elements of Standards-Based Reform

In addition to high contentandperformance standards,standards-based reforms usually encompass the following

1. Curricula aligned with standards;

2. Capacity building of schools and teachers;

3. Regional assessments to measure progress toward standards;

4. Incentive and accountability system that uses assessments and

other variables to provide intervention to schools and school

systems; and

5. Phasing in of student incentive and accountability system based,in (large) part, on assessment result.


7/25

Standards-Based Reform:

A Preliminary Critique

Based on your initial understanding of standards-based

reform, develop a list of advantages and disadvantages for

this approach.

Discuss with a colleague whether your advantages

outweigh your disadvantages? Or vice versa?

Note: We will revisit your list on the last class.


8/25

EVALUATION

PROCESS


9/25

Evaluation

Evaluation is broader than assessment and involves

making judgments about the meritorworth of something

(an evaluand) or someone (an evaluee).

Merit involves judgments about generalizedvalue.

Worth involves judgments about instrumentalvalue.


10/25

Evaluation Terms

Appraise Analyze Assess

Critique Examine Grade

Inspect Judge Rate

Rank Review Score

Study Test Others?


11/25

Purposes of Evaluation

1. Demonstrating accountability

2. Assisting in making a decision

3. Bringing an issue to the attention of others

4. Helping stakeholders elaborate or refine their opinion on an issue

5. Convincing other to take action

6. Exploring and investigating an issue

7. Involving stakeholders in program planning or policy development

8. Gaining support for a program


12/25

Purposes of Evaluation (cont.)

9. Promoting understanding of an issue(s)

10. Changing attitudes and/or individual behaviours

11. Changing the nature of dialogue or interaction among groups

12. Influencing policy

13. Introducing new ways of thinking and researching through

evaluation

Of the previous purposes, which would be the most relevant to the

implementation and maintenance of a standards-based system?


13/25

Research vs. Evaluation

Research Evaluation

Primary purpose is to Add knowledge in a field

and/or contribute to the

growth of theory

Help those who hold a stake in

whatever is being evaluated

make a judgment or decision

Agenda set by Researcher Stakeholders

Results Generalize to broader

population

Specific to particular context

Quality determined by Internal and external validity Accuracy, utility, feasibility,and propriety

Preparation Single discipline Interdisciplinary


14/25

Formative vs. Summative Evaluation

Formative Evaluation: Provide information for program improvement, typically to judge the

merit and worth of a part of a program.

Audience is generally the people delivering the program or those

close to it.

Summative Evaluation:

Provide information for making decisions about program adoption,

continuation, or expansion.

Audience is generally potential consumers (students, teachers,employees, managers, etc).

What problems may arise when an organization focuses too heavily on

either type of evaluation?


15/25

Objectivist vs. Subjectivist Epistemology

Objectivism:

Requires an evaluation study to utilize data collection and analysistechniques that yield results that are reproducible and verifiable byother competent persons using the same techniques.

Subjectivism:

Bases its validity claims on an appeal to experience rather than tothe scientific method.

Provide a list of advantages and disadvantages for each epistemology.Which do you tend to gravitate towards? Why?


16/25

Evaluation Approaches

Objectives-Oriented Approaches:

Focus on specifying goals and objectives and determining the extent towhich they have been attained.

Participant-Oriented Approaches:

Involvement of participants (primarily stakeholders) are central indetermining the values, criteria, needs, data, and conclusions for the

evaluation.

Consumer-Oriented Approaches:

Central issue is developing evaluative information on products andaccountability for consumers.

Expertise-Oriented Approaches: Depend on the direct application of professional expertise to judge quality of

whatever is being evaluated.

Management-Oriented Approaches:

Central concern is identifying and meeting the information needs ofmanagerial decision makers.


17/25

PROGRAM

EVALUATIONSPOTLIGHT


18/25

Objectives-Oriented Evaluation

Defining feature is that the purposes of some activity are

specified, and then the evaluation focuses on the extent to

which those purposes are achieved.

What are some examples of objectives-oriented evaluationin education?

How does your school / institution / organization measure

the attainment of program goals?

Are these procedures formative, summative, or both?


19/25

Tylerian Evaluation Model

1. Establish broad goals or objectives.

2. Classify the goals or objectives.

3. Define objectives in behavioural terms.

4. Find situations in which achievement of objectives can be shown.

5. Develop or select measurement techniques.

6. Collect performance data.

7. Compare performance data with behaviourally stated objectives.

Develop an evaluation plan according to the Tylerian approach for a

curriculum content area (reading, writing, math, etc). Your rough draft

should include steps 1-5. Share your plan with a partner.


20/25

Provus Discrepancy Evaluation Model

As a program is developed it goes through five developmental stages:

1. Definition

2. Installation

3. Process (interim products)

4. Product

5. Cost-benefit analysis (optional)

How might the discrepancy model assist in the reform of the Ontario

education system? Develop a list of advantages and disadvantages of

utilizing this approach within the current context?


21/25

Goal-Free Evaluation

1.

The evaluator purposefully avoids becoming aware of theprogram goals.

2. Predetermined goals are not permitted to narrow the focus of theevaluation study.

3. Focus on actual outcomes rather than the intended outcomes.

4. Evaluator has minimal contact with the program manager andstaff.

5. The likelihood that unanticipated side effects are increased.

How can forms of goal-free evaluation supplement objective-oriented

approaches to evaluation within your school / institution / organization?


22/25


23/25

Education Journals

1. Canadian Open Access Journals:

http://www.oise.utoronto.ca/canedweb/journals.html

2. International Open Access Journals:

http://aera-cr.asu.edu/ejournals/

3. Brock Library Databases:

http://www.brocku.ca/library/databases/index.php?view=subject&subject=Education
http://www.oise.utoronto.ca/canedweb/journals.htmlhttp://aera-cr.asu.edu/ejournals/http://www.brocku.ca/library/databases/index.php?view=subject&subject=Educationhttp://www.brocku.ca/library/databases/index.php?view=subject&subject=Educationhttp://www.brocku.ca/library/databases/index.php?view=subject&subject=Educationhttp://www.brocku.ca/library/databases/index.php?view=subject&subject=Educationhttp://aera-cr.asu.edu/ejournals/http://aera-cr.asu.edu/ejournals/http://aera-cr.asu.edu/ejournals/http://www.oise.utoronto.ca/canedweb/journals.html


24/25

Evaluation Journals

America Journal of Evaluation (formerly Evaluation Practice; AEA)

http://www.eval.org/american_journal_of_evaluation.htm Canadian Journal of Program Evaluation (CES/CSE)

http://www.evaluationcanada.ca/site.cgi?s=4&ss=2&_lang=an

Educational Evaluation and Policy Analysis (AERA)http://www.aera.net/pubs/eepa/

Evaluation Exchange

http://www.gse.harvard.edu/hfrp/eval.html

Evaluation and Program Planning: An International Journal

http://www.elsevier.com/wps/find/journaldescription.cws_home/593/description#description Evaluation Review: A Journal of Applied Social Research

http://www.sagepub.co.uk/journal.aspx?pid=105549

Evaluation: The International Journal of Theory, Research and Practicehttp://www.sagepub.co.uk/journal.aspx?pid=105547

Evidence & Policy: A Journal of Research, Debate and Practicehttp://www.bris.ac.uk/Publications/TPP/tpp.htm

Journal of MultiDisciplinary Evaluation

http://evaluation.wmich.edu/jmde New Directions for Evaluations (AEA)

http://www.eval.org/Publications/NDE.htm

Practical Assessment, Research and Evaluation (online journal)http://pareonline.net/

Studies in Educational Evaluationhttp://www.elsevier.com/inca/publications/store/4/9/7/
http://www.eval.org/american_journal_of_evaluation.htmhttp://www.evaluationcanada.ca/site.cgi?s=4&ss=2&_lang=anhttp://www.evaluationcanada.ca/site.cgi?s=4&ss=2&_lang=anhttp://www.aera.net/pubs/eepa/http://www.gse.harvard.edu/hfrp/eval.htmlhttp://www.elsevier.com/wps/find/journaldescription.cws_home/593/descriptionhttp://www.elsevier.com/wps/find/journaldescription.cws_home/593/descriptionhttp://www.sagepub.co.uk/journal.aspx?pid=105549http://www.sagepub.co.uk/journal.aspx?pid=105547http://www.bris.ac.uk/Publications/TPP/tpp.htmhttp://evaluation.wmich.edu/jmdehttp://evaluation.wmich.edu/jmdehttp://www.eval.org/Publications/NDE.htmhttp://pareonline.net/http://www.elsevier.com/inca/publications/store/4/9/7/http://www.elsevier.com/inca/publications/store/4/9/7/http://pareonline.net/http://www.eval.org/Publications/NDE.htmhttp://evaluation.wmich.edu/jmdehttp://www.bris.ac.uk/Publications/TPP/tpp.htmhttp://www.sagepub.co.uk/journal.aspx?pid=105547http://www.sagepub.co.uk/journal.aspx?pid=105549http://www.elsevier.com/wps/find/journaldescription.cws_home/593/descriptionhttp://www.gse.harvard.edu/hfrp/eval.htmlhttp://www.aera.net/pubs/eepa/http://www.evaluationcanada.ca/site.cgi?s=4&ss=2&_lang=anhttp://www.eval.org/american_journal_of_evaluation.htm


25/25

Evaluation Websites

American Evaluation Association

http://www.eval.org/

Australasian Evaluation Society

http://www.aes.asn.au/

Canadian Evaluation Society

http://www.evaluationcanada.ca

European Evaluation Society

http://www.europeanevaluation.org/
http://www.eval.org/http://www.aes.asn.au/http://www.evaluationcanada.ca/http://www.europeanevaluation.org/http://www.europeanevaluation.org/http://www.evaluationcanada.ca/http://www.aes.asn.au/http://www.eval.org/

Documents

Evaluation Juga