Needs & expectations about an International

Learning Assessment in Higher Education

Washington DC, July 11-12, 2013

1

Dra. Patricia Rosas Chávez, UDGMC Luz María Nieto Caraveo, UASLP

Implementation

Did our expectations were met?

Challenges of a future ILA

2

3

4

Contents

2

Reasons to participate & interest on cross-national learning assessment

1

2

3

4

Reasons to participate & interest on cross-national learning assessment

3

• Main mechanisms to evaluate the quality of Mexican HE:• CENEVAL

• Disciplinary field tests• Generic skills

• CIEES & COPAES• PIFI

• Mechanisms of evaluation focused on inputs & processes mainly. While LOA has been gaining ground internationally.

Reasons to participate & interest on cross-national learning assessment

4

• AHELO opened an opportunity window to emphasize new assessment approaches for higher education system.

• A possibility to know more about the impact of typical indicators such as qualifications of faculty, infrastructure, curriculum updating, tutoring, departmental tests, among many others.

Reasons to participate & interest on cross-national learning assessment

5

• In a country characterized by its regional, cultural & socio economical diversity, its greater demand for higher education, and its many educative subsystems, we wanted to know if

• It is possible to compare LO among educative subsystems at a national level and with other international HEIs?

Implementation

14 participating HEIsUniversidad Autónoma de San Luis Potosí (UASLP)

Universidad de Guadalajara (UDG)

Universidad Autónoma de Yucatán (UADY)

Tecnológico de Monterrey

Instituto Politécnico Nacional (IPN)

Instituto Tecnológico Superior de Irapuato (ITESI)

Universidad Autónoma de Chihuahua (UACH)

Universidad Autónoma de Coahuila (UAC)

Universidad Autónoma de Colima (UCOL)

Universidad Autónoma de Zacatecas (UAZ)

Universidad de Ciencias y Artes de Chiapas (UNICACH)

Universidad Politécnica de Aguascalientes (UPA)

Universidad Tecnológica de la Mixteca (UTM)

Universidad Veracruzana (UV)7

Implementation

• Mexico joined the international consensus to evaluate GS, Economics & Engineering strands.

• There is interest in evaluating specific domains which:– have an international vocation – have the greater enrollment and demand– coming from emergent fields– have recognition of quality, or those that attend specific social problems.

8

Implementation

• In assessing specific field domain capacities, we must be careful of not evaluating all of them, but only those that were common and comparable among countries.

9

Did our expectations were met?

The good• We knew that this were a feasibility

study and that it could result in a viable or non viable AHELO.

• The conceptual idea of AHELO was innovative and suggested many challenges about which we have been reflecting and learning.

• The 1st face of the project resulted the best one since there were enough time for discussing, reaching consensus, and adjusting instruments (GS).

10

Did our expectations were met?

The bad

• Due to the budget gap, the 2nd fase of the project happened in a rush.

• In GS there were no time for pilot. The results of a pilot could be used in the international workshop to calibrate scorers.

• Integrating MCQs was valuable, but there was no time to discuss among countries this instrument that neither was integrated to the conceptual framework. Besides, the report from the Consorsium added the scores from CRTs and MCQs as if they were coming from the same constructo. 11

Did our expectations were met?

The bad

• Comparisons could not be given properly since there were no the same RR among HEIs.

• The contextual dimension was not exhaustive.

• Because of costs, value added study had to be abandoned.

• The evidence about the validity of the instrument may not be contundent as expected.

12

Did our expectations were met?

The ugly

• The way the Consortium reported the results.

13

Challenges for a future ILA

• The first thing to have in mind is that expectations of governments and nations will vary in function of the economic, social and cultural context.

• The financial crisis is an important constraint in designing public policies for HE, tempting governments to use this type of evaluations as a way to allocate budgets.

14

Challenges for a future ILA

• Having evaluation for rankings or allocating budgets puts in risk many HEIs that could be in clear disadventage for their own context.

• Therefore it is needed to find an equilibrium among transparency, accountability, budget constraint and those learnings that HEIs could obtain by comparing themshelves with others in order to improve their quality.

15

Challenges of a future ILA

• It is clear that everyone wishes to learn from the best practices but proper parameters must be found.

• An ILA is not the panacea to solve the problems of the HE, nor the only way to improve its quality, but it is useful to benchmark what others has been doing to get good results.

16

Challenges of a future ILA

What an ILA would do better than isolated nations may be:• Reach consensus about common

generic capacities needed for this global era.

• Help to improve assessing national instruments by considering this global perspective.

17

Challenges of a future ILA

What an ILA would do better than isolated nations may be:• Provide insights to make more

appropriated ranking systems according to the different HEIs profiles.

• Contributing to change the culture of high stakes assessing to a low stakes, reducing pressure for allocating budget in function of rankings, and creating a new culture of learning for its own sake.

18

Challenges of a future ILA

• If there is agreement about that students in a global era share some atributes that help them to survive, the main challenge consist in identifying those attributes and reach an international consensus about what and how to assess GS as well as domain skills.

• So far, we are working with the data and we know for sure that further data must be collected in order to interpret properly what we found at a national level. But very little we got from an international perspective.

19

Concluding remarks

• The feasibility study was valuable since it allowed us to know the good, the bad, and the ugly. By taking care of these aspects, any future main study should be better for sure.

• Mexico wants to continue its international participation but in a low stake ahelo.

• If a main study were taken place, further involvement of participants in definning conceptual frameworks & instruments, is needed.

20

21

Thank you!http://ahelo.uaslp.mx

Proyecto coordinador en México por la UASLP, UDG y UADYcon financiamiento y apoyo de la Secretaría de Educación Pública, Subsecretaría de Educación Superior, Dirección General de Educación Superior Universitaria (PADES 2009, 2010, 2011 y 2012 de la UASLP, UADY y UDG).