View
217
Download
2
Category
Tags:
Preview:
Citation preview
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
The Implementation of National Educational Standards in Austria
Michael BruneforthBIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System
EURASIAN EDUCATIONAL DIALOGUE17.-19. April, Yaroslavl
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System
Areas of work International Studies (PISA, TIMSS, PIRLS, TALIS)
Implementation of national education standards
Formative and summative Evaluation of standards
Central examination for upper secondary education preparing for tertiary education
National Education Reports
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Spotlight on Austrian education
Very rich country (4th highest GDP per Capita in EU)
High expenditure per student (140% of EU Average) in primary and secondary education
Low student teacher ratio and class size
But PISA shock in 2003 and 2009: Very low PISA results in reading (470, similar to Russia
459), no EU country scores less
Strong relation between education outcomes and family background
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Changing in paradigm in educational governance
Since more than a decade there is a change in governance of schools and education in German speaking countries from input oriented control to evidence based output oriented governance and extension of school autonomy
In Austria at the centre is shift towards competence oriented teaching based on the definition of nationwide compulsory education standards and a mechanism of formative and summative evaluation
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Legal Basis and Conditions
From 2004: pilot phase with selected pilot schools
2008: alteration of the law of teaching in schools to allow for introduction of educational standards
2009: educational standards become compulsory for German, Math and English
2009 (grade 8) and 2010 (grade 4): baseline-tests in samples of about 10.000 students each
From 2012 onwards yearly assessment of national standards (one subject per year)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Annex of the legal regulation
Competency Models and descriptors are formulated as Can-Do-Statements
Competency Models and descriptors are legally established
Educational Standards can be interpreted as mandatory and agreed upon social objectives of instruction
Educational Standards are derived from national curricula and clearly define learning outcome.
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Areas coveredGerman (language of instruction) – grades 4 & 8- Listening & Speaking- Reading - Writing texts- Spelling/Grammar
Math – grades 4 & 84 content dimensions4 process dimensions
English, grade 8- Listening- Reading- Speaking- Writing
English Assessment:Common European Framework of Reference for Languages (CEF)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Competencies are …
„fashionable term with a vague meaning“ (Weinert 2001)
„cognitive prerequisites to coping with a specific range of situations“ (Klieme, Hartig & Rauch, 2008)
„cognitive abilities and skills available to/acquired by individuals that enable them to solve specific problems, as well as the related motivational, volitional and social readiness and skills to utilize the solutions successfully and responsibly in variable situations“ (Weinert, 2002)
BIFIE use: subject (domain) specific competency models, focusing on cognitive skills and proficiencies
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Competency Models
Models of competence structures
Dimensions of competence: E.g. TIMSS, grade 8: 4 content domains (number, algebra, geometry, data and chance); 3 cognitive domains (knowing, applying, reasoning)
Models of competence levels
Describe specific skills and proficiencies usually shown by individuals located at specific points of the ability continuum
Used to provide a criterion-referenced interpretation of measurement results
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Competency Models
Models of competence structures
Models of competence levels
Development models
Models describing the acquisition of skills and proficiencies
May be linked to neurological/psychological theories of development
Only a few exist
Would be great to have a theoretical foundation for tests in different grades! (IKM – BIST)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
EXAMPLEMATHEMATICS, GRADE 8
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
M8 – Competency Model
Process domains(Handlungsbereiche)
Content domains (Inhaltsbereiche)
Level of complexity(Komplexitätsbereiche)
demonstrating, modelling
numbers and units use of basic skills and knowledge
calculating, operating variable, functional dependency
build connections/connect
interpreting geometric figures and shapes
reflect, use knowledge of reflection
explaining, reasoning statistical representations, measures of central tendency and variance
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
M8 – Competency Model
The 4x4 grid leads to 16 nodes, using complexity as a 3rd dimension = 48 nodes
Not very handy and the theoretical concept of complexity does not lend itself to assessment
Content
Process
Complexity
Competency (H3, I2, K2)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Example: Competency Model M8
P: „demonstrating, modelling “ C: „numbers and measures“
Formulated as can-do-statements: The students are able to
transform given arithmetical data into (another) mathematical representation. In doing so, direct use of basic skills is necessary
transform given arithmetical data into (another) mathematical representation. In doing so, connections to other mathematical contents (terms, definitions, representations) or activities have to be established
make and evaluate statements about the appropriateness as well as weaknesses and strengths of different mathematical representations (models) of arithmetical data
P1/C1/K1
P1/C1/K2
P1/C1/K3
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Educational Standards –The Austrian Version
Three areas of activities Implementation of standards at school and capacity
building for teacher: Support of teachers and school heads in implementation
Formative evaluation to support teachers diagnostic: On-line evaluation system to be used by teachers
Census to monitor and summative evaluation at system, sub-system, school and classroom level
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Educational Standards –The Austrian Version
School-specific development of instruction Mandatory educational standards
Census-survey of all students and schools
Feedback to all students, teachers, schools
Feedback moderation („Rückmeldemoderation“)
Stimulation of quality development and support by MoE
Strong linkage of monitoring and practical school and instructional development
Strengthening of schools, school management and regional quality development
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Main functions of standards according to the legal regulation
Output orientation: Sustainable output orientation in planning and implementation of instruction
Promotion („Förderfunktion“): Diagnosis and specific individual remediation and challenge
Evaluation: reliable information about the output of instruction, schools and education system
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Implementation of Standards
Supporting materials for instruction Praxishandbücher
Themenhefte
Aufgabenbeispiele, Interaktive Beispiele
Best-Practice-Beispiele
Integration with teacher training institutions In-service teacher training (partially mandatory)
Pre-Service teacher trainin
Research and development work by members of teacher training institutions
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Diagnosis tool for Informal Monitoring of Competencies
Teacher administered on-line system
Grade 3 and 7, one year before Assessment
All Areas of standards
Only teachers themselves have access to results
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Assessment of Educational Standards
Legal assessment cycle: three (school) years
In each cycle, the whole target population has to be assessed in all prescribed competency subdomains Grade 4: German, Mathematics
Grade 8: German, Mathematics, English
Students, teachers, schools and regional education authorities must get feedback about the results (to be used for quality improvement)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Domain-orientated Test Design
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Pilot testing: about 10% of population, up to 9000 Students
Main test: Census up to 83000 students Speaking domain only with sample for system
monitoring
10% of census with external administration and items for next census to ensure linking over time
Example: Test session Mathematics Grade 4
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
This pictureshows about 1/10 of the test materials
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstrating
Dimension „complexity“ is not used as such, since it can‘t be measured empirically
We have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstrating
Dimension „complexity“ is not used as such, since it can‘t be measured empirically
We have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Test blueprint specified the items to be developed: Mathematical node
Item format: Multiple Choice, Constructed response, …
Estimated difficulty of item
Further requirements: adequat for age group,
no biased towards gender, race, religion, also taking into account rural/urban differences.
does not require other knowledge than maths contents.
Items should be uni-dimensional
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Validation
In order to secure validity of the test items, they were … written by school and university teachers
reviewed by teachers (peer review and outsider review)
trialled on a sample of grade 8 students – each item was answered by at least 200 students
item properties were reviewed by a team of psychometricians and school teachers
items with less than desirable or surprising properties were retired and not used in the main test
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Test Design
Multiple matrix sampling
Two-step approach for the test design Allocation of „empty“ blocks to forms that satify several
global prerequisites for the design:• positional and contextual balance, item security (S90
vs. S10), longitudinal link (BL and BIST-UE 2015)
Allocation of items to blocks that satisfy several local prerequisites:• Balanced difficulty, format, representative for the
competence construct, dependency (enemy items), test motivation
• Test information, especially along cutoff-scores
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Scaling
Using Rasch model
Allows to express difficulties of items and students ability on one common scale
Allows to match test cycles over time
Allows for complex test design
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Standard Setting
Standard setting goal: meaningfully described levels
criterion-referenced
Phases: Determination of number of levels
Labelling of levels (PLLs)
Description of levels (PLDs)
Establishing the cut scores between levels
4 levels in Austria in most domains0: standards not reached1: partially reached standards2: reached standards3: exceeded standards Easy to understand for „laymen“
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Proficiency Level Descriptions M8
3 – standards exceededStudents possess fundamental knowledge and skills in all parts of the mathematics curriculum and advanced knowledge structures, which exceed the requirements of level 2, specifically more pronounced abilities of abstraction and higher proficiency in combining parts of knowledge, methods or rules. They are able to apply these independently in novel situations in a flexible way.2 – standards reachedStudents posess fundamental knowledge and skills in all parts of the mathematics curriculum and can use these in a flexible way. They are able to find and apply problem solving strategies, to describe and reason an approach. They are able to handle verbal, graphical and formal representations of mathematical facts in a flexible way and can apply these appropriately. Theay are able to extract relevant information from a differently represented facts (e.g. texts, data material, graphics) and can interpret them in the respective context. They are able to relate their mathematical knowledge and can check, evaluate and/or reason mathematical statements.
1 – standards partially reachedStudents possess fundamental knowledge and skills in all parts of the mathematics curriculum and can master reproductive tasks and carry out routine procedures.
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Standard Setting
„ A standard setting method is a mechanism for helping panelists translate their intentions to the score scale.“ (Reckase, 2006)
Goal: Match predetermined descriptors of competency levels to an empiric scale defined by test items
Determine cut scores between levels of competence
Bifie invited a group of stakeholders to set the cut scores: teachers (lower and upper secondary), teachers of didactics of math; representatives of parents associations; representatives of the MoE; representatives of chamber of commerce; bifie staff
Method used: Item descriptor matching method
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Ordered Item Booklet
Easy Item
Difficult Item
Level 2
Level 3
Level 1
Item-descriptor matches
Level 2
Level 3
Level 1
(Modified) Item descriptor matching method
Freunberger & Yanagida, in press.
Difficulty-ordered items
Cut Score 1
Cut Score 2
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Standard Setting
Standard setting is more about finding a societal consensus than finding a constant
The goal should be clear before test development etc starts
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Feedback and ReportsInformation about results of Educational Standards
SCHOOL LEVEL
School report (for school management) per school, online
Feedback to teachers per teacher, per class/group; online
170.000 studentsabout 85.000 in grade 4 and 8 each
10.000 classesabout 6000 primary and
4000 secondary I
School and instruction development
4600 schoolsabout 3200 primary and
1400 secondary I
Feedback to students per participant, online
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Province reports (pres & vice-) for each procince and grade
SYSTEM LEVEL
99 school districts15 Cities – 84 districts
National report (Minister for Education) aggregated
9 provincesPrimary/4
Lower sec/8, academic sec/8
9 provinces
MoE
School development
System development
Supervisory reports (federal superv. authorities for PS & LS/AS) aggregated on fed. state and supervised-school type
Feedback and ReportsInformation about results of Educational Standards
Supervisory reports (district superv. authorities for PS and LS) per district and grade
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
School report: overview of content
Information about school results:
compared to AUT
compared to the defined goals (in the regulation)
compared to other schools with similar basic conditions („fair comparison“)
Areas of conflict:- Educational Standards apply to all students –
and therefore to all schools
- Conditions of schools vary widely
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
FEEDBACK
The school takes centre stage
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (school report)„Competency levels“
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (teacher feedback)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (school report)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Support system for schools
A SCHOOL
School feedback
Teacher feedback
School authorities
Feedback Moderation
Quality developmen
t- process supervision
IKM (informal
self evaluation)
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
OutlookBringing IKM (grade(s) before BIST) and BIST assessments closer together. Using items in both assessments could lead towards a better understanding of learning
Backwash from doing the assessments towards research in the areas of item development, standard setting, feedback … through evaluation of results
Outcomes of the assessments should not only allow us to give feedback to schools, teachers and students but also to improve the competence models we use for the assessment
Prepare for computer based testing
Open databases for external researchers and improve system level monitoring
The Implementation of National Educational Standards in Austria, 18/05/2013 © BIFIE Salzburg
Thank you for your attention!
MichaeI Bruneforth
m.bruneforth@bifie.at
Bundesinstitut BIFIESalzburg | Zentrum für Bildungsmonitoring & Bildungsstandardswww.bifie.at
Recommended