2 ASSET is a registered trademark and Mindspark is a trademark of EI

WHY THIS IDEA HAS LARGER RELEVANCE

It is widely recognized that education is essential for economic and social development of a country as well as for personal development of its citizens. It is being more commonly accepted (though some disagree) that regular assessment of student achievement is important to make decisions and improve educational systems. Trends in International Mathematics and Science Study (TIMSS) is one of the most influential international assessments of student achievement that is conducted every four years since 1995. TIMSS also includes information about key instructional, curricular and resource related variables with reference to the teaching-learning process in Mathematics and Science. Equivalent studies (e.g., Progress in Reading Literacy Study - PIRLS) are also conducted in reading literacy. The stated aims of TIMSS are to improve the teaching and learning of Maths and Sciences around the world. It provides important information to aid policy development, to foster public accountability and to identify areas of progress or decline in achievement with a view to addressing the concerns for equity. For the first time in many decades, in 2010, India is participating in ‘Programme for International Student Assessment (PISA), yet another international benchmarking study for 15 year old students. This working paper discusses TIMSS with a special focus on the Maths results from the latest study (2007) and its relevance to India.

SALIENT FEATURES OF TIMSS • Time: Conducted on a regular 4 year-cycle. • Number of rounds: One round each in 1995, 1999, 2003 and 2007.

The next study will be in 2011. • Number of participant countries: In 2007, 59 countries and 8 regional entities (like Indian states) participated - the countries are

marked on the cover page. • Subjects tested: Mathematics and Science. • Classes tested: Grades 4 and 8. (Grade 12 was tested in 1995) • Sample: Each country sampled about 4000 students in 150 schools. A total of about 4,25,000 students were assessed in TIMSS 2007. • Organized by: International Association for the Evaluation of Educational Achievement (IEA) - an international organization of national

research institutions and governmental research agencies located in Amsterdam. • Language: Assessment instruments were developed in many different languages, involving a substantial effort in translation and

standardisation. The translations underwent a complex verification procedure coordinated by the IEA Secretariat. • Statistical techniques used: Sophisticated techniques including Item Response Theory, multilevel analysis, etc were implemented to

develop papers and analyse the data. • Data: Datasets are released for research purposes. A lot of detailed TIMSS information is available freely on the TIMSS site. • Complementary studies: Background analysis and video study are undertaken.

FRAMEWORK OF TIMSS

TIMSS has based its framework on the school curriculum. The TIMSS curriculum model has three aspects: the intended curriculum (what is planned at the national or local level), the implemented curriculum (what is taught by teachers), and the attained curriculum (what students have actually learned). The TIMSS tests assess the attained curriculum, i.e. they check what students have actually learned in Maths and Science. Development of tests of student learning outcomes for each student population begins with an analysis of curriculum guides and textbooks from many (but not all!) countries to 'identify priority topics' for the tests. An international panel of curriculum specialists then produces a

framework to guide test development. The TIMSS 2007 Mathematics assessment framework is organized around two dimensions as below: 1. Content dimension: Specifies the domains or subject matter to be assessed within

Mathematics. 2. Cognitive dimension: Details out the domains or thinking processes to be assessed. The domains by dimension and the percentages of questions by domain for 4th and 8th grades are shown in the table.

Grade 2007 2003 1999 1995 4th 37 23 - 22 8th 50 40 30 31

Type Domains 4th

grade 8th

grade

Content Domains

Number 50% 30% Geometric shapes and measures

35% ---

Data display 15% ---Algebra --- 30%Geometry --- 20%Data and chance --- 20%

Cognitive Domains

Knowing 40% 35% Applying 40% 40%Reasoning 20% 25%

Number of Countries Participating in TIMSS

3 © Educational Initiatives Pvt. Ltd.

TEST DESIGN AND INSTRUMENTS

Two question formats are used in the TIMSS assessment – multiple-choice (where the student selects the correct option) and constructed-response (where the student writes the answer). At least half of the total points represented by all the questions in TIMSS are multiple-choice questions. The test development involves several steps to arrive at the final paper. The chart here describes the various steps undertaken to produce the final tool.

TIMSS QUESTIONS – TESTING UNDERSTANDING, NOT JUST RECALL

A notable feature of the TIMSS study is that detailed technical information about how the tests are developed and conducted, and how the results are analysed, is freely released through the internet. This is of great use to both researchers and practitioners worldwide. In addition, a good percentage of items (though not all) from the tests are also released. A few are reproduced below. It may be noted that most of the questions below require the child to understand the concept involved. In India, not only is ‘mechanical’ learning common (i.e. children are able to perform procedures without real understanding) but many educators also consider it alright. This encourages rote learning! Furthermore, school/state level tests usually assess only rote learning reinforcing the illusion that children are learning well.

The square is cut into 7 pieces. Put an X on each of the two triangles that are of the same shape and size

What fraction of this rectangle is shaded?

A. B. C. D.

What number equals 3 ones + 2 tens + 4 hundreds? A. 432 B. 423 C. 324 D. 234

The questions above are from the class 4 Mathematics paper while the ones below are from class 8. Correct answers are in bold.

What is equivalent to 4x – x + 7y – 2y?

A. 9 B. 9xy C. 4 + 5y D. 3x + 5y

The figure shows a shaded triangle inside a square.

What is the area of the shaded triangle?

Which of these shows 1080 as a product of prime factors? A. 1080 = 8 x 27 x 5 B. 1080 = 2 x 4 x 3 x 9 x 5 C. 1080 = 2 x 2 x 2 x 3 x 3 x 3 x 5 D. 1080 = 22 x 32 x 6 x 5

Multiply 0.402 x 0.53

4 ASSET is a registered trademark and Mindspark is a trademark of EI

MAIN FINDINGS: COMPARATIVE PERFORMANCE OF THE COUNTRIES The graph below shows the performance of students from 36 countries in TIMSS 2007 (Maths Grade 4). The black horizontal line represents the international TIMSS average corresponding to a score of 500. (TIMSS uses Item Response Theory (IRT) methods to summarize the achievement for each grade on a scale with a mean of 500 and a standard deviation of 100.) The performance of each country is represented by a vertical bar having black, white and blue portions as explained below. The average student achievement for each country is indicated by the black part of the vertical bar. For example, while the mean score for Hong Kong was 607 (above the international average), the average for Yemen was 224 (below the international average). The white portion indicates the performance of the students in the 25th to 75th percentiles of that country, while the blue portion indicates the performance range of the 5th to 95th percentile performers of the country. Countries are shown in decreasing order of average (mean) scale score.

The bars thus point out the dispersion of the scores within countries. The difference between the 95th and 5th percentiles within countries is often approximately 300 scale points. Observe that Slovenia and Armenia reached similar scores to the international average (500), but Armenia’s bar is longer than Slovenia’s. This indicates that the scores of students from Slovenia were more homogeneous than those of students from Armenia (or that Armenia has greater diversity of performance in its schools than Slovenia). It is interesting to observe that the bars on the right (indicating lower performing countries) are mostly longer than those on the left. In other words, the countries performing below the international average are also the ones which have an equity problem in terms of the wider variations in the quality of schooling within the country. The graph shows that Hong Kong achieved the highest average score in Maths at Grade 4. Note that almost the entire population from Hong Kong is above the international average. At Grade 4, Singapore, Chinese Taipei, Japan, Kazakhstan, the Russian Federation, England, Latvia and the Netherlands lead the averages above the 500 scale average. At Grade 8, TIMSS 2007 found five Asian countries with the highest average achievement in Mathematics (Chinese Taipei, Korea, Singapore, Hong Kong SAR and Japan). At the other end of the achievement continuum, students at Grade 4 from Qatar achieved the lowest average performance in Maths (224). Observe that even the students who had scored high scores in Qatar were below the international average. Not all rich nations perform well, either: the US Grade 4 average was 529 and Grade 8 average was 508 in Mathematics. This has been a major source of concern and action in the United States. Though the comparative performance of countries is often the most visible and highlighted part of the TIMSS study (especially in the media and in political debate), the real value of such studies lies in the deep insights they provide on a number of dimensions. Two of these have been highlighted on the next page. Further, the most common objections to studies like these have been enumerated, accompanied by our views on them.

In other words, the countries performing below the international average are also the ones which have an equity problem in terms of wider variations in the quality of schooling within the country.

5 © Educational Initiatives Pvt. Ltd.

BACKGROUND FACTORS ASSOCIATED WITH LEARNING

The TIMSS 2007 collected information about key variables that are fundamental to understanding the teaching and learning process. A School Questionnaire was used to collect information about the intended and implemented curriculum, in addition to a range of school characteristics such as location, size, resources, curriculum offerings, etc. A Teacher Questionnaire asked about qualifications, approaches to planning and teaching, use of resources, views on current curricular issues, etc. Finally, a Student Questionnaire gathered demographic details, how students spend time, attitudes to subjects, etc. This table summarizes the variables that were found to be linked to higher scores in Maths.

SCALE ANCHORING: WHAT STUDENTS KNOW AND ARE ABLE TO DO

A benchmark is a point of reference that allows us to explain student performance in terms of what students know and are able to do. TIMSS used four points on the scale as international benchmarks and described achievement at those benchmarks in relation to students’ performance on the test questions. The four points were: Advanced (at a score of 625), High (550), Intermediate (475), and Low (400).

The table above is from TIMSS 2007 report. It shows country-wise performance at the respective benchmark levels. For instance, in Japan, for 8th grade Mathematics, while 26% students are at or above the Advanced Benchmark, 97% students are at or above the Low Benchmark. The data indicate that 26% of students from Japan can organize and draw conclusions from information, make generalizations and solve non-routine problems; they can apply their knowledge of geometry in complex problem situations1. Likewise, the graph means that 97% of students from Japan have some knowledge of whole numbers and decimals, operations and basic graphs (Low Benchmark).

OBJECTIONS TO STUDIES LIKE TIMSS AND SOME RESPONSES

1. Such tests are unfair to children since gaps in learning could be due to resources available and quality of teaching. Tests like the TIMSS and other national variations like tests conducted by Educational Initiatives or ASER in India are always low-stake tests for children. They are also usually sample studies. It is not the intention of these studies to ‘blame’ students (or for that matter, teachers or any component of the system). Rather the aim is to provide clear information and goad people at multiple levels into action based on the detailed information and insights available.

2. What is needed is not more testing, but better quality teaching and teacher training to help teachers. This view stems from a perception that students (for example in India) are already tested too much. This view may be true of elite private schools. But, even in these schools, tests based on rote memorisation are not comparable with tests and studies like these. It is such studies that can actually help provide specific information on what kind of teacher training is most needed.

3. It is pointless to compare countries, schools or individuals. Each should strive to excel in a non-competitive manner. The purpose of benchmarking is not comparison but providing insights on how children and education systems elsewhere are performing. The evidence is that detailed information on learning is also provided. Knowing how others are doing can sometimes motivate positive action.

1 A detailed description of benchmarks could be found in TIMSS 2007 report, pages 73 to 100.

Level Factors positively associated to high student performance in Math

4th grade

8th grade

Family

Speaking the language of test at home Higher number of books at home Higher levels of parents education --- Computer at home Internet connection

Student Positive attitudes towards Math Higher self-confidence in learning Math

School

Principal and teachers’ positive view of school climate Principal reports that resources shortages were not a problem Schools with a few attendance problem --- Schools with fewer students economically disadvantaged home

6 ASSET is a registered trademark and Mindspark is a trademark of EI

GOOD FOLLOW-ON PRACTICES BASED ON TIMSS RESULTS

Regular student assessments of reliable quality and statistical validity have extremely high utility in providing the right direction for educational reforms. TIMSS contains valuable information about student achievement in various countries around the world. But these findings will not be meaningful unless they are used by policymakers, educators and society to develop strategies and convert them into concrete programs for changing education quality over a pre-defined timeframe. A few interesting case studies of good follow-on action based on TIMSS results are illustrated below. Norway: Norway participated in TIMSS in 1995 and 2003. TIMSS 2003 revealed a gloomy scenario: not only did Norwegian students score lower than the international average but their score had decreased dramatically since 1995. Other studies showed that teachers demonstrated low competence in Maths and a lack of specialization; homework was not followed up; and schools had problems of noise and disorder. Following a broad discussion using media to raise the national awareness, action was taken around curriculum reform, plans were made for strengthening teaching, implementation of a new system of quality assessment and policy changes for teacher recruitment. In 2007 TIMSS, student achievement for 4th grade increased significantly by 22 points from 451 to 473 points in 4th grade and by 8 points from 461 to 469 points in 8th suggesting that the reforms were in the correct direction. US States of Massachusetts and Minnesota: These two American states decided to participate individually in TIMSS twice as benchmarking participants. They followed all the rigorous quality standards established by TIMSS. Thus, their results are comparable to the participating countries. After the first participation, both states used the TIMSS data to evaluate their programs and policies at curriculum, professional assessment and school leadership levels. In 2007, both states showed significant improvement in student learning. In Minnesota, for the 4th grade, the average score increased from 516 in 1995 to 554 in 2007, while in Massachusetts, for the 8th grade performance grew from 513 in 1999 to 547 in 2007. The achievement average for 4th and 8th grades of Massachusetts and Minnesota are much higher than the achievement average of the United States as a whole (529 at Grade 4 and 508 at Grade 8 in 2007).

TAKEAWAYS AND LESSONS FOR INDIA

1. Our focus in India has remained on inputs (school buildings, teacher training, mid-day meals, etc.) while mostly ignoring outcomes, which is what studies like the TIMSS focus on. Quality education needs to be redefined as students learning with understanding and not merely as recruiting teachers with the requisite qualifications on paper and teaching in good buildings. This, in effect, is what the recent Right to Education (RtE) Act seeks to guarantee.

2. Well-designed assessments like the TIMSS are a strong way of fighting ‘rote learning’ which is probably the cause of a lot of educational problems in India. Learning needs to be seen as something that is useful and continuous and not something that is represented by marks, exams and certificates. While few may dispute that statement, the action required is a shift to exams that test understanding- inspired by studies like the TIMSS.

3. Assessments like the TIMSS help to scientifically diagnose the problem so that an appropriate solution can be found: simply investing in teacher training or development of more curricular material without such research is not useful. Though there is availability of resources in India to conduct studies like this, their importance has not yet been realised possibly because such a study does not seem to directly benefit a large number of students or teachers.

4. India must not only participate in studies like the TIMSS, states must institute their own periodic, low-stakes, sample assessments to regularly monitor the levels of student learning. Two Indian states, Himachal Pradesh and Tamil Nadu are participating in the Programme for International Student Assessment (PISA) – another TIMSS-like study. This is a welcome move. Nevertheless, more well-designed national studies also need to be taken up.

5. Steps need to be taken to encourage research in the areas of student learning and student assessments as a mission of

national importance. Acts like the RtE need to be far more research-based than they currently are. India has an opportunity to focus on research in this area and become a world leader. Though it means building on a negligible base, this area can transform education, and consequently, the nation itself. Can anything be more important?

7 © Educational Initiatives Pvt. Ltd.

EDUCATIONAL INITIATIVES AND TIMSS The work of Educational Initiatives has been inspired in many ways by the TIMSS studies. EI appreciates the way detailed information as well as research techniques are shared in the TIMSS Technical Reports and strives to share its own learning and data in a similar manner. In the absence of much work in India in advanced topics like Item Response Theory, Scale Anchoring, developing knowledge and ability benchmarks, etc., EI has drawn upon parts of TIMSS work in these areas. EI’s large scale assessment studies including the Student Learning in Metros (India 2006), Municipal School Benchmarking (India 2007), Annual Status of Student Learning (Bhutan 2008) and Student Learning Study (India 2009) not only used these techniques, but also used released TIMSS items to benchmark performance with international levels as detailed below.

1. Anchor Items from TIMSS: In some of the EI studies, a few carefully selected items from the TIMSS Released Items have been chosen, translated where required and incorporated in the study. These questions are mostly used without modification, except that the names are changed to local names.

Example: The Student Learning in the Metros Study shows that students from the ‘top’ schools of our metros are performing below average international levels. The performance of the class 4 students is significantly lower than the international average in each one of the 11 questions included from TIMSS. More strikingly, students of class 6 performed only marginally better than the international average for class 4 students.

2. Benchmarking: Scale Anchoring technique was extensively used in the Student Learning Study to analyse and interpret the data. For example, the graph on the right shows that in class 4 Maths 56.5% students in a particular state reached the Low Benchmark, whereas the remaining 43.5% students did not even acquire the knowledge and skills specified by the low benchmark. The state has to therefore ensure that the teacher training programmes enable and equip the teachers to target the low performers. A representation of the benchmark graphs released in the study is given alongside.

The purpose of applying such internationally accepted statistical methods is to highlight the areas of strength and weaknesses in student-learning on which remedial action can be taken.

REFERENCES • Adams, R.J. and Gonzalez, E.J. (1996) “The TIMSS Test Design” in M.O. Martin and D.L. Kelly (eds.), Third International Mathematics

and Science Study (TIMSS) Technical Report, Volume I: Design and Development. Chestnut Hill, MA: Boston College. • Educational Initiatives (2010). Student Learning Study. Status of Student Learning across 18 States of India in Urban and Rural

schools. Ahmedabad: Educational Initiatives. • Harmon, M. and Kelly, D.L. (1996) “Development and Design of the TIMSS Performance Assessment” in M.O. Martin and D.L. Kelly

(eds.), Third International Mathematics and Science Study (TIMSS) Technical Report, Volume I: Design and Development. Chestnut Hill, MA: Boston College

• IEA (2007) TIMSS Impact: The Case of Jordan • http://www.iea.nl/fileadmin/user_upload/administrative_docs/46th_GA/IEA_Studies/TIMSS_Impact/TIMSS_Impact_Jordan.doc • Kavli, Anne-Berit. Impact of TIMSS in Norway. Directorate for Primary and Secondary Education. • http://onlineassociate.net/ppt/Impact-of-TIMSS-in-Norway/ • Mullis, I., Martin, M. and Foy, P. (2008) TMSS 2007, International Mathematics Report. Findings from IEA’s Trends • in International Mathematics and Science Study at the Fourth and Eighth Grades. Chesnut Hill, MA: TIMSS & PIRLS International

Study Center, Lynch School of Education, Boston College • Mullis, Ina et al (2005) TIMSS 2007 Assessment Frameworks. Chesnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch

School of Education, Boston College • http://timss.bc.edu/ • http://www.invalsi.it/ric-int/timss2007/index.php?lang=e • http://www.dest.gov.au/sectors/school_education/publications_resources/schooling_issues_digest/perf_aus_schools/why.htm • http://www.nap.edu/openbook.php?record_id=9605&page=10

8 ASSET is a registered trademark and Mindspark is a trademark of EI

ABOUT EI

This working paper is about the Trends in International Mathematics and Sciences Study (TIMSS) which is an international assessment of student achievement in Maths and Science conducted every four years since 1995. This working paper explains the main characteristics and results provided by TIMSS with a special focus on the Maths results from the latest 2007 study. Good practices based on the TIMSS results and the relevance of these type of studies for India are also discussed. The assessment work of Educational Initiatives (EI) is inspired in parts by studies like TIMSS. EI encourages both research-based studies in general and periodic external student assessment studies specifically. Reports, technical reports, data and items related to TIMSS can be downloaded from the following link: http://timssandpirls.bc.edu/

We are an educational research organization that focuses on learning research through assessments. EI has been started by a group of IIM Ahmedabad alumni with first-hand experience of setting up and running educational institutions. It has been formed with a mission to work towards qualitative improvement in India’s educational system and our vision is “A world where children everywhere are ‘Learning with Understanding’”. PROJECTS OF EI: Andhra Pradesh Randomised Evaluation Study (2004 onwards): Done in partnership with Harvard University, Azim Premji Foundation, World Bank and the Government of Andhra Pradesh, this is a longitudinal study across 8-9 years and covers currently 100,000 elementary school kids and measures the impact of various inputs (e.g., block grants, additional teachers) with outcome-based teacher incentives. Assessment of Student Learning in Sarva Shiksha Abhiyan – RGSM, Chhattisgarh (2008 onwards): The test was developed in Hindi and administered to approximately 3 lakh students in about 1900 schools in 16 districts in Chhattisgarh states. The tests have already been conducted for students of class 3 to 8 for Language and Maths and the report submitted to RGSM. Municipal School Benchmarking Study (2004-2007): Supported by ICICI Bank, this study assessed learning in 35,000 municipal school students from class 2, 4 and 6 across the 6 biggest towns in five states- Gujarat, Andhra Pradesh, Rajasthan, Chhattisgarh and Uttarakhand. UNICEF Learning Assessment Study for Quality Education (2005-2006): assessed mathematics and language acquisition among primary school children in the UNICEF quality package schools in 13 states of India. The tests were standardised across 9 languages and involved very intricate development cycle involving language experts from all over India. Teacher Needs Assessment (2008 onwards): is a census study that has been initiated by the Royal Education Council, Government of Bhutan. In this project all teachers of Bhutan are assessed for their general ability, competence in subject knowledge and pedagogical practices. Bhutan Annual Status Student Learning Study (2008-2009): This study was done in partnership with Royal Education Council and Ministry of Education Bhutan. Nearly 34000 students in classes 4, 6 and 8 are tested for learning in Language, Maths and Science in 424 schools. EI’S PRODUCTS AND LEARNING SOLUTIONS: ASSET: is an objective-type, multiple-choice test for students of Classes 3 to 10. It is a scientifically designed, skill based assessment developed in India for Indian schools. It assesses students’ level of proficiency in the skills and concepts underlying the school syllabus and provides them feedback about their strengths and weaknesses. Know more about ASSET at www.ei-india.com Mindspark: is a computer based self-learning programme that helps the child improve her skills. It allows each student to follow a learning path that is based on her need. Mindspark is currently available for Maths for classes 1-10 in English version. Mindspark can be accessed at www.mindspark.in. Rural Mindspark: Hindi version is currently available on demand for some Maths modules. Contact EI to know more about other language versions and modules.

Some Partners / Clients Google Inc. Government of Andhra Pradesh Michael and Susan Dell Foundation Rajiv Gandhi Shiksha Mission, Chhattisgarh Royal Government of Bhutan Suzlon Foundation WIPRO Applying Thought World Bank UNICEF, India CONTACT DETAILS: Educational Initiatives Pvt. Ltd

Head Office Ahmedabad

Bangalore Office

Delhi Office

Hyderabad Office

Mumbai Office

613-615 JB Towers, Opp. Doordarshan Tower, Drive-in Road, Ahmedabad 380054, INDIA Tel: 079 – 40269696, 40269625 Fax: 079-26841400

No.97, 2nd Floor, Robertson Road, Frazer Town, Bangalore-560005 INDIA Tel: 080 – 41657715, 64567049, 41237162

3rd Floor, Lakshey Deep Plaza, A-252A, Sant Nagar, East of Kailash, New Delhi-110065 INDIA Tel: 011-26462264, 40584952

Flat No 401/A, Shri S.K.Vihar, H.No:3-4-578/1, Beside Sri Chaitanya College, Narayanaguda, Hyderabad-500029, INDIA Tel: 040 – 32419534

Row House Number 1,Sea Coast 2, (Near Cidco Guest House), Kille Gaothan, Belapur, Navi Mumbai-400614 INDIA Tel: 022 - 32686329, 27561430

E-mail us at: assessment@ei-india.com Website: www.ei-india.com September 2010