High-Stakes Language Testing and Young English ... High-Stakes Language Testing and Young English Language

  • View

  • Download

Embed Size (px)

Text of High-Stakes Language Testing and Young English ... High-Stakes Language Testing and Young English...

  • High-Stakes Language Testing and Young English Language Learners: Exploring

    Motivation and L2 Test Validity in South Korea


    John F. Haggerty

    A thesis submitted to the Faculty of Graduate Studies and Research

    in partial fulfillment of the requirements for the degree of

    Master of Arts

    Applied Linguistics

    Carleton University

    Ottawa, Ontario

    December, 2010

    © John F. Haggerty

  • 1*1 Library and Archives Canada Published Heritage Branch

    395 Wellington Street Ottawa ON K1A 0N4 Canada

    Bibliotheque et Archives Canada

    Direction du Patrimoine de Pedition

    395, rue Wellington Ottawa ON K1A 0N4 Canada

    Your file Votre reference ISBN: 978-0-494-79568-2 Our file Notre reference ISBN: 978-0-494-79568-2


    The author has granted a non- exclusive license allowing Library and Archives Canada to reproduce, publish, archive, preserve, conserve, communicate to the public by telecommunication or on the Internet, loan, distribute and sell theses worldwide, for commercial or non- commercial purposes, in microform, paper, electronic and/or any other formats.

    L'auteur a accorde une licence non exclusive permettant a la Bibliotheque et Archives Canada de reproduire, publier, archiver, sauvegarder, conserver, transmettre au public par telecommunication ou par I'lnternet, preter, distribuer et vendre des theses partout dans le monde, a des fins commerciales ou autres, sur support microforme, papier, electronique et/ou autres formats.

    The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission.

    L'auteur conserve la propriete du droit d'auteur et des droits moraux qui protege cette these. Ni la these ni des extraits substantiels de celle-ci ne doivent etre imprimes ou autrement reproduits sans son autorisation.

    In compliance with the Canadian Privacy Act some supporting forms may have been removed from this thesis.

    Conformement a la loi canadienne sur la protection de la vie privee, quelques formulaires secondares ont ete enleves de cette these.

    While these forms may be included in the document page count, their removal does not represent any loss of content from the thesis.

    Bien que ces formulaires aient inclus dans la pagination, il n'y aura aucun contenu manquant.



  • Abstract

    This thesis explores the L2 attitudes, beliefs, and self-conceptions of young

    language learners in South Korea, based on six learner-background characteristics, in an

    attempt better understand their L2 motivational patterns. In addition to considering

    gender, educational level, and self-reported L2 proficiency, this study will also consider

    the time learners spent preparing for L2 tests, the level of L2 test completed (middle-

    school or university-entrance), and the frequency of university-entrance level test

    completion. The inclusion of the last three research questions is an explicit attempt to

    widen conceptions of L2 motivation to more properly include the effects of L2 testing,

    especially in test-driven educational environments. A questionnaire was developed and

    administered to middle school students enrolled in a private language school in Seoul

    (N=341). An exploratory factor analysis (EFA) approach was utilized in order to identify

    factors for further comparison of subgroups. The results indicated that middle school

    students differed significantly on a majority of L2 motivational factors depending on: 1)

    their perceived L2 proficiency level; 2) the amount of time they spent preparing for L2

    tests; and 3) the completion of university-entrance level L2 tests. The implications of

    these findings are discussed in relation to some of the major motivational theories

    advanced in the fields of psychology and second language acquisition (SLA) and the

    potential implications this has for modern notions of validity in language testing. Finally,

    some recommendations are offered to improve the administration of high-level, high-

    stakes tests that affect English language learners (ELLs) around the world.


  • Acknowledgements

    I would like to first thank my thesis advisor, Dr. Janna Fox, for her invaluable

    guidance in navigating the troubled waters of field research in a foreign country. I

    benefitted tremendously from her wealth of experience in language testing and her wise

    words of encouragement when things did not go quite as I had planned. She was often the

    only sympathetic ear I could find once my closest friends grew weary of my ramblings.

    Without her, I am quite sure I could not have completed this research. I would also like to

    thank my graduate studies supervisor, Dr. Devon Woods, for having the faith to allow me

    to do off-campus research and for inspiring me to follow my heart and pursue research

    where I felt it was necessary most. His dogged pursuit to uncover what lies beneath our

    beliefs, assumptions, and knowledge has had a far greater impact on this inquiry than he

    may have realized. I am also greatly indebted to Joan Grant for her excellent

    administrative support and for fully explaining all the options I had available to me. A

    special thanks to Karen Rishel for getting necessary paperwork to the correct people on

    campus when I could not be there myself. I would also like to thank Young Jin for not

    only being an invaluable translator, but also a dear friend whose insight into education

    and motivation in Korea helped me to better organize and understand my research. I

    would also like to thank the Kiervin family for opening up their home and providing

    support during my thesis defence in Ottawa, Canada. Finally, this research would not be

    possible without the support of Seong Hye. When I started this study she was my fiancee,

    but when I finished she was my wife. I think this speaks the loudest to not just how

    important she was in this study, but as it turns out, in my life as well.


  • Table of Contents

    Abstract ii Acknowledgements iii Table of Contents iv List of Tables vi List of Figures and Illustrations vii List of Symbols, Abbreviations and Nomenclature viii


    CHAPTER TWO: VALIDITY IN LANGUAGE TESTING 7 2.1 The 'Early' Years: Criterion-related and Content Validity 9 2.2 The Ascendance of Construct Validity 12 2.3 Positivism, Psychometrics, and Validity 13 2.4 The Messick Model 15 2.5 The Bachman and Palmer Model 18 2.6 Kane's Argument-Based Approach 20 2.7 McNamara and the Social Context of Language Testing 22 2.8 Consequential Validity: A Role for Motivation? 24

    CHAPTER THREE: MOTIVATION RESEARCH 27 3.1 Motivation in Psychology 29

    3.1.1 Achievement Motivation Theory 30 3.1.2 Goal-Setting and Goal-Orientation Theory 31 3.1.3 Attribution Theory 32 3.1.4 Self-Efficacy 33 3.1.5 Self-Worth Theory 34 3.1.6 Self-Determination Theory 35 3.1.7 Weiner's Social Motivation 36

    3.2 Motivation in SLA 37 3.2.1 Gardner and Lambert's 'Integrativeness' 37 3.2.2 1990 to the Present Day: Rethinking and Revisualizing L2 Motivation 38 3.2.3 Dornyei's L2 Motivational Self-System 40 3.2.4 A Greater Appreciation of Young ELLs in L2 Motivation 43 3.2.5 A Greater Role for Language Testing in L2 Motivation Research 45

    3.3 Development of L2 Motivational Constructs 46 3.3.1 Dornyei -Inspired Constructs 47 3.3.2 Pilot Study Constructs 50

    3.4 Learner-background Characteristics (Variables) 51 3.4.1 Gender 51 3.4.2 Education Level 52 3.4.3 Self-reported L2 Proficiency 53 3.4.4 L2 Test Preparation 54 3.4.5 Level and Frequency of L2 Test Completion 54


  • CHAPTER FOUR: RESEARCH METHOD 56 4.1 Participants 57 4.2 Setting 57 4.3 Instrument 58 4.4 Procedure 59 4.5 Data Analysis 60

    CHAPTER FIVE: RESULTS 64 5.1 Exploratory Factor Analysis Results 64

    5.1.1 Factor 1: Ought-To L2 Self + Instrumentality (prevention) 67 5.1.2 Factor 2: Attitudes to L2 Community 68 5.1.3 Factor 3: Attitudes to L2 Learning 69 5.1.4 Factor 4: Ideal L2 Self 69 5.1.5 Factor 5: Attitudes to L2 Testing 70 5.1.6 Theoretical Constructs Not Replicated by EFA Procedure 70

    5.2 Factor Scores 71 5.2.1 Correlation among Factor Score Variables 71 5.2.2 Results for Gender and Education Level 72 5.2.3 Results for Self-Reported L2 Proficiency 73 5.2.4 Results for L2 Test Preparation 74 5.2.5 Results for Level and Frequency of L2 Test Completion 77

    CHAPTER SIX: DISCUSSION 80 6.1 L2 Motivational Factors Identified 80 6.2 Gender and Education Level 82 6.3 Self-Reported L2 Proficiency 84 6.4 L2 Test Preparation 86 6.5 Level and Frequency of L2 Test Completion 87

    CHAPTER SEVEN: CONCLUSIONS AND RECOMMENDATIONS 90 7.1 Limitations 90 7.2 Conclusions about L2 Motivation 91 7.3 Conclusions about Validity in Language Testing 95



  • List of Tables

    Table 1: Theoretical Constructs Utilized for Present Study 49

    Table 2: Pattern Matrix for 5-Factor Solution (EFA) 66

    Table 3: Pearson Product-Moment Correlations for Factor Score Variables 72

    Table 5: Factor Scores and Self-Reported L2 Proficiency Level (N=329)