TEST OF WRITTEN ENGLISH - Educational Testing … TOEFL TEST OF WRITTEN ENGLISH GUIDE Fifth Edition The TOEFL Test of Written English Guide has been prepared for deans, admissions

G u i d e

TEST OFWRITTENENGLISH

FI F T H ED I T I O N

G u i d e

TEST OFWRITTENENGLISH

Listening.Learning.

Leading.

®

Educational Testing Service is an Equal Opportunity/Affirmative Action Employer.Copyright © 2004 by Educational Testing Service. All rights reserved.

EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, TOEFL, the TOEFL logo, TSE, TWE, and the TWElogo are registered trademarks of Educational Testing Service.

No part of this guide may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopy, recording, or any information storage and retrieval system, withoutpermission in writing from the publisher. Violators will be prosecuted in accordance with both US andinternational copyright laws.

1

TOEFLTEST OF WRITTEN

ENGLISH GUIDEFifth Edition

The TOEFL Test of Written English Guide has been prepared for deans, admissionsofficers, and graduate department faculty; administrators of scholarship programs; ESL teachers;foreign student advisers; and others responsible for interpreting Test of Written English (TWE )scores. In addition to providing information about the interpretation of TWE scores, the Guidedescribes the TWE test and scoring guide, and includes samples of actual TWE essay questionsand TWE essays.

This edition of the Guide contains updated information about the TWE program and TWE-related research developments. Statistical tables show the frequency distributions of TWEscores for ranges of TOEFL scores, TWE and TOEFL section and total score correlations bygeographic region, TWE score means for examinees who are applicants to undergraduate andgraduate programs, and TWE score means for graduate school applicants classified by department.

Your suggestions for improving the usefulness of the TOEFL Test of Written EnglishGuide are most welcome. Please feel free to send any comments to us. Additional copies of theGuide are available upon request by contacting ETS Order Services by e-mail([email protected]) or by telephone (1-609-771-7243).

Test of Written EnglishTOEFL Programs and ServicesPO Box 6155Princeton, NJ 08541-6155USA

®

®

2

Page

TWE Guide Highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Overview of the TWE Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

The TOEFL Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

TWE Developmental Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

TWE Item Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

The TWE Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Test Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Development of the TWE Scoring Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6TWE Essay Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

TWE Essay Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Reader Qualifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Scoring Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

TWE Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Guidelines for Using TWE Test Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Statistical Characteristics of the TWE Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Standard Error of Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Performance of TWE Reference Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

TWE Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Appendixes

A. TWE Scoring Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21B. TWE Test Book Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22C. Sample TWE Essay Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23D. Sample TWE Essays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

TABLE OF CONTENTS

3

Page

Table 1 Reader Reliabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 2 Frequency Distribution of TWE Scores for TOEFL TotalScaled Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Table 3 Correlations Between TOEFL and TWE Scores . . . . . . . . . . . . . . . . . . . . . 12

Table 4 Frequency Distribution of TWE Scores for All Examinees . . . . . . . . . . . . 12

Table 5 TWE Score Means –– All Examinees Classified by NativeLanguage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 6 TWE Score Means –– All ExamineesClassified by Country . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Table 7 TWE Score Means –– Applicants to Undergraduate andGraduate Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Table 8 TWE Score Means –– Graduate School Applicants Classifiedby Department . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

LIST OF TABLES

4

POINTS OF SPECIAL INTEREST TOTWE SCORE USERS

The TWE test...

is a direct measure of the writing ability of nonnative speakers of English

is used by some TOEFL score users to obtain information about the writtenEnglish language proficiency of their foreign student applicants

is administered with the TOEFL test

is offered at no additional cost to TOEFL examinees

is supported by an ongoing program of research

is scored independently under standardized conditions by two or morequalified essay raters

✔

✔

✔

✔

✔

✔

5

Overview of the TWE TestThe Test of Written English (TWE®) is the essay componentof the Test of English as a Foreign Language™ (TOEFL®),the multiple-choice test used by more than 4,400 institutionsto evaluate the English proficiency of applicants whose nativelanguage is not English. As a direct, productive skills test, theTWE test is intended to complement TOEFL Section 2(Structure and Written Expression). The TWE test isholistically scored, using a criterion-referenced scale toprovide information about an examinee’s ability to generateand organize ideas on paper, to support those ideas withevidence or examples, and to use the conventions of standardwritten English.

Introduced in July 1986, the TWE test is currently (2004)offered as a required component of the TOEFL test.

The TOEFL TestFirst administered in 1963-64, the TOEFL test is primarilyintended to evaluate the English proficiency of nonnativespeakers who wish to study in colleges or universities inEnglish-speaking countries. Section 1 (Listening Compre-hension) measures the ability to recognize and understandEnglish as it is spoken in North America. Section 2 (Structureand Written Expression) measures the ability to recognizeselected structural and grammatical points in English. Section3 (Reading Comprehension) measures the ability to read andunderstand short passages similar in topic and style to thosethat students are likely to encounter in North American univer-sities and colleges.

TWE Developmental ResearchEarly TOEFL research studies (Pike, 1976; Pitcher & Ra,1967) showed that performance on the TOEFL Structure andWritten Expression section correlated positively with scoreson direct measures of writing ability. However, some TOEFLscore users expressed concern about the validity of Section 2as a measure of a nonnative speaker’s ability to write foracademic purposes in English. The perception among manygraduate faculty was that there might be little actualrelationship between the recognition of correct writtenexpression, as measured by Section 2, and the production ofan organized essay or report (Angelis, 1982).

In surveys conducted in a number of studies (Angelis,1982; Hale and Hinofotis, 1981; Kane, 1983) college anduniversity administrators and faculty, as well as English as asecond language (ESL) teachers, requested the developmentof an essay test to assess directly the academic writing skillsof foreign students.

As an initial step in exploring the development of an essaycomponent for the TOEFL test, Bridgeman and Carlson(1983) surveyed faculty in undergraduate and graduatedepartments with large numbers of foreign students at 34major universities. The purpose of their study was to identifythe types of academic writing tasks and skills required ofcollege and university students.

Following the identification of appropriate writing tasksand skills, a validation study investigating the relationship ofTOEFL scores to writing performance was conducted(Carlson, Bridgeman, Camp, and Waanders, 1985). It wasfound that, while scores on varied writing samples and TOEFLscores were moderately related, the writing samples and theTOEFL test reliably measured some aspect of Englishlanguage proficiency not assessed by the other. The researchersalso found that holistic scores, discourse-level scores, andsentence-level scores of the writing samples were all closelyrelated. Finally, the researchers reported that correlations ofscores were as high across writing topic types as within thetopic types, suggesting that the different topic types used inthe study comparably assessed overall competency inacademic composition.

These research studies provided the foundation for thedevelopment of the Test of Written English. Early TWEtopics were based on the types of writing tasks identified inthe Bridgeman and Carlson (1983) study. Based on the findingsof the validation study, a single holistic score is reported forthe TWE test. This score is derived from a criterion-referencedscoring guide that encompasses relevant aspects ofcommunicative competence.

TOEFL TEST OF WRITTEN ENGLISH GUIDE

6

The TWE CommitteeTests developed by Educational Testing Service must meetrequirements for fair and accurate testing, as outlined in theETS Standards for Quality and Fairness.

In accordance with these ETS standards, in July 1985 theTOEFL program established the TWE Core Reader Group,later known as the TWE Committee. The committee was aconsultant group of college and university faculty andadministrators who were experienced with the intended testpopulation, current writing assessment theory and practice,pedagogy, and large-scale essay testing management. Thecommittee developed the TWE essay questions, evaluatedtheir pretest performance using the TWE scoring criteria, andapproved the items for administration. Members alsoparticipated in TWE essay readings throughout the year.

As a consequence of the move to a computer-deliveredTOEFL test in 1998, the TWE Committee has been disbanded.Test content issues are now handled by the Committee ofExaminers.

Test SpecificationsTest specifications outline what a test purports to measureand how it measures the identified skills. The purpose ofTWE is to give examinees whose native language is notEnglish an opportunity to demonstrate their ability to expressideas in acceptable written English in response to an assignedtopic. Topics are designed to be fair, accessible, andappropriate to all members of the international TOEFLpopulation. Each essay is judged according to lexical andsyntactic standards of English and the effectiveness withwhich the examinee, organizes, develops, and expresses ideasin writing. A criterion-referenced scoring guide ensures thata level of consistency in scoring is maintained from oneadministration to another.

Development of the TWE Scoring GuideThe TWE Scoring Guide (see Appendix A) was developed toprovide concise descriptions of the general characteristics ofessays at each of six points on the criterion-referenced scale.The scoring guide also serves to maintain consistent scoringstandards and high interrater reliability within and acrossadministrations. As an initial step in developing theseguidelines, a specialist in applied linguistics examined 200essays from the Carlson et al. (1985) study — analyzing therhetorical, syntactic, and communicative characteristics at

each of the six points — and wrote brief descriptions of thestrengths and weaknesses of the group of essays at each level.This analysis, the TWE Committee’s analysis of pretest essays,and elements of scoring guides used by other large-scaleessay reading programs at ETS and elsewhere were used todevelop the TWE Scoring Guide.

The guide was validated on the aforementioned researchessays and on pretest essays before being used to score thefirst TWE essays in July 1986. To maintain consistency in theinterpretation and application of the guide, before each TWEessay reading TWE essay reading managers review a sampleof essays that are anchored to the original essays from thefirst TWE administration. This review helps to ensure that agiven score will consistently represent the same proficiencylevel across test administrations.

In September 1989 the TWE Scoring Guide was revisedby a committee of TWE essay reading managers who wereasked to refine it while maintaining the comparability ofscores assigned at previous TWE essay readings. The revisionswere based on feedback from TWE essay readers, essayreading managers, and the TWE Committee.

The primary purpose of the revision was to make theguide a more easily internalized tool for scoring TWE essaysduring a reading. After completing the revisions, the committeeof essay reading managers rescored essays from the firstTWE administration to see that no shift in scoring occurred.

The revised scoring guide was reviewed, used to scorepretest essays, and approved by the TWE Committee inFebruary 1990. It was introduced at the March 1990 TWEreading.

TWE Essay QuestionsThe TWE test requires examinees to produce an essay inresponse to a brief question or topic. The writing taskspresented in TWE topics have been identified by research astypical of those required for college and university coursework. The topics and tasks are designed to give examineesthe opportunity to develop and organize ideas and to expressthose ideas in lexically and syntactically appropriate English.Because TWE aims to measure composition skills ratherthan reading comprehension skills, topics are brief, simplyworded, and not based on reading passages. Samples ofTWE essay questions used in past administrations are includedin Appendix C.

TWE ITEM DEVELOPMENT

7

TWE questions are developed in two stages. The TWEtest developers write, review, revise, and approve essay topics.In developing topics, the following criteria are considered:

• the topic (prompt) should be accessible to TOEFLexaminees from a variety of linguistic, cultural, andeducational backgrounds

• the task to be performed by examinees should beexplicitly stated

• the wording of the prompt should be clear andunambiguous

• the prompt should allow examinees to plan, organize,and write their essays in 30 minutes

Once approved, each TWE question is further reviewedby ETS test developers and fairness reviewers to ensure thatit is not biased, inflammatory, or misleading, and that it doesnot unfairly advantage or disadvantage any subgroup withinthe TOEFL population.

Reader QualificationsReaders for the TWE test are primarily English and ESLwriting specialists affiliated with accredited colleges,universities, and secondary schools in the United States andCanada. To be invited to serve as a reader, an individual musthave read successfully for at least one other ETS program orqualify at a TWE reader training session.

TWE reader training sessions are conducted as needed.During these sessions, potential readers receive intensivetraining in holistic scoring procedures using the TWE ScoringGuide and TWE essays. At the conclusion of the training,participants independently rate 50 TWE essays that werescored at an operational reading. To qualify as a TWE rater,participants must demonstrate their ability to evaluate TWEessays reliably and accurately using the TWE Scoring Guide.

Scoring ProceduresAll TWE essay readings are conducted in a central locationunder standardized procedures to ensure the accuracy andreliability of the essay scores.

TWE essay reading managers are English or ESL facultywho represent the most capable and experienced readers. Inpreparation for a TWE scoring session, the essay readingmanagers prepare packets of sample essays illustrating thesix points on the scoring guide. Readers score and discussthese sets of sample essays with the essay reading managersprior to and throughout the reading to maintain scoringaccuracy.

Small groups of readers work under the direct supervisionof reading managers, who monitor the performance of eachscorer throughout the reading. Each batch of essays isscrambled between the first and second readings to ensurethat readers are not unduly influenced by the sequence ofessays.

Each essay is scored by two readers working independently.The score assigned to an essay is derived by averaging thetwo independent ratings or, in the case of a discrepancy ofmore than one point, by the adjudication of the score by areading manager. For example, if the first reader assigns ascore of 5 to an essay and the second reader also assigns it ascore of 5, 5 is the score reported for that essay. If the firstreader assigns a score of 5 and the second reader assigns ascore of 4, the two scores are averaged and a score of 4.5 isreported. However, if the first reader assigns a score of 5 to anessay and the second reader assigns it a 3, the scores areconsidered discrepant. In this case, a reading manager scoresthe essay to adjudicate the score.

Using the scenario above of first and second reader scoresof 3 and 5, if the reading manager assigns a score of 4, thethree scores are averaged and a score of 4 is reported. However,if the reading manager assigns a score of 5, the discrepantscore of 3 is discarded and a score of 5 is reported. To date,more than 2,500,000 TWE essays have been scored, resultingin some 5,000,000 readings. Discrepancy rates for the TWEreadings have been extremely low, usually ranging from 1 to2 percent per reading.

TWE ESSAY READINGS

8

Six levels of writing proficiency are reported for the TWE test.TWE scores range from 6 to 1 (see Appendix A). A scorebetween two points on the scale (5.5, 4.5, 3.5, 2.5, 1.5) can alsobe reported (see “Scoring Procedures” on page 7). The followingcodes and explanations may also appear on TWE score reports:

1NR Examinee did not write an essay.OFF Examinee did not write on the assigned topic.

Because language proficiency can change considerably ina relatively short period, the TOEFL office will not reportTWE scores that are more than two years old. Therefore,individually identifiable TWE scores are retained in a databasefor only two years from the date of the test. After two years,information that could be used to identify an individual isremoved from the database. Information such as score dataand essays that may be used for research or statistical purposesmay be retained indefinitely; however, this information doesnot include any individual examinee identification.

TWE SCORES

TWE scores and all information that could identify anexaminee are strictly confidential. An examinee's officialTWE score report will be sent only to those institutions oragencies designated by the examinee on the answer sheet onthe day of the test, or on a Score Report Request Formsubmitted by the examinee at a later date, or by other writtenauthorization from the examinee.

Examinees receive their test results on a form titledExaminee’s Score Record. These are not official TOEFLscore reports and should not be accepted by institutions. If anexaminee submits a TWE score to an institution or agency andthere is a discrepancy between that score and the official TWEscore recorded at ETS, ETS will report the official score to theinstitution or agency. Examinees are advised of this policy inthe TOEFL Information and Registration Bulletin.

A TWE rescoring service is available to examinees whowould like to have their essays rescored. Further informationon this rescoring process can also be found in the Bulletin.

GUIDELINES FOR USING TWE TEST SCORES

An institution that uses TWE scores should consider certainfactors in evaluating an individual’s performance on the testand in determining appropriate TWE score requirements.The following guidelines are presented to assist institutionsin arriving at reasonable decisions.

1. Use the TWE score as an indication of English writingproficiency only and in conjunction with other indicatorsof language proficiency, such as TOEFL section and totalscores. Do not use the TWE score to predict academicperformance.

2. Base the evaluation of an applicant’s readiness to beginacademic work on all available relevant information andrecognize that the TWE score is only one indicator ofacademic readiness. The TWE test provides informationabout an applicant’s ability to compose academic English.Like TOEFL, TWE is not designed to provide informationabout scholastic aptitude, motivation, language learningaptitude, field specific knowledge, or cultural adaptability.

3. Consider the kinds and levels of English writing proficiencyrequired at different levels of study in different academicdisciplines. Also consider the resources available at theinstitution for improving the English writing proficiencyof students for whom English is not the native language.

4. Consider that examinee scores are based on a single 30-minute essay that represents a first-draft writing sample.

5. Use the TWE Scoring Guide and writing samplesillustrating the guide as a basis for score interpretation(see Appendixes A and D). Score users should bear inmind that a TWE score level represents a range ofproficiency and is not a fixed point.

6. Avoid decisions based on small score differences. Smallscore differences (i.e., differences less than approximatelytwo times the standard error of measurement) should notbe used to make distinctions among examinees. Basedupon the average standard error of measurement for thepast 10 TWE administrations, distinctions amongindividual examinees should not be made unless theirTWE scores are at least one point apart.

7. Conduct a local validity study to assure that the TWEscores required by the institution are appropriate.

As part of its general responsibility for the tests it produces,the TOEFL program is concerned about the interpretation anduse of TWE test scores by recipient institutions. The TOEFLoffice encourages individual institutions to request its assistancewith any questions related to the proper use of TWE scores.

9

STATISTICAL CHARACTERISTICS OF THE TWE TEST

Table 1Reader Reliabilities

(Based on scores assigned to 213,221 essays in the 10 TWE administrations from October 2001 through November 2003)

Correlation SEM2

Admin. TWE TWE Discrepancy 1st & 2nd Indiv. Score Date N Mean S.D. Rate1 Readers Alpha Scores Diffs.

Oct. 2001 30,912 3.81 0.74 0.006 0.81 0.89 0.24 0.34Jan. 2002 25,956 3.73 0.74 0.007 0.79 0.88 0.26 0.36May 2002 31,683 3.90 0.70 0.006 0.79 0.88 0.24 0.34Aug. 2002 19,821 3.74 0.69 0.006 0.76 0.87 0.25 0.36Sep. 2002 14,446 3.83 0.69 0.007 0.76 0.86 0.26 0.36Oct. 2002 25,114 3.84 0.69 0.005 0.81 0.89 0.23 0.32Jan. 2003 24,099 3.72 0.75 0.010 0.78 0.88 0.26 0.37May 2003 8,761 3.67 0.91 0.008 0.85 0.92 0.26 0.37Oct. 2003 19,895 3.90 0.81 0.007 0.82 0.90 0.26 0.36Nov. 2003 12,535 3.80 0.74 0.005 0.81 0.90 0.24 0.34

second measure reported is coefficient alpha, which providesan estimate of the internal consistency of the final scoresbased upon two readers per essay. Because each reportedTWE score is the average of two separate ratings, the reportedTWE scores are more reliable than the individual ratings.Therefore, coefficient alpha is generally higher than the simplecorrelation between readers, except in those cases where thecorrelation is equal to 0 or 1. (If there were perfect agreementon each essay across all raters, coefficient alpha would equal1.0; if there were no relationship between the scores given bydifferent raters, coefficient alpha would be 0.0.)

Table 1 contains summary statistics and interrater reliabilitystatistics for the 10 TWE administrations from October 2001through November 2003. The interrater correlations andcoefficients alpha indicate that reader reliability is acceptablyhigh, with correlations between first and second readersranging from .76 to .85, and the values for coefficient alpharanging from .86 to .92.

Table 1 also shows the reader discrepancy rate for each ofthe 10 TWE administrations. This value is simply theproportion of essays for which the scores of the two readersdiffered by two or more points. These discrepancy rates arequite low, equal to or less than 1 percent. (Because all essayswith ratings that differed by two or more points were given athird reading, the discrepancy rates also reflect the proportionsof essays that received a third reading.)

ReliabilityThe reliability of a test is the extent to which it yieldsconsistent results. A test is considered reliable if it yieldssimilar scores across different forms of the test, differentadministrations, and, in the case of subjectively scoredmeasures, different raters.

There are several ways to estimate the reliability of a test,each focusing on a different source of measurement error.The reliability of the TWE test has been evaluated byexamining interrater reliability, that is, the extent to whichreaders agree on the ratings assigned to each essay. To date, ithas not been feasible to assess alternate-form and test-retestreliability, which focus on variations in test scores that resultfrom changes in the individual or changes in test contentfrom one testing situation to another. To do so, it would benecessary to give a relatively large random sample ofexaminees two different forms of the test (alternate-formreliability) or the same test on two different occasions (test-retest reliability). However, the test development proceduresthat are employed to ensure TWE content validity (discussedlater in this section) would be expected to contribute toalternate-form reliability.

Two measures of interrater reliability are reported for theTWE test. The first measure reported is the Pearson product-moment correlation between first and second readers, whichreflects the overall agreement (across all examinees and allraters) of the pairs of readers who scored each essay. The

2 Standard errors of measurement listed here are based on theextent of interrater agreement and do not take into account othersources of error, such as differences between test forms.Therefore, these values probably underestimate the actual errorsof measurement.

1 Proportions of papers in which the two readers differed by two ormore points. (When readers differed by two or more points, theessay was adjudicated by a third reader.)

10

Standard Error of MeasurementAny test score is only an estimate of an examinee’s knowledgeor ability, and an examinee’s test score might have beensomewhat different if the examinee had taken a differentversion of the test, or if the test had been scored by a differentgroup of readers. If it were possible to have someone take allthe editions of the test that could ever be made, and havethose tests scored by every reader who could ever score thetest, the average score over all those test forms and readerspresumably would be a completely accurate measure of theexaminee’s knowledge or ability. This hypothetical score isoften referred to as the “true score.” Any difference betweenthis true score and the score that is actually obtained on agiven test is considered to be measurement error.

Because an examinee’s hypothetical true score on a test isobviously unknown, it is impossible to know exactly howlarge the measurement error is for any individual examinee.However, it is possible statistically to estimate the averagemeasurement error for a large group of examinees, basedupon the test’s standard deviation and reliability. This statisticis called the Standard Error of Measurement (SEM).

The last two columns in Table 1 show the standard errorsof measurement for individual scores and for score differenceson the TWE test. The standard errors of measurement that arereported here are estimates of the average differences betweenobtained scores and the theoretical true scores that wouldhave been obtained if each examinee’s performance on asingle test form had been scored by all possible readers. Forthe 10 test administrations shown in the table, the averagestandard error of measurement was approximately .25 forindividual scores and .35 for score differences.

The standard error of measurement can be helpful in theinterpretation of test scores. Approximately 95 percent of allexaminees are expected to obtain scores within 1.96 standarderrors of measurement from their true scores andapproximately 90 percent are expected to obtain scores within1.64 standard errors of measurement. For example, in theNovember 2003 administration (with SEM = .24), about 90percent of examinees with true scores of 3.0 are expectedto obtain a score within the range of 2.61 [i.e.,(3.0) – 1.64*(.24)] and 3.39 [i.e., (3.0) + 1.64*(.24)].

When the scores of two examinees are compared, thedifference between the scores will be affected by errors ofmeasurement in each of the scores. Thus, the standard errorsof measurement for score differences are larger than thecorresponding standard errors of measurement for individualscores (about 1.4 times as large). In approximately 95 percentof all cases, the difference between obtained scores is expectedto be within 1.96 standard errors above or below the differencebetween the examinees’ true scores; in approximately 80percent of all cases, the difference between obtained scores isexpected to be within 1.28 standard errors above or below thetrue difference. This information allows the test user to evaluatethe probability that individuals with different obtained TWE

scores actually differ in their true scores. For example, in theNovember 2003 administration (with SEM = .34), among allpairs of examinees with the same true score, about two-thirdsof examinees' TWE scores are expected to differ from eachother by one standard error of measurement (.34).

ValidityBeyond being reliable, a test should be valid; that is, it shouldactually measure what it is intended to measure. It is generallyrecognized that validity refers to the usefulness of inferencesmade from a test score. The process of validation is necessarilyan ongoing one, especially in the area of written composition,where theorists and researchers are still in the process ofdefining the construct.

To support the inferences made from test scores, validationshould include several types of evidence. The nature of thatevidence should depend upon the uses to be made of the test.The TWE test is used to make inferences about an examinee’sability to compose academically appropriate written English.

Two types of validity evidence are available for the TWEtest: (1) construct-related evidence and (2) content-relatedevidence. Construct-related evidence refers to the extent towhich the test actually measures the particular construct ofinterest, in this case, English-language writing ability. Content-related evidence refers to the extent to which the test providesan adequate and representative sample of the particular contentdomain that the test is designed to measure.

Construct-related Evidence. One source of construct-related evidence for the validity of the TWE test is therelationship between TWE scores and TOEFL scaled scores.Research suggests that skills such as those intended to bemeasured by both the TOEFL and TWE tests are part of amore general construct of English language proficiency (Oller,1979). Therefore, in general, examinees who demonstratehigh ability on TOEFL would not be expected to performpoorly on TWE, and examinees who perform poorly onTOEFL would not be expected to perform well on TWE.

This expectation is supported by the data collected overseveral TWE administrations. Table 2 displays the frequencydistributions of TWE scores for five different TOEFL scoreranges over 10 administrations.

As the data in Table 2 indicate, across the 10 TWEadministrations from October 2001 through November 2003it was rare for examinees to obtain either very high scores onthe TOEFL test and low scores on the TWE test or very lowscores on TOEFL and high scores on TWE. It should bepointed out, however, that the data in Table 2 do not suggestthat TOEFL scores should be used as predictors of TWEscores.

Although there are theoretical grounds for expecting apositive relationship between TOEFL and TWE scores, therewould be no point in administering the TWE test to examineesif it did not measure an aspect of English language proficiencydistinct from what is already measured by TOEFL.

11

Table 2Frequency Distribution of TWE Scores for TOEFL Total Scaled Scores

(Based on 213,935 examinees who took the TWE test from October 2001 through November 2003)

TOEFL Scores TOEFL Scores TOEFL Scores TOEFL Scores Between 477 Between 527 Between 577 TOEFL Scores Below 477 and 523 and 573 and 623 Above 623

6.0 2 0.0+ 11 0.01 75 0.04 372 0.17 649 0.305.5 6 0.0+ 44 0.02 233 0.11 919 0.43 1,072 0.505.0 70 0.03 590 0.28 3,345 1.56 8,867 4.14 6,063 2.834.5 168 0.08 1,047 0.49 4,314 2.02 7,761 3.63 3,594 1.684.0 3,305 1.54 13,973 6.53 34,450 16.10 38,254 17.88 10,420 4.873.5 2,029 0.95 5,260 2.46 7,938 3.71 5,442 2.54 930 0.433.0 8,792 4.11 12,951 6.05 12,193 5.70 5,610 2.62 865 0.402.5 2,161 1.01 1,531 0.72 924 0.43 337 0.16 48 0.022.0 3,495 1.63 1,360 0.64 544 0.25 138 0.06 33 0.021.5 708 0.33 86 0.04 23 0.01 9 0.0+ 2 0.0+1.0 815 0.38 75 0.04 25 0.01 6 0.0+ 1 0.0+

Thus, the correlations between TWE scores and TOEFLscaled scores should be high enough to suggest that TWE ismeasuring the appropriate construct, but low enough to supportthe conclusion that the test also measures abilities that aredistinct from those measured by TOEFL. The extent to whichTWE scores are independent of TOEFL scores is an indicationof the extent to which the TWE test measures a distinct skillor skills.

Table 3 presents the correlations of TWE scores withTOEFL scaled scores for examinees at the 10 administrations.The correlations between the TOEFL total scores and TWEscores range from .61 to .68, suggesting that the productivewriting abilities assessed by TWE are somewhat distinctfrom the proficiency skills measured by the multiple-choiceitems of the TOEFL test.

Table 3 also shows the correlations of TWE scores witheach of the three TOEFL section scores. Construct validitywould be supported by higher correlations of TWE scoreswith TOEFL Section 2 (Structure and Written Expression)than with Section 1 (Listening Comprehension) or Section 3(Reading Comprehension) scores.

Content-related Evidence. As a test of the ability tocompose in standard written English, TWE uses writingtasks similar to those required of college and universitystudents in North America. As noted earlier, TWE items andprompts are developed to meet detailed specifications thatencompass widely recognized components of written languagefacility. Thus, each TWE item is constructed by subject-matter experts to assess the various factors that are generallyconsidered crucial components of written academic English.

Finally, the actual scoring of TWE essays is done byqualified readers who have experience teaching Englishwriting to native and nonnative speakers of English. TheTWE readers are guided in their ratings by the TWE ScoringGuide and the standardized training and scoring proceduresused at each TWE essay reading.

TWE Score N Percent N Percent N Percent N Percent N Percent

Performance of TWE Reference GroupsTable 4 presents the overall frequency distribution of TWEscores based on the 10 administrations from October 2001through November 2003.

Table 5 lists the mean TWE scores for examinees tested atthe 10 administrations, classified by native language. Table 6lists the mean TWE scores for examinees classified by nativecountry. These tables may be useful in comparing the testperformance of a particular student with the averageperformance of other examinees who are from the samecountry or who speak the same native language.

It is important to point out that the data do not permit anygeneralizations about differences in the English writingproficiency of the various national and language groups. Thetables are based simply on the performance of those examineeswho have taken the TWE test. Because different selectivefactors may operate in different parts of the world to determinewho takes the test, the samples on which the tables are basedare not necessarily representative of the student populationsfrom which the samples came. In some countries, for example,virtually any high school, university, or graduate student whoaspires to study in North America may take the test. In othercountries, government regulations permit only graduatestudents in particular areas of specialization, depending onnational interests, to do so.

12

Table 4Frequency Distribution of TWE Scores for All Examinees


Percentile CumulativeTWE Score N Percent Rank Percent

6.0 1,109 0.52 99.74 0.525.5 2,274 1.06 98.95 1.585.0 18,935 8.85 94.00 10.434.5 16,884 7.89 85.63 18.324.0 100,402 46.93 58.22 65.253.5 21,599 10.10 29.70 75.353.0 40,411 18.89 15.21 94.242.5 5,001 2.34 4.59 96.582.0 5,570 2.60 2.12 99.181.5 828 0.39 0.63 99.571.0 922 0.43 0.22 100

Oct. 2001 30,913 0.62 0.55 0.62 0.57

Jan. 2002 25,956 0.61 0.56 0.59 0.55

May 2002 31,686 0.61 0.56 0.59 0.56

Aug. 2002 19,893 0.62 0.55 0.62 0.56

Sep. 2002 14,452 0.63 0.55 0.62 0.57

Oct. 2002 25,128 0.62 0.55 0.60 0.57

Jan. 2003 24,705 0.65 0.56 0.63 0.59

May 2003 8,772 0.67 0.60 0.66 0.60

Oct. 2003 19,895 0.68 0.58 0.67 0.63

Nov. 2003 12,535 0.64 0.55 0.63 0.58

Total Section 1 Section 2 Section 3Admin. Date N r r r r

Table 3Correlations Between TOEFL and TWE Scores1


TOEFL

1 Correlations have been corrected for unreliability of TWE and TOEFL scores.

13

Table 5TWE Score Means –– All Examinees Classified by Native Language1

(Based on 211,631 examinees2 who took the TWE test from October 2001 through November 2003)

1 Because of the unreliability of statistics based on small samples, means are not reported for groups with fewer than 30 examinees.2 Excludes 2,304 examinees who did not specify native language.

Afrikaans 29 —Albanian 608 3.7Amharic 116 3.7Arabic 1,223 3.4Armenian 13 —Assamese 6 —Azeri 220 3.8

Basque (Euskara) 6 —Belarusian 22 —Bemba 8 —Bengali 357 3.8Berber 3 —Bhili 1 —Bikol 5 —Bulgarian 110 4.2Burmese 1,653 3.3

Catalan (Provencal) 11 —Cebuano (Visayan) 281 4.0Chichewa 68 4.7Chinese 175,331 3.8Chuvash 1 —Czech 65 4.2

Danish 66 4.1Dutch 316 3.9

Efik - Ibibio 38 4.3English 633 4.3Estonian 197 4.2Ewe 153 3.4

Farsi (Persian) 124 3.5Fijian 30 3.9Finnish 37 4.0French 2,305 3.8Fula (Peulh) 53 3.2

Galician 5 —Ganda (Luganda) 1 —Georgian 6 —German 1,500 4.2Greek 572 3.8Guarani 1 —Gujarati 51 4.3

Hausa 39 3.8Hebrew 9 —Hindi 146 4.5Hungarian (Magyar) 49 4.2

Ibo (Igbo) 294 4.4Icelandic 367 4.1Ilocano 22 —Indonesian 324 3.5Italian 540 3.7

Japanese 4,110 3.1Javanese 34 2.9

Kannada (Kanarese) 17 —Kanuri 2 —Kashmiri 4 —Kazakh 33 3.7Khmer 355 3.4Kikuyu 17 —Kinyarwanda 38 4.1Kirundi 15 —Konkani 8 —Korean 1,102 3.6Kurdish 6 —Kurukh (Oraon) 3 —Kusaiean 37 3.6

Lao 27 —Latvian (Lettish) 270 4.0Lingala 28 —Lithuanian 11 —

Luba-Lulua 4 —Luo 12 —

Macedonian 249 3.7Madurese 12 —Malagasy 75 3.4Malay 45 4.0Malayalam 140 4.3Malinke-Bambara-Dyula 143 2.8Maltese 5 —Marathi 11 —Marshallese 101 3.6Mende 12 —Mongolian 120 3.2More 35 3.8

Nepali 45 4.0Norwegian 765 4.0Nyanja 2 —

Oriya 3 —Oromo 7 —

Palauan 287 3.4Panay-Hiligaynon 29 —Pashtu 4 —Pidgin 15 —Polish 783 4.0Ponapean 22 —Portuguese 418 3.5Punjabi 41 4.1

Romanian 354 4.2Russian 900 3.9

Samoan 2 —Santali 1 —Serbo-Croatian 252 3.8Sesotho 6 —Setswana 3 —Shona 20 —Sindhi 42 4.2Sinhala 247 4.4Siswati 16 —Slovak 179 4.1Slovenian 37 4.1Somali 10 —Spanish 5,132 3.8Sundanese 4 —Swahili 93 4.1Swedish 681 4.0

Tagalog 592 4.0Tajik 17 —Tamil 102 4.4Tatar 3 —Telugu 38 4.4Thai 283 3.2Tibetan 30 3.5Tigrinya 273 3.7Tongan 1 —Trukese 35 3.4Tulu 1 —Turkish 648 3.8Turkmen 54 3.9Twi 16 —

Ukrainian 73 3.9Urdu 503 4.3Uzbek 5 —

Vietnamese 3,185 3.5

Wolof 138 3.6

Yapese 31 3.4Yoruba 103 4.3

Zulu 4 —

Language N Mean Language N Mean

14

Country N Mean Country N Mean

Afghanistan 12 —Albania 536 3.7Algeria 18 —American Samoa 3 —Andorra 1 —Angola 47 3.4Anguilla 0 —Antigua and Barbuda 2 —Argentina 541 3.8Armenia 7 —Aruba 125 3.5Australia 21 —Austria 262 4.2Azerbaijan 232 3.8Azores 3 —

Bahamas 1 —Bahrain 125 4.1Bangladesh 39 3.9Barbados 8 —Belarus 50 4.0Belgium 283 4.0Belize 26 —Benin 122 3.4Bermuda 0 —Bhutan 34 4.7Bolivia 363 3.8Bosnia and Herzegovina 97 3.7Botswana 2 —Brazil 332 3.5British Virgin Islands 0 —Brunei Darussalam 16 —Bulgaria 106 4.2Burkina Faso 76 3.7Burundi 18 —

Cambodia 350 3.4Cameroon 302 4.0Canada 153 4.1Cape Verde 5 —Cayman Islands 1 —Central African Republic 2 —Chad 25 —Chile 533 3.8China, People’s Republic of 175,084 3.8Colombia 476 3.8Comoros 0 —Congo - DRC (Formerly Zaire) 159 3.3Congo Republic 10 —Cook Islands 0 —Costa Rica 60 3.8Cote d’Ivoire (Ivory Coast) 53 3.4Croatia 19 —Cuba 10 —Cyprus 919 3.8Czech Republic 66 4.2

Denmark 61 4.1Djibouti 9 —Dominica, Commonwealth of 8 —Dominican Republic 4 —

Ecuador 338 4.0Egypt 20 —El Salvador 116 4.4England 8 —Equatorial Guinea 0 —Eritrea 272 3.7Estonia 229 4.2Ethiopia 125 3.8

Faroe Islands 4 —Fiji 72 4.0Finland 38 4.0France 1,105 4.0French Polynesia 3 —

Gabon 4 —Gambia, The 90 3.9Gaza Strip 68 3.4Georgia 7 —

Table 6TWE Score Means –– All Examinees Classified by Country1

(Based on 212,915 examinees2 who took the TWE test from October 2001 through November 2003)

Germany 1,017 4.2Ghana 29 —Greece 19 —Greenland 1 —Grenada 2 —Guadeloupe 6 —Guam 0 —Guatemala 5 —Guinea 58 2.8Guyana 13 —

Haiti 346 3.6Honduras 215 4.1Hong Kong 146 3.7Hungary 27 —

Iceland 370 4.1India 449 4.5Indonesia 334 3.5Iran 115 3.5Iraq 15 —Ireland 1 —Isle of Man 0 —Israel 14 —Italy 538 3.7

Jamaica 7 —Japan 4,190 3.1Jordan 22 —

Kazakhstan 11 —Kenya 83 4.5Kiribati 1 —Korea (DPR) 21 —Korea (ROK) 1,029 3.6Kuwait 54 3.6Kyrgyzstan 70 4.0

Laos 27 —Latvia 385 4.0Lebanon 44 3.7Lesotho 5 —Liberia 99 3.9Libya 19 —Liechtenstein 2 —Lithuania 11 —Luxembourg 41 4.3

Macau 51 3.7Macedonia, Former Yugoslav

Republic of 268 3.7Madagascar 83 3.4Madeira Islands 2 —Malawi 76 4.7Malaysia 109 4.0Maldives 2 —Mali 178 2.8Malta 7 —Marshall Islands 103 3.5Martinique 16 —Mauritania 7 —Mauritius 5 —Mexico 530 3.5Micronesia, Federated States of 147 3.5Moldova 132 4.0Monaco 3 —Mongolia 106 3.1Montserrat 0 —Morocco 67 3.3Mozambique 31 3.6Myanmar (Burma) 1,673 3.3

Namibia 23 —Nauru 2 —Nepal 29 —Netherlands 24 —Netherlands Antilles 57 4.1New Caledonia 21 —New Zealand 32 4.2Nicaragua 123 4.1Niger 73 3.6Nigeria 628 4.4

1 Because of the unreliability of statistics based on small samples, means are not reported for groups with fewer than 30 examinees.2 Excludes 1,020 examinees who did not specify country.

15

Table 6 (continued)

Niue Island 0 —Northern Ireland 1 —Northern Mariana Islands 8 —Norway 783 4.0

Oman 137 3.9

Pakistan 518 4.2Palau 294 3.3Panama 258 3.7Papua New Guinea 8 —Paraguay 254 3.7Peru 362 3.5Philippines 956 4.0Poland 804 4.0Portugal 197 3.7Puerto Rico 4 —

Qatar 154 3.4

Reunion 8 —Romania 290 4.3Russia 521 3.9Rwanda 130 4.0

Samoa 3 —Sao Tome and Principe 1 —Saudi Arabia 279 3.1Scotland 1 —Senegal 168 3.3Seychelles 0 —Sierra Leone 33 4.2Singapore 10 —Slovakia 180 4.1Slovenia 32 4.2Solomon Islands 4 —Somalia 2 —South Africa 24 —Spain 653 3.9Sri Lanka 305 4.4St. Lucia 1 —St. Vincent and the Grenadines 3 —

1 Because of the unreliability of statistics based on small samples, means are not reported for groups with fewer than 30 examinees.2 Excludes 1,020 examinees who did not specify country.

Country N Mean Country N Mean

Sudan 85 3.4Suriname 68 3.5Swaziland 19 —Sweden 694 4.0Switzerland 314 4.1Syria 92 3.5

Tahiti 26 —Taiwan 294 3.7Tajikistan 68 3.9Tanzania 55 4.2Thailand 287 3.2Togo 212 3.4Tonga 0 —Trinidad and Tobago 53 4.8Tunisia 56 3.3Turkey 291 3.7Turkmenistan 90 3.9Turks and Caicos Islands 1 —Tuvalu 1 —

Uganda 10 —Ukraine 116 3.8United Arab Emirates 1 —United Kingdom 21 —United States of America 103 4.1Uruguay 268 3.9Uzbekistan 6 —

Vanuatu 1 —Venezuela 67 3.8Vietnam 3,209 3.5

Wales 0 —West Bank 10 —

Yemen 30 3.6Yugoslavia 173 3.8

Zambia 13 —Zimbabwe 30 4.7

Table 7TWE Score Means –– Applicants to

Undergraduate and Graduate Programs(Based on 191,275 examinees who took the TWE test

from October 2001 through November 2003)

N Mean

Undergraduate 41,428 3.66

Graduate 149,847 3.89

Table 7 shows the mean TWE scores and numbers ofexaminees who indicated that they were taking the TWE testfor admission to undergraduate or graduate degree programs.As the table indicates, there was no substantial differencebetween the performance of self-identified undergraduateand graduate applicants. Zwick and Thayer (1995) found,

however, that after matching undergraduate and graduateexaminees on TOEFL total score, undergraduate TWE meanswere higher than graduate means in 63 of 66 data sets analyzed.

Of the 149,847 examinees who indicated that they wereapplying to graduate programs, 89,135 requested at the timeof testing that their scores be sent to specific graduatedepartments in the United States and Canada. Table 8 showsthe mean TWE scores for examinees who requested that theirscores be sent to graduate departments in the United Statesand Canada, classified by major field of study. (Examineeswho requested their scores be sent to programs in more thanone major field were classified on the basis of the firstdepartment code they specified.) The mean TWE score forthis subgroup of graduate applicants was 3.97, which issomewhat higher than the mean for the total group of graduateapplicants in Table 7. On average, applicants in the humanitiesand social sciences scored slightly higher than applicants inbiological and physical sciences.

16

Table 8TWE Score Means –– Graduate School Applicants Classified by Department1


Department N Mean Department N Mean

1 Because of the unreliability of statistics based on small samples, means are not reported for groups with fewer than 30 examinees.

HumanitiesArchaeology 98 3.8Architecture 923 3.8Art History 125 3.9Classical Languages 32 3.8Comparative Literature 263 4.2Dramatic Arts 61 3.9English 513 4.0Far Eastern Languages 156 4.2

and LiteratureFine Arts, Art, Design 520 3.7French 79 4.0German 32 4.1Linguistics 386 4.2Music 261 3.5Near Eastern Languages & 33 4.0

LiteraturePhilosophy 143 4.0Religious Studies or Religion 196 3.9Russian/Slavic Studies 43 4.0Spanish 66 4.1Speech 18 —Other Foreign Languages 60 4.1Other Humanities 108 4.1

Social SciencesAmerican Studies 90 4.1Anthropology 200 4.1Business and Commerce 7,250 3.9Communications 1,426 4.1Economics 6,134 4.1Education (incl. M.A. in Teaching) 1,886 4.2Educational Administration 1,852 3.8Geography 307 4.0Government 268 4.2History 246 4.1Industrial Relations and Personnel 128 4.0International Relations 619 4.1Journalism 779 4.1Library Science 274 4.0Physical Education 80 3.7Planning(City, Community, Urban, 365 4.0

Regional)Political Science 231 4.2Psychology, Clinical 107 3.8Psychology, Educational 313 4.0Psychology, Other 183 4.0Psychology, Social 209 4.0Psychology,Experimental/ 122 4.0

DevelopmentPublic Administration 672 4.1Public Health 463 4.0Social Work 215 3.9Sociology 588 4.2Other Social Sciences 448 4.1

Biological SciencesAgriculture 424 3.9Anatomy 64 3.9Audiology 51 3.6Bacteriology 51 3.9Biochemistry 1,432 4.0Biology 746 4.0Biomedical Sciences 869 4.1Biophysics 105 4.0Botany 142 3.9Dentistry 146 4.0Entomology 48 3.7Environmental Science 1,002 3.9Forestry 119 3.9Genetics 428 4.0Home Economics 29 —Hospital & Health Services 161 3.8

AdministrationMedicine 629 3.9Microbiology 383 4.0Molecular & Cellular Biology 1,219 4.1Nursing 284 3.8Nutrition 289 4.0Occupational Therapy 18 —Pathology 176 4.0Pharmacy 791 4.0Physical Therapy 62 3.8Physiology 172 4.0Speech-Language Pathology 12 —Veterinary Medicine 64 3.8Zoology 65 3.9Other Biological Sciences 392 4.0

Physical SciencesApplied Mathematics 571 4.0Astronomy 124 3.9Chemistry 3,115 4.0Computer Sciences 9,271 3.9Engineering, Aeronautical 313 3.9Engineering, Chemical 1,781 4.0Engineering, Civil 2,335 3.9Engineering, Electrical 9,964 4.0Engineering, Industrial 815 4.0Engineering, Mechanical 3,674 3.9Engineering, Other 1,984 4.0Geology 396 3.9Mathematics 1,272 3.9Metallurgy 126 4.0Oceanography 145 3.9Physics 1,912 4.0Statistics 1,117 4.1Other Physical Sciences 351 3.9

Graduate Schools of 7,299 4.1Management

Law Schools 2,626 4.2

17

� An Investigation of the Appropriateness of the TOEFLTest as a Matching Variable to Equate TWE Topics. GeraldDeMauro. May 1992. TOEFL Research Report 37. Thisstudy explored the feasibility of using linear andequipercentile equating methods to equate forms of theTWE test by using the TOEFL test as an anchor. Thedifferences between equated and observed scores (equatingresiduals) and differences among the mean equated scoresfor examinee groups were further examined in terms ofcharacteristics of the TWE topics. An evaluation of theassumptions underlying the equating methods suggeststhat TOEFL and TWE do not measure the same skills andthe examinee groups are often dissimilar in skills. Therefore,use of the TOEFL test as an anchor to equate the TWE testsdoes not appear appropriate.

�Scalar Analysis of the Test of Written English. GrantHenning. August 1992. TOEFL Research Report 38. Thisstudy investigated the psychometric characteristics of theTWE rating scale employing Rasch model scalar analysismethodology with more than 4,000 scored TWE essaysacross two prompts. Results suggest that the intervalsbetween TWE scale steps were surprisingly uniform, andthe size of the intervals was appropriately larger than theerror associated with assignment of individual ratings. Theproportion of positively misfitting essays was small andapproximately equal to the proportion of essays that requiredadjudication by a third reader. This latter finding, alongwith the low proportion of misfitting readers detected,provides preliminary evidence of the feasibility ofemploying Rasch rating scale analysis methodology for theequating of TWE essays prepared across prompts.

�Effects of Amount of Time Allowed on the Test of WrittenEnglish. Gordon Hale. June 1992. TOEFL Research Report39. This study examined students’ performance on TWEprompts under two time limits – 30 minutes, as on thecurrent TWE, and 45 minutes. Mean scores on the six-point TWE scale were found to be significantly higher byabout 1/4 to 1/3 point under the 45-minute condition,indicating that allowing additional time produced a modestbut reliable increase in scores. The magnitude of the effectwas roughly comparable for students of low versus highproficiency, and for students in intensive English programsversus students in academic coursework. The correlationbetween scores for both time conditions was relativelyhigh; both parallel-form reliability and interrater reliabilitywere approximately the same for the two time conditions.Provision of additional time apparently had little effect onthe relative standing of students on the test. Results arediscussed in relation to the literature on time effects and topractical implications for the TWE test.

Ongoing research studies related to the TWE test continue toaddress issues of importance to the TWE program. Thisresearch, reviewed and approved by outside specialists fromthe academic and testing communities, is essential to continualevaluation and improvement of the technical quality andutility of the test. To date 12 TWE-related research projectshave been completed. The results of research efforts arepublished as reports and are available by writing to TOEFLResearch Reports, PO Box 6161, Princeton, NJ 08541-6161or online at www.ets.org/toefl.

Research Reports Available (by date of completion)

�Survey of Academic Writing Tasks Required of Graduateand Undergraduate Foreign Students. Brent Bridgemanand Sybil Carlson. September 1983. TOEFL ResearchReport 15. This report describes a survey of faculty in 190departments at 34 US and Canadian universities with highforeign student enrollments; respondents indicated a desireto use scores on a direct writing sample to supplementadmissions and placement decisions.

�Relationship of Admissions Test Scores to WritingPerformance of Native and Nonnative Speakers ofEnglish. Sybil Carlson, Brent Bridgeman, Roberta Camp,and Janet Waanders. August 1985. TOEFL Research Report19. This study investigated the relationship between essaywriting skills and scores on the TOEFL test and the GREGeneral Test obtained from applicants to US institutions.

� A Preliminary Study of the Nature of CommunicativeCompetence. Grant Henning and Eduardo Cascallar.February 1992. TOEFL Research Report 36. This studywas conducted to survey the theoretical literature relatedto communicative competence; to identify major variablessaid to comprise the construct(s); to test for comparativepresence and measurability of such variables as in typicalna t ive /nonna t ive speake r un ive r s i t y academiccommunication; to propose a tentative model of communi-cative competence as a synthesis of these variables; and toexamine the relationship of TOEFL, TSE®, and TWE scoreswith the various elements of the tentative model. Resultsprovide information on the comparative contributions ofsome theory-based communicative competence variablesto domains of linguistic, discourse, sociolinguistic, andstrategic competencies. In turn, these competency domainswere investigated for their relation to components of lan-guage proficiency as assessed by the TOEFL, TSE, andTWE tests. Twelve oral and 12 written communicationtasks were also analyzed and rank ordered for suitability ineliciting communicative language performance.

TWE RESEARCH

18

� Topic and Topic Type Comparability on the Test of WrittenEnglish. Marna Golub-Smith, Clyde Reese, and KarinSteinhaus. March 1993. TOEFL Research Report 42. Thisstudy addressed the question of how comparable scores arefor TWE essays written on different topics and/or differenttopic types, particularly compare-contrast and chart-graphtopic types. It compared TWE mean scores across eightequivalent groups of examinees in an operational TWEadministration and also reported on differences observedacross prompts in the number of examinees at each scorelevel. Additional analyses by gender were also conducted.

� A Comparison of Performance of Graduate andUndergraduate School Applicants on the Test of WrittenEnglish. Rebecca Zwick and Dorothy T. Thayer. May1995. TOEFL Research Report 50. The performance ofgraduate and undergraduate school applicants on the Testof Written English was compared for each of 66 data sets,dating from 1988 to 1993. The analyses compared theaverage TWE score for graduates and undergraduates aftermatching examinees on the TOEFL total score. The mainfinding was that, for matched examinees, undergraduateTWE means were higher than graduate means in 63 of the66 data sets. Although these standardized mean differences(SMDs) never exceeded 0.3 of a TWE score point (withstandard errors that were typically between 0.01 and 0.02),the results are noteworthy because they give a differentpicture than do simple comparisons of means for unmatchedgraduates and undergraduates, which showed higher meanTWE scores for graduate applicants in the majority ofcases.

�Reader Calibration and Its Potential Role in Equatingfor the Test of Written English. Carol Myford, DianaMarr, and J. Michael Linacre. Spring 1996. TOEFLResearch Report 52. When judges use a rating scale to rateperformances, some may rate more severely than others,giving lower ratings. Judges may also differ in theconsistency with which they apply rating criteria. Thisstudy pilot tested a quality control procedure that providesa means for monitoring and adjusting for differences inreader performance. FACETS, a Rasch-based rating scaleanalysis procedure, was employed to calibrate readers withinand across two TWE administrations. The study had fourgeneral foci: (1) to determine the extent to which individualreaders can be considered interchangeable, both within andacross TWE administrations; (2) to investigate readercharacteristics and their relationships to the volume andquality of ratings; (3) to examine the efficacy of the use of athird reading to adjudicate rating discrepancies; and (4) tomake a preliminary determination of the feasibility of usingFACETS Reader Severity Measures as a first step towardequating TWE scores across different topics.

� A Study of Writing Tasks Assigned in Academic DegreePrograms. Gordon Hale, Carol Taylor, Brent Bridgeman,Joan Carson, Barbara Kroll, and Robert Kantor. Spring1996. TOEFL Research Report 54. Writing tasks assignedin 162 undergraduate and graduate courses in severaldisciplines at eight universities were collected. Using asample of the assignments, key dimensions of differencewere identified, and a classification scheme based on thosedimensions was developed. Application of the classificationscheme provided data on the prevalence of various types ofassignments and, for essay tasks, showed the degree towhich the assignments were characterized by each of severalfeatures. Differences among the kinds of writing tasksassigned in different groups of disciplines were examined.

� Adjustment for Reader Rating Behavior in the Test ofWritten English. Nicholas T. Longford. Spring 1996.TOEFL Research Report 55. This report evaluated theimpact of a potential scheme for score adjustment usingdata from the administrations of the Test of Written Englishin 1994. It is shown that, assuming noninformativeassignment of readers to essays, the adjustment due toreader differences would reduce the mean squared error forall essays except those graded by readers with smallworkloads. The quality of the rating process as describedby the variances due to true scores, severity, andinconsistency, as well as the distribution of workloads wassimilar across the administrations. This would enable areliable prediction of the optimal score adjustment in futureadministrations. Two approximations to the optimaladjustment are proposed, and an array of diagnosticprocedures for the engaged raters are presented. The reporthighlights the relevance of shrinkage estimators to problemsin which a large number of quantities is to be estimated andindicates how combining information across rating exercisescould lead to further gains in the precision of assignedscores.

�Computer Analysis of the Test of Written English.Lawrence Frase and Joseph Faletti with consultants DougBiber, Ulla Connor, Gerard Dalgish, and Joy Reid. Seeksto conduct a variety of automated text analyses of TWEessays to summarize, analyze, and compare linguisticproperties of TWE essays written by examinees fromdifferent language groups and to determine how TWEscores relate to linguistic text properties. Database ofanalyzed essays is now being used in other studies.

19

REFERENCES

Kane, H. (1983). A study of practices and needs associatedwith intensive English language programs: report of findings.Internal report submitted to the TOEFL Program Office. NewYork: Kane, Parsons, and Associates, Inc.

Livingston, S. A., and Zieky, M. J. 1982. Passing Scores: AManual for Setting Standards of Performance on Educationaland Occupational Tests. Educational Testing Service:Princeton, NJ.

Oller, J. W. (1979). Language tests at school. London:Longman Group Ltd.

Pike, L. (1976). An evaluation of alternate item formats fortesting English as a foreign language (TOEFL ResearchReport No. 2). Princeton, NJ: Educational Testing Service.

Pitcher, B., and Ra, J. B. (1967). The relationship betweenscores on the Test of English as a Foreign Language andratings of actual theme writing (Statistical Report 67-9).Princeton, NJ: Educational Testing Service.

Way, W. D. (1990). TOEFL 2000 and Section II: Relationshipsbetween structure, written expression, and the Test of WrittenEnglish (Internal Report, March 1990). Princeton, NJ:Educational Testing Service.

Zwick, R., and Thayer, D. T. (1995). A comparison ofperformance of graduate and undergraduate school applicantson the Test of Written English (TOEFL Research Report No.50). Princeton, NJ: Educational Testing Service.

American Educational Research Association, AmericanPsychological Association, and National Council forMeasurement in Education. (1985). Standards for educationaland psychological testing. Washington, DC: AmericanPsychological Association.

Angelis, P. J. (1982). Academic needs and priorities fortesting. American Language Journal, 1, 41-56.

Bridgeman, B., and Carlson, S. (1983). Survey of academicwriting tasks required of graduate and undergraduate foreignstudents (TOEFL Research Report No. 15). Princeton, NJ:Educational Testing Service.

Carlson, S. B., Bridgeman, B., Camp, R., and Waanders, J.(1985). Relationship of admission test scores to writingperformance of native and nonnative speakers of English(TOEFL Research Report No. 19). Princeton, NJ: EducationalTesting Service.

Conlan, G. (1976). Suggestions for writing essay questions.Princeton, NJ: Educational Testing Service.

Educational Testing Service. (1987). ETS guidelines fordeveloping and scoring free-response tests. Princeton, NJ:Author.

Educational Testing Service. (1981, 2000). ETS standardsfor quality and fairness. Princeton, NJ: Author.

Educational Testing Service. (1991). TWE readingmanagement guidelines. Princeton, NJ: Author.

Hale, G. A., and Hinofotis, F. (1981). New directions inEnglish language testing. Internal report submitted to theTOEFL Research Committee. Princeton, NJ: EducationalTesting Service.

21®

APPENDIX ATEST OF WRITTEN ENGLISH (TWE) Revised 2/90

SCORING GUIDE

Readers will assign scores based on the following scoring guide. Though examinees are asked to write on a specific topic, partsof the topic may be treated by implication. Readers should focus on what the examinee does well.

Scores6 Demonstrates clear competence in writing on both the rhetorical and syntactic levels, though it may have

occasional errors.A paper in this category––effectively addresses the writing task––is well organized and well developed––uses clearly appropriate details to support a thesis or illustrate ideas––displays consistent facility in the use of language––demonstrates syntactic variety and appropriate word choice

5 Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will probablyhave occasional errors.A paper in this category––may address some parts of the task more effectively than others––is generally well organized and developed––uses details to support a thesis or illustrate an idea––displays facility in the use of language––demonstrates some syntactic variety and range of vocabulary

4 Demonstrates minimal competence in writing on both the rhetorical and syntactic levels.A paper in this category––addresses the writing topic adequately but may slight parts of the task––is adequately organized and developed––uses some details to support a thesis or illustrate an idea––demonstrates adequate but possibly inconsistent facility with syntax and usage––may contain some errors that occasionally obscure meaning

3 Demonstrates some developing competence in writing, but it remains flawed on either the rhetorical orsyntactic level, or both.A paper in this category may reveal one or more of the following weaknesses:––inadequate organization or development––inappropriate or insufficient details to support or illustrate generalizations––a noticeably inappropriate choice of words or word forms––an accumulation of errors in sentence structure and/or usage

2 Suggests incompetence in writing.A paper in this category is seriously flawed by one or more of the following weaknesses:––serious disorganization or underdevelopment––little or no detail, or irrelevant specifics––serious and frequent errors in sentence structure or usage––serious problems with focus

1 Demonstrates incompetence in writing.A paper in this category––may be incoherent––may be undeveloped––may contain severe and persistent writing errors

Papers that reject the assignment or fail to address the question must be given to the Table Leader. Papers that exhibit absolutelyno response at all must also be given to the Table Leader.

Copyright © 1986, 1990 by Educational Testing Service, Princeton NJ, USA. All rights reserved.

22

APPENDIX B TWE TEST BOOK COVER

®

Do NOT open this test bookuntil you are told to do so.

Read the directions that follow.

1. The TWE essay question is printed on the inside of this test book. Youwill have 30 minutes to plan, write, and make any necessary changes toyour essay. Your essay will be graded on its overall quality.

2. Read the topic carefully. You may want to read it more than once to besure you understand what you are asked to write about.

3. Think before you write. Making notes may help you to organize youressay. Below the essay topic is a space marked NOTES. You may use thisarea to outline your essay or make notes.

4. Write only on the topic printed on the inside. If you write on a differenttopic, your essay will not be scored. Write clearly and precisely. Howwell you write is much more important than how much you write, but tocover the topic adequately, you may want to write more than oneparagraph.

5. Start writing your essay on the first line of Side 3 of the TWE answersheet. Use Side 4 if you need more space. Extra paper will not beprovided. Write neatly and legibly. Do not skip lines. Do not write in verylarge letters or leave large margins.

6. Check your work. Allow a few minutes before time is called to read overyour essay and make any changes.

7. After 30 minutes, you will be instructed to stop and put your pencildown. You MUST stop writing. If you continue to write, it will beconsidered cheating.

Do NOT break the seal on this book until you are told to do so.

When you have finished reading the directions, look up.

Copyright © 1996 by Educational Testing Service. All rights reserved.Princeton, NJ 08541-0001, USA

EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, TOEFL, the TOEFL logo,TWE, and the TWE logo are registered trademarks of Educational Testing Service.

Form: 3RTF12

Test of Written English

TWE® Test Book

Topic A

®

23

APPENDIX C

SAMPLE TWE ESSAY QUESTIONS

SAMPLE 1

Some people say that the best preparation for life is learning to work with others and becooperative. Others take the opposite view and say that learning to be competitive is thebest preparation. Discuss these positions, using concrete examples of both. Tell whichone you agree with and explain why.

SAMPLE 2

Supporters of technology say that it solves problems and makes life better. Opponentsargue that technology creates new problems that may threaten or damage the quality oflife. Using one or two examples, discuss these two positions. Which view of technologydo you support? Why?

SAMPLE 3

Some people believe that automobiles are useful and necessary. Others believe thatautomobiles cause problems that affect our health and well-being. Which position do yousupport? Give specific reasons for your answer.

SAMPLE 4

Inventions such as eyeglasses and the sewing machine have had an important effect onour lives. Choose another invention that you think is important. Give specific reasons foryour choice.

SAMPLE 5

Do you agree or disagree with the following statement?

Teachers should make learning enjoyable and fun for their students.

Use reasons and specific examples to support your opinion.

24

This appendix contains actual TWE essay responses. Sample essays are presentedfor each TWE score point for each of three TWE prompts.

APPENDIX D SAMPLE TWE ESSAYS

25

SET 1

TWE Essay Question

Supporters of technology say that it solves problems and makes life better.Opponents argue that technology creates new problems that may threaten ordamage the quality of life. Using one or two examples, discuss these twopositions. Which view of technology do you support? Why?

26

TWE Score 1

1 Demonstrates incompetence in writing.

A paper in this category

—may be incoherent

—may be undeveloped

—may contain severe and persistent writing errors

27

TWE Score 1

28

TWE Score 2

2 Suggests incompetence in writing.

A paper in this category is seriously flawed by one or more of the followingweaknesses:

—serious disorganization or underdevelopment

—little or no detail, or irrelevant specifics

—serious and frequent errors in sentence structure or usage

—serious problems with focus

29

TWE Score 2

30

TWE Score 3

3 Demonstrates some developing competence in writing, but it remainsflawed on either the rhetorical or syntactic level, or both.

A paper in this category may reveal one or more of the following weaknesses:

—inadequate organization or development

—inappropriate or insufficient details to support or illustrate generalizations

—a noticeably inappropriate choice of words or word forms

—an accumulation of errors in sentence structure and/or usage

31

TWE Score 3

32

TWE Score 4

4 Demonstrates minimal competence in writing on both the rhetoricaland syntactic levels.


—addresses the writing topic adequately but may slight parts of the task

—is adequately organized and developed

—uses some details to support a thesis or illustrate an idea

—demonstrates adequate but possibly inconsistent facility with syntax and usage

—may contain some errors that occasionally obscure meaning

33

TWE Score 4

34

TWE Score 5

5 Demonstrates competence in writing on both the rhetorical and syntacticlevels, though it will probably have occasional errors.


—may address some parts of the task more effectively than others

—is generally well organized and developed

—uses details to support a thesis or illustrate an idea

—displays facility in the use of language

—demonstrates some syntactic variety and range of vocabulary

35

TWE Score 5 (continued)

36

TWE Score 5

37


38

TWE Score 6

6 Demonstrates clear competence in writing on both the rhetorical andsyntactic levels, though it may have occasional errors.


—effectively addresses the writing task

—is well organized and well developed

—uses clearly appropriate details to support a thesis or illustrate ideas

—displays consistent facility in the use of language

—demonstrates syntactic variety and appropriate word choice

39


40

TWE Score 6

41


42

SET 2

TWE Essay Question

Inventions such as eyeglasses and the sewing machine have had an importanteffect on our lives. Choose another invention that you think is important. Givespecific reasons for your choice.

43

TWE Score 1






44

TWE Score 2






—serious problems with focus

45

TWE Score 3

3 Demonstrates some developing competence in writing, but itremains flawed on either the rhetorical or syntactic level, or both.






46

TWE Score 4

4 Demonstrates minimal competence in writing on both the rhetoricaland syntactic levels.







47


48

TWE Score 5

5 Demonstrates competence in writing on both the rhetorical andsyntactic levels, though it will probably have occasional errors.







49


50

TWE Score 6

6 Demonstrates clear competence in writing on both the rhetoricaland syntactic levels, though it may have occasional errors.







51


52

SET 3

TWE Essay Question

Do you agree or disagree with the following statement?

Teachers should make learning enjoyable and fun for their students.

Use reasons and specific examples to support your opinion.

53

TWE Score 1






54

TWE Score 2






55


56

TWE Score 3

3 Demonstrates some developing competence in writing, but it remainsflawed on either the rhetorical or syntactic level, or both.






57

TWE Score 4

4 Demonstrates minimal competence in writing on both therhetorical and syntactic levels.







58


59

TWE Score 5

5 Demonstrates competence in writing on both the rhetorical andsyntactic levels, though it will probably have occasional errors.







60


61

TWE Score 6

6 Demonstrates clear competence in writing on both therhetorical and syntactic levels, though it may have occasionalerrors.







62


63

64

Test of Written EnglishPO Box 6155

Princeton, NJ 08541-6155USA

E-mail: [email protected] site: www.ets.org/toefl

Educational Testing Service72702-01186 • U74M2.5 • Printed in U.S.A.

725446

®

®

Documents

TEST OF WRITTEN ENGLISH - Educational Testing … TOEFL TEST OF WRITTEN ENGLISH GUIDE Fifth Edition The TOEFL Test of Written English Guide has been prepared for deans, admissions