25p. · DOCUMENT RESUME. ED 413 349 TM 027 657. AUTHOR Land, Robert TITLE Moving Up to Complex Assessment Systems. Proceedings from. the CRESST Conference (Los Angeles, CA, September

DOCUMENT RESUME

ED 413 349 TM 027 657

AUTHOR Land, RobertTITLE Moving Up to Complex Assessment Systems. Proceedings from

the CRESST Conference (Los Angeles, CA, September 5-6,1996) .

INSTITUTION National Center for Research on Evaluation, Standards, andStudent Testing, Los Angeles, CA.

SPONS AGENCY Office of Educational Research and Improvement (ED),Washington, DC.

PUB DATE 1997-00-00NOTE 25p.

CONTRACT R305B60002AVAILABLE FROM UCLA Center for the Study of Evaluation, 10920 Wilshire

Boulevard, Suite 900, Los Angeles, CA 90024-6511.PUB TYPE Collected Works Proceedings (021) Collected Works

Serials (022) Reports Evaluative (142)JOURNAL CIT Evaluation Comment; v7 n1 p1-22 Sum 1997EDRS PRICE MF01/PC01 Plus Postage.DESCRIPTORS Conferences; *Educational Assessment; Educational Research;

Educational Technology; Elementary Secondary Education;*Evaluation Utilization; Models; Reliability; Research andDevelopment; *Standards; *Test Use; Validity

IDENTIFIERS *Center for Research on Eval Standards Stu Test CA

ABSTRACTAssessment systems to measure high educational standards

emerged as the major theme at the 1996 conference of the National Center forResearch on Evaiuntion, Standa:ds, and Sr.:d1,.nt: Tesi:in::; (CR:2;56T), "iquring Up

to Complex Assessment." This feature article, providing a summation of theproceedings of the conference, reports that the approximately 250 educatorsand community leaders were in general agreement that challenging standardsare the key to the improvement of American education. In opening remarks, thecodirectors of CRESST, Eva L. Baker and Robert L. Linn explained theconceptual model that is guiding CRESST assessment research and developmentin the next 5 years. This model focuses on the utility of assessment systemsfor various purposes and establishes long-range goals for the Center'sresearch. The CRESST model highlights three qualities that are essential tothe productive use of assessment: validity, fairness, and credibility.Conference presentations centered on broad areas related to these qualities:(1) developing valid, fair, and credible assessments; (2) enhancing theutility of assessments; and (3) exploring the role that technology can playin creating new possibilities in developing and using assessment systems.(SLD)

********************************************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

********************************************************************************

O.

ri)

EVALUATION COMMENTA PUBLICATION OF UCLA's CENTER FOR THE STUDY OF EVALUATION AND

GRADUATE SCHOOL OF. EDUCATION & INFORMATION STUDIES

NATIONAL CENTER FOR RESEARCH ON EVALUATION,.STANDARDS, AND STUDENT TESTING

FEATURE ARTICLE

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDU TIONAL RESOURCES INFORMATIONCENTER (ERIC)

This document has been reproduced asreceived from the person or organizationoriginating it.Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

Summer 1997, Vol. 7, No. 1

MOVING UP TO COMPLEX ASSESSMENT SYSTEMS

PROCEEDINGS FROM THE 1996 CRESST CONFERENCE

ROBERT LAND UCLA/CRESST

ASSESSMENT systems to measurehigh educational standards emergedas the major theme at this year's CRESST

conference, Moving Up to Complex AssessmentSystems, September 5-6, 1996, at UCLA's Sun-set Village Conference Center. The broad appeal

of the agenda was reflected by the diversity ofthe conference participants. Approximately 250researchers, community leaders, teachers, prin-cipals, school board members, state and federal

h._ education officials, and representatives of pri-vate and commercial interests attended two full

,31 days of presentations and assessment forums.

N Opinion was strong from many conference

c( presenters that challenging standards were theO backbone to the improvement of American

education.

MAILING LIST UPDATE

We will be updating the CRESST mailinglist during the next few months. Postcards willbe mailed to everyone who currently receivesfree copies of CRESST Line and EvaluationComment. To remain on our list, please returnthe postcard promptly.

Just a reminder that you or your associatesmay register to be placed on our publicationsmailing list at any time through our Web site,www.cse.ucla.edu.

Thank you for your cooperation.

BEST COPY AVAILABLE

2


"I'd rather set the [standards] bar high,"said Sidney Thompson, superintendent of theLos Angeles Unified School District, "and haveus look at how we're going to help the studentget over that bar, than set the bar low and know

that when she or he got over it, it didn't meana darn thing."

Thompson noted that the Los AngelesUnified School District has joined all 50 statesand many large school districts in setting stan-dards for what children should know and beable to do across multiple grade levels andtopics. The District and CRESST are workingtogether to develop a new standards-basedassessment system comprised of a commer-cial standardized test, performance-based as-sessments based on CRESST instructionalmodels, and classroom tests to improve in-struction, learning, and student performance.

"Our current emphasis on high, challeng-ing standards for all students," said CRESSTCo-director Eva Baker in her conference pre-sentation, "can be traced to the 1989 Gover-nors' Education Summit and was reinforcedin the 1994 Goals 2000 legislation and therecent Improving America's Schools Act'which reauthorized federal Title I programs."

"By 1997-98," explained Baker, "Title I

schools must have in place challenging con-tent and performance standards in at least

'Improving America's Schools Act of 1994, Confer-ence Report 103-761. Regarding Public Law 103-382, signed October 20, 1994, (pp. 6-33). Washing-ton, DC: House of Representatives.

Evaluation Comment

reading and mathematics, followed by high-quality assessments of those standards in the2000-2001 school year. The assessmentsmust involve multiple approaches and mea-sure complex thinking skills and understand-ing of rigorous content. Performanceassessments will be part of the system butpose some unique challenges in terms of the

UCLA's

CenterCenter for theStudy of Evaluation

The National Centerfor Research on Evaluation,

Standards, and Student Testing

Eva L. Baker, Co-DirectorRobert L. Linn, Co-Director

Joan L. Herman, Associate DirectorPamela Aschbacher, Assistant Director

Ronald Dietel, CRESST Director ofCommuncations

Katharine Fry, Editorial AssistantMary Wilby, Design Layout

The work reported herein was supported under theEducational Research and Development CentersProgram, PR/Award Number R305B60002, asadministered by the Office of Educational Researchand Improvement, U.S. Department of Education.The findings and opinions expressed in thispublication do not reflect the position or policiesof the National Institute on Student Achievement,Curriculum, and Assessment, the Office ofEducational Research and Improvement, or theU.S. Department of Education.

3


time, costs and technical quality required todevelop accurate measures of individual ac-

complishment.""One of our biggest challenges is to learn

how performance assessments can work withtraditional assessments," noted CRESST Co-

director Robert Linn in his opening confer-ence remarks, "and how these assessmentsfit into the larger education reform picture,from the classroom to the national level. We've

moved from a focus on single instruments toa system perspective."

In other opening remarks, both Baker andLinn explained the new conceptual model that

is guiding the CRESST assessment researchand development efforts for the next five years.

The CRESST model (Figure 1) focuses on the

utility of assessment systems for various pur-

poses and audiences and establishes threeimportant, long-range social goals for theCenter's research:

providing new knowledge andunderstanding about educa-tional quality;

contributing to educational im-provement in policy, accounta-bility, and teaching and learning;and

encouraging productive publicengagement in education.

The CRESST model highlights threequalities that are essential to the productiveuse of assessment, namely, validity, fairnessand credibility, and focuses the CRESSTresearch agenda on understanding therelationships among and between thesequalities and effective assessment systems.

Teaching 8. LearningClassrooms & Schools

Accountability & PolicyPublic Understanding

Figure 1. CRESST Conceptual Model

4 Summer 1997


Conference presentations focused on threebroad areas related to the CRESST conceptual

model: developing valid, fair, and credible as-sessments; enhancing the utility of assess-ments; and finally, the role that technology can

play in creating new possibilities in both de-veloping and utilizing assessment systems.

DEVELOPING VALID, FAIR, AND

CREDIBLE ASSESSMENTS

N the past, researchers, educators, andpolicy makers seemed satisfied ifassessments met relatively narrow cri-

teria of technical quality. But as the pur-poses of assessment have grown and thedemand for more inclusive and informa-tive tests has increased, both test devel-opers and test users have recognized thatnarrow technical criteria are not enough.Echoing themes from the CRESST model,conference presenters took a comprehen-sive view of what is required for goodassessment: an expanded view of valid-ity, including attention to intended pur-poses and consequences; heightenedconcerns for inclusion of and fairness toall students; and recognition of the im-portance of public credibility.

Please note that beginning with this issue ofEvaluation Comment we shall resume the use of

volume and issue numbers.

Evaluation Comment

VALIDITY

TODAY'S assessment systems are inten-

ended to serve multiple audiences atthe federal, state, local, classroom, and

student levels and likewise are intended toserve a range of purposes, from communi-cating standards and promoting accountabil-ity, to contributing to school improvement, in-

forming teaching and learning, and improv-ing student performance. These demandsbring new complexity to assuring that assess-ment systems provide accurate informationfor decision making. Conference participantsparticularly highlighted three areas warrant-ing sustained effort: alignment, the measure-ment of progress, and linking the results frommultiple measures.

"...assessment is both a very cen-tral part of reform and the index forjudging the success of that reform."

Alignment

THAT assessment is to be aligned with

rigorous standards for studentachievement is a defining feature of

today's assessments. As Ed Reidy, deputycommissioner of the Kentucky Department ofEducation expressed it, "Assessment is both

5


a very central part of reform and the index forjudging the success of that reform." Assess-ment is intended to stimulate reform by com-municating these standards, holding educa-tors and students accountable for achievingthem, and to provide an accurate measure ofstudents' performance on the standards.

"It seems simpleadopt standards andmake assessments that are aligned withthembut there is a lot more involved," notedRobert Linn.

What does such alignment really mean?How do states, districts, and schools knowwhether their assessments are aligned? "Howthe major elements of an education systemwork together to guide the process of helpingstudents achieve higher levels of mathemati-cal and scientific understanding," said NormanWebb, Wisconsin Center for Educational Re-search, "goes beyond a simple content analy-

sis."Citing results of a study by the Council of

Chief State School Officers (CCSSO), Webbreported that few states have addressed thesequestions with much rigor and even fewer have

examined broader alignment issues.Contributing to the complex alignment pictureis that states lack a formal and systematicprocess to develop assessments directly based

on their standards, and instead have developed

assessments prior to or at the same time astheir standards.

Based on his review of current practicesand relevant literature, Webb presented fivecategories of criteria that states and local dis-tricts can use to evaluate alignment.

These categories include:

pedagogical implications;

equity and fairness;

articulation across grades andages;

system applicability; and

content focus.

Content focus includes topic coverage, depthand range of student knowledge, and balance

of representation.Several presenters reported that even

when assessments are developed directly from

standards, alignment can be complicated.David Wiley, technical director of the NewStandards Project, summed up the generalproblem by stating that standardseven per-formance standards anchored in students'workare not specified well enough for pur-poses of test development. They do not ad-equately guide the concrete decisions thatneed to be made on what is to be measured,how it is to be measured, and what specifictasks and criteria will be used.

6 Summer 1997


To develop the New Standards mathemat-

ics assessments, Wiley found it was neces-sary to create an "infrastructure that wouldprovide another level of construct definition."The New Standards' seven measurable math-ematics standards, for example, were clusteredunder three constructs: "concepts," "skills,"and "problem solving"; these in turn were "de-fined in terms of student capabilities and theprocesses needed for successful task perfor-mance." The constructs were used to assure abalanced test instrument and were the basisfor standards-setting and reporting.

Eva Baker recommended the use of an in-termediate strategy to provide a "crosswalk be-tween standards and assessments," but onewith the added advantage of providing gener-alizable assessment models that increase thecost-effectiveness and classroom utility oflarge-scale assessments. Based in cognitivetheory, the CRESST models focus on coretypes of learning that recur across the curricu-

Based in cognitive theory, theCRESST models focus on coretypes of learning that recur acrossthe curriculum...

lum: conceptual understanding, knowledgerepresentation, problem solving, communica-tion, and team work. The models providespecifications for developing assessment tasks

Evaluation Comment

and scoring rubrics that are operationalizedin specific subject areas and customized to lo-

cal curriculum emphases and grade levels tobe tested. Teachers can use the CRESST mod-

els to create classroom instruction and assess-

ment, and to align their practices withestablished standards and assessments.

David Niemi, University of Missouri, andZenaida Aguirre-Munoz, CRESST/UCLA,described the application of the CRESSTmodels in Hawaii and elsewhere, where theyhave been used for assessing history andmathematics. Among the advantages Niemiand Aguirre-Munoz noted were improvedreplicability and comparability of tasks andresults, enhanced system alignment, greaterefficiency, and improved engagement ofteachers and the public in all aspects of large-scale assessment.

A number of conference participantsstressed that aligning standards and assess-ments is only one piece of what's required forthe success of current reforms. Professionaldevelopment, curriculum and instruction, in-centives and sanctions, resource allocation,district and school infrastructureall thesemust be aligned with standards if real progressis to be made.

Aligning standards and assessment withcurriculum is essential if student learning isto be affected. But determining such alignment

can be complex, as discussed by WilliamSchmidt, Michigan State University. In his re-search on the Third International Mathemat-ics and Science Study (TIMSS), Schmidt


found that even deciding whether or not acountry covered a particular mathematics topic

turned out to be anything but simple."From a curricular perspective," Schmidt

explained, "math is not math everywhere.There is little overall overlap in the countrieswe studied."

To create the assessment, it was neces-sary to develop sets of curricularly sensitiveitems that address areas where groups of

"...math is not math everywhere.There is little overall overlap in thecountries that we studied."

countries show overlapping curricula and asophisticated methodology for characterizing

curriculum.

Measuring Progress

THE challenge of accurately assessing

student progress was highly salient toa number of conference participants

who were struggling with Title I requirements.The key issue was the mandate that schoolsmust show adequate yearly progress sufficientto enable all students to achieve high stan-dards of accomplishment within a reasonableperiod of time. Like the alignment of standards

and assessment, accurately measuringprogress is easier said than done.

"From a validation point of view," said BobLinn, "we have to ask the question: How dowe know when we see improvement?"

Linn provided an example from Kentucky,where student achievement as measured bystatewide assessments showed significantyear-to-year increases while the National As-sessment of Educational Progress (NAEP) in-dicated little or no significant change.

"There may be problems of test compara-bility from year to year," explained Linn,"caused by differences in conditions of testadministration or the degree of alignment be-tween the test and the state's instructionalgoals." Linn noted other factors that might ac-

count for the increases in student achievementon the statewide test including the substan-tial incentives and sanctions associated withstudent test performance, possible changesin school populations, students becomingmore familiar with the test format, teacherswho changed instruction to match the statetest content, or some technical design issueassociated with performance assessment.

Bengt Muthen, CRESST/UCLA, under-scored the difficult analytic challenges of mea-suring student progress by identifying somekey problems including:

8 Summer 1997


selecting an analysis methodthat gives the best picture ofperformance;

analyzing and interpretingthe interaction among multiplegrowth processes;

determining the size and dura-tion of instructional and othertreatment effects; and

understanding the aggregateimpact of various school experi-ences and programs and the con-tributions of individual studentcharacteristics.

Muthen described CRESST's researchprogram to address these problems. Usinglongitudinal student performance data in avariety of skills areas, Muthen is developingnew, multilevel modeling tools to analyzestudent progress and the contributions ofschool, classroom, and other factors to suchprogress. In addition to providing newtechnical knowledge useful to researchers, this

research should provide policy makers withpractical guidance for formulating, reporting,and interpreting assessment results within andacross schools.

Evaluation Comment

Linking

LINKING, a third validity challenge incurrent assessment systems, was alsosubject to wide discussion at the con-

ference. States and local districts are devel-oping unique assessments based on their ownstandards. Yet their publicsparents, com-munity, policy makers, and studentsstillwant to know how their students' performancecompares with that of othersfrom other lo-calities, other states, nationally, and interna-tionally. If students meet local or state stan-dards, does it mean they are nationally or in-ternationally competitive? What are the groundrules for linking results, especially across dif-ferent types of measures, such as norm-refer-enced tests and performance assessments?

Conference speakers discussed a numberof efforts to link results between the NationalAssessment of Educational Progress (NAEP),state-by-state NAEP assessments, and inter-national tests such as the Third InternationalMathematics and Science Study (TIMSS).

For example, Sharif Shakrani, NationalCenter for Education Statistics, presented anumber of technical and practical challengesin making links that might permit states tocompare their test results nationally and inter-nationally.

"The market-basket approach appears tobe one promising option," explained Shakrani,"where states could administer representativesamples of items drawn from the full set of


NAEP items along with their state assessments

to get good estimates of how students in theirstate compare with students nationally."Shankrani added that the market-basket ap-proach, applied to NAEP and state assessment,

"would have the additional advantage of pro-viding rapid turnaround time, perhaps asquickly as three months, and would allow more

frequent NAEP testing in more subjects."In discussing his agenda for the National

Center for Education Statistics, CommissionerPascal Forgione endorsed research on themarket-basket approach and efforts to linkstate, national, and international data. He alsodiscussed plans to study the feasibility ofembedding robust NAEP items in states' non-NAEP assessments to generate more timelyand cost-effective state-level NAEP scores.Further, Forgione revealed plans for linkingNAEP with other national testing data to builda more comprehensive, cost-effective, andconsistent database for policy makers and re-searchers.

FAIRNESS

RILE fairness is an integral compo-

nent of validity, the CRESST modeland conference participants identi-

fied it as needing concerted attention in cur-rent assessment systems. Prompted in partby recent Title I legislation, which will affectapproximately 67% of schools in the UnitedStates, states and districts are increasingly in-

0

terested in assessments that will contributeto educational improvements for all students,and they are committed to the testing of allstudents.

"Results from a recent survey ofCouncil schools...show that 85% ofthe Council districts have changedor are changing their assessmentsto align with new national, state, orlocal content standards."

"The challenge," said Adrienne Bailey,senior consultant for the Council of the GreatCity Schools (CGCS), "is to get all studentsto perform at higher levels than they havebefore."

But the urgency to improve schoolsunderscores the need to make sure that theprocess is fair. "Results from a recent surveyof Council schools," said Bailey, "show that85% of the Council districts have changed orare changing their assessments to align withnew national, state, or local content stan-dards." Based on the volume of reform in pro-cess, Bailey emphasized that broadcommunity involvement is essential in orderto produce standards and assessments thatare supported by diverse, urban communitiesand that help urban students to achieve world-

class levels of performance.

Summer 1997

10


"Equity and fairness are no longer simplyissues of morality," warned CRESST partnerand Yale Professor Emeritus Edmund Gordon;

"in educational measurement, they emerge asbeing at the very core of what our work isabout."

Ideally, the quality of inferences from as-sessments will be judged in terms of their ac-curacy and appropriateness for all people, ofall backgrounds and needs. For this ideal tobe approached, every aspect of the assessment

process must be fairfrom assessment de-velopment through administration and inter-pretation of results. However, although oftenheralded as a bridge to opportunity, testing and

assessment have more often been viewed asunfair to underrepresented minorities and asbarriers to educational access.

"To help improve education for all stu-dents," added Gordon, "tests must go beyondtelling how a student is performing, to givinguseful information about how to improve thatperformance. The system must be committedto adapting to the diverse needs of all stu-dents."

Gordon identified four broad categories offairness issues:

the political economy ofeducational assessment;

limitations in the politicaland technical capacities ofpedagogy and assessment;

Evaluation Comment

epistemological andtheoretical contexts foreducational assessment;and

the technological de-mands of equitablesystems of assessment.

"We must learn to factor into our peda-gogical and assessment practices the condi-tional and situational correlates of humanperformance," said Gordon. "Why, for ex-ample," he asked, "do Black students do bet-ter on tests when the test administrator is Black

rather than White? To make fair tests we willhave to change, expand, and achieve bettersymmetry among our concepts of knowledge,pedagogy, and intelligence. We will have tohonor and accommodate diversity. And we will

certainly have to move beyond traditionalmultiple-choice tests."

"This task," concluded Gordon, "willrequire a strategic plan for assessmentdevelopment, and the new CRESST modelpoints us in that direction."

In addition to addressing issues of fair-ness for students who are currently includedin testing, CRESST research is focusing ontwo groups who traditionally have been ex-cluded from large-scale assessments: students

with disabilities and language minority stu-dents who are not fully proficient in English.

11

11


Students With Disabilities

How many students with disabilities are

currently excluded from testing? Whatare the characteristics of these stu-

dents? What kinds of accommodations arecurrently being used? The answers to thesekey questions are far from clear according toLinda Bond, North Central Regional Educa-tional Laboratory. In a national survey of state-

In general, anywhere from 5% to10% of all students were excludeddepending on the particular assess-ment and the state.

wide testing programs, Bond found tremen-dous variability in states' estimates of howmany students with disabilities were excludedfrom the state test. In general, anywhere from5% to 10% of all students were excluded de-pending on the particular assessment and the

state.

Similarly, Daniel Koretz, CRESST/RAND,cited U.S. Department of Education statisticsindicating that states' estimates of studentswith disabilities in their state vary widely, from5.5% to 15%, with an overall average of about10%. Estimates of specific disabilities suchas mental retardation or learning disabilitiesare even more variable.

T2

"Differences among states," Koretz sug-gested, "result mostly from differences in defi-nitions of students with disabilities, rather thanreal differences in where students with dis-abilities live."

Problems with counting and defining stu-dents with disabilities are not surprising giventhat states lack clear guidelines defining thosestudents who should and should not be in-cluded in their assessments. JamesYsseldyke, National Center on EducationalOutcomes (NCEO), found that state guidelineson inclusion range from a single descriptivesentence to 60 pages of directions. To helpstates include more students with disabilitiesin their testing programs, NCEO is revisingand clarifying their guidelines for inclusionpractices.

Good descriptive information and soundguidelines, however, may not be enough.Some districts and states are still reluctant toinclude students with disabilities for fear thatlow performance will hurt the child's self-esteem and lower overall state scores.

"Nearly 70% of fourth-grade students cur-rently excluded from state NAEP assessments

could take those tests," claimed presenter FranStancavage, American Institutes for Research.

Stancavage urged increased adaptive testingstrategies because current assessments arenot providing the necessary information aboutthe performance of the least able and the most

advanced students. Ysseldyke agreed, observ-

ing that "the most difficult measurement is-sues get addressed at the margins of the

Summer 1997

12


distributionvery high and very low perform-ing students." These factors suggest that as-sessment developers must make testssimultaneously more challenging and more ac-cessible.

Accommodations will be a key to increas-ing accessibility, at least for some students.But deciding who needs accommodations,what kinds of accommodations are feasible,and whether accommodations create an un-fair advantage for test takers will be a majorundertaking. Bond reported that most states

"The purpose of accommodationfrom a measurement standpoint isto offset bias in order to make themeasurement more valid than itwould otherwise be."

have little problem modifying testing condi-tions for physically disabled students, offer-ing Braille tests for blind students, for example.

But accommodations for cognitively disabledstudents are not as common; and the higherthe stakes, the fewer the accommodations be-cause of test validity concerns.

"The purpose of accommodation from ameasurement standpoint," responded Koretz,"is to offset bias in order to make the mea-surement more valid than it would otherwisebe."

Evaluation Comment

Discussing CRESST's program of research

on accommodations and adaptations for spe-cial needs students, Koretz described researchon the use of paraphrasing for mildly men-tally retarded students taking Kentucky's sci-ence tests. Issues Koretz is struggling with arethe meaning of the scores obtained with andwithout accommodations and policies for as-sessing students whose classifications as dis-abled are ambiguous.

Scott Trimble, Kentucky Department ofEducation, spelled out his state's strong com-mitment to include all students in their stateassessments. Kentucky districts and schoolsare advised to use the same accommodationsin the assessment that they use in instruction.

"Instructionally relevant accommoda-tions," added Trimble, "such as the use of para-

phrasing, extended time frames, and smallergroup settings, seem to work fairly well."

But he cautioned that accommodationsare not magic solutions to all problems asso-ciated with assessing students with disabili-ties. For example, students may receive special

attention in the classroom one year, but not inanother year. Trimble also questions whetheryear-to-year scores for students with disabili-ties are fair measures of their progress whenone year they are tested with accommodations

and another year they are not.

13

13


Language Minority Students

LIMITED English proficient [LEP] students

are more likely to be tested than stu-dents with disabilities," said Charlene

Rivera, George Washington University. "ButLEP students are less likely to be given ac-commodations," she added. As an example,Rivera reported that in 1994, 17 states required

students to pass a high school graduation testto earn a diploma. While 13 of the 17 statespermitted accommodations, only 2 states of-fered tests specifically designed for LEP stu-dents. "Scores on tests that assume Englishproficiency are likely to grossly underestimateLEP students' academic achievement," sug-gested Rivera.

"For purposes of accountability, improvedteaching, and student learning," said LorrieShepard, CRESST/University of Colorado atBoulder, "we need assessment systems thatcan identify student performance on relevant

\continua of proficiency. For LEP students,"a\Ided Shepard, "this requires multiple mea-ail:es that distinguish English language pro-ficiency, native language proficiency, andacademic achievement. Such systems are farfrom being available now. In particular, cur-rent measures of English language proficiency

are few and limited, yet are essential to under-

standing LEP students' performance.""Chicago and the entire state of Illinois

are working intensively to develop good mea-sures of English language proficiency," saidpresenter Carole Perlman, Chicago Public

14

Schools. The state is developing large-scaleassessments for LEP students in Grades 3-11

that will be used in 1997. They will includemultiple-choice reading tests and writing as-sessments with both textual and graphicprompts. "In the Chicago Public Schools,"added Perlman, "the focus is on giving teach-ers of bilingual classes the tools to developand use performance assessments to docu-ment both native and English languageachievement."

Efforts to develop appropriate accommo-dations for LEP students are just beginning.Translating tests into students' native language

...if students haven't learned the con-tent in their native language, the value

of a translated test is debatable.

seems like a straightforward solution; and in-deed, the two states Rivera identified as offer-ing alternatives to their high school graduation

tests use translations. But if students haven'tlearned the content in their native language,the value of a translated test is debatable. "Fur-

thermore," Shepard warned, "linguistic andcultural differences make exact translation im-possible, casting doubt on the equivalency ofscores such tests yield." She called for moresmall-scale, focused research to provide goodinformation for making accommodation deci-sions. Agreeing that translations do not meetthe needs of all LEP students, Rivera nonethe-

Summer 1997

14


less called for their increased use, at least inthe short run, "especially for students whoenter schools with a high degree of literacy intheir native language."

Simplifying test language is another ac-commodation option. Based on his analysisof NAEP data, Jamal Abedi, CRESST/UCLA,cited evidence that complexly worded math-ematics questions depress LEP students'scores. Abedi discussed CRESST research in-dicating that simplified wording significantlyimproved scores, at least for some students.Additional projects are underway to examinethe effects of linguistic and other adaptationson limited English proficient and fully Englishproficient students of varying abilities.

Using mixed-ability, collaborative workgroups also may be an effectiveandinstructionally relevantform of accommo-dation. During her research in classrooms withhigh LEP populations, presenter Noreen Webb,

CRESST/UCLA, found that low-ability studentsscored better on performance assessmentswhen they worked on similar tasks in groupswith high-ability students prior to the test."High-ability students' scores were not af-fected," said Webb.

Although there are hints of promising di-rections, it is clear that there is much work tobe done to develop valid, fair, and useful testsfor all students. A complicating factor is thatthe work must be done with an eye toward mak-

ing assessments that the public understands

and trusts.

Evaluation Comment

CREDIBILITY

THE most valid.and fair assessment sys-

tems imaginable will fail if they lackpublic credibility," said Eva Baker dur-

ing her conference presentation. "Validity with-

out credibility produces assessments that haveno life span and whose findings are contended,

diminished, or dismissed," added Baker.Several presenters addressed the impor-

tance of CRESST's third prerequisite for util-ity, arguing that public communication andengagement are essential in establishing as-sessment system credibility.

Secrecy, driven partly by the legitimateneed for test security, has long been a trade-mark of the measurement community. But thepublic is increasingly reluctant to accept as-sessmentsnew or otherwiseon blindfaith. As a result, many members of the as-sessment community have found that theyneed improved communication and public re-lations skills to complement their technicalskills.

Lorraine McDonnell, CRESST /Universityof California, Santa Barbara, addressed thebroader social and political context, which de-mands better communication of assessmentinformation. Citing recent polls, McDonnellnoted that only 25% of the public trusts gov-ernment institutions to do the right thing allor most of the time, and only 25% of the vot-ers, who decide whether or not to support pub-

lic education with taxes, even have children in

15

15


school. Consequently, new assessment sys-tems must survive in a context of mistrust andlimited public understanding of education. "Inthis environment," McDonnell noted, "curricu-lar standards and assessments become the fo-

cal point for many contested social values, notjust about what is important to learn, but abouthow we define the good society and how those

ideals should be passed on to succesive gen-erations." Based on her research on the poli-tics of education reform in several states,McDonnell suggested several guidelines formaking assessments politically and publiclycredible.

...leadership has to come, at leastpartially, from people who areelectorally accountable..."

"First, where political will is lacking tomake a needed long-term investment," saidMcDonnell, "an incremental approach may ac-

tually yield better results than a comprehen-sive approach."

"Second, if a state decides to engagein substantial reform," McDonnell argued,"strong political leadership is necessary. Thatleadership has to come, at least partially, from

people who are electorally accountable,"added McDonnell, "not just from the educa-tion establishment and non-elected officials.Elected officials," she pointed out, "are in regu-

lar contact with constituents, and the con-straint of their two- or four-year electoral cyclehelps them bring a valuable, real-world per-spective to the process."

"Third," McDonnell asserted, "the devel-opment of new curriculum standards and as-sessments cannot solely be a technicalprocess with participation limited to experts.... Public participation in open, two-way dia-logues is very important," McDonnell noted,"because it involves public deliberation aboutwhat skills and knowledge are most impor-tant for a productive life and active citizen-ship." Acknowledging that building publicconsensus is a very difficult process,McDonnell warned that to avoid it would beto make a mockery of the notion of commonstandards.

"Communicating a topic as complex asassessment," said Leah Lievrouw, CRESST/UCLA, "is a formidable challenge in any mod-

ern, diverse society.... Despite improvementsin mass communication, we live in an era ofseparation characterized by high levels of in-tra-group conversation and low levels of in-ter-group communication," she added.Consequently, we may tend to target largemedia outlets while ignoring the types ofsmaller market electronic and print media tai-lored to many linguistic and ethnic minori-ties. "As a result," explained Lievrouw, "ourmessage may not get across to many of theseimportant groups, and we need to rethink our

media strategies."

Summer 1997

16


Richard Colvin, Los Angeles Times, em-phasized the crucial value in building publiccredibility, something that was not done by theCalifornia State Department of Education dur-ing the California Learning Assessment Sys-tem crisis. "CLAS was a forty-million-dollarmistake," said Colvin, "not because it produced

invalid or unfair results, but because of wide-spread public distrust caused by perceptionsthat there were weird things on the test." Cit-ing the bunker mentality that led to the demiseof CLAS, Colvin urged researchers and testmakers to engage in open and understandable

communication with the public. He offered sev-eral tests that he feels assessments must passto achieve public credibility:

The "barber chair" test. Ordinarypeople should be able to discuss theassessment in ordinary social situations.Assess familiar content that the public thinkschildren need to get along in the world.

The "realtor" test. Test scores affecthousing values and, consequently, influencehomeowners' support for local schools. Reportscores in a format simple enough that realtorscan use them as a closer.

The "newspaper" test. Newspapershave limited space for even the most important

stories. Give us results that we can fit into twoor three columns.

Evaluation Comment

Challenging the research community tomake testing understandable to the generalpublic, Colvin advised that "there is a paceyou can walk at that the public can follow, oryou can run out ahead and lose everybody."

McDonnell's, Lievrouw's, and, particularly,Colvin's remarks challengedeven nettledsome of the conference attendees, but thethemes were strongly endorsed by otherCRESST presenters who have been workinghard to build public support for high standardsand improved assessments. Moreover, thesepresenters universally argued that credibilityand successful implementation was not pos-sible without active, broad-based, public par-ticipation.

In a roundtable presentation, four stateeducation officialsDuncan MacQuarrie,Washington; Wayne Martin, Colorado; DorisRedfield, Virginia; and Catherine Smith, Michi-

ganidentified key constituencies that mustbe formally included in the process for anystate-level reform to be successful. The listincludes teachers, students, parents, schooladministrators, school board members, highereducation officials and admissions officers,education researchers, state legislators, thegovernor, representatives of the business com-

munity, and the media. Presenters noted that

teachers are almost always deeply involved inthe process from the beginning, but that othergroups, particularly school administrators andparents, should be involved more directly andearlier than usual.

17

17


Echoing Colvin's advice, these presentersalso stressed the importance of making cer-tain that standards and assessments includeitems that represent the public's general un-derstanding of a subject and what's important,for example, a computation standard in math-ematics or a grammar/spelling standard inwriting.

Discussing district-level reform, Los An-geles Board of Public Education PresidentMark Slavkin identified several keys in build-ing and maintaining credibility at the districtlevel.

Slavkin identified jargon as amajor threat to credibility...

"To keep public and political credibility,"said Slavkin, "it is necessary to keep one footin the old [norm-referenced assessments] aswe move to the new standards-based perfor-mance assessments."

Slavkin identified jargon as a major threatto credibility, recommending that everythingbe publicly disclosed in the process of devel-opment, and emphasizing the importance ofkeeping the media updated about progressalong the way.

But foremost, in Slavkin's opinion, is "buy-in" from the very beginning by parents, teach-ers, and community members. One exampleof a successful, if not always smooth, effort to

8

get such commitment has been a three-yearproject to develop language arts standards,curriculum, and performance assessments inthe Los Angeles Unified School District. De-scribed by Charlotte Higuchi, a CRESST part-

ner and LAUSD teacher, this project involveda large and diverse group of teachers, par-ents, and community members whose inputshaped the reform from the beginning.

Los Angeles Unified School District Su-perintendent Sidney Thompson also empha-sized the importance of buy-in at the schoollevel. "The people who have to do it at theschool site have to own it; they have to be-lieve it can be done, and if they believe thatand bring the parents into it, then we have apathway to get us there."

UTILITY

VALIDITY, fairness, and credibility are

necessary, but not sufficient, forsystem utility. Because assessment

systems usually serve multiple purposes andusers, often with different and competingneeds, it becomes very difficult to design asystem useful for all purposes and people.Randy Bennett, Educational Testing Service,expressed concern that the utility of a com-plex, multipurpose assessment system wouldbe similar to the Swiss Army Knife, service-able in a pinch for all sorts of jobs from re-moving screws to opening cans, but not idealfor any one of them.

Summer 1997

18


Like a sensitive ecosystem, the utility ofan assessment system will likely be greatestwhen the individual components areharmonious with the processes that link them.If any process or component is disrupted, the

"...if you don't give us good infor-mation about what works, and soon,those of us responsible for imple-menting assessment reforms maywell perish."

entire system suffers. Unfortunately, thepolitical environment in which assessmentexists is volatile and urgent as Judith Billings,then superintendent of education for the stateof Washington, noted.

"You [academicians] may publish," saidBillings, "whether new assessments work ornot. But if you don't give us good informationabout what works, and soon, those of usresponsible for implementing assessmentreforms may well perish."

In Washington, as in virtually all states, one

of the purposes of assessment reform has been

to change teaching and, thus, to improvestudent learning. Many conference presentersthis year focused on an element central to theCRESST assessment model, teacher capacitybuilding and teachers' changing instructionalpractices as a result of changes in assessmentmethods.

Evaluation Comment

ISSUES IN IMPROVING INSTRUCTION

AND TEACHER CAPACITY BUILDING

TEACHERS are a key to the credibility of

assessment reform and essential to itssuccess," said presenter Sid Thompson.

But Marilyn Monahan, secretary-treasurer ofthe National Education Association, warnedagainst the assumption that teachers will beable to immediately embrace reform. Stan-dards-based assessment reform demandschange in instructional practices. Monahanstated that teachers welcome this, but that those

who want reform must invest in teacher knowl-edge through professional development.

"The journey from what takes place at thisconference to what takes place in teachers'classrooms is long, complex, and unpredict-able," cautioned Monahan.

Agreeing with Monahan, CRESST partnerHilda Borko found in her research with teach-ers and students that teachers were interestedin assessment and instructional reform, but felt

they didn't have the time or expertise to de-velop alternative assessments on their own.Borko found that teachers initially inserted per-formance assessment into their instruction; that

is, the new assessments were added into analready busy instructional program. Whilesome teachers began to incorporate these as-sessments into their ongoing instruction by theend of the first year of the reform effort, it oftenwas not until the third year of the project, with

9

19


help from Borko and others, that teachers were

able to integrate assessment and instructionalreforms into their daily classroom activities.

Maryl Gearhart and Megan Franke,CRESST/UCLA, reported on action researchprojects focusing on assessment reform inmathematics. The integration of assessmentinto instruction, they argued, is essential tothe realization of mathematics reform envi-sioned by the National Council of Teachers ofMathematics standards. Gearhart and Franke's

research showed that teachers need deeper un-

derstanding of mathematics and children'smathematical reasoning in order to implementnew pedagogies at more than a superficiallevel. For example, although many teachersparticipating in their projects began to ask chil-

dren to share their thinking, only some teach-ers were able to probe with specific and

The journey from what takes place atthis conference to what takes placein teachers' classrooms is long, com-plex, and unpredictable...

substantive questions or to guide analyticaldiscussions of student problem-solving strat-egies.

"Classroom instruction and performanceassessment should be inseparable aspects ofthe education experience for language minor-ity students," urged Richard Duran, CRESST/

University of California, Santa Barbara. He rec-

ommended that teachers gather multiple forms

of evidence of student performanceclass-room tests, graded projects, student self-as-sessments, and videotapes revealing students'fluency with learning tools.

Agreeing with Duran, Thomas Romberg,University of Wisconsin, Madison, argued that"teachers need multiple assessment strategies

to accurately measure student performance,"adding that "in mathematics, teachers needto know if students can add, subtract, multi-ply and divide, and if they can put these piece-

meal skills together to solve routine andnonroutine problems."

"But teachers also need to know if students

can explain why an answer is correct," addedRomberg, "and how students navigate theproblem-solving process." He suggested cre-ating situations that enable teachers to gatherinformation from listening to students' expla-nations, observing students at work, and ex-amining the products of their work.Large-scale assessment can signal students'level of performance, but teachers need con-tinuous, detailed evidence of students' learn-ing processes and understanding to fullysupport the multiple assessment process.

That large-scale, standards-based perfor-mance assessments are not typically designed

to meet the information needs of the class-room teacher was also noted by Phil Daro,director for assessment development for theNew Standards Project. Daro recommended

20 Summer 1997

20


that one method to help teachers understandwhat standards demand of their students is tomake performance assessments look like well-constructed teacher-tests.

"Teachers do standards-based instructionand performance testing all the time," saidDaro. "The trick is to clarify the standards andassessments sufficiently for teachers torecognize the parallels with their own practice,"

added Daro, "but not so much that the truecomplexity of the reform effort is lost."

"We need to use large-scale performanceassessment systems to define standards andfocus school-level efforts on student worklinked to standards," agreed Lynn Winters,Long Beach (CA) Unified School District. Sheemphasized the need to engage teachers incontinuing professional activities to enablethem to implement effective, ongoing,standards-based classroom instruction andassessment.

"Professional development is the hardestsell in the reform marketplace," admittedCatherine Smith, Michigan State Departmentof Education, but she added that it was a vitalcomponent to the success of any reform effort.

USING TECHNOLOGY TO

CREATE NEW POSSIBILITIES

AT least a few of the major problemspresented by complex assessmentsystems might be resolved by improve-

ments in technology according to several con-

Evaluation Comment

ference presenters. Clearly, technology canhelp with managing, linking, and disseminat-ing assessment results. But technology alsomay permit test developers to increase authen-

ticity and open up the range of modalities andsystems of representation used for assess-ment.

"Future generations of tests will need totap nontraditional constructs, base testdesigns on cognitive principles, and increasethe diversity of problems types," noted RandyBennett, Educational Testing Service. In spiteof current logistic problems, Bennett predicted

that large-scale assessments would sooninclude computer-based presentations ofproblem types not possible with paper-and-pencil tests. Bennett shared multimediaprototype items using historical speeches andnewscasts to illustrate the potential ofpresenting and asking students to respond to"dynamic stimuli."

Ron Stevens, CRESST/UCLA, demon-strated the use of neural network technologyto permit real-time assessment of complexproblem solving. In one of Stevens' prototypes,

medical students were presented with realis-tically sketchy information about a patient'ssymptoms, a set of diagnostic tests that theycould order, and a "library" of reference mate-rials. As the students worked through the op-tions presented by the computer program, their

choices were recorded and could be compared

with patterns of hypotheses generated by ex-pert diagnosticians investigating the sameproblem.

21

21


Based on lessons learned from his effortsto develop computer-based assessments ofgroup and teamwork processes, Harold O'Neil,CRESST/University of Southern California,noted a number of problems endemic totechnology projects. These include the costsand time for software design and development,

inadequacies of existing telecommunicationstechnology, unavailability of sophisticatedtechnologies in public schools, and thecomplexity and cost of maintaining testsecurity. O'Neil also shared the substantialprogress his group has made in usingtechnology to measure the quality and quantityof individual contributions and group problemsolving. He pointed particularly to the potentialof collaborative concept mapping wherestudents work in teams through networkedtechnology to create and revise concept maps.

Collaborative concept mapping makespossible the real-time assessment andreporting of deep understanding and teamwork

performancean example of a potentiallyuseful and cost-effective near-term applicationof technology.

CONCLUSION

N their closing remarks to the 1996CRESST conference, Eva Baker andRobert Linn acknowledged the formidable

challenges ahead. Among them are assuringsystem validity; supporting the alignment be-tween standards and assessments; promoting

02

fairness; addressing the technical chal-lenges of measuring progress and linking dif-ferent assessments to address the needs andpurposes of many audiences at multiple lev-els; improving schools' and teachers' capac-ity; and productively engaging the public."These are all priorities for the research com-munity," said Baker and Linn. "They will callon the best of our technical skills along withvery considerable sociopolitical prowess."

"I think a key lesson of the past two days,"

concluded Baker, "is that we must approachthe assessment challenges we've discussedthrough greater collaboration. It's clear that weshare a commitment to improve education.Let's move forward together."

1997 CRESST CONFERENCE

September 4 through September 5,1997

Sunset Village Conference CenterUCLA Campus

Registration materials and complete detailswill be available in the Summer 1997 CRESSTLine and on the CRESST World Wide Website, www.cse.ucla.edu. Anyone requiringearly registration may contact Mary Wilby,CRESST/UCLA, 10920 Wilshire Blvd., Ste.900, Los Angeles, CA 90024; e-mail:[email protected]; phone: (310) 206-1532.

Summer 1997

CRESST REPORTS AVAILABLE Now!

The following CRESST reports areavailable by calling Kim Hurst, (310) 206-1532, or sending a message to Kim at:[email protected].

Reforming Schools by Reforming As-sessment: Consequences of the ArizonaStudent Assessment Program (ASAP):Equity and Teacher Capacity BuildingMary Lee SmithCSE Technical Report 425, 1997 ($9.00)

In this study, Mary Lee Smith and otherresearchers focused on how schools changedas a result of state-mandated standards andassessments.

The Politics of State Testing: Imple-menting New Student AssessmentsLorraine McDonnellCSE Technical Report 424, 1997 ($5.00)

Lorraine McDonnell continues her synthe-ses of innovative state assessment programsin Kentucky (Kentucky Instructional Results In-

formation System), California (CaliforniaLearning Assessment System), and NorthCarolina.

Teachers' Developing Ideas and Prac-tice About Mathematics PerformanceAssessment: Successes, StumblingBlocks, and Implications for Profes-sional Development

Evaluation Comment

CRESST TECHNICAL REPORTS

Hilda Borko, Vicky Mayfield, Scott Marion,Roberta Flexer, and Kate CumboCSE Technical Report 423, 1997 ($3.00)

This study focuses on the change processexperienced by a group of third-grade teach-ers as they implemented mathematics perfor-mance assessments in their classrooms.Based on workshop conversations and inter-views between teachers and the research/staffdevelopment team throughout a single schoolyear, the team reached five major conclusions.

New Writing Assessments: The Chal-lenge of Changing Teachers' BeliefsAbout Students as WritersShelby Wolf and Mary/ GearhartCSE Technical Report 422, 1997 ($3.00)

During a two-year collaboration with el-ementary school teachers, Wolf and Gearhartexamined ways that teachers' beliefs abouttheir students as writers mediated their invest-

ment in new methods of assessing students'writing.

Teachers' Beliefs About Assessmentand Instruction in LiteracyCarribeth Bliem and Kathryn DavinroyCSE Technical Report 421, 1997 ($3.00)

Bliem and Davinroy further investigateteachers' beliefs about assessment and itsconnection to instruction in literacy. Most ofthe data were drawn from transcripts of bi-weekly meetings between the research teamand third-grade teachers using performanceassessments in their classrooms.

23

CRESST TECHNICAL REPORTS

The Politics of Assessment: A ViewFrom the Political Culture of ArizonaMary Lee SmithCSE Technical Report 420, 1996 ($3.00)

Mary Lee Smith traces the events of theArizona Student Assessment Program (ASAP),

an innovative multiple assessment programthat grew out of discontent with mandatedstandardized testing in Arizona.

Implications of the OECD ComparativeStudy of Performance Standards forEducational Reform in the UnitedStatesEva L. BakerCSE Technical Report 419, 1996 ($3.00)

In this report, Eva Baker explores the im-plications for education reform in the UnitedStates of an OECD study of performance stan-

dards. Using a general model of educational

Fold here and mail.

reform, Baker analyzes the meaning of perfor-mance standards in the United States, ad-dressing key influences of tradition, diversity,control, and participation.

Assessment and Instruction in the Sci-ence ClassroomGail Baxter, Anastasia Elder, and Robert Glaser

CSE Technical Report 418, 1996 ($2.50)Findings from this study of fifth-grade stu-

dents provided further evidence that criticaldifferences exist between students who thinkand reason well with their knowledge andthose who do not.

CSE Technical Reports may also be found on

the CRESST Web site: www.cse.ucla.edu.

= O= 0 0 0 0 0 =I CI == = CI 0 = = 0 0 0= 0 0 0 =7 0 0 CM =I I=

UCLA Center for the Study of Evaluation10920 WILSHIRE BLVD SUITE 900

LOS ANGELES CA 90024-6511

C=I I=3

Place

postagehere.

24 Summer 1997

I I 1

CSE Number Title No. of Copies Price Each Total

Postage and Handling Order Subtotal

Postage & Handling

Residents add 8.25%

TOTAL

Subtotal of: Up to $10 add $1.50California

$10.01 $20 add $2.50

$20.01 $50 add $3.50over $50 add 10% of Subtotal

(Please prepay orders less than $10.00.)

NAME AND MAILING ADDRESS PLEASE PRINT OR TYPE

0 I would like to receive

free copies of TheCRESS T Line and

Evaluation Comment.

UCLA Center for the Study of Evaluation1320 Moore Hall/Mailbox 951522Los Angeles CA 90095-1522

www.cse.ucla.edu

ADDRESS CORRECTION REQUESTED (ED 4M)

Dick Hays

Deputy Asst. Secretary

US Dept. or Education

555 New Jersey Ave., NW

Washington DC 20208

NON PROFIT ORG.

U.S. POSTAGE

PAID

U.C.L.A.

25

[21

(9/92)

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement (OEM)

Educational Resources Information Center (ERIC)

NOTICE

REPRODUCTION BASIS

ERIC

This document is covered by a signed "Reproduction Release(Blanket)" form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Releaseform (either "Specific Document" or "Blanket").

Documents

25p. · DOCUMENT RESUME. ED 413 349 TM 027 657. AUTHOR Land, Robert TITLE Moving Up to Complex Assessment Systems. Proceedings from. the CRESST Conference (Los Angeles, CA, September