Embed Size (px)
Citation preview
DOCUMENT RESUME
ED 418 102 TM 028 196
AUTHOR Chung, Gregory K. W. K.; Herl, Howard E.; Klein, Davina C.D.; O'Neil, Harold F., Jr.; Schacter, John
TITLE Estimate of the Potential Costs and Effectiveness of ScalingUp CRESST Assessment Software.
INSTITUTION National Center for Research on Evaluation, Standards, andStudent Testing, Los Angeles, CA.
SPONS AGENCY Office of Educational Research and Improvement (ED),Washington, DC.
REPORT NO CSE-TR-462PUB DATE 1997-12-00NOTE 33p.
CONTRACT R305B60002-97PUB TYPE Numerical/Quantitative Data (110) Reports Evaluative
(142)
EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS Computer Assisted Testing; *Computer Software; *Concept
Mapping; *Educational Assessment; Integrated Activities;Problem Solving; Scoring; *Simulation; *TechnologicalAdvancement; *Test Construction
IDENTIFIERS Center for Research on Eval Standards Stu Test CA
ABSTRACTThis report examines issues in the scale-up of assessment
software from the Center for Research on Evaluation, Standards, and StudentTesting (CRESST). "Scale-up" is used in a metaphorical sense, meaning addingnew assessment tools to CRESST's assessment software. During the past severalyears, CRESST has been developing and evaluating a series of softwareproducts to measure various types of learning. Collectively, these are calledthe Integrated Assessment System. The integrated simulation CRESST hasdeveloped includes both collaborative and individual concept mapping tasks, aproblem-solving search task, an explanation task, and a metacognitivequestionnaire. New tools should measure one or more aspects of the CRESSTmodel of learning, be stand-alone or integrate with other tools, haveauthoring capacity, use automated scoring, and be Internet-deployable.Specifications are listed for eight proposed tools, and each is evaluated forthe CRESST requirements. The tools are: (1) Collaborative Concept Mapper, a
tool for group use; (2) Flowcharter, a tool to allow students to sequenceevents; (3) Idea Generator and Evaluator, a tool for group problem solving;(4) Model Simulator, a model building tool; (5) Networked Team Simulator, a
team model builder; (6) Multimedia Concept Mapper, a multimedia tool forconcept mapping; (7) Outliner, an organizing tool; and (8) Problem Solver, aninstrument to measure problem solving processes. (Contains 4 tables and 13references.) (SLD)
********************************************************************************
Reproductions supplied by EDRS are the best that can be madefrom the original document.
********************************************************************************
NO?-4
00
et
A.1. MR MR. .
. .
Estimate of the Potential Costs and Effectiveness ofScaling Up CRESST Assessment Software
CSE Technical Report 462
Gregory K W. K Chung, Howard E. Herl, Davi C.CRESST/University of California, Los Anrtageles
D. Klein
Harold F. O'Neil, Jr.University of Southern California /CRESST
John SchacterCRESST /University of California, Los Angeles
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL
HAS BEEN GRANTED BY
e"%-( fNA 4.
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
U S DEPARTMENT OF EDUCATIONOttic f Educational Research and Improvement
ED ATIONAL RESOURCES INFORMATIONCENTER (ERIC)
This document has been reproduced asreceived from the person or organization
originating itMinor changes have been made to
improve reproduction quality
Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position cr policy
4.
.
UCLA Center for theStudy of Evaluation
in collaboration with:
University of Colorado
NORC, University of Chicago
LRDC, Universityof Pittsburgh
University of California,Santa Barbara
University of SouthernCalifornia
11. The RAND
Corporation
AIM BEST COPY AVAILABLE
Estimate of the Potential Costs and Effectiveness ofScaling Up CRESST Assessment Software
CSE Technical Report 462
Gregory K. W. K. Chung, Howard E. Herl, Davina C. D. KleinCRESST/University of California, Los Angeles
Harold F. O'Neil, Jr.University of Southern California/CRESST
John SchacterCRESST/University of California, Los Angeles
December 1997
Center for the Study of EvaluationNational Center for Research on Evaluation,
Standards, and Student TestingGraduate School of Education & Information Studies
University of California, Los AngelesLos Angeles, CA 90095-1522
(310) 206-1532
Copyright © 1997 The Regents of the University of California
The work reported herein was supported under the Educational Research and Development CenterProgram, PR/Award Number R305B60002, as administered by the Office of Educational Researchand Improvement, U.S. Department of Education.
The findings and opinions expressed in this report do not reflect the positions or policies of theNational Institute on Student Achievement, Curriculum, and Assessment, the Office of EducationalResearch and Improvement, or the U.S. Department of Education.
ESTIMATE OF THE POTENTIAL COSTS AND EFFECTIVENESS OF SCALING
UP CRESST ASSESSMENT SOFTWARE1
Gregory K. W. K. Chung2, Howard E. Her12,Davina C. D. Kleine, Harold F. O'Neil, Jr.3, and John Schacter2
In this report, we examine issues in the scale-up of CRESST assessmentsoftware. We use the term "scaling up" in a metaphorical sense, meaning what
new assessment tools should be added to our assessment software. Criteria fornew assessment tools are based upon the need to develop additional evaluationtools that would be useful and relevant to students, teachers and the business
community.
During the past several years CRESST has been developing and evaluating aseries of software products to measure various types of learning. These arecollectively called the Integrated Assessment System. This system has been
documented in a series of reports to the Department of Defense's ComputerAided Education & Training Initiative (CAETI) (Baker & O'Neil, 1996; Herl &O'Neil, 1996; Herl et al., 1996; Klein, O'Neil, & Baker, 1996) and a recent series ofconference papers presented at the 1997 annual meeting of the AmericanEducational Research Association (Chung, O'Neil, Herl, & Dennis, 1997; Herl,O'Neil, Chung, & Dennis, 1997; Klein, O'Neil, Dennis, & Baker, 1997; O'Neil,1997; Schacter et al., 1997). These products are model-based in the sense ofreflecting the five types of learning that are specified in the CRESST model oflearning (Baker, 1995; see Figure 1).
CRESST has assembled a suite of performance assessment tasks (ourintegrated simulation) onto which we have mapped the types of learningexpected of students. The design of integrated simulation performanceassessment has the following characteristics: (a) relevant, project-based scenariosthat include meaningful, real-world tasks; (b) individual and team processes andproducts; (c) a technology base using Web-based, networked systems; and (d)model-based assessments that integrate types of cognitive learning, grade level,and content area.
1 Authors are listed alphabetically.2 CRESST/University of California, Los Angeles3 CRESST/University of Southern California
1
ContentUnderstanding
1,
Collaboration Learning Problem Solving
I-.
Communication Metacognition
Figure 1. The CRESST model of learning.
The integrated simulation we have developed includes both collaborativeand individual concept mapping tasks, a problem-solving search task, anexplanation task, and a metacognition questionnaire. It is designed to be useful tostudents at different ages, given different levels of understanding, and withdifferent content areas. Its administration is computer-based; the softwarecontrols and documents progress, while permitting real-time scoring andreporting. For assessing content understanding we use a concept mappingapproach that is computer-based. To assess collaborative skills, we use acollaborative mapper (Chung et al., 1997), and to assess problem solving, we use aWeb-based approach to support a problem-solving search task (Schacter et al.,1997). To assess metacognition, we use a questionnaire to measure both trait andstate metacognition (e.g., see O'Neil & Abedi, 1996, for a description of the stateversion). In general, the evaluation data for our assessment tools show them tobe feasible. However, there is limited reliability and validity information. Suchstudies are planned. Moreover, there is an issue of scaling up such an integratedsimulation: What new assessment tools should be added to the integratedsimulation?
New Computer-Based Assessment Tools
The proposed assessment tools will support Project 1.3 goals of the CRESSTgrant proposal to the Office of Educational Research and Improvement (Baker,
2
Linn, & Herman, 1995), particularly the goals for Advanced Technology and
Quality Improvement and Utility Enhancement. We suggest that basic
assessment research of such tools through prototype development would befunded by Project 1.3, and product refinement and possible commercializationwould be funded through other sources (e.g., NCES or commercial partners).
In general, two classes of software are needed to design and developcomputer-based automated assessment tools: (a) individual assessment tool
applications software, and (b) application programming interfaces (APIs).
Application programming interfaces provide transparent layers for collecting,
parsing, managing, analyzing, and outputting data. Application programminginterfaces provide assistance for tool evaluation and performance (e.g., data
logging and front-end and back-end interface configuration). End users (e.g.,
students and trainees) interact directly with the application software assessment
tools (for example, the Multimedia Concept Mapper) but indirectly with theapplication programming interfaces.
Tool Design Strategy
Our strategy is to design assessment tools that (a) measure one or morecomponents of the CRESST model of learning, (b) are stand-alone or integrate
with other tools, (c) have authoring capability, (d) employ automated scoring and
reporting, and (e) are Internet-deployable in both Windows and Macintosh
environments.
Tool integration. Each tool will be designed as both a stand-alone
instrument and part of an integrated assessment suite. To illustrate this point,the software package Microsoft Office is analogous to a tool suite or ourintegrated simulation, and the applications within Microsoft Office, such asWord, Power Point, and Excel, are analogous to individual component tools. To
maximize integration, all tools will be designed around a set of application
programming interfaces. These application programming interfaces are sets of
code and services that are independent of any specific tool. All tools will collect
and log users' performance and process data. Instead of creating (and recreating)
code to perform data logging on a per-tool basis, the data logging application
programming interface will provide one set of standard data logging routinesaccessible by all tools. Authoring, scoring, reporting, and analysis are other areas
3
that would benefit from the development of application programming interfacesoftware.
Authoring. All tools will be authorable. That is, all tools will provide userswith the capability to define content, establish scoring criteria, and customize theinterface of the tool to adapt to audiences of different ages. For younger users,interfaces would be adjusted to be sparse and iconic, whereas for older, moreexperienced users, interfaces can be adjusted to show the more complex featuresof the software. An example of an adjustable interface is the Microsoft WordTool Bar. Users can customize this tool bar such that either very few or manyoptions appear on it. Our authoring capabilities will take a similar approach.
Automated scoring and reporting. A goal of all CRESST applications is real-time automated scoring and reporting. Automated scoring and reportingprocesses involve real-time data collection, management, and analysis.
Therefore, underlying application programming interfaces will be written tohandle these real-time processes for all tools. Although each tool will havespecific reporting requirements, many of the underlying functions (e.g.,
calculating statistics) will be handled by an application programming interface.
Internet-deployable and platform-independent. All tools will be designed tobe deployable across the Internet for both the Windows and Macintoshenvironments. Thus, tools will be developed in an architecture-neutral languagesuch as JavaTM or in applications that create downloadable plug-ins that run onboth platforms. In addition, all tools will be compatible with NetscapeNavigatorTM and Microsoft Explorer"' Internet browsers.
Spiral model of software development. Our assumption is that we will bedeveloping software. Given thisand because such software will likely haveevolving requirementswe believe a spiral model is the appropriatedevelopment model. The spiral model minimizes risks through incrementaldevelopment of software. In concrete terms, periodic deliveries of prototypes areplanned. The goal of each prototype cycle is to design and implement only thehighest priority features. Each successive prototype cycle incorporates morefeatures. The benefit of the spiral model is that technical and functionalproblems are identified early in the development cycle. This early riskidentification means that we can make informed decisions about whether tocancel or continue a project. An added benefit of this development model is that
4
there is high product visibility: There is always a working prototype that can be
demonstrated.
Proposal assessment tools. Table 1 lists each proposal assessment tool andthe component of the CRESST model that serves as the measurement target.Table 2 presents a list of assessment tools and their respective applicationprogramming interfaces. Detailed descriptions of each tool are found in the Tool
Specifications section.
Tool Specifications
In this section, specifications for a series of tools are proposed. Table 3
provides a first cut of the specifications for each proposed tool, and Table 4provides a first cut of specifications for each proposed application programminginterface.
Next Steps
We need to prioritize the development activities.
Table 1
Tools Proposed and Cognitive Families of Learning
Tools
Contentunder-
standingProblemsolving
Communi-cation
Collabor-ation
Metacog-nition
Collaborative Concept Mapper X X X X X
The Flowcharter X X
Idea Generator and Evaluator X X X
Model Simulator X X
Networked Team Simulator X X X X
Multimedia Concept Mapper X X
The Outliner X X
Problem Solver X X X X X
5
Tab
le 2
App
licat
ion
Prog
ram
min
g In
terf
ace
(API
) Su
ppor
tfo
r A
sses
smen
t Too
ls
Dat
aC
ol la
bor-
Sim
u-T
ext
Proc
ess
Aut
hori
nglo
ggin
gR
epor
ting
ativ
eSe
arch
ing
latio
ncl
assi
fier
Scor
ing
anal
yzer
API
API
API
API
API
API
API
API
API
Col
labo
rativ
e C
once
pt M
appe
rX
XX
XX
XX
X
The
Flo
wch
arte
rX
XX
XX
X
Idea
Gen
erat
or &
Eva
luat
orX
XX
XX
XX
Mod
el S
imul
ator
XX
XX
XX
XX
Net
wor
ked
Tea
m S
imul
ator
XX
XX
XX
Mul
timed
ia C
once
pt M
appe
rX
XX
XX
X
The
Out
liner
XX
XX
XX
Prob
lem
Sol
ver
XX
XX
XX
X
Tab
le 3
Spec
ific
atio
ns f
or A
sses
smen
t Too
ls
Ass
essm
ent T
ool:
Col
labo
rativ
e C
once
pt M
appe
r
Bri
ef P
rodu
ct S
tate
men
t: T
he J
ava-
base
d C
olla
bora
tive
Con
cept
Map
per
will
use
the
Java
-bas
ed I
ndiv
idua
l Con
cept
Map
per
and
build
into
itbo
th s
ynch
rono
us a
nd a
sync
hron
ous
colla
bora
tive
supp
ort c
apab
ility
. A g
roup
of
stud
ents
, bus
ines
s pe
ople
, par
ents
, and
so
on,c
ould
use
this
tool
colla
bora
tivel
y to
dev
elop
con
cept
ual u
nder
stan
ding
of
an a
rea.
Pro
visi
ons
will
be
mad
e to
neg
otia
te w
ho le
ads
the
colla
bora
tion
and
whe
n. A
fron
t-en
d us
er in
terf
ace
will
be
deve
lope
d fo
r en
teri
ng th
e co
ncep
t ter
ms
and
links
of
choi
ce. T
his
fron
t-en
d in
terf
ace
will
hav
e sy
nchr
onou
sand
asyn
chro
nous
col
labo
rativ
e ca
pabi
litie
s su
ch th
at u
sers
can
agr
ee o
n th
e te
rms
and
links
to u
se.
CR
ESS
T le
arni
ng f
amili
esC
omm
unic
atio
n, C
olla
bora
tion,
Con
tent
Und
erst
andi
ng, P
robl
em S
olvi
ng
Proj
ect 1
.3 a
rea
Impr
ovem
ent a
nd U
tility
Enh
ance
men
t
Rat
iona
leT
his
is a
mul
ti-us
er a
sses
smen
t and
lear
ning
tool
that
is d
omai
n- a
nd a
udie
nce-
inde
pend
ent.
It is
adap
tabl
e to
any
sub
ject
or
cont
ent a
rea
and
can
be s
core
d in
rea
l tim
e us
ing
auto
mat
ed te
chni
ques
.
Dom
ain
Dom
ain-
and
aud
ienc
e-in
depe
nden
t
Tas
kC
onst
ruct
a c
once
pt m
ap c
olla
bora
tivel
y
Perf
orm
ance
mea
sure
sC
once
pt m
appi
ng; c
olla
bora
tive
and
com
mun
icat
ive
inte
ract
ions
Proc
ess
mea
sure
sSt
ate
map
con
stru
ctio
n ov
er ti
me;
use
of
colla
bora
tive
inte
ract
ions
Con
figu
rabi
lity
Ada
ptab
le f
ront
-end
use
r in
terf
ace
and
expe
rt s
cori
ng te
mpl
ate
Scor
ing
Aut
omat
ed; e
xper
t per
form
ance
(H
erl m
etri
c: H
erl,
Bak
er, &
Nie
mi,
1996
)
Rep
ortin
gA
utom
ated
; fin
al p
erfo
rman
ce, p
roce
ss p
erfo
rman
ce (
i.e.,
use
of in
tera
ctio
n, m
ap c
onst
ruct
ion
over
time,
pro
duct
ive
colla
bora
tive
beha
vior
s)
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le5-
10 m
onth
s pr
oduc
tion
wor
k an
d te
stin
g
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)C
ompl
etel
y sc
alab
le
4.
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Flow
char
ter
Bri
ef P
rodu
ct S
tate
men
t: Fl
owch
artin
g to
ol th
at a
llow
s st
uden
ts to
seq
uenc
e ev
ents
. Usa
ble
in a
var
iety
of
cont
ent
dom
ains
and
acr
oss
grad
ele
vels
(e.
g., h
isto
rica
l flo
w, p
rogr
amm
ing,
mat
hem
atic
al a
lgor
ithm
s, e
xper
imen
tal m
etho
dolo
gy).
Stu
dent
flo
wch
arts
can
be s
core
dau
tom
atic
ally
onl
ine.
CR
ESS
T le
arni
ng f
amili
esC
onte
nt U
nder
stan
ding
, Pro
blem
Sol
ving
Proj
ect 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leW
ithou
t the
com
plic
atio
ns o
f na
tura
l lan
guag
e sc
orin
g, f
low
char
ting
allo
ws
for
insp
ectio
n of
stu
dent
unde
rsta
ndin
g of
bas
ic p
roce
dure
s, p
rinc
iple
s, s
eque
ntia
l ord
erin
g, a
nd p
robl
em s
olvi
ng.
Dom
ain
Dom
ain-
and
aud
ienc
e-in
depe
nden
t
Tas
kSt
uden
ts c
reat
e fl
owch
arts
to s
how
the
prog
ress
ion
of e
vent
s fo
r a
give
n pr
oble
m s
tate
men
t.
Perf
orm
ance
mea
sure
sSt
uden
t flo
wch
art
Proc
ess
mea
sure
sN
one
Con
figu
rabi
lity
Com
plet
ely
adap
tabl
e to
con
tent
and
aud
ienc
e ne
eds;
ada
ptab
le f
ront
-end
use
r in
terf
ace
as w
ell a
sba
ck-e
nd s
cori
ng a
nd r
epor
ting
inte
rfac
e
Scor
ing
Aut
omat
ed; b
ased
on
teac
her
inpu
t
Rep
ortin
gA
utom
ated
on-
line
and
off-
line
repo
rtin
g av
aila
ble;
in a
dditi
on, s
tude
nt r
espo
nses
may
be
prin
ted
for
teac
her
insp
ectio
n
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le6-
10 m
onth
s; r
equi
res
wor
king
ver
sion
of
essa
y-sc
orin
g A
PI f
or b
est r
esul
ts
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)C
ompl
etel
y sc
alab
le1
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Idea
Gen
erat
or a
nd E
valu
ator
Bri
ef P
rodu
ct S
tate
men
t: T
he s
hell
for
this
pro
duct
is a
mul
ti-us
er J
ava-
base
d bu
lletin
boa
rd. A
gro
uppr
oble
m is
pos
ted.
Gro
up m
embe
rs a
reas
ked
to c
ome
up w
ith p
oten
tial s
olut
ions
or
idea
s to
sol
ve th
e pr
oble
m. A
ll id
eas
are
post
ed to
the
bulle
tin b
oard
. One
to tw
o id
eas
are
then
rand
omly
ass
igne
d an
d m
aile
d to
an
indi
vidu
al g
roup
mem
ber
for
eval
uatio
n an
d re
view
. Ide
as a
reev
alua
ted
usin
g th
e C
RE
SST
pro
blem
-so
lvin
g ru
bric
. Qua
lity
of id
eas
and
qual
ity o
f ev
alua
tions
of
idea
s ar
e as
sess
ed.
CR
ESS
T le
arni
ng f
amili
esC
omm
unic
atio
n, C
olla
bora
tion,
Pro
blem
Sol
ving
Proj
ect 1
.3 a
rea
Impr
ovem
ent a
nd U
tility
Enh
ance
men
t
Rat
iona
leT
his
is a
mul
ti-us
er a
sses
smen
t and
lear
ning
tool
that
is d
omai
n- a
nd a
udie
nce-
inde
pend
ent.
It is
adap
tabl
e to
any
sub
ject
or
cont
ent a
rea
and
can
be s
core
d us
ing
auto
mat
ed te
chni
ques
com
bine
dw
itha
hist
ory
data
base
of
prob
lem
s pr
opos
ed a
nd e
xem
plar
yso
lutio
ns a
nd e
valu
atio
ns.
Dom
ain
Dom
ain-
and
aud
ienc
e-in
depe
nden
t
Tas
kU
sers
pro
pose
, rev
iew
, and
eva
luat
e id
eas
for
solv
ing
a co
mpl
ex p
robl
em.
Perf
orm
ance
mea
sure
sW
ritte
n id
eas,
eva
luat
ion
of o
ther
s' id
eas
Proc
ess
mea
sure
sC
lear
com
mun
icat
ion
of id
eas;
ana
lysi
s of
idea
s; d
evel
opm
ent o
f id
eas
afte
ran
alys
es
Con
figu
rabi
lity
Com
plet
ely
adap
tabl
e to
con
tent
and
aud
ienc
e ne
eds
Scor
ing
Aut
omat
ed; e
xper
t-ba
sed
eval
uatio
ns; c
ompi
led
hist
ory
of p
robl
em s
olut
ions
and
eva
luat
ions
libra
ryR
epor
ting
Aut
omat
ed; q
ualit
y of
idea
s an
d qu
ality
of
eval
uatio
ns a
re r
epor
ted
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le4-
10 m
onth
s
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)C
ompl
etel
y sc
alab
le
IE
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Mod
el S
imul
ator
Bri
ef P
rodu
ct S
tate
men
t: T
his
tool
will
let s
tude
nts
build
a c
once
ptua
l mod
el a
nd in
clud
e a
sim
ulat
ion
optio
n.St
uden
ts w
ill b
uild
som
e sy
stem
(a s
yste
m is
a s
et o
f co
mpo
nent
s th
at r
elat
e to
eac
h ot
her
caus
ally
). W
ith th
e si
mul
atio
n op
tion
they
will
be
able
to r
un th
e sy
stem
and
get
feed
back
on
thei
r m
odel
. The
sim
ulat
ion
(or
"run
") o
ptio
n ca
n al
so b
e us
ed to
sco
re th
e m
odel
. The
stu
dent
s w
illbu
ild th
eir
mod
el v
isua
lly u
sing
a to
olki
t of
wid
gets
and
obj
ects
.
CR
ESS
T le
arni
ng f
amili
esPr
oble
m S
olvi
ng, M
etac
ogni
tion
PRO
JEC
T 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leM
ost a
sses
smen
t mea
sure
s do
not
req
uire
stu
dent
s to
con
stru
ct s
omet
hing
that
they
hav
e to
get
wor
king
. Usu
ally
, the
ass
essm
ent m
easu
re r
equi
res
them
to r
epre
sent
thei
r un
ders
tand
ing
in s
tatic
term
s as
a s
et o
f in
terr
elat
ed c
once
pts.
Rar
ely
are
stud
ents
req
uire
d to
app
ly th
eir
unde
rsta
ndin
g.
Dom
ain
Any
are
a th
at c
an b
e re
pres
ente
d as
a c
ause
-eff
ect s
yste
m o
f in
tera
ctin
g co
mpo
nent
s
Tas
kSt
uden
ts a
re g
iven
wid
gets
rep
rese
ntin
g m
ajor
com
pone
nts
of s
yste
m. T
hey
are
requ
ired
tode
mon
stra
te th
eir
unde
rsta
ndin
g of
the
syst
em b
y bu
ildin
g a
wor
king
mod
el.
Perf
orm
ance
mea
sure
s(1
) R
unna
bilit
y of
sys
tem
; (2)
num
ber
of c
ompo
nent
s us
ed; (
3) q
ualit
y an
d si
gnif
ican
ce o
fre
latio
nshi
ps b
etw
een
com
pone
nts;
(4)
par
sim
ony
of m
odel
; (5)
com
pany
to e
xpan
d m
odel
Proc
ess
mea
sure
s(1
) T
rial
s to
suc
cess
; (2)
pat
h to
sol
utio
n (t
rial
-err
or, s
yste
mat
ic);
(3)
use
of
feed
back
; (4)
pro
blem
appr
oach
(bo
ttom
-up,
top-
dow
n)
Con
figu
rabi
lity
Abl
e to
set
par
amet
ers
of s
imul
atio
n m
odel
; abl
e to
con
figu
re c
ompo
nent
s of
mod
el; a
ble
to a
ssig
nic
ons
to m
odel
Scor
ing
(1)
Run
nabi
lity
of m
odel
; (2)
com
pari
son
with
exp
ert
Rep
ortin
gA
utom
ated
; qua
lity
of m
odel
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le3
mon
ths
for
build
-onl
y; 1
2 m
onth
s fo
r si
mul
atio
n co
mpo
nent
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)Sc
alab
le b
ut u
se is
lim
ited
by s
imul
atio
n co
nten
t
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Net
wor
ked
Tea
m S
imul
ator
Bri
ef P
rodu
ct S
tate
men
t: T
his
tool
will
be
a ne
twor
ked
sim
ulat
ion
with
in a
team
wor
k fr
amew
ork
that
will
req
uire
gro
upm
embe
rs to
bui
ld a
runn
able
mod
el o
f a
syst
em. T
he ta
rget
sys
tem
is c
ompr
ised
of
a se
t of
inte
rrel
ated
"bl
ack
boxe
s."
Eac
h m
embe
r ge
ts a
blac
k bo
x to
bui
ld a
nd h
asto
inte
grat
e w
ith o
ther
team
mem
bers
. The
sys
tem
wor
ks o
nly
if e
ach
blac
k bo
x w
orks
.
CR
ESS
T le
arni
ng f
amili
esC
olla
bora
tion,
Pro
blem
Sol
ving
, Met
acog
nitio
n
Proj
ect 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leT
he c
urre
nt te
amw
ork
mea
sure
s of
pro
cess
es a
nd s
kills
are
bas
ed o
n ta
sks
that
wea
kly
coup
le g
roup
mem
bers
the
succ
ess
of th
e te
am is
not
dep
ende
nt o
n al
l mem
bers
. A b
lack
box
/sys
tem
sim
ulat
ion
mea
ns th
at th
e su
cces
s of
the
team
dep
ends
on
the
succ
ess
ofea
ch m
embe
r (h
igh
face
val
idity
for
mea
suri
ng w
orkf
orce
rea
dine
ss).
Dom
ain
Any
thin
g th
at c
an b
e re
pres
ente
d as
a n
etw
ork
of c
ausa
lly r
elat
ed c
ompo
nent
s. P
roba
bly
scie
nce/
engi
neer
ing-
type
pro
blem
s. M
ay a
lso
be a
pplic
able
to c
lass
ic s
urvi
val p
robl
ems.
Tas
kG
roup
mus
t bui
ld a
wor
king
sys
tem
. Eac
h m
embe
r is
ass
igne
d on
e co
mpo
nent
of
the
syst
em. M
embe
rsw
ork
toge
ther
to in
tegr
ate
com
pone
nts
into
sys
tem
. Sys
tem
run
s w
hen
all c
ompo
nent
s ar
e w
orki
ng.
Perf
orm
ance
mea
sure
s(1
) W
heth
er s
yste
m w
orks
; (2)
qua
lity
of s
olut
ion
(e.g
., to
lera
nce
of s
yste
mdo
es s
yste
m r
un u
nder
all c
ondi
tions
or
does
sys
tem
wor
k on
ly u
nder
cer
tain
con
ditio
ns);
(3)
if s
yste
m d
oesn
't w
ork,
sco
re b
yqu
ality
of
com
pone
nts.
Proc
ess
mea
sure
sT
eam
wor
k pr
oces
ses;
tria
ls to
sol
utio
n; ty
pe o
f hy
poth
esis
/test
ing
stra
tegy
Con
figu
rabi
lity
Abl
e to
set
par
amet
ers
of s
imul
atio
n m
odel
; abl
e to
con
figu
re c
ompo
nent
s of
mod
el; a
ble
to a
ssig
nic
ons
to m
odel
Scor
ing
Aut
omat
ed; h
igh-
ver
sus
low
-com
pari
son
Rep
ortin
gT
eam
wor
k pr
oces
ses,
sim
ulat
ion
resu
lts
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le9-
12 m
onth
s
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)Sc
alab
le b
ut u
se is
lim
ited
by s
imul
atio
n co
nten
t
Tab
le 3
(co
ntin
ued)
Bri
ef P
rodu
ct S
tate
men
t: M
ultim
edia
Con
cept
Map
per
Bri
ef P
rodu
ct S
tate
men
t: T
his
conc
ept m
appi
ng to
ol is
a v
aria
tion
of th
e In
divi
dual
Con
cept
Map
per.
Stu
dent
s co
nstr
uct c
once
pt m
apsu
sing
the
indi
vidu
al c
once
pt m
appe
r's c
onst
ruct
ion
syst
em. M
ultim
edia
imag
es (
grap
hics
, sou
nd, a
nd v
ideo
) ca
n be
impo
rted
by
stud
ents
to s
erve
as
inst
ance
s of
con
cept
s an
d lin
ks. M
ultim
edia
imag
es (
i.e.,
phys
ical
spa
ce r
epre
sent
atio
ns, g
eogr
aphi
cal m
aps,
and
timel
ines
) ca
n al
so b
e us
ed to
add
dim
ensi
onal
ity to
the
map
ping
are
a.
CR
ESS
T le
arni
ng f
amili
esC
onte
nt U
nder
stan
ding
Proj
ect 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leC
once
pt m
aps'
con
stru
ctio
n sy
stem
s ar
e cu
rren
tly te
xt-b
ased
. Thi
s to
ol w
ill a
llow
stu
dent
s to
cons
truc
t and
enh
ance
thei
r co
ncep
t map
s th
roug
h th
e us
e of
gra
phic
s an
d so
und.
Stu
dent
s po
sses
sing
mor
e lim
ited
voca
bula
ries
may
ben
efit
from
vis
ual a
nd/o
r au
rals
timul
i.
Dom
ain
Dom
ain-
inde
pend
ent
Tas
kSt
uden
ts u
se th
e in
divi
dual
con
cept
map
ping
con
stru
ct s
yste
m. T
hey
can
acce
ss a
nd im
port
imag
esus
ing
the
mul
timed
ia im
age
data
base
bui
lder
to e
labo
rate
thei
r co
ncep
t map
s. S
tude
nts
cons
truc
ting
hist
ory
conc
ept m
aps
may
be
prom
pted
to p
ositi
on c
once
pts
with
in ti
mel
ine
and
geog
raph
ical
dim
ensi
ons.
Stu
dent
s co
nstr
uctin
g sc
ienc
e m
aps
may
be
prom
pted
to m
ap c
once
pts
onto
phy
sica
l spa
cedi
men
sion
s.
Perf
orm
ance
mea
sure
sSe
man
tic c
onte
nt a
nd o
rgan
izat
iona
l str
uctu
re s
core
s ca
n be
obt
aine
d us
ing
expe
rt-b
ased
sco
ring
crite
ria.
Proc
ess
mea
sure
sN
one
Con
figu
rabi
lity
Ada
ptab
le to
con
tent
dom
ain
Scor
ing
Exp
ert-
cons
truc
ted
conc
ept m
aps
can
be u
sed
to p
rovi
de a
utom
ated
sco
ring
of
stud
ents
' map
s.
Rep
ortin
gA
utom
ated
rep
ortin
g of
indi
vidu
al m
ap s
core
s
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 100
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le6-
9 m
onth
s to
dev
elop
sof
twar
e
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)D
epen
dent
upo
n us
er h
ardw
are
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Out
liner
Bri
ef P
rodu
ct S
tate
men
t: O
utlin
ing
tool
to a
llow
stu
dent
s to
org
aniz
e im
port
ant
info
rmat
ion
with
in th
e fr
amew
ork
of a
pre
wri
ting
activ
ity.
Usa
ble
in a
var
iety
of
cont
ent d
omai
ns a
nd a
cros
s gr
ade
leve
ls. S
tude
nt o
utlin
es c
anbe
sco
red
auto
mat
ical
ly o
nlin
e.
CR
ESS
T le
arni
ng f
amili
esC
onte
nt U
nder
stan
ding
, Com
mun
icat
ion
Proj
ect 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leSi
mila
r to
ess
ay w
ritin
g ta
sks,
but
sco
ring
issu
es a
re s
impl
ifie
d by
the
outli
ne s
truc
ture
.K
ey te
rms
can
be h
ighl
ight
ed b
y in
stru
ctor
, as
can
rele
vant
ord
erin
g, li
nks,
and
so
on.
Dom
ain
Dom
ain-
and
aud
ienc
e-in
depe
nden
t
Tas
kG
iven
a to
pic
(tea
cher
-inp
utte
d), s
tude
nts
crea
te o
utlin
es (
whi
ch c
ould
subs
eque
ntly
be
used
to w
rite
essa
ys, l
ab r
epor
ts, p
robl
em s
olut
ions
, etc
.).
Perf
orm
ance
mea
sure
sO
utco
me
outli
ne
Proc
ess
mea
sure
sPo
ssib
ly s
teps
tow
ards
cre
atio
n of
out
line
Con
figu
rabi
lity
Com
plet
ely
adap
tabl
e to
con
tent
and
aud
ienc
e ne
eds;
out
line
topi
c, a
mou
ntof
info
rmat
ion
avai
labl
eto
the
stud
ent,
stud
ent l
evel
of
expe
rtis
e, a
nd r
ange
of
appr
opri
ate
resp
onse
sal
l ada
ptab
le;
adap
tabl
e fr
ont-
end
user
inte
rfac
e as
wel
l as
back
-end
sco
ring
and
rep
ortin
gin
terf
ace
Scor
ing
Aut
omat
ed; b
ased
on
teac
her
inpu
t (se
e au
tom
ated
ess
ay s
cori
ng)
Rep
ortin
gA
utom
ated
on-
line
and
off-
line
repo
rtin
g av
aila
ble;
in a
dditi
on, s
tude
nt r
espo
nses
may
be p
rint
edfo
r te
ache
r in
spec
tion
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le6-
8 m
onth
s; n
eeds
sim
ple
wor
king
ver
sion
of
essa
y-sc
orin
g A
PI f
or o
utlin
e sc
orin
g
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)C
ompl
etel
y sc
alab
le
Tab
le 3
(co
ntin
ued)
Ass
essm
ent T
ool:
Prob
lem
Sol
ver
Bri
ef P
rodu
ct S
tate
men
t: Pr
oble
m s
olvi
ng in
volv
es e
xtra
ctin
g re
leva
nt in
form
atio
n, s
impl
ifyi
ng a
nd o
rgan
izin
g it,
and
bei
ng a
ble
to th
enpr
esen
t and
exp
lain
the
solu
tion
clea
rly
to o
ther
s. M
easu
rem
ent o
f th
ese
proc
esse
s in
depe
nden
t of
the
cont
ent o
rdo
mai
n is
the
goal
of
this
high
ly a
dapt
able
tool
. Mod
els
and
info
rmat
ion
can
be p
lace
d in
to th
is to
ol (
e.g.
, dia
gram
s, te
xt, i
mag
es, v
ideo
), o
r in
divi
dual
s ca
n us
eth
isto
ol to
acc
ess
exis
ting
info
rmat
iona
l res
ourc
es s
uch
as th
e In
tern
et. T
his
tool
will
mea
sure
how
wel
l ind
ivid
uals
ext
ract
rel
evan
tin
form
atio
n,si
mpl
ify
and
orga
nize
it, a
nd th
en p
rese
nt it
to o
ther
s. D
eter
min
ants
for
eac
h of
thes
e ca
tego
ries
are
bas
ed e
ither
on
expe
rtpe
rfor
man
ce o
r on
auto
mat
ed s
cori
ng te
chni
ques
. The
se p
roce
sses
are
join
ed w
ith p
erfo
rman
ce in
the
form
of
an in
divi
dual
con
cept
map
, col
labo
rativ
e co
ncep
tm
ap, e
ssay
task
, pre
sent
atio
n, s
hort
-ans
wer
task
, mul
tiple
-cho
ice
task
, dia
gram
con
stru
ctio
n, o
ror
al p
rese
ntat
ion.
Int
egra
tion
of a
ny o
f th
ese
perf
orm
ance
out
com
e m
easu
res
will
fit
with
in th
is to
ol.
CR
ESS
T le
arni
ng f
amili
esPr
oble
m S
olvi
ng
Proj
ect 1
.3 a
rea
Adv
ance
d T
echn
olog
y
Rat
iona
leT
his
gene
ric
prob
lem
-sol
ving
tool
mea
sure
s th
e pr
oces
ses
of in
form
atio
n se
ekin
g (e
.g.,
dete
rmin
ing
rele
vant
fro
m s
puri
ous
info
rmat
ion;
org
aniz
atio
n an
d si
mpl
ific
atio
n of
info
rmat
ion)
. It c
an f
it w
ithot
her
asse
ssm
ent t
ools
to m
easu
re th
ese
proc
esse
s ag
ains
t out
com
e pe
rfor
man
ce.
Dom
ain
Dom
ain-
inde
pend
ent
Tas
kU
sers
sea
rch,
ana
lyze
, ext
ract
, sim
plif
y, a
nd r
eorg
aniz
e in
form
atio
n.
Perf
orm
ance
mea
sure
sIn
form
atio
n se
ekin
g; r
elev
ance
det
erm
inan
ts; i
nfor
mat
ion
sim
plif
icat
ion
and
orga
niza
tion
Proc
ess
mea
sure
sIn
form
atio
n se
ekin
g; in
form
atio
n si
mpl
ific
atio
n an
d or
gani
zatio
n; a
ttent
ion
Con
figu
rabi
lity
Ada
ptab
le to
con
tent
and
aud
ienc
e ne
eds
Scor
ing
Aut
omat
ed; e
xper
t per
form
ance
; lex
ical
libr
arie
s in
tegr
ated
with
late
nt s
eman
tic a
naly
ses
Rep
ortin
gA
utom
ated
; pro
blem
-sol
ving
pro
cess
es r
epor
ted
alon
g w
ith o
utco
me
mea
sure
per
form
ance
Soft
war
e re
quir
emen
tsN
etsc
ape
3.0,
Mac
OS
7.5.
3, W
indo
ws
95 o
r N
T
Har
dwar
e re
quir
emen
tsM
acin
tosh
Pow
er P
C o
r Pe
ntiu
m P
C, 1
6 M
B o
f R
AM
, 20
MB
of
disk
spa
ce
Dev
elop
men
t cyc
le9-
12 m
onth
s; th
is to
ol n
eeds
to b
e de
velo
ped
conc
urre
ntly
with
oth
er to
ols
to e
nsur
e th
at o
ther
perf
orm
ance
tool
s w
ork
with
in it
. 14-
24 m
onth
s be
fore
this
tool
is f
ully
ope
rabl
e w
ith o
ther
tool
sac
ross
ass
essm
ent s
uite
Lar
ge-s
cale
ass
essm
ent o
utlo
ok(s
cala
bilit
y)C
ompl
etel
y sc
alab
le
Tab
le 4
Spec
ific
atio
ns f
or A
sses
smen
t App
licat
ion
Prog
ram
min
g In
terf
aces
(A
PIs)
API
Des
crip
tion
Aut
hori
ng A
PIPu
rpos
e: T
his
API
will
pro
vide
aut
hori
ng s
ervi
ces
to to
ols.
Aut
hori
ng s
ervi
ces
incl
ude
crea
te/e
dit/d
elet
e ca
pabi
lity
for
spec
ifyi
ng a
sses
smen
t cri
teri
a, c
onte
nt, a
nd to
ol p
aram
eter
s.
Dev
elop
men
t Cyc
le: 6
-12
mon
ths
Dat
a lo
ggin
gA
PIPu
rpos
e: T
his
API
will
pro
vide
dat
a lo
ggin
g se
rvic
es f
or to
ols.
The
dat
a w
ill b
e lo
gged
to a
data
base
in a
com
mon
log
form
at. S
ervi
ces
will
be
prov
ided
to lo
g, e
xtra
ct, p
arse
, sor
t, an
d fi
lter
data
.
Dev
elop
men
t Cyc
le: 2
-6 m
onth
s
Rep
ortin
g A
PIPu
rpos
e: T
his
API
will
pro
vide
rep
ortin
g se
rvic
es to
tool
s. R
epor
ting
serv
ices
incl
ude
retr
ievi
ngpe
rfor
man
ce d
ata
and
proc
ess
data
, rep
ortin
g st
atis
tics
on th
e re
trie
ved
data
, and
repo
rtin
g sc
orin
g re
sults
.
Dev
elop
men
t Cyc
le: 6
-12
mon
ths
Col
labo
rativ
ene
twor
ked
API
Purp
ose:
Thi
s A
PI w
ill p
rovi
de c
olla
bora
tive
serv
ices
to to
ols.
Net
wor
k se
rvic
es a
rere
quir
ed f
or to
ols
that
req
uire
onlin
e co
llabo
ratio
n, te
amw
ork,
or
othe
r ne
twor
k-ba
sed
func
tions
.
Dev
elop
men
t Cyc
le: 6
-8 m
onth
s
Sear
ch A
PIPu
rpos
e: T
his
API
will
pro
vide
sea
rch
serv
ices
to to
ols.
Sear
ch s
ervi
ces
incl
ude
a se
arch
eng
ine
with
key
wor
d se
arch
ing
capa
bilit
y, B
oole
an o
pera
tors
, and
res
ults
ret
urne
d as
key
wor
d hi
ts o
r si
mila
rity
ran
king
s.
Dev
elop
men
t Cyc
le: 4
-8 m
onth
s
Sim
ulat
ion
API
Purp
ose:
Thi
s A
PI w
ill p
rovi
de s
imul
atio
n se
rvic
es to
tool
s. S
imul
atio
n se
rvic
esin
clud
e se
tting
and
ret
riev
ing
mod
elpa
ram
eter
s, r
unni
ng th
e m
odel
, que
ryin
g co
mpo
nent
s fo
r st
ate
info
rmat
ion,
and
prov
idin
g ac
cess
to w
idge
t obj
ects
and
libra
ries
.
Dev
elop
men
t Cyc
le: 8
-18
mon
ths
2E
Tab
le 4
(co
ntin
ued)
Tex
t cla
ssif
ier
API
Purp
ose:
Thi
s A
PI w
ill p
rovi
de te
xt c
lass
ific
atio
n se
rvic
es to
tool
s. G
iven
a s
et o
f te
xt, t
his
API
will
pro
vide
an
estim
ate
of s
imila
rity
bet
wee
n th
e gi
ven
text
and
oth
er te
xt in
the
data
base
. Thi
s A
PI w
ill m
ost l
ikel
y be
use
d fo
r m
atch
ing
stud
ent w
ritte
n te
xts
to p
resc
ored
text
s an
d ca
tego
rizi
ng s
hort
ans
wer
s or
use
r-ty
ped
mes
sage
s.
Dev
elop
men
t Cyc
le: 1
8-24
mon
ths.
Thi
s A
PI w
ill n
eed
a pr
ogra
m o
f re
sear
ch.
Com
men
ts: A
ny a
utom
ated
, onl
ine
syst
em m
ust h
ave
text
pro
cess
ing
capa
bilit
y. O
ther
wis
e, th
e sy
stem
will
lack
the
capa
bilit
y to
han
dle
a ve
ry c
omm
on a
nd im
port
ant f
orm
of
asse
ssm
ent.
May
not
be
scal
able
if th
e nu
mbe
r of
req
uest
ed c
ompa
riso
ns is
larg
e, o
r if
the
text
base
use
d fo
r co
mpa
riso
n is
larg
e.
Scor
ing
API
Purp
ose:
Thi
s A
PI w
ill p
rovi
de s
cori
ng s
ervi
ces
to to
ols.
Sco
ring
ser
vice
s in
clud
e co
mpu
ting
perf
orm
ance
dat
a an
d pr
oces
sst
atis
tics,
car
ryin
g ou
t sco
ring
fun
ctio
ns (
e.g.
, com
pari
ng s
tude
nt a
nd e
xper
t con
cept
map
s).
Dev
elop
men
t Cyc
le: 6
-12
mon
ths
cr)
Proc
ess
anal
yzer
Purp
ose:
Thi
s A
PI w
ill p
rovi
de th
e ca
pabi
lity
to c
hara
cter
ize
the
kind
s of
cog
nitiv
e pr
oces
sing
stu
dent
s ar
e us
ing
as th
eyA
PIin
tera
ct w
ith c
ompu
ter-
base
d to
ols.
Thi
s A
PI w
ill p
rovi
de e
stim
ates
of
stud
ents
' pro
cess
ing.
If
even
mod
erat
ely
succ
essf
ul, t
his
effo
rt w
ill p
rovi
de s
ubst
antia
l util
ity f
or r
eal-
time
feed
back
, dia
gnos
is, a
nd p
resc
ript
ion.
Will
adv
ance
stat
e of
the
art i
n co
mpu
ter-
base
d co
gniti
ve p
roce
ss r
esea
rch.
Dev
elop
men
t Cyc
le: 1
8-36
mon
ths.
Nee
d to
gat
her
thin
k-al
oud
prot
ocol
dat
a an
d lin
k it
to c
ompu
ter-
base
d da
ta. N
eed
to d
evel
op m
odel
of
proc
esse
s fi
rst a
t the
tact
ical
leve
l and
then
at t
he s
trat
egic
leve
l. T
his
API
will
nee
d a
prog
ram
of
rese
arch
.
Com
men
ts: W
e kn
ow v
ery
little
abo
ut w
hat s
tude
nts
actu
ally
do
whi
le th
ey ta
ke te
sts
in c
ompu
teri
zed
envi
ronm
ents
.T
he in
fere
nces
we
draw
abo
ut c
ogni
tive
proc
essi
ng a
re r
arel
y ba
sed
on s
tron
g pr
oces
s da
ta. Y
et c
oncl
usio
ns a
bout
cog
nitiv
epr
oces
sing
and
its
rela
tions
hip
to p
erfo
rman
ce a
re ty
pica
lly b
ased
on
self
-rep
orte
d da
ta o
r da
ta a
ggre
gate
d ov
er th
eta
sk.
Est
imat
ing
lear
ners
' pro
cess
es w
ill b
e tr
acta
ble
on ta
sks
that
req
uire
com
plex
rea
soni
ng, d
ecis
ion
mak
ing,
eva
luat
ion,
and
self
-reg
ulat
ion.
The
task
mus
t pro
vide
opp
ortu
nitie
s fo
r th
ese
kind
s of
pro
cess
es to
occ
ur. A
str
ong
coup
ling
is r
equi
red
betw
een
over
t act
ion
and
thes
e pr
oces
ses.
References
Baker, E. L. (1995, September). Finding our way. Presentation at the annualconference of the Center for Research on Evaluation, Standards, andStudent Testing, University of California, Los Angeles.
Baker, E. L., Linn, R. L., & Herman, J. L. (1995). Institutional grant proposal forOERI Center on Improving Student Assessment and EducationalAccountability: Integrated assessment systems for policy and practice:Validity, fairness, credibility, and utility. Los Angeles: University ofCalifornia, Center for Research on Evaluation, Standards, and StudentTesting.
Baker, E. L., & O'Neil, H. F., Jr. (1996). Design report for CAETI assessment plans(Year 1) (Report to ISX/DODEA). Los Angeles: University of California,Center for Research on Evaluation, Standards, and Student Testing.
Chung, G. K. W. K., O'Neil, H. F., Jr., Herl, H. E., & Dennis, R. A. (1997, March).Use of networked collaborative concept mapping to measure team processesand team outcomes. In H. F. O'Neil, Jr. (Chair), An integrated simulationapproach to assessment. Symposium presented at the annual meeting of theAmerican Educational Research Association, Chicago.
Herl, H. E., Baker, E. L., & Niemi, D. (1996). Construct validation of an approachto modeling cognitive structure of U.S. history knowledge. Journal ofEducational Research, 89, 206-219.
Herl, H. E., & O'Neil, H. F., Jr. (1996). Final report. CAETI STS1 technologiesbaseline data collection (Year 1) (Report to ISX/DODEA). Los Angeles:University of California, Center for Research on Evaluation, Standards, andStudent Testing.
Herl, H. E., O'Neil, H. F., Jr., Chung, G. K. W. K., & Dennis, R. A. (1997, March).Feasibility of an on-line concept mapping construction and scoring system.In H. F. O'Neil, Jr. (Chair), An integrated simulation approach toassessment. Symposium presented at the annual meeting of the AmericanEducational Research Association, Chicago.
Herl, H. E., O'Neil, H. F., Jr., Chung, G. K. W. K., Dennis, R. A., Klein, D. C. D.,Schacter, J., & Baker, E. L. (1996). Measurement of learning across five areasof cognitive competency: Design of an integrated simulation approach tomeasurement. Year 1 report (Report to ISX/DODEA). Los Angeles:University of California, Center for Research on Evaluation, Standards, andStudent Testing.
13
Klein, D. C. D., O'Neil, H. F., Jr., & Baker, E. L. (1996). A cognitive demandsanalysis of innovative technologies (Report to ISX/DODEA). Los Angeles:University of California, Center for Research on Evaluation, Standards, andStudent Testing.
Klein, D. C. D., O'Neil, H. F., Jr., Dennis, R. A., & Baker, E. L. (1997, March). Thefive families of cognitive learning; A context in which to conduct cognitivedemands analyses of innovative technologies. In H. F. O'Neil, Jr. (Chair),An integrated simulation approach to assessment. Symposium presented atthe annual meeting of the American Educational Research Association,Chicago.
O'Neil, H. F., Jr. (Chair). (1997, March). An integrated simulation approach toassessment. Symposium presented at the annual meeting of the AmericanEducational Research Association, Chicago.
O'Neil, H. F., Jr., & Abedi, J. (1996). Reliability and validity of a statemetacognitive inventory: Potential for alternative assessment. Journal ofEducational Research, 89, 234-245.
Schacter, J., Herl, H. E., Chung, G. K. W. K., O'Neil, H. F., Jr., Dennis, R. A., & Lee,J. J. (1997, March). Feasibility of a Web-based assessment of problem solving.In H. F. O'Neil, Jr. (Chair), An integrated simulation approach toassessment. Symposium presented at the annual meeting of the AmericanEducational Research Association, Chicago.
18
U.S. Department of EducationOffice of Educational Research and Improvement (OERI)
Educational Resources Informarion Center (ERIC)
REPRODUCTION RELEASE
I. DOCUMENT IDENTIFICATION:
(Specific Document)
ERIC
Tolle:Estimate of the Potential Costs and Effectiveness of Scaling UpCRESST Assessment Software
A401011S): Gregory K.W.K. Chung, Howard E. Herl, Davina C.D. Klein, Harold F. O'Neil, Jr., John Schacter
Corporate Source:
UCLA Center for the Study of Evaluation
Publication Date:
December 1997
II. REPRODUCTION RELEASE:In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced
in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproducedpaper copy, and electronic/optical mode, and sold through the ERIC Document Reproduction Service (EDRS) or other ERIC vendors. Crack isgiven So the source of each document, and, d reproduction release is granted, one of the blowing notices is affixed to the document
If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following two options and sign atthe botbm of the page.
Check hereFor Level 1 Release:Permitting reproduction inmicrofiche (4' x 6' film) orother ERIC archival media(e.g.. electronic or optical)and paper copy.
The sample sticker shown below will be The sample sticker shown below will beaffixed to of Level 1 documents
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL
HAS BEEN GRANTED BY
Qua
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
Level 1
affixed to a1 Level 2 documents
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS
MATERIAL IN OTHER THAN PAPERCOPY HAS BEEN GRANTED BY
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
Level 2
Documents will be processed as indicated provided reproduction quality permits. If permissionto reproduce is granted, but neither box is checked, documents will be processed at Level 1.
0Check here
For Level 2 Release:Permitting reproduction inmicrofiche ( x 6' film) orother ERIC archival media(e.g.. electronic or optical).but not in paper copy.
Sign Si
here)please APIIIL Aka
an ahem/ raes
hereby grant to the Educational Resources Information Center (ERIC) nonexdusive permission to reproduce and disseminatethis document as indicated above. Reproduction from the ERIC microfiche or electronic optical mocks by persons other thanERIC employees and Is system contractors requires permission from the copyright holder. Exception is made for non-profitreproduction by libraries and other service agencies to satisfy information needs of educators n response to discrete inquiries.
510 -0c6 -1613a-- 310-g-5- .5T-Mal Address: rUirti7
IK.CnsC(50-U0-0edt41 a-6/9
