Embed Size (px)
Citation preview
SIG Newsletter 1/2015 1 / 25 4/7/2015
AERA SIG 167 ‐ Cognition & Assessment
Volume 3, Issue 1 Newsletter March 2015
From the Chair
André A. Rupp, Educational Testing Service
In this newsletter issue we have a few highlights to share with you, which include:
1. AERA 2015 SIG conference program
2. Interviews with Jonathan Templin, Janice Gobert, and Russell Almond, our 2015 award winner
and honorable mentions
3. Bios of graduate student presenters for business meeting
4. Bios of our SIG officers
5. Moderated blog
Finally, I continue to encourage you to check out the AERA website for the SIG,
http://www.aera.net/SIG167/CognitionandAssessmentSIG167/tabid/12259/Default.aspx
where we have basic information about the SIG such as contact information for all SIG officers and award
winners, a list of all sessions and papers from annual meetings, and a copy of our by‐laws. The website
has been expanded by AERA recently and we are working on adding materials. Please also consider joining
our LinkedIN group,
http://www.linkedin.com/groups?gid=4514305&trk=hb_side_g
which allows you to create discussions, communicate with other members, and ask for informal votes
on issues.
We are looking very forward to shaping this newsletter and initiatives of the SIG with my officers in the
service of you, our membership, in the upcoming years. If you have any suggestions for how we can serve
you better please do not hesitate to contact any of us!
Sincerely,
(André A. Rupp)
Dear members of the SIG 167 ‐ Cognition & Assessment,
I would like to welcome you to the fifth issue of our SIG newsletter, right before our
annual meeting of AERA in Chicago, IL! I hope to see many – if not all of you – in person
in Chicago and wish you a wonderful conference! And please consider coming to our
business meeting – there will be free food and a cash bar this year!
SIG Newsletter 1/2015 2 / 25 4/7/2015
Topic 1 – AERA 2015 Finalized Conference Program Program Chair: Donna L. Sundre
Date: Friday, April 17, 2015
Time: 10:35 am to 12:05 pm
Room: Marriott Fifth Level, Denver/Houston
Session Type: Invited Speaker Session Title: The Promise of Cognition and Assessment
Chair: Donna L. Sundre, James Madison University
Presenters:
Laine Bradshaw, University of Georgia – Athens
Susan Embretson, Georgia Institute of Technology
Paul D. Nichols, Pearson
Jonathan Templin, The University of Kansas
Date: Friday, April 17, 2015
Time: 4:05 to 5:35 pm
Room: Sheraton Fourth Level, Chicago VI & VII
Session Type: Poster Session
Title: Applications of Cognition and Assessment Presenters: Analysis of Learning Map Structure for a Dynamic Assessment (Feng Chen, The University of Kansas; Amy Clark,
The University of Kansas; Russell E. Swinburne Romine, The University of Kansas) Developmental Changes in the Effects of Minimal Social Presence on Children’s Executive Functioning (Jason C.
Chin, Boston University; Rachel Bell, Boston University; Grace Min, Boston University; Katie Kao, Boston University; Stacey Doan, Boston University; Kathleen C. Corriveau, Boston University)
Does Block Location Influence Children’s Performance during the Forward and Backward Adminstration of the Corsi…? (Andrea Palmay, Queen’s University – Kingston; Derek H. Berg, Queen’s University)
Utility of a Working Memory Task in Early Detection of Alzheimer’s Disease for Chinese Older Adults in Hong Kong (Jiafang Chang, Chinese University of Hong Kong; Kit‐Tai Hau, Chinese University of Hong Kong; Chi‐Shing Tse, Chinese University of Hong Kong; Linda C.W. Lam, Chinese University of Hong Kong)
SIG Newsletter 1/2015 3 / 25 4/7/2015
Date: Saturday, April 18, 2015
Time: 10:35 am to 12:05 pm
Room: Sheraton Ballroom Level, Sheraton IV
Session Type: Structured Poster Session
Title: Conversation‐based Assessments Session Organizer: Diego Zapata‐Rivera, Educational Testing Service Chair: Diego Zapata‐Rivera, Educational Testing Service Discussant: Arthur C. Graesser, The University of Memphis Presenters: Using Conversation‐based Tasks to Diagnose the English Language Skills and Math Knowledge of Middle School
English Learners (Alexis Lopez, Educational Testing Service; Non‐presenting author ‐ Danielle Guzman‐Orth, Educational Testing Service)
Assessing Argumentation Skills using Conversations: Comparing Constrained and Open Response Formats in Game‐Based Assessment
Assessing Mathematical Argumentation and Algebraic Reasoning through Automated Conversations (Gabrielle Alexis Cayton‐Hodges, Educational Testing Service; Non‐presenting authors – Malcom Bauer, ETS; Maria Bertling, ETS; Irvin R. Katz, ETS; E. Caroline Wylie, ETS)
Evaluating Conversation Patterns to Elicit Evidence of Students’ Science Reasoning Skills (Tanner Jackson, Educational Testing Service; Non‐presenting authors – Diego Zapata‐Rivera, ETS; Lei Liu, ETS; Margaret Vezzu, ETS)
Evaluating Different Types of Scaffolds to Assess English Language Skills (Youngsoon So, Educational Testing Service; Non‐presenting authors – Keelan Evanini, ETS; Laura Battistini, ETS; Christine Luce, ETS; Jidong Tao, ETS)
Comparing Trialogue‐based Tasks with other Assessment Tasks (Haiying Li, The University of Memphis; Non‐presenting authors – Diego Zapata‐Rivera, ETS; Blair Lehman, The University of Memphis; Tanner Jackson, ETS)
Exploring Scalability Issues of Conversation‐based Assessments (Lei Liu, Educational Testing Service; Non‐presenting authors – Isaac I. Bejar, ETS; Diego Zapta‐Rivera, ETS; Tanner Jackson, ETS)
Automated Testing of Conversation‐based Assessments (Irvin R. Katz, Educational Testing Service; Non‐presenting authors – Diego Zapata‐Rivera, ETS; Maria Bertling, ETS; Tanner Jackson, ETS; Nathan Lederer, ETS; Keith Kisser, ETS)
SIG Newsletter 1/2015 4 / 25 4/7/2015
Date: Saturday, April 18, 2015
Time: 2:45 to 4:15 pm
Room: Marriott Fifth Level, Los Angeles/Miami
Session Type: Paper Session
Title: Applications of Cognition and Assessment Discussant: Elizabeth L. Pier, University of Wisconsin‐Madison; Jaime Dice, University of Rhode Island Presenters: A Diagnostic Comparison of Turkish and Korean Students’ Mathematics Performances on the TIMSS (Trends in
International Mathematics and Science Study) 2011 Assessment (Muhammet Arican, University of Georgia‐Athens; Sedat Sen, Harran University)
Cultural Competence and Teacher Efficacy in Novice/Career Teachers: Examining Injustice Through Variations in Beliefs (Ranjini Mahinda JohnBull, Johns Hopkins University)
Developing and Evaluating Learning Progression‐Based Assessments in Mathematics(Emily R. Lai, Pearson; Paul D. Nichols, ACT, Inc.; Jennifer L. Korbin, Pearson)
How Students Process Multiple‐Choice Questions with Pictorial Information: Evidence from Eye Movement Recordings (Marlit Annalena Linder, IPN‐Leibinz Institute for Science and Mathematics Education; Olaf Koeller, Leibniz Intstitute for Science and Mathematics Education)
Date: Saturday, April 18, 2015
Time: 6:30 to 8:00 pm
Room: Marriott Fourth Level, Clark
Session Type: SIG Business Meeting
Chair: André A. Rupp, ETS The Business Meeting will also offer complimentary light refreshments and beverages (cash bar).
SIG Newsletter 1/2015 5 / 25 4/7/2015
Topic 2 ‐ Outstanding Contribution to Research in Cognition and Assessment Award
Interviews with Award Recipient and Honorable Mentions
Dr. Jonathan Templin, University of Kansas 2014‐2015 Outstanding Contribution to Research in Cognition and Assessment
(Winner)
First, congratulations on the award. How does it feel to be recognized as a leader in the field of cognition
and assessment?
Well I think that’s a leading question. Number one, I don’t know that I’m a leader in the field, but I
appreciate the recognition. I would like to thank the person who nominated me. I like to think those who
wrote me letters for consideration and above all else I would like to thank the SIG for taking the time to
recognize us. I’m humbled by the award but I don’t think I’m a leader in the field.
When and how did you become interested in psychometrics?
It’s kind of a twisted story but I’ll give you the best of it. I was in community college doing my best to avoid
mathematics as I failed it in high school. Took a math class, took a statistics course to get out of taking
business calculus. Learned I loved statistics so I wanted to figure out a way to do statistics in the social
science as research. So I moved to my undergraduate institution, which is Sacramento State in California.
I had a professor mention quantitative psychology as a field that I could be interested in. And in his class
we had done factor analysis. Exploratory factor analysis. And out of the computer came an eigenvalue.
Something called an eigenvalue. I had no idea what the hell the word eigenvalue meant. And that’s what
spurred me to get into psychometrics because it said eigenvalue. And just as a little more filler to the
story, the professor said “Eigenvalue‐I think that’s matrix algebra, you should go take a course on it.” So I
took a course on it. The first day I said, “what’s an eigenvalue?” The professor on the first day I said “it’s
a scalar that when multiplied by vector sends a vector to the scalar space” or something along those lines.
And I said “Whoah, my head is blown” so I knew I had to go to grad school and do psychometrics at that
moment.
What about the field of psychometrics continues to motivate you to work in this area?
I think it’s an interesting and fascinating problem really. So there’s no shortage of difficulty in the attempt
to define and then measure things that don’t exist. In a way it’s kind of like if you remember the first
Ghostbusters movie. And how hard it was for them to capture the ghosts. We’re like, you know, Peter
Venkman and so we’re trying to capture the ghosts that don’t exist but we’re using psychometrics instead
of the little traps in the floor. There you have it.
So when you say they don’t exist could you say a little more about that?
Absolutely. The psychological constructs and abilities that we’re trying to measure through psychometrics
don’t exist. The only things that exist are the test items, the survey items, or the things that we create.
And so, our methodology is there built to test whether or not our assumptions about whether they exist
SIG Newsletter 1/2015 6 / 25 4/7/2015
are true but they don't exist. No matter what we do they don’t exist. There’s no way that we can easily
measure intelligence or anything else. It has to be conceptualized and then developed through a set of
items. So I think that presents a ton of different issues as we go forward and all the nuances as we go
forward.
You’ve been known to tell your students that all day you think about psychometrics, you dream about
psychometrics, and you wake up and read about psychometrics. So this may be hard to imagine, but if
you weren’t a psychometrician, what would you be doing?
Oh that’s a tough one because I learned at an early age this is what I wanted to do. But I have two options
for this and so basically I like to think if one career doesn’t work out, you have a back up. So my first would
be, I’d love to be a roadie for the Marshall Tucker band. I think it be great putting together sets and sound
stages, I think that’d be awesome. And I feel like that a little when we go to NCME we have our conference
presentations, it’s sort of like, okay maybe not. The second thing would be, I’d love to be a meteorologist
of some sort, study weather. And for the same reason it uses high data, it has some impact on our lives.
It’s just fascinating to see all the factors that play into it and understand how to do it well and see all the
complexity there is in the world.
You are currently working on a project that is blending features of DCMs and BINs to estimate 1000s
of attributes in an operational assessment. What have been the challenges you have faced
implementing such new methodology in an operational setting?
A number of challenges. Let me just describe my role first. I am developing the software to estimate these
very high dimensional models. And to do so in a way that blends the diagnostic model world which comes
about from IRT with the Bayesian inference network world which comes form more of a computer science
type of framework. And so, first and foremost, is the language, the languages between the two fields. I
feel somewhat late to the party on this, but for every topic in the psychometric world, there’s a similar
name and entity in the Bayesian inference computer science world. So there really is a one to one
connection of everything.
Okay, so once you get past the language, the next part is really trying to figure out what type of
architecture, like computer hardware and software that really can handle this type of information. On
that point, to architecture, so to really do this, the hard part is, how do we build a software program to
take advantage of a super computer so that we can get estimates as fast as possible? So that is really the
challenge I have been struggling with. Beyond that, most of our challenges, I feel very confident in that
we will solve the computer science angle of things. And, people have solved it. We are not doing
theoretical physics. We are not doing the particle collisions. We are using models that are very
computationally intensive. So we can do this.
Our bigger problem is dealing with people who are not as aware of this technology and its use in
education. I wouldn’t say educators, I wouldn’t say people who are on the ground in education. But it’s
people in the middle. It’s the people who are the state boards of education and the technical advisory
committees seem to be the biggest problem in dealing with this. And it’s essentially a framework that is
hard and fast: a score is like a height and it’s something that is real and rather than something that doesn’t
exist that causes us bigger problems.
SIG Newsletter 1/2015 7 / 25 4/7/2015
In the future, where do see DCM‐based methodology fitting into the educational assessment
landscape?
Well, for diagnostic models themselves, when you’re talking about the measurement of categorical latent
variables, they are clearly not as elegant as a nice continuum. You have a high group and a low group.
They are really good for assessing at least a rough enough idea of what a person knows with more
certainty than a continuum. It takes a lot of items and a lot of time and effort to measure one thing and
do it well. So the trade off has to be, if you want to measure more than one thing, how do you do it? And
in a lot of education, we are really faced with how do we get people to learn material, better, faster, and
bigger. And a single score that takes 80 items to achieve is just infeasible to make it work. And we need
something to fill that gap. It's the methodology. Who would have guessed to making a switch in an
assumption would have started all this. This is where I see it fitting into our assessment landscape.
What other changes do you foresee happening in assessment over say the next 10 years?
Oh assessment. Such a broad topic. That’s a tough one to answer. It’s a hunch, but the way we are
assessing people now, we are so big, with so many tests. I feel like we need to give more for how much
we are paid to do it and how much it time takes out of classroom instruction. I worry that we are near the
testing and assessment analog of peak oil—when oil production only declines from that point forward.
Call it peak testing.
So, what we are starting to see, and particularly the project I’m working on, is testing programs are trying
to give more information from of tests as we go forward. Not just a single score. I would expect to see
that trend to continue. However, if it weren’t to continue, I would fear that what we would also see is a
contraction of testing practices. We really aren’t getting our money’s and time’s worth currently. What
we’re really getting from these tests is not that much information at the end of the day. So there’s two
ways we can go. Either we get more information for the same cost or we get the same amount of
information and we shrink. I think that’s the way the field is going.
This may be a tough question, because part of idea generation could be subliminal and it may be hard
to define the source. How do you come up with research ideas? And how do you assess their feasibility
and determine which research ideas to pursue?
That’s a really good question. I think some of it comes from, you know, trying to be interested in statistics
or psychometrics in a broad sense so you try to read things outside your area, inside your area. But that
naturally spurs questions about how is it different, why is it different. That starts to become the nature of
it. Another part of it is just inspiration. And for me, inspiration comes by listening to Katy Perry. So I found
a lot of times I’m just listening to her and singing along and the next thing I think, “hey I should test that.”
In all honesty, it really is trying to get outside of my own personal comfort zone, my pursuit is always trying
to learn. There is such commonality in what we do, and what we are trying to learn, but it doesn’t always
seem like that. For me, it’s putting myself outside the boundary of what I know it happens to be and try
to find my way back and that usually bring back a lot of questions.
As for feasibility, I am not a good person to ask about that. I don’t know what makes a good study. I can
think an idea may be an interesting idea, I can mostly know if it has it done before. I think it comes from
a hunch, but it also comes from still doing literature reviews and being up to date on what people are
publishing, what people are presenting, and what people are talking about.
SIG Newsletter 1/2015 8 / 25 4/7/2015
What areas of research in our field need more attention? What areas would you encourage graduate
students to pursue to maximally contribute to the field’s needs?
I think we need to have a critical attention played to the interplay between model fit and what that means
regarding the assumptions we can come about. And I don’t mean that in terms of research. I think it's the
interplay on the research side on the policy side. I think in general, that’s kind of my biggest thing. Our
tests don’t fit as well as we think they do. But we keep claiming these results that they do really well. And
we have policies built around them assuming that they are perfectly fitting and that’s all kind of a mess.
But that being said, that’s kind of a tried and true measure.
I think bigger picture than that, the interplay between our discipline and so many others statistically. All
of what we do. If we show up in other disciplines, the labels change a little bit. All these disciplines have
something to contribute. Each of these fields have something to contribute.
Advice for a younger Jonathan Templin, or a current graduate student?
Oh, that’s a good question. There was a time in grad school where I had a conversation with my first
advisor. And he said “you know it’s great that you want to learn everything, but at some point you have
to home in on something.” And it’s true. You have to kind of have a depth of knowledge about one area.
But I think having that boarder focus while you’re deep is important. It’s hard to do. That’s why I talk about
psychometrics being my hobby in addition to my career. I do a lot of my work in diagnostic models, but I
am interested in so many other things. So, stay interested in other things. Stay broad. Find a focus that is
interesting to you. That’s where the true knowledge. And there are a few other things I would advise:
Don’t do as much service as I started to take on. They are things that derail you from the bigger purpose.
Keep doing research.
SIG Newsletter 1/2015 9 / 25 4/7/2015
Dr. Janice Gobert, Worcester Polytechnic Institute
2014‐2015 Outstanding Contribution to Research in Cognition and Assessment (Honorable Mention)
How did you become interested in intelligent tutoring systems, as well as in cognition and assessment
as a whole?
The recognition that assessment of science skills was highly problematic was the driver. So I first became
interested in this, I would say really even in the late 80s and early 90s when I was trying to characterize
students’ science learning using various methodological techniques like think aloud protocols, student‐
generated diagrams, and then I eventually went on to things like students’ explanations, and then log files
from student interactions in microworlds. Log files really opened up an entire area of thinking about how
they could be analyzed and how they could be utilized in real‐time for assessment.
Could you elaborate a little bit further on log files?
Log files are generated in real time as a person is working in a virtual learning environment. So it’s all their
mouse clicks, their time on task(s), and all the decisions to inputs that they make in a simulation‐based
environment; when they set a certain value for a variable and then run that simulation, for example. So
all those very fine grained mouse clicks create a log file or trace of what a student did in solving the
problem or executing an inquiry task. And they become a conceptual and procedural trace of what the
student did during reasoning. Then we can distill those data and use it to develop performance assessment
metrics to assess how students are forming a hypothesis, how they're designing controlled experiments
(or not), how they’re interpreting their data, how they are warranting their claims, and how they're
communicating their findings. And so this becomes a really powerful technique to analyze students’ skill
levels aligned to the Next Generation Science Standards where the onus is really on researchers to develop
techniques that can capture and analyze students’ skills in a high‐fidelity manner.
In addition, what we really need to do is be able to scale these assessments because as a pedagogical or
instructional model, I think the Next Generation Science Standards are incredibly rich; however, from an
assessment point of view, developing and scaling assessments for these skills is not straightforward, and
that’s an understatement. It’s extremely difficult and we need to be working at the rich intersection of
learning sciences and in computer science to develop computational techniques that are guided by
theoretical frameworks about how people learn and what we know about how students “come to” science
tasks and how they bring their prior knowledge to the science task.
What type of specific modeling have you found to be the most successful?
With respect to assessment, we’ve had success with educational data mining techniques – machine
learning techniques where you hand score a log file – what you can think of as protocol analysis of a log
file with some categories that are derived from the literature a priori, about what teachers are going to
need, variables that reflect the construct validity of the of the skill that you're looking at. Then we use
SIG Newsletter 1/2015 10 / 25 4/7/2015
machine learning to train the system recognize those patterns so that the algorithms can be applied in
real time. Once you do that, you can not only assess in real time, but you can also then adaptively instruct
in real time. You can have a pedagogical agent; we have a pedagogical agent named Rex who reacts in
real time to support students as they conduct inquiry. And that way you can keep the student at a level
they are comfortable with.
Csikszentmihalyi talks about flow state and I won't elaborate too much on that, but if a student perceives
the task as far beyond what they think they can do, their self‐efficacy is highly affected and they're going
to disengage from the task. In fact, we can measure students’ disengagement and we have a new paper
in Educational Psychologist that just came out – myself, Ryan Baker from Columbia, and Mike Wixon here
at WPI – where we were show that we are able to detect when students become disengaged from inquiry
in a microworld. If you have that critical knowledge, you can keep the task scaffolded so that students can
work within their zone of proximal development, which we all understand very well from of Vygotsky’s
work.
What about the field of cognition and assessment motivates you to work in this area?
It's important for STEM and for the future. Many students are very disenfranchised because the current
teaching and assessment models in science are selecting out our creative and innovative thinkers.
Innovation really needs people who can flexibly apply their knowledge to novel problems because the
problems that we have solutions for are not the problems of the future, they’re the problems of the past.
I really feel, and this idea is not unique to me, Robert Sternberg has written about this extensively, that
the standardized testing movement that's so oriented around multiple choice questions has a trickling
down effect on classroom culture such that what gets rewarded is rote learning and rote memorization.
And when people are engaged in rote learning and rote memorization, they're not engaged in constructing
rich mental models upon which inferencing can be done. So that is a necessary, but not sufficient,
condition for innovation. So that's in part what I think really motivates me: thinking about students who
are highly creative and who could potentially play a big role in science in the future but they are being
disenfranchised by the way in which science is taught, which is being driven by the way in which science
is currently being assessed.
If you weren't in educational research scientist, what would you likely be doing?
So this is an interesting question because, although my career path has been very steadfast, my degrees
are in experimental psychology at the undergraduate level, educational psychology and cognitive science
at the Masters level, and cognitive science at the PhD level, and then science education at the postdoc
level. So despite that extremely steadfast approach, in the past I’ve had thoughts about ‘if I wasn't doing
this, what would I do?’ One is a medical doctor. The human body fascinates me, medical science fascinates
me. I like people, I’m a very people‐oriented person. So I think I could have been a medical doctor. In fact,
my undergraduate supervisor told me that I have all the characteristics of a medical doctor. It was kind of
interesting. The other thing that I’ve had a real interest in, and that has been reflected in my research, is
architecture because I’m very interested in spatial visualizations. I’ve always been very interested in three
dimensions, how we understand three dimensions, how we understand three dimensions from two
dimensional diagrams, etc. I did my Masters and Ph.D. research with architects looking at expertise in the
ways in which they understand a building from its plans. So this interest in visual representation, visual
learning, and visual cognition runs very deep through my research and my interests.
SIG Newsletter 1/2015 11 / 25 4/7/2015
And then lastly, this is a more kind of ‘If you could do anything, what might you do?’ I’m a former a cappella
singer. I love to sing and singing puts me in a zone that transcends space and time in the way that they
talk about with flow states because it’s a complex activity that requires you to deeply engage to manage
your breathing, manage your memory for the words, manage your presentation, and execute the song in
the best way possible. So despite not being very talented, I would have loved to have been a singer and I
do sing a little bit still.
That’s amazing!
Thank you! It was a hobby, but I did compete for a few years in the United States with a really fun group
of women called Women of Note in Massachusetts. It was really very engaging, and it required a very
different set of skills and a “different part of my brain”.
Your research spans many areas of cognition assessment, for someone not familiar with your work,
how would you describe it?
My work uses visual simulations to engage, teach, and assess science understanding. So it’s undergirded
by what we know about how people learn science and how people learn from visualizations in particular.
It’s undergirded by knowledge acquisition from visual information sources and the technique that we use
to get at that is real‐time eye‐tracking. Actually, myself and Ermal Toto, a Senior Software Engineer in my
group, have a patent application pending on an eye tracking system that detects where students’ eyes are
focused on the screen and then directs them according to their knowledge states and their knowledge
acquisition patterns. Lastly, my work is undergirded by rigorous computer science‐based analytical
techniques like knowledge engineering and educational data mining that allow us to do real‐time
assessment of students’ science skills and real time scaffolding of students’ science skills using a
pedagogical agent named Rex.
So this may be tough question because part of idea generation is subliminal and hard to define the
source you come up with research ideas and how do you assess their feasibility?
Our group does a lot of tech development. So we think about what might be possible in 10 years, based
on what we know about human learning and what we know about the projected state of technology;
policy needs are considered too. Then we bound and concretize the problem and then design a system,
and select the relevant variables to study and how to study them.
How do you see assessment changing over the next 10 years?
Hopefully, the ideal, the Platonic ideal is that stealth assessments that incorporate rich computational
techniques like educational data mining will be the norm and the standard. And that would be my most
sincere and greatest hope. But I also think it's really important that we start thinking about providing an
associated metric of how good or how valid a particular measure is of a student’s engagement in that task.
So for example, if Johnny is not engaged in the task, the skill metrics that are going to be generated are
not going to be a good reflection of what he is capable of doing. So we really do need, and this is not
impossible – my group is in the process of doing this – an associated metric that will say, ‘On balance,
given the performance assessment of Johnny in the past, we can say that this is a valid set of metrics on
what Johnny is capable of doing with respect to science inquiry.’
SIG Newsletter 1/2015 12 / 25 4/7/2015
You call this an ideal, do you think this is realistic within the next ten years?
So there's an interesting question: is it realistic? It’s absolutely realistic from a technical point of view. But
I think it is necessary for learning scientists and computer scientists, and all other disciplines that play a
role, but largely learning scientists and computer scientists, to “come across the aisle” and work deeply
at the intersection of these different disciplines. So right now, you see many people working on the
computational side, and you see many people working on the learning sciences side, but the future is at
the intersection of learning sciences and computer science with respect to ed tech development and
technology‐based assessments.
Here’s a concrete example of what we need: we need to use science‐based theoretical frameworks about
how people learn in order to distill meaningful categories out of our log data. So it's not enough to say
‘Suzie did better than Mark and Susie generated 45,000 mouse clicks and Mark only generated 25,000
mouse clicks” because, of course, that is not pedagogically meaningful on any level to the student, the
teacher, the policy holder, et cetera. What we really need is to be able to generate metrics that are based
on deep understanding of human cognition and also then allow us to concretize skills into sub‐skills, which
then allow capability to auto‐score data using computational techniques. There might be some richness
lost in this approach, but, right now, we are throwing the baby out with the bathwater by using multiple
choice questions to assess students’ skill level in science inquiry. The fidelity of those multiple choice
measurements is poor to none. And I'm not being facetious when I say “poor to none”.
The fidelity of real time log files, by contrast, is extremely good. And its analytical capacity for assessment
really is based on the degree to which we can apply theoretical frameworks to figure out what’s
meaningful in logs from an assessment point of view and from an instructional point of view, so that we
can eventually scale the assessments and replace multiple choice tests. Jim Pellegrino and I both quote
Voltaire on this issue where we say, “Don't let the perfect interfere with the good.” We have these stealth
assessments that are capable of high fidelity measures of students’ skills and we really need to get these
in play broadly to replace multiple choice tests. With this approach, we can continue to tweak the
assessments with the data; and given that these assessments are digital, the re‐production costs are less
than for paper‐ and pencil based tests. What I have been describing is far better than multiple choice tests.
Do you have any advice for a younger Janice Gobert, or a current graduate student?
I think people should be working at the intersection of technology and learning science because this is where the future of this field is. Secondly, they really need to dive deep on both sides. It's not enough to just be on the tech side and it's not enough to just be on the learning sciences side. We really need to develop more graduate programs that train people on both the computational and the learning science side to think deeply about the future of technology in education, both for learning and for assessment. Innovation is possible when two fields “bump up” against each other in this way. The field of Cognitive Science was “born” this way when Herbert Simon an economist, and Allen Newell, a computer scientist began working together on the Logic Theorist, a program that was able to prove mathematical theorems. And then more on a personal note, I would say “Pick a research topic that you're passionate about but be
realistic about what you're willing to sacrifice for it and what you’re not. Because you don't want to get
halfway down the path and then decide that you can’t do what you set out to do because you're not
willing to make the personal sacrifices that are needed to do it.
SIG Newsletter 1/2015 13 / 25 4/7/2015
Any other thoughts that you would like to share?
I guess what I think is important and that I didn’t highlight to the extent that I should have is the
combination of techniques to measure and triangulate data in the service of guiding learners in real time.
For example, eye‐tracking is used to trace their knowledge acquisition patterns. This could be important
because we can’t get a good skill metric if they're not actually looking at the material. We can also detect
when students are disengaged or when they're careless like we do in our work with Ryan Baker and his
post‐doc Luc Paquette. In our system, we are capable of detecting when they're disengaged and when
they are careless. So they may know the skill, but they can get careless and not pay close enough attention.
These two things are important because they can allow us to specify parameters to adaptively change the
system for a student who is working really well and then gets bored because it's become too easy. We
can use that data to change the nature of the tasks and questions for the student, so that they are
appropriately challenged. Similarly, if the task is too difficult, students are going to disengage. If they feel
like they're never going to succeed at the task, they’re going to disengage.
We can also measure when they're bored, we can measure when they are confused, we can measure
when they are frustrated. In some earlier work that Ryan and I did with our post doc Arnon Hershkovitz,
we looked at these affective states and how they relate to a student’s learning. In the future, a system
that’s undergirded by various different kinds of technological techniques with different data and then
triangulated in really rich ways will allow us to truly adapt to individual differences; this is really what I
think we need to be doing in this field.
I think that eye tracking is just really fascinating and amazing technology. What it allows us to do in
terms of assessing these different facets of student engagement is incredible.
Yes, I think there’s a lot of recognition of its power and applicability. Similarly, the educational data mining
approach is gaining recognition for its power. We – myself, Ryan Baker, and Mike Sao Pedro – have a
patent application pending for our EDM‐based algorithms for assessment as well. This has been the main
topic of my work for some years now, generously funded by the National Science Foundation and the US
Department of Education.
SIG Newsletter 1/2015 14 / 25 4/7/2015
Dr. Russell Almond, Florida State University
2014‐2015 Outstanding Contribution to Research in Cognition and Assessment (Honorable Mention)
First, congratulations on being recognized for your research contributions. How does it feel to be
recognized for your work in assessment?
Actually, I feel relieved. That answer probably requires an explanation. Recently I heard a musician
express a sentiment that I've heard from a number of artists: you should do what you consider to be good
work and not worry about what the critics/public will think about it. If you are doing good work, they will
discover it eventually. I think the same thing applies in science, and I've always tried to work on things I
thought were important, trying not to worry about the opinion of others. After all, I can't control what
others will think. However, I'm going up for tenure in the fall. Tenure is all about what others think of
your work. That part was making me nervous because it is often hard to tell what others think of your
work. So I'm happy that the judging committee thought well of my work, and relieved because this speaks
well of my chances for tenure.
When and how did you become interested in cognition and assessment?
My dissertation research was about graphical belief functions (an extension of Bayesian networks) and I
was doing work related to the use of Bayes nets for calculating the reliability of large engineering systems.
I had a contract with NASA for this work, but that contract was ending and new contracts were not coming
in. At an American Statistical Association meeting, I spoke to an old classmate, Eric Bradlow, who was
working for ETS. He asked me if I knew Bob Mislevy, who was doing work with Bayes nets in education. I
said no, but maybe I should introduce myself. Bob introduced me to Linda Steinberg, who had just
completed the HyDRIVE project, and that was the start of Evidence‐centered Design. I was trained in
Statistics. I basically learned psychometrics by talking with Bob, Howard Wainer, Charlie Lewis, Mike Zieke,
and lots of other people who were around ETS at the time. This was 1995 and Bob had recently finished
his Psychometric Society address on the topic of evidence and assessment. That was one of the first papers
on assessment I read, and it was all about how measurement models should reflect models of cognition.
I was therefore surprised a few years later to learn that not everybody thinks about cognition when they
think about assessment. Also, I've had a long standing interest in artificial intelligence, so I'd already read
a lot about models of human cognition, so that seemed natural to blend in with assessment from the start.
What about the field of psychometrics motivates you to work in this area?
It is a natural area where Bayesian thinking can help. In particular, latent variables are not identifiable
from data alone. At the very least, you need an expert to supply interpretations for the factors you
uncover in your exploratory factor analysis. Therefore, to build psychometric models, especially
cognitively diagnostic models, you need a mixture of data‐driven learning and domain expertise. There
are a lot of challenging problems in representing and managing the information that goes into building
and maintaining such models.
SIG Newsletter 1/2015 15 / 25 4/7/2015
You somewhat recently transitioned into academics, how has that change impact your work on
cognition and assessment and/or your perspectives about cognition and assessment?
Actually, it is the other way around. It is my perception of where cognition and assessment research needs
to go that has caused me to move to academia. In my last couple of years at ETS, I had two insights. One
is that cognitively diagnostic assessments would necessarily be long. If you need between 6 and 10 items
per aspect of proficiency to get reliable measurement, then it is very hard to support the kind of detailed
diagnostic models many cognitive experts want to build. Instead, you would need to build up a complete
profile of a student over many smaller interactions separated in time. The other is that modern computer
technology is going to create a marketplace for electronic educational modules containing some kind of
instructional content and an end‐of‐unit assessment. Efforts like SCORM and the IMS Question and Test
Interoperability specifications would ensure that they would all work together from a computer
hardware/software perspective. However, what is still needed is a model‐based infrastructure for
combining the scores from these learning objects in a meaningful way.
So I became interested in the problem of putting together a profile of a student from the output of a large
number of low‐stakes activities (e.g., class assignments) separated in time. ETS, for all the money it puts
into general psychometric research (including research in cognition and assessment), still gets most of its
money from the big high‐stakes tests (SAT, GRE, TOEFL, Praxis). As there wasn't a natural market for this
work at ETS, it was a low priority. Now that I'm a free agent, I can seek out my own funding, as well as
collaborate with people at Pearson, West Ed, University of Oregon and other places doing interesting
work, as well as continue to collaborate with old colleagues from ETS.
If you weren’t a statistician/psychometrician, what would you likely be doing?
Probably doing computer system analysis. I enjoy large programming projects and have designed a
number of software systems. (You can see the list at my home page at http://ralmond.net/). If I were
independently wealthy, I might become a Renaissance musician. I have a number of historic instruments
which I can play (some better than others) and have spent some time arranging Renaissance dance music.
How would you describe ECD to someone not familiar with your work? And what are some examples
of its application in educational assessment?
Evidence‐centered assessment design is a way to think about designing assessments. It is based on four
steps: (1) the purpose of any assessment is to make claims about what the examinee knows, thinks, can
do, or feels; first identify those claims. (2) Figure out what evidence to support or refute those claims
would be. (3) Figure out how to engineer situations (tasks) in which to gather the evidence. (4) Figure out
how much evidence you need so that you can make the claims you want to make with an appropriate
degree of confidence.
There are a number of example of ECD out now, but two really good ones with lots written about them
spring to mind. The first is Biomass, which is the project we did as we were developing ECD to check to
make sure it was really useful. The development of Biomass is described in Chapters 14 and 15 of the new
book Bayesian Networks in Educational Assessment, just published by Springer. The second is Adaptive
Content with Evidence‐based Diagnosis (ACED) which is a project that Val Shute led. The complete models
and field test data for ACED, along with a list of ACED papers, are available at:
http://ecd.ralmond.net/ecdwiki/ACED/ACED.
SIG Newsletter 1/2015 16 / 25 4/7/2015
Where do Bayesian Networks fit into the educational assessment landscape?
Is this where I get to mention that I recently published a book on this topic? I think the right way of thinking
about Bayes nets are not so much that they are new models, but they are a new notation for thinking
about old models. I think there are three advantages.
First, the ready availability of Bayes net software packages, such as Netica, means that it is natural to start
thinking about the problem of scoring a single student. Actually, that is a big difference in focus: most
psychometric models focus on the question of how I should learn model parameters given data from a
collection of students. But Bayes nets encourage psychometricians to start with the problem of how does
evidence from specific items get combined into proficiency estimates for a single student. I'm currently
teaching a Bayes net class, and I've done a number of exercises in class where I simply convert simple
unidimensional IRT models into Bayes nets and then talked about how they work. The students seem to
learn a lot from that, even the ones who have already studied IRT.
Second, Bayes nets provide a very flexible notation for models. Why are there no software packages that
allow a test designer to mix and match compensatory and conjunctive multidimensional IRT models?
Surely we know the algorithms for both, so it shouldn't be too difficult. The problem is not one of
estimation, but of knowledge management. The software needs to know which observable outcome
variable should have which distribution. Bayes nets, which assume each observable has its own
distribution, confront that problem head on. But they are not enough. This is why you need a tool like ECD
to manage all of the inputs that go into a model. Third, the flexibility of Bayes nets encourages researchers
to think about new ways of modeling the relationship between variables. My current work with the R
package CPTtools (see http://pluto.coe.fsu.edu/RNetica/) is designed to make an easily expandable set of
building blocks for Bayesian networks.
This may be a tough question, because part of idea generation could be subliminal and it may be hard
to define the source. How do you come up with research ideas? And how do you assess their feasibility?
I mentioned my interested in artificial intelligence before. A lot of my ideas come from there. In particular,
I pretty regularly attend the Uncertainty in Artificial Intelligence conference and am a regular contributor
to its application workshop. Often I see interesting methods there and think about how I could apply
them to education. I'm pretty bad at judging their feasibility though. I'll often think that something looks
easy only to discover it is going to take 4 times as long as I though it would. Another good source of ideas
is collaborating with somebody who is doing real work with teachers and students. Often they will have
problems in their research, particularly statistical modeling problems, for which they would like better
methods. A big advantage of working on their problems is that it is sure to be a problem that somebody
cares about.
How do you see assessment changing over the next 10 years?
I think that the biggest change is that assessment will be a lot more computerized, and that there will be
a lot more interesting constructed response tasks, especially ones that use simulation and game‐like
things. On the other hand, I wouldn't trust my track record as a prognosticator. After NCLB came out I
would have predicted that the furor would die down after 10 years, but 15 years latter, RTTP is taking
NCLB one step further. In particular, there is real push to use tests designed for student evaluation for
high‐stakes teacher evaluation without really thinking through the validity and reliability issues.
SIG Newsletter 1/2015 17 / 25 4/7/2015
Advice for a younger Russell Almond, or a current graduate student?
First, don't be afraid to tackle small problems. Small projects are much easier to complete and get
published. If you are tackling a big problem, the risks are a lot greater. Anyway, you need to break it up
into small pieces to get it published, so if you can do that from the start you are better off. Also, learn as
much as you can about computer programming. Not just running stat packages, but the logic behind
computers and how to design and maintain software. It will mean that you have a lot more tools to
implement your ideas.
SIG Newsletter 1/2015 18 / 25 4/7/2015
Topic 3 –Graduate Student Presenters at the Business Meeting
Angela M. Lui
1) SIG Business Meeting Presentation Title: Exploring Cognitive and Affective Processes to Formative Feedback
2) Program and University: Educational Psychology and Methodology, University at Albany/SUNY
3) Research Interests: Formative assessment in K‐16 school settings; cognitive and affective processes in teaching and learning; and validity issues in educational and psychological measurement; the intersection of these three areas to inform effective teaching strategies and curriculum approaches
4) Personal Interests: Educational Psychology and Methodology Graduate Student Organization
membership; health‐ and medical‐related readings; painting and sketching; piano; child care and teaching; informal peer mentoring
5) Fun Question: Books on your shelf are begging to be read when you’re not reading for school?
i) Far from the Tree: Parents, Children, and the Search for Identity by Andrew Solomon ii) A Game of Thrones Series by George R. R. Martin iii) What is the What by Dave Eggers iv) Bird by Bird: Some Instructions on Writing and Life by Anne Lamott
Kristin Morrison
1) SIG Business Meeting Presentation Title: Examining Alternative Methods to Understand the Impact of Linguistic Context on Mathematical Item Difficulty
2) Program and University: Quantitative Psychology, Georgia Institute of Technology
3) Research Interests: Cognitive Complexity, IRT, Item Generation, Mathematics Education, CAT,
MST
SIG Newsletter 1/2015 19 / 25 4/7/2015
4) Personal Interests: Reading (when there is time), Being With Family/Friends
5) Fun Question: Books on your shelf are begging to be read when you’re not reading for school?
A Song of Fire and Ice series (i.e., Game of Thrones)!
Myrah R. Stockdale
1) SIG Business Meeting Presentation Title: Global Engagement and Student Outcomes of Funded
Undergraduate Researchers at the University of North Carolina at Greensboro: An Assessment and Evaluation Framework
2) Program and University: Educational Research Methods; University of North Carolina at Greensboro
3) Research Interests: Educational measurement, Item Response Theory, Program Evaluation, Higher Education Assessment, Educational Reform
4) Personal Interests: Traveling, cycling, strength training, hiking, and volunteering.
5) Fun Question: Which big‐name academic do you aspire to have coffee with and why? Dr. Terry Ackerman. I had the distinct honor to have him as my introductory graduate statistics instructor in 2014. He is the most caring instructor I have ever had in my long college career. He never left any student behind; he'd answer even the most asinine of questions with patience and precision. Dr. Ackerman is always very busy with his duties as a Dean, instructor, and an academic, but I would love to sit down with him and learn more about how to emulate his passion and patience in the classroom. He is such an outstanding scholar with a professional career that has inspired me in my own career.
SIG Newsletter 1/2015 20 / 25 4/7/2015
Umit Tokac
1) SIG Business Meeting Presentation Title: Using Partially Observed Markov Decision Processes (POMDPs) to implement a Response to intervention (RTI) Framework for Early Reading
2) Program and University: Measurement and Statistics, Florida State University
3) Research Interests: Bayesian data analysis and applications in education; adapting artificial intelligence methods to education in order to measure and monitor learners’ current proficiencies, and forecast their future proficiencies
4) Personal Interests: soccer, basketball, rock climbing, listening music, reading, and traveling
5) Fun Question: Books on your shelf are begging to be read when you’re not reading for school? The Theory That Would Not Die by Sharon Bertsch Mcgrayne
Tugba Elif Toprak
1) SIG Business Meeting Presentation Title: Making the Most of Cognitive Diagnostic Assessment in Language Testing: Developing a Cognitive Model for L2 Reading Comprehension
2) Program and University: Gazi University, Institute of Educational Sciences, Ankara, Turkey; Visiting Graduate Scholar at the University of Georgia
3) Research Interests: Diagnostic classification modeling, evidence‐centered design framework for assessment and psycholinguistics
4) Personal Interests: Photography, painting and playing the reed flute (ney)
5) Fun Question: Which big‐name academic do you aspire to have coffee with and why? I would like to have coffee with Dr. Andre A. Rupp and have a chance to benefit from his suggestions and insights because as an advanced doctoral student. I am interested in applying diagnostic classification modeling within an evidence‐centered design assessment framework and from an interdisciplinary perspective. I believe his work in this strand is truly significant, rich, and inspiring.
SIG Newsletter 1/2015 21 / 25 4/7/2015
Topic 4 – Bios of New SIG Officers
Chair (2015‐2017) – Laine Bradshaw
Laine Bradshaw, Ph.D., holds an appointment as a tenure‐track Assistant Professor of Quantitative Methodology in the Educational Psychology Department in the College of Education at the University of Georgia. She received her Ph.D. in Research, Evaluation, Measurement and Statistics in 2011 from the University of Georgia, where she also received an M.Ed. in Mathematics Education. Her expertise is in psychometric modeling and assessment development, with a focus on diagnostic measurement models that support the design of multidimensional assessments capable of providing reliable, timely feedback about student knowledge and reasoning. Her work in developing psychometric methodology for providing probabilistic diagnoses of student misconceptions using a diagnostic classification model (DCM) framework was recognized by the 2012 Outstanding Dissertation Award of the American Educational Research Association Cognition and Assessment Special Interest Group. Dr. Bradshaw is currently collaborating on the design and implementation of a DCM‐based formative assessment system that measures elementary and middle grades mathematics comprehension for the Partnership for Assessment of Readiness for College and Careers (PARCC). Her program of research is currently funded by the National Science Foundation and has been published in journals such as Psychometrika, Educational Measurement: Issues and Practice, and the International Journal for Testing.
Vice‐chair (2015‐2017) – Dubravka (Duda) Svetina
Duda is an Assistant Professor of Inquiry Methodology at Indiana University, Bloomington. Her research interests involve methodological investigations of the performance of current psychometric models and procedures. In particular, she is interested in multidimensional item response theory (MIRT), issues related to dimensionality assessment, and cognition and assessment design. With the rise of various cognitive diagnostic models as a response to the increasing demands from administrators to understand better what students can and cannot do, much is to be investigated with respect to psychometric properties of such models, including dimensionality, estimation, and model fit. Over the years, she was involved in the leadership of AERA at various levels, most recently as this SIG’s Program Chair for the 2014 annual meeting, a reviewer for both this SIG and AERA Division D, as well as a graduate student representative to Division D (2008‐2010).
SIG Newsletter 1/2015 22 / 25 4/7/2015
Program Chair (AERA 2016) – Mahnaz Moallem
Mahnaz is a professor of Instructional Technology and Research, Program Coordinator for the Instructional Technology Program and Grant Coordinator for the Watson School of Education at University of North Carolina, Wilmington (UNCW). From 2006‐2008, Mahnaz also served as Instructional Technology Project Leader (IPA) at National Science Foundation. I received my PhD. in Instructional Systems—Educational Research and Program Evaluation Certification from Florida State University in 1993. Mahnaz has been a member of AERA since 1991 and served in various capacities such as the chair of the Problem‐Based Education SIG as well as discussant, session chair, and proposal reviewer. Mahnaz has also served on editorial board of ten journals and has been a reviewer for five major conferences in education and educational technology. Mahnaz has received several research and teaching excellence awards and published books, book chapters and numerous referred journal articles and has a substantial record of scholarly and institutional achievements both nationally and internationally. This record includes leadership positions involving initiating and implementing improvements in various aspects of human learning and cognition, grant projects and other scholarship and service activities.
Secretary / Treasurer (2015‐2017) – Charles Secolsky
After receiving his doctorate in measurement, statistics, and evaluation from University of Illinois at Urbana‐Champaign,, Charles joined the staff of ETS as a measurement statistician. He then taught statistics at the college level and served as Director of Institutional Research & Planning at County College of Morris in New Jersey. In 2011, he completed editing Handbook on Measurement, Assessment, and Evaluation in Higher Education published by Routledge. Soon after the publication, he moved back to University of Illinois working at the Center for Instructional Research and Curriculum Evaluation. Since 2009, Charles’ research has focused on following up on his 1983 seminal JEM article, on identifying student misconceptions using MDS, Euclidean distances, and cognitive sources of DIF. He has made presentations at IAEA conferences for two of the last three years and was invited and presented at the First International Conference on Assessment and Evaluation in Saudi Arabia. His most recent endeavor was the development of a new civil service test for Saudi Arabia based on intelligence theory using ECD. Charles is a frequent presenter at AERA, NCME, AIR, AEA, and NERA and also serves as reviewer of paper proposals for these associations for their annual conferences. He also reviews manuscripts for publication in AIR journals.
SIG Newsletter 1/2015 23 / 25 4/7/2015
Graduate Student Liaison (2014‐2016) – Matthew Madison
Matthew Madison is a doctoral student in Quantitative Methodology at the University of Georgia. He
previously received an M.A. in mathematics from Central Michigan University, and an M.S. in statistics
from the University of Georgia. His primary research interests lie in the development and application of
advanced psychometric models. Specifically, he is interested in an emerging class of psychometric models
called diagnostic classification models (DCMs) and their application in educational settings. Matthew is
also interested in assessment in mathematics and statistics education. After he graduates in May 2016,
he hopes to become a professor of educational measurement and psychometrics.
SIG Newsletter 1/2015 24 / 25 4/7/2015
Topic 5 – Moderated Blog for SIG‐related Handbook of Cognition and Assessment
Thanks to the work of Russell Almond and Matthew Madison we are excited to announce a new initiative for the SIG – a moderated blog! The site is really easy to access and can be found here:
https://cognitionandassessment.wordpress.com/ Russell has already posted two blog entries, one general welcome entry and an entry entitled “The Purpose of Cognition and Assessment”. For your convenience the welcome post is reproduced here but you will have to go to the blog site to check out the first content post and subscribe to the RSS feed! We have already asked for volunteers from our pool of Handbook authors in order to have a consistent wave of contributions – we are shooting for about 1‐2 contributions a month. In addition we very much welcome contributions from our SIG membership! Thus, if you are interested in contributing to the blog please contact Matthew ([email protected]) or Russell ([email protected]). Let us know what you think – we are looking forward to hearing from you!
Welcome to the Cognitive Science and Assessment Blog
We are the Cognitive Science and Assessment Special Interest Group (SIG) of the American Educational Research Association (AERA). The SIG is made up of people who are interested in the intersection of cognitive scienceand assessment especially as applied to education. If you are interested in any of those topics, then you are among our target readers.The bloggers for this site are mostly professionals in the field along with some graduate students. We expect that many of the readers will be graduate students, so we will try to keep discussions at that level. However, that still means that there will still likely be a lot of jargon, especially about measurement (i.e., assessment or testing) that will be difficult to understand for a lay audience. If you are interested in the topics in this blog, but are having trouble with the jargon, we would like to recommend the collection of educational resources about assessment available at the National Council on Measurement in Education (NCME). Why a blog? The idea came from a conversation I had with SIG President André Rupp, as well as a number of others, about how to better foster a sense of community among SIG members, particularly between the annual meetings of the AERA. My idea was that a technical blog, something at about the level of Andrew Gelman’s Statistical Modeling Blog, something that would encourage discussion about topics that were technical but not so technical that only a few people could understand them, would encourage people to spend time discussing with each other. Furthermore, by encouraging a highly literate and technical audience, we would get technically interesting and useful discussions. A second reason for starting a blog is that there are a high number of semi‐technical topics that we need to work through in our field: things which are too well understood to be the subject of papers, but not well enough worked out to be in the standard textbooks.
SIG Newsletter 1/2015 25 / 25 4/7/2015
Take the issue of how long an assessment should be, especially a cognitively diagnostic assessment which measures multiple aspects of proficiency. The [APA/AERA/NCME Joint] Standards state that the assessment needs to be of a length that is suitable to its purpose. Anybody who has done assessment design knows that there are a lot of complex considerations that the designers need to balance when deciding on the length of the assessment. My goal is not to work this issue now (good topic for a future post), but rather to point out that a lively discussion from multiple experts in the field would benefit both the community of practitioners and students trying to learn both the art and science of assessment. But aren’t blogs dead? I’ve seen on the internet lately (ironically from blogs I follow) several posts indicating that blogging is a dead art. I think that the truth is we are just now discovering what it is that blogs are best at. Mailing lists work well for calls for papers and post‐doc positions, but they don’t really encourage discussions. After all, there are many times in my life when I want less rather than more email. Facebook and LinkedIn work well for keeping friends and colleagues updated about your life, but again are not the right forum for technical discussions. Twitter is very good at capturing reactions to other content, but not on originated the content itself (especially not anything that would take more than 140 characters to explain). So what is the niche for a blog? I think the answer is just what we are proposing to do: short technical articles followed by technical discussion. Unlike media companies, we don’t need an ever wider audience to sell to advertisers to pay our salaries. Instead we need the right audience (if you have read this far I hope that includes you) to keep the discussion lively and interesting. Who are the participants? My goal initially is to have enough entries in the queue to be able to post a new entry every week. At least initially, my goal is to get enough co‐bloggers who commit to one article per month (or even less) that we have a good queue of material to post without anybody going nuts (in my case, that may be too late). Andre and I have already reached out to a number of you (although I have forgotten some of the people I have talked to about this). If you are willing to contribute, contact Russell Almond or Matthew Madison. Of course, some of the most important positions in a blog are the commenters. If you want to join the conversation you are welcome to contribute below. As long as you have an interest in cognition, assessment or both and are willing to maintain an appropriate level of professional courtesy we welcome you. This is not an official publication of the AERA or the Cognition and Assessment SIG, and all opinions are those of the bloggers and commenters and not representative of any particular institution, particularly the places where they are employed. However, we are unofficial representatives of those institutions and we expect that all discussion will follow the standards of professional behaviour we have come to expect. What do you think about this idea? Your comments about the new blog, what its scope and rules should be, what topics we should cover, and so on are welcome below.
