II. Explanation of ASSISTmentsweb.cs.wpi.edu/~nth/pubs_and_grants/papers/2011... · Web viewFigure 2 – A teacher report for the last four ASSISTments completed by students. little

Content and Pedagogy: Chapter 6H Heffernan et al. 14719 words plus 4 figures = 6234 words

EFFECTIVE AND MEANINGFUL USE OF EDUCATIONAL TECHNOLOGY:

THREE CASES FROM THE CLASSROOM

Neil T. Heffernan, Matthew Militello, Cristina L. Heffernan, Michele Bennett Decoteau

I. INTRODUCTION

The shift from steam to electric power was gradual and costly, not just because of

the required investments in technology, but because the technology enabled and

required fundamentally new ways of organizing and conducting work1.

Technology in schools today is ubiquitous. Educational technology makes bold claims of

efficiency, interactivity, and the ability to provide instantaneous, useful information for teachers

to teach better and for students to learn more. Unfortunately, like many educational reformation

predecessors, the research, development, and legislative policies that anchor 21st century reform

in technology remain largely underutilized and unrealized2-5. Nonetheless, digital teaching

platforms (DTP) have a unique opportunity to meet the social and economic demand to

effectively use educational technologies.

The purpose of this chapter is to describe a specific DTP designed to extend already

proven effective and transform current teaching practices. Specifically, relative advantage for

teachers using this technology is garnered through ease-of-use, efficiency, and the ability to

observe and document improvements in student engagement.

II. EXPLANATION OF ASSISTMENTS


ASSISTments is a web-based assessment system that provides tutoring based on student

responses. The system is so named because it is a blend of assessMENT and instructional

ASSISTance. This system collects data efficiently and provides student-level diagnostic results -

what we call cognitively diagnostic - allowing teachers to monitor students’ progressions through

the cognitive model. This is a tool that can be adapted

and used in a variety of manners with different

cognitive models and content libraries. The most well

know library is middle school mathematics and this is

the library we will be using as an example. Nonetheless,

it is important to remember that ASSISTments has

many libraries of content including statistics, science,

and language arts. The middle school math cognitive

model is based on 130 specific math skills. Examples of

the skills are “Venn Diagram,” “Percent of,” “Area of a

Circle,” and “Area of an Irregular Figure.”A list of

knowledge components is available at

www.assistments.org. The system is not designed to

“teacher proof” the classroom. In fact, ASSISTments is

designed to augment, replicate, and promote good

teaching practices including: uncovering detailed

diagnosis of misconceptions, providing immediate,

specific feedback, and monitor student practice. Figure 1 – A screen shot of a single ASSISTments question with associated scaffolding, tutoring and buggy messages.


ASSISTments has over 1000 “tutored” items in middle school math. Tutored means that

if a student gets the problem wrong, they are given scaffolding questions, associated hints, and

buggy messages. Figure 1 shows a screenshot of a single ASSISTment. For this

ASSISTment, the student must know two skills: “Venn Diagram” and “Percent of.” A student

who gets the main problem right will move on to another problem. If not, as in this example,

they will get tutoring. We see that the student incorrectly typed “8,” and the system responded,

“Hmm, no. Let me break this down for you.” The tutor then asked the student a scaffolding

question: … find the “Percent of students in Biology, Algebra, and Band.” We see that the

student had to look at one hint to answer this question. This hint consists of an image to show

that the student had to add up the percents in gray. The next scaffolding question follows up and

this student got it right. The last question re-asks the original question. This student typed in

23400 and was given a “buggy message” reminding the student to check to see that the answer

was reasonable (i.e., less than 900). The system then reminded the student that when multiplying

by 0.27, move the decimal over two points. Once the student actually types in the correct answer

of 234, the student can proceed to the next ASSISTment. The original question is tagged with

two skills where the individual scaffolding questions are tagged with one each allowing the

teacher to assign problems with more than one skill.

Then individual skills can be diagnosed by looking at the results of the scaffold questions.

Note how this student received individualized context-sensitive just-in-time-help, while

the teacher does not have to “do” any grading. By breaking questions into steps, the tutor is able

to provide cognitively diagnostic information to teachers: which items are students getting right,

which scaffold questions are they getting right, and what are common wrong answers. If the

teacher looks at a report of this student’s work on this problem, they would see the student had a


little trouble with the first “Venn Diagram”

question, had no trouble on the second, and had a

little trouble with the “Percent of” skill in the

third scaffolding question.

Figure 2 shows a teacher’s item report

where students have done four ASSISTments in a

row (items #4517, #8842, #4674 and #78). The

teacher can see that over half of the students got

the first question wrong on their first attempt (if

they have to ask for a hint they are marked

wrong). However, they seem to have learned from the tutoring and gotten the next two questions

correct. The fourth item, ASSISTment #78, only 29% of students could get the item correct. The

teacher sees that ASSISTment #78 also has the knowledge component of “Percent Of” in

addition to “Venn Diagram.” The teachers can see the common answers for the Original

Question: 18% of the students answered “0” while 12% answered “657.”

In addition to this cognitively diagnostic data as seen in Figure 2, ASSISTments can be

used for Mastery Learning and nightly homework. While the digital divide is a genuine concern

many communities have public computers that, if given a few days, students can access outside

of school. Parental notification is a feature that allows teachers to easily inform parents about

how their child is performing and what they have, and have not, mastered. This saves the teacher

tremendous amounts of time and enhances teacher, student parent communication. All of these

features are designed to enhance what the teacher is already doing but allow them to do them in a

more efficient manner.


An innovative use of ASSISTments is the Advanced Student Response System. This

feature allows teachers to create their questions (on the fly or prepared), ask the students to

respond, anonymously post the answers using a projector, and generate rich, meaningful

discussions. This provides instantaneous feedback by students to either questions teachers

provide or to others student’s work shown to the class with a projector or smart board.

This system comes with a student response system that also includes essay grading and

provides a teacher-friendly presentation system. As more and more teachers who use

ASSISTments have projectors in their classrooms, the advantages of sharing the data with

students in class have emerged. In fact, when students have access to computers at the same

time, the teacher can project the results on the screen we have seen the classroom light up with

discussion. These discussions can be instantly followed up by more data collection creating a

dialogue loop that involves every student responding to the teachers questions online and having

their answers recorded on ASSISTments and projected for everyone to see.

In summary, this DTP can be used at multiple points in a teacher’s routine.

1) Planning a lesson. ASSISTments can help a teacher look at a scope and sequence, look at

past lesson plan data, and modify lesson plans based on current data.

2) Delivering a lesson. ASSISTments can deliver the lesson, more focused on the coached

practice aspect.

3) Evaluating the lesson. ASSISTments can help a teacher determine the success or

retention of a lesson.

4) Providing homework support. Students receive immediate feedback on their actions

offering assistance where needed and giving teachers immediate feedback on students’

work.


II. ASSISTMENTS IN ACTION IN REAL CLASSROOMS.

In this section we closely examine the work of a number of real teachers who are using

ASSISTments to inform their teaching and help their student achieve. They use it for Nightly

Homework, Mastery Learning, and an Advanced Student Response System. Together these

teachers demonstrate the flexibility and responsiveness of these features to help teachers build on

what they already do in order to teach more effectively.

II.A. Homework

“When I look at my emailed report [regarding ASSISTments homework] in the

morning, I can save time and use the information to drive my instruction for that

morning.” (Christine O’Connor)

Nightly homework is an important part of extended classroom learning. With the help of

ASSISTments, teachers are able to streamline the grading, help to students and assessing that are

all part of the homework process.

The traditional homework routine involves sending a set number of problems home with

students, they do their best and then they come in the next day to see what they got wrong and

have questions answered. With ASSISTments, teachers select the problems for homework,

students get feedback in the form of correctness response and sometimes tutoring, and then the

teachers review and plan around the emailed reports they get the morning before class. They can

even share the data with the class by just pulling up and projecting the item report (see Figure 2)

from the homework. For students without Internet access, teachers have the option of printing a

handout.


We found that simply completing nightly homework on ASSISTments produces more

learning that paper and pencil alone6. This echoes what other researchers have found in other

disciplines7. In this section we will describe how one teacher, Ms. Tignor, uses these pre-made

problems with tutoring and how another teacher Ms. Mulcahy uses ASSISTments along with her

normal homework.

Ms. Tignor works at a technical school where she assigns a problem set of challenging

problems (with ASSISTments tutoring) from the high-stakes statewide assessment test for them

to solve outside of class online. Each student must score a proficient or take review math classes

and take the exam again. Ms. Tignor’s goals for homework are for students to: 1) Work on

problems to practice for the exam, 2) Get help in the form of ASSISTments tutoring on any

problem they get wrong, and 3) Write up solutions for the problems they get wrong. With these

three goals in mind, she gets her students involved in assessing how well prepared they are for

the state exam, they get help and the also have to show her what they understand after reading

through the tutoring.

Ms. Tignor reinforces the need to show their work and gives them a worksheet to fill out

if they get the problem wrong. She knows that the open response portion of the State test where

students must explain their work is the most challenging. She values their explanations so she

gives each student a rubric designed to focus their work on explaining and help her in assessing

their work.

All students who get a problem wrong initially see ASSISTments tutoring to help them

work through the problem (see Figure 1). The problem stays open on the screen as you see in

Figure 1 so students can refer back to the tutoring to help them write their explanations. Ms.


Tignor only has to look at the work of students who could not solve the problem and need to be

monitored on their work.

One great benefit of assigning homework online is that Ms. Tignor does not have to wait

until Monday to monitor how the whole class is doing. As you can see from Figure 2, Ms. Tignor

can get data instantaneously or first thing in the morning before she plans her daily lessons. She

then uses this data to assess her students and make instructional plans for the next week.

Ms. Tignor has two routines that she uses to respond to the data she collects. First, she

adds one or two of the problems from the previous week’s homework to class work as a warm up

problem. For example, one week only a third of the students got a problem correct. Ms. Tignor

finds it beneficial to have a discussion about a topic that was clearly confusing to many students.

Second, once a month during the weekly exam, Ms. Tignor confers with each student on their

progress, and their work on ASSISTments is discussed. During these meetings Ms. Tignor offers

suggestions on their writing.

This activity involves a lot of training of the students to learn to write about their math,

show their work, and manage their time.

Another teacher, Ms. O’Connor, who works at a suburban middle school, sends home

regular nightly homework as ASSISTments and has written ASSISTments to allow students to

input their answers online and get correct feedback. Students are told which problem to do in

their text (for example the ASSISTments will just say Page 12 # 6 and have a space for the

student to put in the answer.) They start by doing the problem on paper because Ms. O’Connor

values having the work organized and referenced in class. Then they input the answer in the

system, click submit and are instantly told if they are correct and to either try again if it is wrong,


or move on to the next problem. Ms. O’Connor can write hints or other support for these

problems as she wishes.

These examples illustrate how teachers use this feature differently. It also highlights how

this feature harnesses technology to (1) link the home and school whereby parents can be

integrally involved in student work, (2) create efficiency in the teacher’s ability to understand

individual student learning prior to the next class meeting, and (3) allow students to engage in

homework in an electronic manner—capturing today’s adolescents desire to work in a

technological environment.

II.B. Mastery Learning

Mastery Learning, similar to the first steps in Gressler’s diagnosis and remediation

framework, assesses a student’s knowledge level on a single skill from the ASSISTments list of

130 skills. A major challenge of Mastery Learning is the bookkeeping. Keeping track of skill

development is a time consuming and detail-oriented task. Teachers need to keep track of the

skills each student has and has not mastered, who is having trouble mastering a skill, and what

the prerequisites are for each skill.

ASSISTments keeps track of all of these things for teachers, students and parents. The

Mastery Learning feature focuses on single skill items like solving an equation or adding

fractions unlike most problems in ASSISTments that require students to use multiple skills (e.g.,

ASSISTment # 78 that requires two skills, Figure 1.) Different students need different amounts

of practice to demonstrate mastery of a skill to themselves and their teachers. Building enough

content for this sort of practice is the most important part of mastery learning. ASSISTments

uses variables in the problem definitions, allowing each problem developed to become a set of


problems measuring the same skill but with different numbers or names. This keeps the students

practicing. Additionally, of the 130 knowledge components we can track, we have a prerequisite

structure already in place. For example, for a student to master Pythagorean Theorem, they need

to have mastered equation solving and square roots. This tagging allows a student who is

struggling in a knowledge component to move back to a prerequisite skill before continuing with

the original skill.

If a problem set in ASSISTments is tagged with Mastery Learning it automatically tracks

the amount of practice a student gets depending on how well they are answering the questions.

Most of the problem sets determine that a student “masters” the skill once they get a set number

of problems correct in a row. Students who know the skill can demonstrate that quickly. Students

who do not know the skill must work until they reach mastery. Students who are getting

problems wrong get help from the ASSISTments tutoring or they can seek help from other

sources. Teachers can monitor students using a simple report (see Figure 3) and seek out students

who are struggling.

Ms. Mulcahey gives out one Mastery Learning problem set each night to her suburban 8th

grade students. At the beginning of the year she selected skills that they should have mastered in

7th grade and monitored their progress. Many students made errors but were able to attain

Mastery quickly. Ms. Mulcahey monitored and addressed the students who did not reach

Mastery. Later in the year she selects skills that she has covered. This simple problem set just ads

a few minutes to her student’s nightly homework and is an important part of her ability to give

her students the practice they need.


Mr. Burnett is a suburban

middle school teacher who uses

ASSISTments exclusively for

Mastery Learning. His students

start their work in the computer lab

and then have to finish the work at home during the week.

“I use Mastery Learning to reinforce learning from classroom and to pinpoint

any problems or weaknesses. I use ASSISTments as a two tiered system: 1) it

allows me to have students work independently using hints when they struggle and

to come to me when they just cannot get over that hump on their own, and 2) it

works as a great motivator to some students – they really work hard to see that

word “mastered” next to the assignment.” (Mr. Andrew Burnett)

Mr. Burnett monitors the students’ progress by referencing a Mastery Learning progress

report. The report tells the teacher if the student mastered or is still working. It tells how many

problems the student did and how long they spent in their quest to master a skill.

Like Ms. Mulcahey, Mr. Burnett selected review content at the beginning of the year.

This allowed him an efficient way to give practice only to those who need it. As a conclusion to

the Mastery Learning activity he put together a regular ASSISTments problem set with one

problem per skill to help him and the students monitor their overall progress on those review

skills.

Figure 3 – Teacher report showing Mastery Status of four students.


One extension of the Mastery Learning feature is the Automatic Reassessment and

Retention System. This assesses students’ retention of skills and forces a student to revisit topics

after Mastery so they can demonstrate retention over many days. Typically students are

reassessed on a topic a week, two weeks, and a month later. If a class has retention rates less than

50%, this is a strong indicator of shallow learning and a topic the teacher needs to revisit.

The Mastery Learning feature provides a platform for teachers to allow students to

practice as much as they need on a skill by skill basis. As students practice, the reports help the

teacher monitor their students’ strengths and weaknesses in these skills. The technology allows

for templates with variables to build large sets of problems. ASSISTments also has an authoring

tool available that, with training, allows anyone to build these problem sets on any skill they

want.

III. C. ADVANCED STUDENT RESPONSE SYSTEM.

In this section we will show how ASSISTments handles the important task of writing and

reading explanations of solution strategies. The NCTM Standards includes communication in

mathematics as one of the standards that can improve student understanding of mathematical

concepts.

Today’s math teachers are being asked to get students to explain their work verbally and

in writing. They are also asked to include writing in their math classrooms. ASSISTments helps

teachers orchestrate discussions and assess student’s explanations of their work. Beyond

declarative, factual knowledge and memorization, the Advanced Student Response System in

ASSISTments provides student with procedural knowledge attainment and communication in


mathematics. Procedural knowledge is a vital component to the deep understanding of

mathematical knowledge and skills8.

When a teacher wants a student to provide an explanation in ASSISTments they select

the question type “Open Response” in contrast to the question types “Fill In [the blank]” or

“Multiple Choice.” In this question type, a box is provided for students to type their explanation.

The one drawback to this answer type is that unlike the others, ASSISTments cannot grade these

items. To facilitate these responses there is a special link to a feature called “Essay Grading.”

Once teachers link to this page they see the question and then the selection of explanations

written by the students next to their names. They are then able to grade the essay from 1 to 4.

Once this is done the grade changes from “ungraded” to the percent in the regular item report

(see Figure 2) and the average changes. If a teacher wants to review the essays with the class,

teachers can select just a few exemplary essays and only have them show up for the discussion

allowing the teacher to focus the discussion on the essays that move the learning objectives

forward.

In the final days of a unit on linear equations, Ms. O’Connor gave an ASSISTments

problem set that included regular “Fill In [the blank]” ASSISTments and one “Open Response”

ASSISTments. The numerical answers are graded automatically freeing up Ms. O’Connor's time

to review and reflect on the students written explanations. Once she collected all the explanations

and read through them on ASSISTments she selected a few of the explanations to use in class.

She wants students to (1) know their explanations will be read and possibly shared with other

students and (2) read other student’s explanations. In reading each other’s explanations students

both learn the math, by being exposed to another student’s solution strategy, and learn how to

write better explanations themselves.


Ms. O’Connor learned a lot by looking at the data that came in from this activity. First,

all but one of her students got the right answers to A - D so she knows they all had a strong

understanding of the procedural parts of this problem. Armed with that information Ms.

O’Connor began to review the responses to E. This question asked students to justify their

answer by explaining how the number of club members going on the trip should affect their

decision. Ms. O’Connor has been working on explanations with her students. She is looking for

clear use of vocabulary, getting the right answer and clearly explaining how they got their

answer. She finds that the best way to improve explanations like this is to have students read and

discuss each other’s work.

To prepare for this work Ms. O’Connor carefully selected the five essays shown in Figure

4 to show a variety of levels of understanding and misunderstanding. For example Student 1

made an error in just

looking at the y-intercept

and not taking note of the

per person rate. Student 2

answers the question as 72

(which are the numbers

calculated in A and B). The

student also mentions the

break-even point of 100

skiers but does not explain

how this number effects the

decision. Student 3 gives a


very nice explanation clearly explaining which number of students should choose each company,

The student is clear that at 100 either company could be chosen and also explains how the set fee

and the rate per person effect this. Student 4 must have done some work to find 100 but does not

clarify anything in the explanation. Student 5 did not even understand that there was a point

where the two companies would be equal. The student just selected 50 students and argued that

for that number Mountain Charter would be cheaper. All of these points and more are discussed

in class as Ms. O’Connor goes over the responses with the class.

In this case, the Advance Student Response system allowed Ms. O’Connor to be more

efficient and more effective as a teacher. The activity described here allowed students to not only

give explanations that were easily read and graded by the teacher but it also allowed the teacher

to share selected explanations with the rest of the class and have them discuss those responses.

As a result, accountability of a lesson is no longer between the teacher and the student. With

these types of advanced technologies work becomes more public or as a “gallery” described by

Weiss and Borderlon’s chapter. In essence, student work becomes interactive with the teacher,

peers, and with themselves. Making student work more public, while safeguarding student

anonymity, is a powerful tool in advancing student learning.

IV. IMPLICATIONS TO THE FUTURE OF TEACHING AND LEARNING PRACTICES IN

SCHOOLS

Educational reforms have promised much and delivered very little for several reasons,

including 1) reform designers are often non-educators, 2) professionals are risk-adverse, 3) the

school and district organization is highly bureaucratic and often dysfunctional, and 4) education

as an institution has a tradition of morphing reform to mirror current practices and routines. As a


result, one should not be surprised that technologies have not had a pronounced impact in

education as it has in other aspects of our lives.

So how do we get real, sustainable technological advances in schools? The examples

provided in this chapter offer a glimpse of what these new practices might look like:

Teachers are freed of routine and mundane administrative tasks such as grading

and organizing data in a way that can be quickly used;

Students are asked to reflect on their own work as well as the work of fellow

students;

Students are asked to work alone and to collaborate with fellow students;

Teachers are able view summaries of learning as well as specific aspect of

possible problem areas;

Students and teachers are able to articulate learning through examinations of both

declarative and procedural knowledge;

Students are presented with problem set similar to future testing efforts; and

The platform is web or cloud-based making access in the new digital divide based

only on connectivity.

Instructional technologies such as ASSISTments have powerful implications for policy

practice, and research. To begin, policy wonks continue to have student assessment and

technology in the crosshairs of school reformation. There is a clear and present press to use

student achievement data to improve teaching and learning in the same ways that the teachers

described above are using data to inform their instruction.


Additionally, this work has major implications for the current and future practice of

teachers and those who lead and train them. If practice is to change to keep pace with the

development of new technologies and the expectation of students then pre-and in-service

development efforts must be altered. Moreover, more cloud-based, interactive instructional

technologies must be developed and implemented in our schools.

Finally, further research is warranted to understand the utility of instructional

technologies like ASSISTments. This research should not only consist of experimental and

quasi-experimental studies of student achievement, but also more robust, detailed examinations

of school-level educators development and implementation and understanding the students’

engagement.

V. CONCLUSION

The evolution of technology and schooling have proceeded at different rates. Technology

has revolutionized offices, stores, airlines, steel plants, hospitals and the military9. Yet schools

have remained largely unaffected. The innovations that had staying power in schools were

simple to manipulate, easy to access and efficient for teachers10,11. Chalkboards, textbooks, the

duplicating machine, and now the interactive white board, have captured these important

characteristics.

Digital Teaching Platforms are well situated to be the next educational innovation that

can impact teaching and learning. Those who implement this new technology should be well

versed in the history of technological innovations of the past. Kling stated, “We do not simply

replace horses and mules with cars and trucks. We have configured an elaborate system of

motorized transport, including new roads, traffic regulations, gas stations, repair shops, insurance


and so on”12. Similarly, systems, structures, and technologies must be developed, implemented,

supported, and sustained in schools. The road to the effective and meaningful use of educational

technology must be paved with the right technology, but also the right people and the right

support and resources. If technologies are to have powerful lasting impacts on the way teachers

teach and how much students learn then the technologies that are developed and how they are

used matter. Instructional technologies like ASSISTments have the potential to fundamentally

alter the normative practices of the teaching profession, and thus the potential to improve student

learning.

REFERENCES

1. Kling, R. "Hopes and horrors: Technology utopianism and anti-utopianism in narratives

of computerization." In Computerization and controversy: Value, conflict and social

choices, edited by R. Kling, 40-59. New York, New York: Academic Press, 1996.

2. Cuban, L. “Computer meets classroom: computer wins.” Teachers College Record 95,

no. 2 (1995): 185-210.

3. —. Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard

University Press, 2001.

4. Ravitch, D. Leftback: A century of battles over school reform. New York: Touchstone,

2000.

5. Tyack, D. and Cuban, L. Tinkering toward utopia: A century of public school reform.

Cambridge, MA: Harvard University Press, 1995.

6. Mendicino, M., Razzaq, L., and Heffernan, N. T. "Comparison of Traditional

Homework with Computer Supported Homework: Improving Learning from Homework


Using Intelligent Tutoring Systems." Journal of Research on Technology in Education

41, no. 3 (2009): 331-359.

7. Warnakulasooriya, R. and Pritchard, D. E. "Learning and problem-solving transfer

between physics problems using web-based homework tutor." EdMedia: World

Conference on Educational Multimedia, Hypermedia, and Telecommunications.

Montreal, 2006. 2976-2983.

8. National Council of Teachers of Mathematics. Principals and Standards for School

Mathematics. Reston, VA: National Council of Teachers of Mathematics, 2000.

9. Tyack, D. and Cuban, L. Tinkering toward utopia: A century of public school reform.

Cambridge, MA: Harvard University Press, 1995.

10. Cuban, L. Teachers and Machines: The Classroom Use of Technology Since 1920. New

York: Teachers College Press.1986.

11. —. Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard

University Press, 2001.

12. Kling, R. "Hopes and horrors: Technology utopianism and anti-utopianism in narratives

of computerization." In Computerization and controversy: Value, conflict and social

choices, edited by R. Kling, 40-59. New York, New York: Academic Press, 1996. p. 44

Documents

II. Explanation of ASSISTmentsweb.cs.wpi.edu/~nth/pubs_and_grants/papers/2011... · Web viewFigure 2 – A teacher report for the last four ASSISTments completed by students. little