Data Collection: Data-driven Decisions in the Area of Assistive Technology Dr. Anna Evmenova George Mason University September 15, 2011

Data Collection:Data Collection:Data-driven Decisions in Data-driven Decisions in

the Area of Assistive the Area of Assistive TechnologyTechnology

Dr. Anna EvmenovaGeorge Mason

UniversitySeptember 15, 2011

AgendaAgenda• Introductions• Data Collection:

• Useful Resources• Summary

So what?

What else?

How?

What?

Why?

Comprehension QuestionsComprehension Questions

1. The data collection and analysis should be done only once during the evaluation process.

2. What behavior dimension or variable should be used to measure how long it takes the user to activate the device after the prompt has been issued?

3. What are the three major data collection techniques? Explain each.

4. What is the research term for examining the user's satisfaction with the AT implementation and outcomes?

5. What are the important elements of the visual data analysis that allow hypothesizing about the effectiveness of the AT tool? (Name at least three)

Why Collect Data?Why Collect Data?

To objectively answer questions:

Is AT needed?How is AT used?Does AT work?Which AT tool is better?Is there a continuous progress

towards the goal?

5 StepsReview, Develop, Examine, Evaluate, Identify

Decision

Current interv. working

AT used is working

AT not used/working

Don’t knowenough

Continue-AT not needed

Continue -record AT in

IEP

Plan for AT (trials)

Stop process,schedule referral

AT in the IEP Consideration AT in the IEP Consideration ProcessProcess

(TAM, (TAM, 2005)2005)

AT Assessment ProcessAT Assessment Process• Assessing Students’ Needs for Assistive

Technology (Reed & Lahm, 2004)

• Education Tech Points (Bowser & Reed, 1995)

• Student, Environment, Tasks, Tools (SETT) Framework (Zabala, 2002)

•Human Activity Assistive Technology (HAAT) Model (Cook & Hussey, 2002)

AT Implementation PlanAT Implementation Planhttp://natri.uky.edu/resources/fundamentals/defined.html

What Data to Collect?What Data to Collect?

• Data - recordings of observable and measurable performance, events, and/or responses

• In order to determine what data to collect, we need to start with …

GOALS and GOALS and OBJECTIVESOBJECTIVES

Setting Up GoalsSetting Up Goals

Specific goals

• E.g., “To try a switch to see if the user’s quality of life will improve”

• E.g., “The user will activate appliances (e.g., computer or TV) by pressing a switch placed on the wheelchair lap tray”

Setting Up Goals (cont.)Setting Up Goals (cont.)

The technology should NOT be the goal itself! It can be an objective though.

• E.g., “Sara will click the correct picture on her AAC device 5 times with 100% accuracy.”

• E.g., Using an AAC device, Sara will (accurately) respond to communication prompts during the play activity in 8 of 10 opportunities.

Setting Up Goals (cont.)Setting Up Goals (cont.)

• Realistic types of change• E.g., gradual increase in vocabulary or gradual

decrease in the time it takes to dress up using adapted tools

• Reasonable mastery criteria• E.g., if collecting data on meaningful switch use,

100% frequency may not be a desired goal• E.g., if collecting data on crossing the street, 5

out of 5 times is crucial

How do you measure success?How do you measure success?Progress towards• Academic standards

• E.g., Virginia Standards of Learning (state DOE sites; software aligned with academic standards)

• Speech-language competences• E.g., Operational ==> Strategic ==> Social

==> Linguistic Competence (Light, 1989)

•Functional skill sets• E.g., (a) Daily Living, (b)

Vocational, (c) Recreation/Leisure, and (d) Community Functioning

Dependent & Independent Dependent & Independent VariablesVariables

Dependent VariableThe behavior targeted for change• e.g., “Use picture prompts with students with severe

disabilities to increase instruction-following skills.”

Independent Variable:The technology intervention being used to change behavior• E.g., “Increase reading skills by two grade

levels following instruction using the “I’m A Better Reader” program.”

Target BehaviorTarget BehaviorThe behavior that is to be changed

What are the target behavior criteria?

ObservableMeasurableQuantifiable

Operationalized (on-task behavioron-task behavior)

Know vs. Point toKnow vs. Point to

Common Behavior DimensionsCommon Behavior Dimensions

Measuring

• Frequency• Rate• Accuracy or fluency• Duration• Latency

What behavior dimensions have you What behavior dimensions have you

measured in your practice? measured in your practice?

Is your goal to Is your goal to improve the number improve the number of? Speed? of? Speed? Quality or Quality or accuracy? Time accuracy? Time behavior lasts? Or behavior lasts? Or the time it takes the the time it takes the user to initiate user to initiate something?something?

Frequency Recording SystemFrequency Recording System

Number of occurrences• E.g., Number of words in the essay or AAC

messages

Adapted from QIAT listserv forms

Frequency w/ Controlled Frequency w/ Controlled OpportunitiesOpportunities

http://www.aiu3.net/Level3.aspx?id=3860

RateRateNumber of occurrences over total number of minutes

• E.g., Math problems per minute, written words per minute

//// //// //// //// ///

//// ///

User: ___________________________________Observer: _______________________________Behavior: _______________________________

23/15=1.510:00 – 10:153/16

8/20=0.410:00 – 10:203/15

RateNotations of Occurrence

TimeStart Stop

Date

Adapted from Alberto & Troutman, 2009

Accuracy or FluencyAccuracy or Fluency

Number of correct or number of correct over time• E.g., Reading or writing fluency

Adapted from Reed, Bowser, & Korsten, 2002

DurationDuration

Time it takes the user to complete the task• E.g., Time it takes to read the book with and

without a text reader

Adapted from Gast, 2010

LatencyLatencyTime between directions and the behavior occurrence

• E.g., time it takes the student to start writing an essay after the prompt with and without the graphic organizer

Student: _________________________Observer: ________________________Behavior/Task: ___________________

Date Device Used

Time Latency

Delivery of Prompt

Response Initiation

Ad

ap

ted

fro

m A

lbert

o &

Tro

utm

an

, 2

00

9

One More Recording System to One More Recording System to ConsiderConsider

How to Collect Data?How to Collect Data?

Examples of data collection techniques include:

• Permanent Products• Direct or Video/Audio Observations• Interviews

Permanent ProductsPermanent ProductsOutcomes

• Finished products completed by students (e.g., worksheets, essays)

• Mouse-click and performance data collection built into the existing AT tools

Built-in Data CollectionBuilt-in Data Collection• General:

• Total and average duration of sessions• # of trials and activities completed as well as attempts• Performance data (% correct, # correct 1st try/total,

comparing to expected outcomes)

• Specific• Scan duration, scans per item, re-prompt time (Laureate)• Include anecdotal observations (Cambium)• Portfolios for “not scored” activities (Cambium)

• Hear students’ pronunciation recordings (DJ)• Language Activity Monitor (LAM) programs that

allow transfer and evaluation of AAC messages

Next Next

Webinar

Webinar

ObservationsObservationsDirect observations of behavior (process)

Video/audio observations of behavior (product)

• Anecdotal Notes (e.g., brief narratives)• Event Recording (e.g., tallies for discreet

behaviors)

• Interval Recording (e.g., tallies for intervals)

Ø Ø √ Ø Ø √

Length of Intervals in Seconds 10” 20” 30” 40” 50” 60”

InterviewsInterviews• Unstructured or open-ended

• E.g., “Tell me all you think about this AT tool”

• Semi-structured • E.g., “What features of this AT tool did you like?

Why?”

• Structured• E.g., Likert rating scales: “How would your rate

your level of satisfaction with the effectiveness of the tried AT tools?” (from 1 – extremely dissatisfied to 5 – extremely satisfied)

Systematic Data CollectionSystematic Data Collection

Single-Subject Research Designs

SSRD involves studying

a single individual by taking repeated

measurementof the targeted behavior whilesystematically applying andwithdrawing the intervention

Key SSR Characteristics Key SSR Characteristics • A-B logic

A = Baseline or control phase (period of no treatment)

B = Intervention or treatment phase (period of introducing an AT tool)

• Stability of data

• Repeated measurements of behavior (e.g., several baseline and treatment sessions)

•Visual analysis of graphical representation of data

•Individual serves as own control

Adaped from Del Siegle, University of Connecticut

Suppose you want to compare user’s performance with and without an AT tool.

Baseline (w/o AT) Treatment (with AT)

Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool.

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task without the AT.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Baseline w/o AT

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task without AT. You would measure how long it took each day

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Baseline w/o AT Treatment with AT

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16



Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0



Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task without AT. You would measure how long it took each day for several days.


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task. You would measure how long it took each day for several days. In the example below, it took the user 35 minutes on the first day

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task. You would measure how long it took each day for several days. In the example below, it took the user 35 minutes on the first day, 30 min. on the second day

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


Suppose you want to compare the time it takes the user to complete the task or the duration with and without an AT tool over many days. First you would need to establish a baseline of how long it takes to complete the task. You would measure how long it took each day for several days. In the example below, it took the user 35 minutes on the first day, 30 min. on the second day, and 35 min. on the third day.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Once a baseline of behavior has been established (when a consistent pattern emerges with at least three data points), the treatment begins.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Once a baseline of behavior has been established (when a consistent pattern emerges with at least three data points), the treatment begins. The observer continues to plot how long it takes to complete the task

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Once a baseline of behavior has been established (when a consistent pattern emerges with at least three data points), the treatment begins. The observer continues to plot how long it takes to complete the task, while using the AT tool.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

In this example, we can see that the time it takes to complete the task decreased immediately once the AT tool was implemented.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

In this example, we can see that the time it takes to complete the task decreased immediately once the AT tool was implemented. The design in this example is known as an A-B design.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


In this example, we can see that the time it takes to complete the task decreased immediately once the AT tool was implemented. The design in this example is known as an A-B design. The baseline period is referred to as A

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

In this example, we can see that the time it takes to complete the task decreased immediately once the AT tool was implemented. The design in this example is known as an A-B design. The baseline period is referred to as A and the treatment period is identified as B.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0


In this example, we can see that the time it takes to complete the task decreased immediately once the AT tool was implemented. The design in this example is known as an A-B design. The baseline period is referred to as A and the treatment period is identified as B.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

A BBaseline w/o AT Treatment with AT

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Another design is the A-B-C design. An A-B-C design involves trying another AT tool in the third phase.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Baseline w/o AT Treatment with AT #1 Treatment with AT #2

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Another design is the A-B-C design. An A-B-C design involves trying another AT tool in the third phase.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

A B C

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Baseline w/o AT Treatment with AT #1 Treatment with AT #2

Another design is the A-B-A design. An A-B-A design (also known as withdrawal design) involves discontinuing the use of AT and returning to baseline.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Baseline w/o AT Treatment with AT Baseline w/o AT

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Another design is the A-B-A design. An A-B-A design (also known as withdrawal design) involves discontinuing the use of AT and returning to baseline.

Day

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

A B A

Tot

al d

urat

ion

in m

inut

es

40

35

30

25

20

15

10

5

0

Baseline w/o AT Treatment with AT Baseline w/o AT

Basic Single-Subject DesignsBasic Single-Subject Designs

• AB – teaching design• ABC – changing conditions

design• ABA – withdrawal design

Note: These are not good research designs!

• Adding another phase – ABAB reversal design

• Alternating treatments design – rapid alternation of the AT tools

Time Series Concurrent and Time Series Concurrent and Differential (TSCD) ApproachDifferential (TSCD) Approach

• Measuring over time• With and without AT

(aided vs. unaided)• Graphing data in a

certain way• Discover data patterns

pointing to the impact of AT

For more information see Smith, 2000

How Often to Collect Data?How Often to Collect Data?

• Trial-by-trial data collection (every time)• Probe data collection (episodic but systematic)

How often do you think we need to collect data?

3 times a week…Weekly…Daily … Multiple times a day…

Either, as long as it is continuous, systematic, and consistent!

What Else to Consider?What Else to Consider?

• Interobserver agreement

• Fidelity of treatment

• Social validity

• Cost analysis

Interobserver Agreement

Accuracy or reliability of the data collection

• Degree to which two separate people independently and simultaneously collecting data agree on what occurred or did not occur

• Less than 80% agreement flags a possible problem with the data collection!

AgreementsAgreements + Disagreements

X 100%

Fidelity of ImplementationFidelity of Implementation

Consistently and precisely implementing the AT tool(s) the way it was intended

• To make sure AT is working as needed• Evaluators and observers behave the way

intended• Data are collected in comparable environments• Data collection is consistent in all conditions

Social ValiditySocial Validity• User satisfaction with the implementation and

outcomes of the AT tool

• User’s preference for and ability to use one or another AT tool is crucial• E.g., using an AAC device with dynamic display or

with semantic compaction

• Collect via interviews and Likert-scale questionnaires

• E.g., the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST; Demers, Weiss-Lambrou, & Ska, 2002).

Cost Analysis in ATCost Analysis in ATIs it worth it? Is it worth it? vs. vs. It doesn’t matter how much it costs!It doesn’t matter how much it costs!

Concept of Practical Significance Concept of Practical Significance

Formula for optimizing independence in rehabilitation (Smith, 2000)

+ Value of Independence- Cost of Assistive Technology- Cost of Personal Assistant- Lost Time Available to the Person with a Disability= Overall Cost

So What Data Tells Us?So What Data Tells Us?Make a decision about whether:

• The user should continue using AT, • Try something else, or • More data are needed to make

a decision

Based on: • Visual analysis of line graphs, bar graphs, pie

graphs• Difference in counts, percentages, means,

and standard deviations

Visual Data Analysis

Analyze data patters across adjacent phases for:

• Change in level (mean)

• Change in trend (slope and magnitude)

• Variability of data within and across phases (high, medium, low)

• Immediacy of change (rapid or gradual)

• Data overlap between phases

• Consistency of data patterns

ExamplesExamples

Useful ResourcesUseful Resources• AT Data Collection Tools -

http://www.aiu3.net/Level3.aspx?id=3860

• Assistive Technology Outcomes Measurement System Project - http://www.r2d2.uwm.edu/atoms/

• Reed, P. Bowser, G. and Korsten, How Do You Know It? How Can You Show It?, Wisconsin Assistive Technology Initiative http://www.wati.org/WatiMaterials

SummarySummary

Don’t be afraid of the data!

Collect observable & measurable data over time

Compare data with and without AT

Look at the data! Does AT work? ANS WERS

ANS WERS

Q & A TimeQ & A Time

Contact Info: [email protected]

ReferencesReferences• Alberto, P. A., & Troutman, A. C. (2008). Applied behavior analysis for teachers (8th ed.).

Upper Saddle River, NJ: Prentice Hall. • Bowser, G. & Reed, P. (1995). Education Tech Points for assistive technology planning.

Journal of Special Education Technology, 12, 325-338.• Demers, L., Weiss-Lambrou, R., & Ska, B. (2002). The Quebec User Evaluation of

Satisfaction with Assistive Technology (QUEST 2.0): An overview and recent progress. Technology and Disability, 14, 101-105

• Gast, D. L. (2010). Single subject research methodology in behavioral sciences. New York, NY: Routledge.

• Light, J. (1989). Toward a definition of communicative competence for individuals using augmentative and alternative communication systems. Augmentative and Alternative Communication, 5, 137-144. doi: 10.1080/07434618912331275126

• Reed, P., & Lahm, E. (Eds.). (2004). Assessing students' needs for assistive technology: A resource manual for school district teams. Oshkosh, WI: Wisconsin Assistive Technology Initiative. Retrieved from www.wati.org/content/supports/free/pdf/ASNAT4thEditionDec08.pdf

• Smith, R. O. (2000). Measuring assistive technology outcomes in education. Assessment for Effective Intervention, 25, 273-290 doi: 10.1177/073724770002500403

• TAM (2005). Assistive Technology Planner and Implementation Plan. Retrieved from http://natri.uky.edu/resources/fundamentals/defined.html

• Zabala, J. S. (2002). The SETT framework: Critical areas to consider when making informed assistive technology decisions. Lake Jackson, TX: Assistive Technology and Leadership. Retrieved from http://sweb.uky.edu/~jszaba0/JoySETT.html

• Cook, A. E., & Hussey, S. M. (2002) (2nd ed.). Assistive technologies: Principles and practice. St. Louis, MO: Mosby.

Documents

Data Collection: Data-driven Decisions in the Area of Assistive Technology Dr. Anna Evmenova George Mason University September 15, 2011