Using Assessment to Using Assessment to Improve and Evaluate Improve and Evaluate Student Learning in Student Learning in

Introductory Introductory StatisticsStatistics

Bob delMas (Univ. of MN)Bob delMas (Univ. of MN)

Slide 2 of 41

GAISE GuidelinesGAISE Guidelines

Assessment needs to be aligned with Assessment needs to be aligned with learning goals learning goals

Focus on learning key ideas (not only Focus on learning key ideas (not only skills, procedures, and computed answers)skills, procedures, and computed answers)

Include formative assessments as well as Include formative assessments as well as summativesummative

Use timely feedback to promote learningUse timely feedback to promote learning It is possible to implement good It is possible to implement good

assessments even in large classroom assessments even in large classroom settingssettings

Slide 3 of 41

Suggestions for TeachersSuggestions for Teachers Integrate assessment as an essential component Integrate assessment as an essential component

of the course (timing of assessment and of the course (timing of assessment and activities).activities).

Use a variety of assessment methods to provide a Use a variety of assessment methods to provide a more complete evaluation of student learning. more complete evaluation of student learning.

Assess statistical literacy using assessments such Assess statistical literacy using assessments such as interpreting or critiquing articles in the news as interpreting or critiquing articles in the news and graphs in media. and graphs in media.

Assess statistical thinking using assessments Assess statistical thinking using assessments such as student projects and open-ended such as student projects and open-ended investigative tasks.investigative tasks.

Slide 4 of 41

Suggestions for Large Suggestions for Large ClassesClasses

Use small group projects instead of Use small group projects instead of individual projects.individual projects.

Use peer review of projects to provide Use peer review of projects to provide feedback and improve projects before feedback and improve projects before grading.grading.

Use items that focus on choosing good Use items that focus on choosing good interpretations of graphs or selecting interpretations of graphs or selecting appropriate statistical procedures. appropriate statistical procedures.

Use discussion sections for student Use discussion sections for student presentations.presentations.

Slide 5 of 41

AAHE 9 Principles of AAHE 9 Principles of AssessmentAssessment

Assessment begins with educational values.Assessment begins with educational values. Treats understanding of learning as multidimensional, Treats understanding of learning as multidimensional,

integrated, and revealed over time.integrated, and revealed over time. Requires clear, explicitly stated purposes.Requires clear, explicitly stated purposes. Requires attention to experiences that lead to Requires attention to experiences that lead to

outcomes.outcomes. Works best when ongoing, not episodic.Works best when ongoing, not episodic. Fosters wider improvement when representatives from Fosters wider improvement when representatives from

across the educational community are involved (all across the educational community are involved (all stakeholders). stakeholders).

Represents issues and questions that people really Represents issues and questions that people really care about.care about.

Most likely to lead to improvement if part of a larger Most likely to lead to improvement if part of a larger set of conditions that promote change.set of conditions that promote change.

Meets responsibilities to students and to the public.Meets responsibilities to students and to the public.

Slide 6 of 41

Assessment TriangleAssessment TriangleNational Research Council (2001), National Research Council (2001), Knowing what Knowing what

Students KnowStudents Know Cognition: the aspects of achievement or competencies that

are to be assessed Observation: the tasks used to collect evidence about

students’ achievement (i.e., the assessments) Interpretation: the methods used to analyze the evidence

resulting from the tasks

The three elements are interdependent

A successful assessment synchronizes all three elements

Slide 7 of 41

COGNITIONCOGNITION A theory or set of beliefs about: A theory or set of beliefs about:

How students represent knowledgeHow students represent knowledge How students develop competenceHow students develop competence

Used to identify important Used to identify important knowledge and skillsknowledge and skills

Based on a learning model that Based on a learning model that provides a level of detail provides a level of detail sufficient to accomplish the sufficient to accomplish the assessmentassessment

Slide 8 of 41

OBSERVATIONOBSERVATION What we typically consider to be the What we typically consider to be the

“assessment”“assessment”

Careful design of tasks that will provide evidence Careful design of tasks that will provide evidence that can be linked to the learning modelthat can be linked to the learning model

Context and Purpose are ImportantContext and Purpose are Important

A national assessment can indicate relative A national assessment can indicate relative standing, but not sensitive to nuances of standing, but not sensitive to nuances of instructioninstruction

Instructor constructed assessment can be tied Instructor constructed assessment can be tied to classroom instruction, but may not to classroom instruction, but may not generalize to a larger populationgeneralize to a larger population

Slide 9 of 41

INTERPRETATIONINTERPRETATION Set of assumptions and models that are used to Set of assumptions and models that are used to

interpret evidence from observationinterpret evidence from observation

Links observations to competencies (cognition)Links observations to competencies (cognition)

National assessments may use formal, National assessments may use formal, statistical models that identify patterns statistical models that identify patterns indicative of competency levelsindicative of competency levels

Classroom assessment is typically more Classroom assessment is typically more qualitative and identifies categories of qualitative and identifies categories of competency based on observationscompetency based on observations

Important to identify what has not developed as Important to identify what has not developed as well as what has developedwell as what has developed

Important to identify misunderstandings and Important to identify misunderstandings and misconceptions as well as correct misconceptions as well as correct understandingunderstanding

Slide 10 of 41

Example: Understanding p-Values

A p-value is the probability of obtaining results as or more extreme than the observed results, given that the null hypothesis is true.

A p-value IS NOT the probability that the null hypothesis is true.

A p-value IS NOT the probability that the alternative hypothesis is true.

Slide 11 of 41

A research article reports the results of a new drug test. The drug is to be used to decrease vision loss in people with Macular Degeneration. The article gives a p-value of .04 in the analysis section. Indicate if the following interpretation of this p-value is valid or invalid.

Statement (N = 1617) Valid Invalid

The probability of getting results as extreme or more extreme than the ones in this study if the drug is actually not effective.

Example: Understanding p-Values

Slide 12 of 41

A research article reports the results of a new drug test. The drug is to be used to decrease vision loss in people with Macular Degeneration. The article gives a p-value of .04 in the analysis section. Indicate if the following interpretation of this p-value is valid or invalid.

Statement (N = 1617) Valid Invalid

The probability of getting results as extreme or more extreme than the ones in this study if the drug is actually not effective.

57.7 42.3

Example: Understanding p-Values

Slide 13 of 41

A research article reports the results of a new drug test. The drug is to be used to decrease vision loss in people with Macular Degeneration. The article gives a p-value of .04 in the analysis section. Items 25, 26, and 27 present three different interpretations of this p-value. Indicate if each interpretation is valid or invalid.

Statement (N = 1617) Valid Invalid

25. The probability of getting results as extreme or more extreme than the ones in this study if the drug is actually not effective.

57.7 42.3

26. The probability that the drug is not effective.

27. The probability that the drug is effective.

Example: Understanding p-Values

Slide 14 of 41

A research article reports the results of a new drug test. The drug is to be used to decrease vision loss in people with Macular Degeneration. The article gives a p-value of .04 in the analysis section. Items 25, 26, and 27 present three different interpretations of this p-value. Indicate if each interpretation is valid or invalid.

Statement (N = 1617) Valid Invalid

25. The probability of getting results as extreme or more extreme than the ones in this study if the drug is actually not effective.

57.7 42.3

26. The probability that the drug is not effective. 39.9 60.1

27. The probability that the drug is effective. 45.2 54.8

9% of all students made an incorrect choice for all 3 items Of those who chose “valid” for Item 25 (N = 933)

55% chose valid for at least one of the other items 7% chose “valid” for all 3 items.

Example: Understanding p-Values

Slide 15 of 41

Assessment CycleAssessment Cycle(from Beth Chance ; see G. Wiggins 1992, 1998)(from Beth Chance ; see G. Wiggins 1992, 1998)

Set goalsSet goals What should students know, be able to do?What should students know, be able to do? At what point in the course?At what point in the course? Identify assessable learning outcomes that match goalsIdentify assessable learning outcomes that match goals

Select methodsSelect methods Identify an assessment that matches the type of learning outcomeIdentify an assessment that matches the type of learning outcome Consider minute papers, article reviews, newspaper assignments, Consider minute papers, article reviews, newspaper assignments,

projects, short answer items, multiple choiceprojects, short answer items, multiple choice Can the assessment be built into the activity?Can the assessment be built into the activity?

Gather evidenceGather evidence (i.e., administer the assessment) (i.e., administer the assessment) Draw inference Draw inference

Don’t use results just to assign a gradeDon’t use results just to assign a grade Consider what responses indicate about student understandingConsider what responses indicate about student understanding

Take actionTake action Provide feedbackProvide feedback What can be done to remedy a misunderstanding (an activity; extra What can be done to remedy a misunderstanding (an activity; extra

reading; more experience with a procedure or a concept)reading; more experience with a procedure or a concept) Re-examine goals and methodsRe-examine goals and methods

Slide 16 of 41

Embedding Assessment Embedding Assessment into Classroom Activitiesinto Classroom Activities

Sorting DistributionsSorting Distributions Goal: Learn to associate labels Goal: Learn to associate labels

with shapes of distributionswith shapes of distributions Normal DistributionNormal Distribution

Goal: Learn to find areas for the Goal: Learn to find areas for the standard normal distributionstandard normal distribution

Sampling DistributionsSampling Distributions Learn the characteristics of Learn the characteristics of

sampling distributionssampling distributions Understand effect of sample size Understand effect of sample size

Slide 17 of 41

ARTIST ARTIST WebsiteWebsite

https://app.gen.umn.edu/artihttps://app.gen.umn.edu/artistst Item Database (Assessment Builder): A collection of about 1100

items, in a variety of item formats, organized according to statistical topic and type of learning outcome assessed.

Resources: Information, guidelines, and examples of alternative assessments. Copies of articles or direct links to articles on assessment in statistics. References and links for other related assessment resources.

Research Instruments: Instruments that can be used for research and evaluation projects that involve assessments of outcomes related to teaching and learning statistics.

Implementation issues: Questions and answers on practical issues related to designing, administering, and evaluating assessments.

Presentations: Copies of conference papers and presentations on the ARTIST project, and handouts from ARTIST workshops.

Events: Information on ARTIST events. Participation: Ways to participate as a class tester for ARTIST materials.

Slide 18 of 41

ARTIST Topic TestsARTIST Topic Tests

There are 11 scales, consisting of 8-12 multiple-choice items, that can be administered online. Our goal is to develop high quality, valid and reliable scales that can be used for a variety of purposes (e.g., research, evaluation, review, or self-assessment).

TOPICS Data Collection (data types, types of study, study design) Data Representation (choose appropriate graphs, interpret graphs) Measures of Center (estimate, when to use, interpret, properties) Measures of Spread (estimate, when to use, interpret, properties) Normal Distribution (characteristics, empirical rule, areas under the curve) Probability (interpret, independence, relative frequency, simulation) Bivariate Quantitative Data (scatterplots, correlation, descriptive and inferential

methods, outliers, diagnostics, influential observations) Bivariate Categorical Data (two-way tables and chi-square test, association) Sampling Distributions (types of samples, sample variability, sampling

distributions, Central Limit Theorem) Confidence Intervals (interpret, confidence level, standard error, margin of error) Tests of Significance (hypothesis statements, p-values, Type I and II error, statistical

and practical significance)

Slide 19 of 41

Comprehensive Assessment of Comprehensive Assessment of Outcomes in Statistics (CAOS)Outcomes in Statistics (CAOS)

Forty item test that can be administered as an online test to evaluate the attainment of desired student outcomes.

CAOS items are designed to represent the big ideas and the types of reasoning, thinking and literacy skills deemed important for all students across first courses in statistics.

Unifying focus is on reasoning about variability: in univariate and bivariate distributions, in comparing groups, in samples, and when making estimates and inferences.

Not intended to be used exclusively as a final exam or as the sole assessment to assign student grades.

CAOS can provide very informative feedback to instructors about what students have learned and not learned in an introductory statistics course (e.g., administered as pretest and posttest).

Slide 20 of 41

Data Representation Item (page 19)A baseball fan likes to keep track of statistics for the local high school baseball team. One of the statistics she recorded is the proportion of hits obtained by each player based on the number of times at bat as shown in the table below. Which of the following graphs gives the best display of the distribution of proportion of hits in that it allows the baseball fan to describe the shape, center and spread of the variable, proportion of hits?

Slide 21 of 41

A baseball fan likes to keep track of statistics for the local high school baseball team. One of the statistics she recorded is the proportion of hits obtained by each player based on the number of times at bat as shown in the table below. Which of the following graphs gives the best display of the distribution of proportion of hits in that it allows the baseball fan to describe the shape, center and spread of the variable, proportion of hits?

RESPONSE

PERCENT

(N = 1643)

Graph A

Graph B

Graph C

Graph D

What percents would you predict for your students?

Data Representation Item (page 19)

Slide 22 of 41

A baseball fan likes to keep track of statistics for the local high school baseball team. One of the statistics she recorded is the proportion of hits obtained by each player based on the number of times at bat as shown in the table below. Which of the following graphs gives the best display of the distribution of proportion of hits in that it allows the baseball fan to describe the shape, center and spread of the variable, proportion of hits?

RESPONSE

PERCENT

(N = 1643)

Graph A 11.1

Graph B 46.4

Graph C 29.1

Graph D 13.4

RESULTS

US undergraduates 2005-2006

Data Representation Item (page 19)

Slide 23 of 41

Data Representation Item (page 20)A local running club has its own track and keeps accurate records of each member's individual best lap time around the track, so members can make comparisons with their peers. Here are graphs of these data. Which of the graphs allows you to most easily see the shape of the distribution of running times?

RESPONSE

PERCENT

(N = 1345)

Graph A

Graph B

Graph C

All of the above

Slide 24 of 41

A local running club has its own track and keeps accurate records of each member's individual best lap time around the track, so members can make comparisons with their peers. Here are graphs of these data. Which of the above graphs allows you to most easily see the shape of the distribution of running times?

RESPONSE

PERCENT

(N = 1345)

Graph A 43.8

Graph B 48.9

Graph C 3.6

All of the above 3.7

Data Representation Item (page 20)

Slide 25 of 41

First Small Group Exercise

Designate one person to be the recorder.

Discuss the following questions (pages 21-22 of handout) with respect to the Data Representation items:

Why do you think students are selecting the incorrect responses for these items? (3-5 minutes)

Outline an instructional activity to help students develop the correct understanding. (10 minutes)

Slide 26 of 41

Second Small Group Exercise

Choose a Topic:Sampling VariabilityConfidence IntervalsTests of SignificanceBivariate Quantitative Data

Discuss the following questions:

Why do you think students are selecting the incorrect responses for each item? (5-10 minutes)

Outline an instructional activity to help students develop the correct understanding. (10-15 minutes)

Slide 27 of 41

Sampling Variability Item 1

A certain manufacturer claims that they produce 50% brown candies. Sam plans to buy a large family size bag of these candies and Kerry plans to buy a small fun size bag. Which bag is more likely to have more than 70% brown candies?

RESPONSERESPONSEPERCENTPERCENT

(N = 1608)(N = 1608)

Sam, because there are more candies, so his bag can have more brown candies.

5.35.3

Sam, because there is more variability in the proportion of

browns among larger samples. 11.611.6

Kerry, because there is more variability in the proportion of

browns among smaller samples. 32.432.4

Kerry, because most small bags will have more than 50% brown candies.

1.71.7

Both have the same chance because they are both random samples.

48.948.9

Slide 28 of 41

Sampling Variability Item 2

Consider the distribution of average number of hours that college students spend sleeping each weeknight. This distribution is very skewed to the right, with a mean of 5 and a standard deviation of 1. A researcher plans to take a simple random sample of 18 college students. If we were to imagine that we could take all possible random samples of size 18 from the population of college students, the sampling distribution of average number of hours spent sleeping will have a shape that is

RESPONSERESPONSEPERCENTPERCENT

(N = 872)(N = 872)

Exactly normal. 18.818.8

Less skewed than the population. 34.434.4

Just like the population (i.e., very skewed to the right). 34.734.7

It's impossible to predict the shape of the sampling distribution. 12.012.0

Slide 29 of 41

Confidence Interval Item 1

Suppose two researchers want to estimate the proportion of American college students who favor abolishing the penny. They both want to have about the same margin of error to estimate this proportion. However, Researcher 1 wants to estimate with 99% confidence and Researcher 2 wants to estimate with 95% confidence. Which researcher would need more students for her study in order to obtain the desired margin of error?

RESPONSE

PERCENT

(N = 1296)

Researcher 1. 51.9

Researcher 2. 25.9

Both researchers would need the same number of subjects. 9.1

It is impossible to obtain the same margin of error with the two different confidence levels. 13.1

Slide 30 of 41

Confidence Interval Item 2A high school statistics class wants to estimate the average number of chocolate chips per cookie in a generic brand of chocolate chip cookies. They collect a random sample of cookies, count the chips in each cookie, and calculate a confidence interval for the average number of chips per cookie (18.6 to 21.3). Indicate if the following interpretations are valid or invalid.

Statement (N = 1609) Valid Invalid

We are 95% certain that each cookie for this brand has approximately 18.6 to 21.3 chocolate chips.

51.2 48.8

We expect 95% of the cookies to have between 18.6 and 21.3 chocolate chips.

34.1 65.9

We would expect about 95% of all possible sample means from this population to be between 18.6 and 21.3 chocolate chips.

53.1 46.9

We are 95% certain that the confidence interval of 18.6 to 21.3 includes the true average number of chocolate chips per cookie.

75.7 24.3

Slide 31 of 41

Test of Significance Item 1

A newspaper article claims that the average age for people who receive food stamps is 40 years. You believe that the average age is less than that. You take a random sample of 100 people who receive food stamps, and find their average age to be 39.2 years. You find that this is significantly lower than the age of 40 stated in the article (p < .05). What would be an appropriate interpretation of this result?

RESPONSE

PERCENT

(N = 1101)

The statistically significant result indicates that the majority of people who receive food stamps is younger than 40.

33.8

Although the result is statistically significant, the difference in age is not of practical importance.

50.5

An error must have been made. This difference is too small to be statistically significant.

15.7

Slide 32 of 41

Test of Significance Item 2

A researcher compares men and women on 100 different variables using a two-sample t-test. He sets the level of significance to .05 and then carries out 100 independent t-tests (one for each variable) on data from the same sample. If, in each case, the null hypothesis actually is true for every test, about how many "statistically significant" findings will this researcher report?

RESPONSE

PERCENT

(N = 1160)

0 30.2

5 45.7

10 7.1

None of the above 17.1

Slide 33 of 41

Bivariate Quantitative Data Item 1The number of people living on American farms has declined steadily during the last century. Data gathered on the U.S. farm population (millions of people) from 1910 to 2000 were used to generate the following regression equation: Predicted Farm Population = 1167 - .59 (YEAR). What method would you use to predict the number of people living on farms in 2050.

RESPONSE

PERCENT

(N = 1591)

Substitute the value of 2050 for YEAR in the regression equation, and compute the predicted farm population.

19.8

Plot the regression line on a scatterplot, locate 2050 on the horizontal axis, and read off the corresponding value of population on the vertical axis.

15.6

Neither method is appropriate for making a prediction for the year 2050 based on these data.

28.4

Both methods are appropriate for making a prediction for the year 2050 based on these data.

36.236.2

Slide 34 of 41

Bivariate Quantitative Data Item 2A statistics instructor wants to use the number of hours studied to predict exam scores in his class. He wants to use a linear regression model. Data from previous years shows that the average number of hours studying for a final exam in statistics is 8.5, with a standard deviation of 1.5, and the average exam score is 75, with a standard deviation of 15. The correlation is .76. Should the instructor use linear regression to predict exam scores from hours studied?

RESPONSE

PERCENT

(N = 850)

Yes, there is a high correlation, so it is alright to use linear regression. 21.2

Yes, because linear regression is the statistical method used to make predictions when you have bivariate quantitative data. 27.1

Linear regression could be appropriate if the scatterplot shows a clear linear relationship. 46.2

No, because there is no way to prove that more hours of study causes higher exam scores. 5.5

Slide 35 of 41

Assessment Builder: SearchAssessment Builder: Search

Slide 36 of 41

Assessment Builder: ResultsAssessment Builder: Results

Slide 37 of 41

Assessment Builder: ResultsAssessment Builder: Results

Slide 38 of 41

Assessment Builder: Item SetAssessment Builder: Item Set

Slide 39 of 41

Assessment Builder: DownloadAssessment Builder: Download

Slide 40 of 41

Assessment Builder: DownloadAssessment Builder: Download

Slide 41 of 41

ARTIST ARTIST WebsiteWebsite

https://app.gen.umn.edu/artihttps://app.gen.umn.edu/artistst

We invite you to contact the ARTIST team with any comments and suggestions you have regarding this presentation, or any of the materials at the ARTIST website.

Thank you for your participation in today’s session.