FIRST II faculty are experimenting with inquiry-based teaching to promote deep understanding by students.Key to these instructional changes and innovations is determining their effectiveness. Hence, this newsletterfocuses on assessment. After a brief introduction to types of assessment, we provide a specific example ofhow to use multiple assessments to collect substantive data about student understanding.

FOCUS ON ASSESSMENT

DATA COLLECTION WITH A PURPOSE

Assessment is data collection with the purpose ofanswering questions. The questions fall into threecategories:

(1) Description – What is happening?(2) Cause – Does “x” (a treatment) affect “y” (an

outcome)?(3) Process or Mechanism – Why or how does

“x” cause “y”?

Once the question is clearly stated, the types of dataand methods of collection are determined (Figure 1).

Direct data collection requires that studentsdemonstrate their knowledge and skills, which arethen objectively measured. Examples of instrumentsused to collect direct data are tests, homeworkassignments, oral presentations, research or positionpapers, and concept maps. Indirect data includes

/

Newsletter - August 2003Volume 1, Issue 2Supported by the National Science Foundation grant number DUE 0088847.Editor: Debra Linton

students’ reflection on their learning, attitudes orconfidence. These data are most commonlycollected by self-report surveys. Interviews andportfolios can be used to collect both direct andindirect data. For example, in an interview studentsmay be asked questions about their attitudes as wellas their understanding of a concept.

Data collection designs must also be practical interms of time and implementation. Realistically, theease of assessment techniques and their potentialvalue for assessing understanding are inverselyproportional (Figure 2). However, we can still getmeaningful data about learning if we are consistentin aligning our goals with assessment tools. Thefollowing example illustrates the use of multipleassessments to collect data about students’understanding of evolution and natural selection.

end text

Page 2

In this example, an inquiry-based activity isincorporated into a large class meeting (~200students). Students use a simulation of changes inmale guppy coloration patterns in response todifferent levels of predation (see page 7 fordetails). The learning goals are for students to beable to:

1. Explain what "fitness" is and why it isnecessary for judging the adaptiveness of atrait.

2. Apply the concept of evolution by naturalselection to a specific case.

Student understanding is assessed using:(1) a pretest, (2) formative assessment, and(3) summative assessment.

Pretest Examples:Pre-instruction assessment of students’ priorknowledge and alternative conceptions isnecessary to guide creation of a student-centered,active learning module. Pretest data also serves asa point of comparison with posttest data forassessment of student learning gains. Pretests cantake the form of multiple-choice questions,extended response writings, or concept maps ofrelevant terms. A survey or consensogram (withpost-its or Personal Response System) could alsobe used to collect students’ self-report data onprior knowledge or attitudes.

Multiple Choice

Multiple-choice questions from the ConceptualInventory of Natural Selection (CINS).

The Conceptual Inventory of Natural Selection(Anderson et al., 2002) provides three validatedsets of multiple-choice questions dealing with theconcepts of fitness, evolution and naturalselection. Each set of questions is preceded by abrief description of the results of an actualevolutionary research study [Galapagos Finches,Venezuelan Guppies, Canary Island Lizards]. Theauthors of the CINS have documented common

misconceptions of students and used them for thefoils in the questions. Students’ incorrect answersprovided data on the specific misconceptions aboutthese concepts. Results of the multiple-choicequestions are displayed as percentages of studentresponses (Figure 3). These data point to theprevalence of specific misconceptions.

Assessment Examples - Natural Selection

Extended Response

“Explain the changes that occurred in the treeand animal in the figure (Figure 4). Address indetail your current understanding of evolution bynatural selection.”

The instructor can quickly read through theresponses to determine students’ priorknowledge and misconceptions. Responses canbe scored using a 5-3-1 rubric(http://www.msu. edu/course/isb/202/ebertmay/rubrics.html)

or counted as “+” or “-”.

Page 3

A coding rubric provides more detailed frequencydata for individual concepts. To develop a codingrubric, start with the “ideal” answer and also list allthe known misconceptions (add others as they arisein student responses). Group these concepts intocategories and assign each concept a uniqueidentifier. As you read through the response, markeach concept with the appropriate identifier, and thencount the number of responses containing eachconcept and calculate the frequency.

“When the cook tastes the soup, that’s formative;when the guests taste the soup, that’ssummative”- Bob Stake (NSF, 1996)

Formative Assessment Example

Formative assessment is the diagnostic use ofassessment to provide feedback to instructors andstudents during the course of instruction (i.e. ,prior to testing). Formative assessment data showstudents’ current understanding of a concept andsupplies data on which to base instructionaldecisions. Any classroom assessment techniqueis a formative assessment and should informongoing instruction.

In-class writing

After completing the guppy simulations incooperative groups, students complete this in-class assignment:

In your groups, discuss the following questions,then write your individual responses oncarbonless paper and place them in your foldersfor collection.

1. What determines the color pattern of anindividual guppy?

2. What role(s) does color play in guppysurvival and reproduction?

3. Explain the results in terms of fitness andnatural selection. Describe how sexualselection and natural selection push inopposite directions.

The instructor can read the responses and noteareas of weakness, assign grades using a scoringrubric, and/or code for detailed data analysis.These data can be used to make instructionaldecisions. If the instructor is not satisfied withstudents’ understanding, further instructionshould occur (Figure 5).

Page 4

Summative Assessment Examples

After all instruction is completed, studentsdemonstrate their understanding of conceptsthrough summative assessment. The key toimproving this practice is to match theassessments to the stated objectives. Thefollowing condensed version of Bloom’s (1959)taxonomy provides guidelines to generatequestions at the appropriate cognitive level for theobjectives.

Cognitive Levels

1. Knowledge - remember

2. Understanding and Application - graspmeaning, use, interpret

3. Critical Analysis - original thinking, open-ended answers, whole to parts, parts towhole, evaluation

4. Synthesis - make connections

Some objectives are properly assessed byknowledge level questions, but higher-levelunderstanding cannot be inferred from studentresponses to knowledge questions. For example,on one exam, 95% of students correctly answeredthat plants obtain carbon from CO2 in theatmosphere (knowledge-level multiple choice).However, when asked to trace the path of carbonthrough an ecosystem in an extended responsequestion (critical analysis), 46% of studentsreverted to their misconception of plants takingup carbon through their roots.

Final Exam Questions

(1) analogous set of multiple-choice questions fromthe CINS, (2) dinosaur problem, and (3) conceptmap of 10 key terms relating to fitness, naturalselection, and evolution.

If the assessment data is to be used only for gradeassignment, then the analysis is familiar. Multiple-choice are graded right/wrong and extendedresponses are scored by rubric. Concept maps can bescored according to the method of Novak (1984). Ifthe data are to be used for course evaluation, or forscience education research, further analysis isrequired.

Examples of Analysis of Pre/Post-test Data

CINS questions

The CINS has analogous question pairs for keyconcepts in evolution. Two such questions arepresented in Table 2. In this example, we used one ofeach pair in the pretest and its analog in the posttest.If we use the same, or analogous questions on thepretest and posttest, we can compare the scores usinga paired t-test. Proportions of students choosingspecific answers on individual question pairs can becompared using the Chi-Square test of independence(Sokal and Rohlf, 1995).

Page 5

Extended Response

After coding both pretest and posttest extendedresponses and counting frequencies, a histogram(Figure 6) is a quick way to look for trends instudents’ understanding. Also examine pretestvs. posttest frequencies for any single concept.In the example, 21.3% of students demonstratedcorrect understanding of the concept of fitnesson the pretest. On the posttest that number hadrisen to 59%. Frequency data of this type can becompared statistically by the nonparametric Chi-Square test (Table 3).

Table 3. Dinosaur Extended Response Analysis

Concept: Correct Explanation of Fitness

Pretest Posttest TotalsCorrect 17 47 64Incorrect / None 63 33 96Totals 80 80 160

Chi-Square Test of Independence:

Research Designs

Page 6

In a research design with paired treatment andcontrol classes, a common practice is to comparenormalized gain on the pre/post test [see Hake,1998 for a discussion of normalized gain]between the two classes using a t-test. ANCOVAcomparison of the posttest scores, using thepretest score as the covariate, can also be used.Results can be compared between sections of thesame class in different years or semesters. Withlarge enough sample sizes and the normalizinguse of the pretest, uncontrollable differencesbetween the student pools in either of those casescan potentially be held to acceptable levels.

Pre/post-test analysis provides data on a singleclass, but standing alone it is a limited researchdesign for testing theory. Due to the lack of acontrol group, pre/post-test designs do notprovide sufficient evidence for making

causal inference, as they cannot reject possiblealternative explanations. Even if the data showimprovement in students’ understanding of aconcept, determining the cause is difficult.

However, when used in the framework of adesign experiment pre/post-testing can be avaluable data collection technique. Designexperiments are classroom-based research, inwhich the instructor/researcher designsinstruction and makes predictions about studentlearning, based on learning theory (Brown, 1992).The test of the prediction is a test of the theory aswell as an informed effort to improve studentlearning. A design experiment is an elegantapproach that minimizes the “control” difficultiesinherent in educational research and providespractical opportunities for FIRST II participantsto contribute to the science education literature.

Anderson DL, Fisher KM, Norman GJ. 2002. Development and Evaluation of the Conceptual Inventory of NaturalSelection. Journal of Research in Science Teaching 39(10): 952-978.

Bloom BS (Ed.). 1956. Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook I,Cognitive Domain. New York: Longmans, Green.

Brown A. 1992. Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventionsin Classroom Settings.

Hake RR. 1998. Interactive-engagement vs Traditional Methods: A Six-thousand-student Survey of Mechanics TestData for Introductory Physics Courses. American Journal of Physics

National Science Foundation. 1996. User-Friendly Handbook for Project Evaluation: Science, Mathematics,Engineering, and Technology. NSF-93152

Novak JD and Gowin DD. 1984. Learning How to Learn. New York: Cambridge University Press.Sokal RR, Rohlf FJ. 1981. Biometry. New York: W. H Freeman and Company.

April 3-5 FS Team Leaders Workshop - MSU

May 22–23 Archbold Workshop

May 28–30 Hancock Workshop

July 8–11 Akron Workshop

July 11–12 Kellogg Workshop

August 27 LUMCON IT videoconference

Sept 4–7 University of Washington Workshop

Sept 18-20 Archbold Workshop

Sept 19-20 Washington DC Workshop

References

FIRST II Team Activities - 2003

Page 7

Guppy Activity - http://www.first2.org/resources/tools/guppy_activity.htm

FIRST II NewsletterOregon Institute of Marine BiologyUniversity of OregonP.O. Box 5389Charleston OR 97420

ADDRESS

Page 8

Classroom Resources

This area of the website has thepotential to become a valuableresource and lasting contribution forthe project. Many of you are doinginnovative and effectiveinquiry–based instruction in yourclasses. We want to document themand make them available to the rest ofthe FIRST II participants and otherinterested faculty members. Theformal for reporting instructionalactivities includes annotation ofinstructional design, suggestions forassessment, and results if available.

FIRST II Websitewww.first2.org

The guppy natural selectionsimulation (see page 7) is posted asan example:[Resources>ClassroomResources>InquiryActivities>Guppy Problem].Please consider submitting your best1 or 2 inquiry-based instructionalactivities, used in class meetings orlaboratory. Use the guppysimulation annotation as a templatefor your submission. Submissionswill be reviewed by the project PIs,formatted and posted to the websitefor dissemination.

FIRST II Website – Submit a Resource