CMU - Teaching Portfolio Guide

8/21/2019 CMU - Teaching Portfolio Guide

1/175

Eberly Center for Teaching Excellence | 5000 Forbes Avenue, Cyert Hall 125 | Pittsburgh, Pennsylvania 15213

www.cmu.edu/teaching | 412.268.2896

www.cmu.edu/teaching

Guidelines for Teaching Portfolios

C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S

What is a Teaching Portfolio? ..............................Page 1

Why do a Teaching Portfolio? ..............................Page 1

What goes in a Teaching Portfolio? ..................Page 1-2Reflective statement

Statement of teaching responsibilities

Supporting documents

Statement of teaching goals for the future

What kinds of supporting documentsshould I include? ............................................... Page 2-5

Your own teaching

Your students

Your colleagues, department, and institution

Your discipline or teaching in general

What kind of support is available to meas I compile my Teaching Portfolio? .....................Page 5

Examples of documents for theTeaching Portfolio ............................................Page 6-8

C O N T E N T S
http://www.cmu.edu/teaching/http://www.cmu.edu/teaching/


2/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 1

What is a Teaching Portfolio?T e teac ing port o io is a escription o

a professors major strengths and teaching achieve-

ments. It describes documents and materials which

collectively suggest the scope and quality of a profes-

sors teaching performance. (Seldin 1997). Because

teaching efforts and accomplishments are difficult to

capture solely by means of numerical summaries, the

concept o a port o io ecomes use u in t is arena. A

teac ing port o io goes eyon a ist to inc u e e a o-

rations an re ections. In t is sense it is simi ar to an

artists portfolio, because it showcases your best work

and conveys a sense of yourself and your vision.

Why do a Teaching Portfolio?The goal for a teaching portfolio is twofold.

At its core, compiling a teaching portfolio is essen-

tially a reflective activity. The very process of writ-

ing down your teaching philosophy and corroborat-

ing it with the appropriate evidence automatically

causes you to reflect on your teaching. It can help

clarify your goals, underscore your development asan e ucator, an ig ig t areas or urt er growt .

However, port o ios are a so use y t e a minis-

tration for the assessment of your teaching perfor-

mance. Therefore, you are writing both for yourself

and for an external audience, for the purposes of

self-reflection and summative evaluation.

What goes in a Teaching Portfolio?Teaching portfolios generally exhibit a simi-

ar structure, ut t e in ivi ua ocuments inc u ecan vary signi icant y. You wi want to p ay to your

strengths as you showcase the breadth of your accom-

plishments, so you should think about what they are

and how to best represent them. The structure of a

teaching portfolio consists of four broad categories:

{ Reflective statement} Also called the teaching or

ducation philosophy, this document frames the

whole portfolio. It communicates who you are as a

teac er: ow you e ine e ucation, ow you concep-

tua ize your ro e an t e earners ro e in an out o

the classroom, and how you translate your beliefs

nto action (typically 1 or 2 pages, but you shold

heck with your Department Head).

{ Statement of teaching responsibilities } This doc-

ument should communicate at a glance the breadth

f the instructors teaching activities. It should be

presented in a tabular format, and possibly support-

d by a narrative in the appendix. The table should

nclude every teaching assignment over the past 3

ears, inc u ing:

O icia teac ing assignments

Ot er teac ing activities e.g., An rew s Leap

Supervisory activities e.g., in epen ent stu -

es, SURG grants)

For each teaching assignment, the information that

s typically included consists of:

type of course (e.g., studio, lab, large lecture)

umber of students enrolled

breakdown of majors/non-majors or under-

raduate/graduate if relevant

numerical summary of FCE ratings

whether the course has been developed or

u stantia y revampe y t e instructor.

The other 2 categories should include information

appropriate to eac activity. T e ta es rom yourannua reports are a goo p ace to start to create t is

ocument. In addition, check with your Department

Head for departmental guidelines (up to 3 pages).

{ Supporting documents } This section will provide

vidence for the claims in the philosophy and is


3/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 2

really the portfolios body. This is the section most

similar to the artists portfolio. For guidelines on

how to organize the documents in this category, see

t e Supporting Documents section e ow.

{ Statement of teaching goals for the future } T e

port o io s ou e orwar - oo ing, wit i eas

or uture action ue e y re ections on t e past

and the present. Just like a researcher needs to

have a research agenda, an educator should be able

to articulate how she wants her teaching to grow.

For instance, you might talk about working on a

specific challenge in the classroom, or the wish to

create a summer program to reach out to talented

high-school students, and so on. Some instructors

pre er to integrate t is statement into t e teac ing

p i osop y. In t is case, t e goa s s ou e c ear y

highlighted within that document (1 page).

What kinds of supporting documentsshould I include?

ust like with an artists portfolio, the aim is

not to be comprehensive. A novice instructor might

not have copious material to showcase, whereas

some o y w o s een teac ing or a ong time mig t

c oose to e more se ective in er c oices. Some

people might opt to showcase the most significant

contributions thorough the years, others might wish

to emphasize their most recent work. Others still,

might choose to emphasize development, starting

from a challenge in their own teaching, steps they

took to overcome it, and finally evidence of having

successfully navigated the challenge. Whatever theirection c osen, t e port o io is a ynamic ocu-

ment an it wi evo ve over t e course o your ca-

reer. U timate y, it s ou emonstrate commitment

to and/or impact on:

. Your own teaching. This section aims to demon

trate that the instructor is engaged in an ongoing

reflection about her teaching.

. Your students. T ere is no teac ing wit ou

earning. The best teachers inspire, motivate, and

reate an environment w ere earning can occur

T is section aims to emonstrate a t is.

. Your colleagues, department and institution

Teaching takes place in a broader context than the

lassroom. Therefore, it should be informed by and

nform the departments curricular goals and by thenstitutions pedagogical mission. This section aims

o demonstrate all this.

4. Your discipline or teaching in general. The bes

eaching is also informed by a broader scholarship

whether disciplinary (e.g., Science Education), cross

isciplinary (e.g., First Year Programs) or specific ped

agogies e.g., Service Learning . T is section aims to

emonstrate your participation in t ose ia ogues.

In a t ese imensions t ere is a trajectory

rom commitment to impact, and instructors wil

be at different points along the four continua de

ending on where they are in their career. For in

tance, beginning instructors (years 1 to 3) would

be expected to show commitment to reflecting on

heir own teaching, to their students, and to learn

ng about educational goals and practices of the in

titution and the discipline. At a later stage (years

4 to 6 , t e process o re ection s ou resu t in

oncrete c anges imp emente in t e course, suc

as the development of new materials, or the integra

ion of new technologies and pedagogies. Instructors

hould have some preliminary measures of the im

act of such changes. Advanced instructors should

be able to document a demonstrated impact on

tudent learning and motivation, as well as broader


4/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 3

participation in pedagogical activities outside their

own classroom, possibly up to the national level

(e.g., taking active roles in the education section

o t eir iscip inary association . Some exceptiona

instructors mig t a so e a e to emonstrate recog-

nition of their impact. This could take the form of

awards, internal or national, being invited to speak

on teac ing topics, an so on. The figure below

synthesizes this model.

Different types of evidence are appropriate

at each stage. For instance, a record of attendance

to teaching development seminars might be enough

to demonstrate commitment to your own teaching,

but in order to demonstrate impact you will need toarticu ate ow t e now e ge rom t ose seminars

trans ates into pe agogica practices in your courses.

The type of evidence presented can vary even with-

in the same stage, depending on what is appropriate

to the type of course and discipline. For instance,

if you are trying to document student learning and

performance, you might have indirect measures

(e.g., the words of an external observer saying that

the students appeared to be learning) and more di-

rect ones (e.g., samples of student work, or pre-posttests in appropriate omains .

So how do you select documents for the

portfolio?As a genera ru e, as you gat er more evi-

dence of impact, this evidence gradually replaces the

evidence of commitment. Likewise, more direct mea-

sures should be preferred to indirect ones when avail-

able and appropriate. Following is a list of suggested

indicators for the four dimensions. Sometimes t e

way to ocument t ose in icators is se -evi ent e.g.,

stu ents scores on tests ; or cases where it is not im-

ediately clear how to document them, possible evidence

ollows in italics.

. Your Own Teaching

Participation in an association con-

erned with the improvement of

eaching and learning (e.g., AAC&U,AERA, SENCER, Education section of

isciplinary association)

Taking/auditing courses relevant to

eaching (e.g., cognitive psychology,

group ynamics, pu ic spea ing,

u tura iversity

See ing ee ac rom co eagues or E er y

enter

tatement from colleagues or Eberly Center records

Observing other classes Evidence of reflection on your own teaching

(Course logs)

Participation in seminars and workshops to

mprove teac ing

Records from Eberly Center or other relevant

rgan zat on

. Your Students

a. Student Learning and Performance

Stu ents scores on tests, i e pre-post tests Stu ents a oratory wor oo s an ogs

Student essays, creative work and projects or

ieldworks reports

Demonstration of change in student per-

ormance (e.g., successive drafts in light of

rading criteria or rubrics)

Ones Own

Teach ng tudents

olleagues |Department |

Inst tut on

D sc pl ne oreach ng n

eneralommitment

Impact


5/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 4

Research on the impact of changes to thecourse

Reports or published papers

Instructional and assessment materials

developed (e.g., handouts, projects, grading

rubrics, pre-post tests)

Student ratings of own learning (From FCEs)

Student comments from FCEs related to

their own learning

Unsolicited letters from students about their

own learning

Interview data collected from students, suchas SGIDs or focus groups

Made available to instructor confidentially by

Eberly Center; her choice whether to submit them

or not

Statements from colleagues who have ob-

served the instructor teach or who have

observed the quality of student work

Statements from teaching consultants who

observed the instructor teaching



or not

Statements on the preparedness of instructors

students by instructors who teach courses

downstream

Letters from alumni about impact

Reports on students preparedness by their

employers

b. Student Inspiration/Motivation

Record of students who elect another course

with same instructor (e.g. letters from students)

Evidence of the effect of courses on student

major/career choices

From student letters

Letters from alumni about impact Student ratings of satisfaction with course/

instructor (from FCEs)

Student comments from FCEs

Unsolicited letters form students

Interview data collected from students, such

as SGIDs or focus groups



or not

Statements from colleagues who have ob-

served the instructor teach Statements from teaching consultants who

have observed the instructors teaching



or not

Early evaluations and evidence of responsive-

ness to student feedback

Evidence of effective supervision of honors,

masters or PhD theses, independent studies,

5th year scholars, SURG grants (from student

letters)

Information on instructor availability to

students (above and beyond office hours, in

person or online) (Self-report, or from students

letters and comments on FCEs)

3. Your Colleagues/Department/Institution

Internal education awards

Letters from Dean praising the instructors

teaching

Adoption of instructors innovation bybroader curriculum (e.g. template syllabus for

all sections of intro course, labs developed by

instructor still used in the course when other

people teach it)


6/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 5

Instructional or assessment materials devel-

ped (e.g., syllabi, educational software)

Developing new courses or revamping old

nes (self-report, syllabi)

Deve oping course port o ios to aci itate t e

task of future instructors teaching the course

Evidence of effectiveness of help given to

olleagues or graduate students on teaching

mprovement, informally or through Eberly

enter or other workshops (from faculty and

graduate student letters, Eberly Center database)

Serve on educational or curriculum develop-ment revision committees

Organizing or ea ing teac ing eve opment

pportunities TA training or orientation,

seminars in instruction, teaching teas) (self-

report, letters from participants)

Informal help given to colleagues or graduate

students on teaching improvement (self-report

of time and task involved, colleague and graduate

student letters)

Evidence of success of internship, co-op, sum-mer aca emy programs set up y instructor

Setting up interns ip program, or co-op, sum-

mer academy (self-report or program documentation)

4. Your Discipline or Teaching in General

Awar s rom externa institutions

Invitations to teach for outside agencies

Instructional or assessment materials (e.g.,

textbooks and instructors manuals)

Publications on teaching (research articles, op-

d columns) (published articles, internal reports

for work not yet accepted for publication, or

nvitations to contribute for work not done yet)

Other kinds of invitations based on ones

reputation as a teac er (e.g., a magazine or radio

interview)

Recor s o a option o own text oo y

t er peop e

Editing or contributing to a professional

ournal on teaching ones subject

Evidence of success of internship, co-op sum-

mer program with external students (number

f applicants, student evaluation of satisfaction,

uture course selection if tracked)

What kind of support is available to me as Icompile my Teaching Portfolio?

ec wit your Department Head or epart-

mental or college guidelines. Some departments have

written documents. Your entorsmight be another

possible source of support. Finally, the Eberly Center

s available for consultations as well.


7/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 6

Examples of documents for the TeachingPortfolio

{ Reflective statements/teaching philosophies }

The following pages include three examples of re-

flective statements, selected from both tenure and

teaching tracks. They are from multiple disciplines

and all look different, but they all convey a sense of

the authors teaching philosophy.

Steven Rudich, Computer Science

This statement is very concise, but it clearly

communicates what the instructor values in

teaching, and how he translates his values into

classroom pedagogy.

Pamela Lewis, Heinz School

This statement blends text and diagrams to pro-

vide a sense of the evolution of the instructors

self-concept through the years. It also com-

municates that she is familiar with some of the

teaching literature. This statement includes the

statement of teaching goals as well.

Peggy Knapp, English

This statement demonstrates development

through a different model, an initial philosophy

(dated 1999) plus an addendum to reflect her

recent accomplishments(dated 2003).

{ Supporting documents }

The following documents highlight some points

across the continuum from commitment to impact.

The list is not comprehensive, but is meant to get

you started thinking about the possibilities. Not all

documents have actually appeared in a packet yet,

but all could be included. Again, the selection draws

from both tenure and teaching track faculty.

Instructional and assessment materials

developed

Handout: How to write a philosophy

paper by Maralee Harrell, Philosophy

This handout reveals an understanding of

the level of writing and argumentation skills

first-year students possess, and a commitment

to proactively helping them develop further.

Grading rubric by Shelley Evenson, Designand HCII

The significance of the rubric is to clarify

expectations and grading criteria for the

students in order to facilitate their study and

their learning.

Grading According to a Rubric, Maralee

Harrell, Philosophy (published in Teaching

Philosophy)

Publishing a paper on your own grading

rubric is evidence of impact on the disciplin-ary discourse.


8/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 7


Research on the impact of changes to the

course

Using Argument Diagrams to Improve

Critical Thinking Skills in Introductory

Philosophy, Maralee Harrell, Philosophy,

technical report

This report documents the positive impact of

a specific innovation the instructor intro-

duced in her class by means of a statistical

data analysis of student performance.

Implementing a Computerized Tutor in a

Statistical Reasoning Course: Getting the

Big Picture, Oded Meyer, statistics, Paper

presented at the American Association for

Educational Research conference.

In this case, the claim of impact is validated

by acceptance for presentation and by the

educational/scientific community.

Adoption of instructors innovation by broader

curriculum

A Successful Peer Writing Assistant

Program, Bonnie Youngs and Anne Green,

Modern Languages, paper published in the

Foreign Language Annals

This article documents the key features of a

program the two faculty members designed,

which has been very successful and is now

used by instructors in all seven languages

taught in the Modern Languages department.

Developing New Courses or Revamping Old

Ones

Tutoring for Community Outreach: A

Course Model for Language Learning and

Bridge Building between Universities and

Public Schools, Susan Polanski, Modern

Languages, paper published in the Foreign

Language Annals

This article describes a new course marry-

ing disciplinary content and service-learning

pedagogy.

Organizing or leading teaching development

opportunities

Have your cake and eat it too, lecture

on how to give lectures by Steven Rudich,

Computer science

This seminar, given to faculty and graduate

students in Computer Science, documentsthe instructors commitment to the depart-

ment. Evaluations of the seminar (not shown

here) serve as evidence how of well received

the lecture was by participants.

TA Handbook 2005-2006, Maralee

Harrell, Philosophy

The handbook documents the instructors

effort to clarify roles and responsibilities for

TAs in the department, as well as to equip

them with some valuable pedagogical knowl-

edge.


9/175




C

R

E

A

T

I

N

G

A

C

O

M

M

U

N

I

T

Y

O

F

E

D

U

C

A

T

O

R

S


Page 8


Setting up internship, co-op, summer academy

etc

Andrews Leap, Steven Rudich, Computer

Science

This is a summer program for talented high-

school students to explore the frontiers of

computer science. The materials presented

document the instructors commitment to

outreach in Computer science education.

Records of advanced studies of participants

(not shown here) give evidence of impact on

students motivation and learning.

Allegheny Intermediate Unit Outreach

Program, Scott Matthews, Engineering and

Public Policy

This is a campus-based outreach program for

high-school students to familiarize them with

environmental principles and generate inter-est in related careers. The materials presented

document some of the impact in terms of

students increased environmental awareness.

Publications on teaching

A Statics Concept Inventory: Development

and Psychometric Analysis, Paul Steif et

al., Mechanical Engineering

This paper, published in the Journal of

Engineering Education, documents a test

developed by the author and used at many

universities to record conceptual progress in

the learning of statics.

Humorous Engineering 101, Larry

Cartwright, Civil and Environmental

Engineering (award-winning paper at the

ASEE conference 2001)

This paper is evidence not only of national

impact but also national recognition.

What We Know about Learning, Herb

Simon, Psychology (1997 Frontiers inEducation conference keynote address)

This paper too is evidence of national impact

and recognition, across fields.

Other kinds of invitations based on ones

reputation as a teacher

Laura Lee: Bridging the Gap between

School and Practice, in Direct Connection,

volume 3, issue 2, December 2000

This interview demonstrates the instructors

influence on architecture education nationally.


10/175


11/175


12/175


13/175


14/175


15/175


16/175


17/175


18/175


19/175


20/175


21/175


22/175


23/175


24/175


25/175


26/175


27/175

1

How to Write a Philosophy PaperBy Mara Harrell

How do I know what I think until I see what I say? EM Forster

Writing is important, especially to a liberal arts education, because in many ways the act ofwriting is the embodiment of the process of critical thinking. Consider what Douglas Soccio

1has

said about the subject:

Critical thinking is the conscious, deliberate, rational assessment of claims according to clearlydefined standards of proof (Soccio 2001, 39).

One of the signs of good critical thinking is the willingness to accept the best evidence, even

when it requires modifying or rejecting a cherished belief of highly desired conclusion (Soccio2001, 41).

Just as we could say that philosophy is the pursuit of wisdom (for philosophers are, literally,lovers of wisdom), we can say that philosophy is a deep and comprehensive inquiry into humanexperience. In order to perform this inquiry, philosophy relies on knowledge from other

disciplines, but also may go further in its investigation by analyzing the assumptions made byother disciplines, like what it means to be human, what it means to have knowledge, etc.

As such, philosophical inquiry is a special brand of critical thinking. As philosophers we are

intensely interested in what we should believe, what we should do, and how we are to know whatwe should believe and what we should do. The process of creating, analyzing, and evaluating

arguments helps us to determine what claims are most likely to be true, and thus helps us achieveour philosophical goals.

The goal of this handout is to help you to become comfortable writing philosophy papers. Many

of the ideas, directives, and admonitions are idiosyncratically my own, and may not be shared byall philosophy teachers. However, I do think that much of it will be useful to you not only in my

class, but in other philosophy courses you may take, and indeed in other college courses you maytake.

I. Vocabulary and Logic

In philosophy, we take precise definitions of words very seriously. Many philosophers havespent a good chunk of their lives arguing about what a particular word or phrase means. In this

spirit, I am going to introduce several technical terms that have particular meaning in logicalphilosophical discourse. These words may have different colloquial uses, so be conscientious

about using them properly in your philosophy papers.

Statement: A statementis a sentence that can either be true or false. Example: It will raintomorrow. We say that this sentence has a truth-value (either true or false), and perhaps we can

1Soccio, Douglas J. (2001)How to Get the Most Out of Philosophy, 4th

Edition. Belmont, CA: Wadsworth.


28/175

2

even know what the truth-value is. Not all sentences in English are statementsquestions,commands, and propositions, for example. Some non-statement sentences can, however, be

transformed into statements with some re-wording. Example: Be a doctor! can be usefullytransformed into You should be a doctor if the context permits.

Conditional Statement: A conditional statementis also a sentence that can be either true orfalse. They are special, though, because they occur so often in arguments. A conditionalstatement has two parts: the antecedentand the consequent. A conditional statement generally

has the form of an if, then statement, in which we have If [the antecedent], then [theconsequent]. Consider the following example:

Ifyou earned an A on the final paper, thenyou earn an A in the class.The entire statement is the conditional, you earned an A on the final paper is the antecedent,

and you earn an A in the class is the consequent.

Argument: An argumentis a set of statements, one of which is the conclusion, and the othersarepremises. The premises provide support for the conclusion. In other words, the conclusion

asserted to be true on the basis of the premises.Example: Premise: Either it will rain tomorrow, or it will be sunny tomorrow.

Premise: It will not rain tomorrow.Conclusion: It will be sunny tomorrow.

An argument can be good or bad based on (1) how well the premises support the conclusion, and(2) whether the premises are actually true.

Validity: A validargument is one in which it is not possible for the conclusion to be false if the

premises are true. This is a very bold statement, not about what is actually the case, but aboutwhat could possibly be the case. It is helpful, when considering validity, to consider the notion of

possible worlds. I can imagine a possible world in which grass is blue, and I can imagine apossible world in which trees are blue. But consider the following argument:

Premise: If grass is blue, then trees are blue.Premise: The grass is blue.

Conclusion: Trees are blue.There is no possible world in which the premises are true, but the conclusion false. Thus, this is a

valid argument.

Conversely, an invalidargument is one in which it is possible for the premises to be true and theconclusion false. Consider the following argument:

Premise: If you earned an A on the final paper, then you earn an A in the class.Premise: You earn an A in the class.

Conclusion: You earned an A on the final paper.It is possible for the premises to be true, but the conclusion false, if there are other ways to get an

A in the class, and you achieved one of them. For example, it may also be a policy in the classthat if your homework average is an A, then you earn an A in the class. In this case you can earn

an A in the class without earning an A on the final paper.On the other hand, the following argument is valid:

Premise: If you earned an A on the final paper, then you earn an A in the class.Premise: You earned an A on the final paper.


29/175

3

Conclusion: You earn an A in the class.Here, if the premises are true, then the conclusion must be true.

*It is important to note that, given our definitions, a statement cannot be valid or invalid, and an

argument can neither be true nor false.

Soundness: A sound argument is a valid argument in which all the premises are actually true inour world. This means that any argument that is either invalid, or valid with at least one false

premise, is unsound. Consider this example again:Premise: If grass is blue, then trees are blue.

Premise: The grass is blue.Conclusion: Trees are blue.

This is a valid but unsound argument because at least one of the premises is not actually true.Consider again the invalid argument from above:


Conclusion: You earned an A on the final paper.This argument is unsound because it is invalid, regardless of whether the premises are actually

true.

Formal Fallacy: An invalid argument may be a bad argument. Consider again the followingargument:


Conclusion: You earned an A on the final paper.This is considered to be a bad argument because it exemplifies the formal fallacy ofAffirming

the Consequent. No consider a similar argument:Premise: If you earned an A on the final paper, then you earn an A in the class.

Premise: You did not earn an A on the final paper.Conclusion: You did not earn an A in the class.

This is also an invalid argument, because it is possible that the premises could be true while theconclusion false. Recall the example above: it may also be a policy in the class that if your

homework average is an A, then you earn an A in the class. In this case you can earn an A in theclass without earning an A on the final paper. This is also considered to be a bad argument

because it exemplifies the formal fallacy isDenying the Antecedent.

Strength: An invalid argument may be a good argument. Consider the following argument:Premise: 90% of Americans are afraid of snakes.

Premise: Jane is an American.Conclusion: Jane is afraid of snakes.

This argument is invalid because it is certainly possible that Jane is part of the 10% of Americanswho are not afraid of snakes. Thus, it is possible for the premises to be true and the conclusion to

be false. However, it is unlikely that Jane is a part of the 10% rather than the 90%, so it isunlikely that the conclusion would be false if the premises are true.


30/175

4

A strongargument, then, is an invalid argument in which is likely that the conclusion istrue, given that the premises are true. Unlike validity, strength can come in degrees. Consider a

similar argument:Premise: 99% of Americans are afraid of snakes.

Premise: Jane is an American.

Conclusion: Jane is afraid of snakes.Here, it is even more likely that the conclusion is true given that the premises are true. And sinceit is more likely, we say that this argument is stronger than the first, although they are both

considered strong.

Conversely, a weakargument is an invalid argument in which it is not likely that the conclusionis true, given the truth of the premises. Consider the following argument:

Premise: 30% of Americans speak French.Premise: Jane is an American.

Conclusion: Jane speaks French.While it is possible that Jane is part of the 30% of Americans who speak French, it is more likely

that she is part of the 70% who do not. Thus, this is a weak argument.

Strong arguments are not always those with premises that assert percentages. Many (if not all)scientific laws are actually the conclusions of strong arguments, the premises of which are

assertions about what we have experienced so far, and the conclusions of which are assertionsabout what we will continue to experience in the future. These are often referred to as arguments

by induction.

In addition, many arguments by analogyare strong arguments. This type of argument is one wecome across quite frequently in philosophy. The premises generally are (1) that two situations

are analogous (or alike in some important respects), and (2) that certain things are true of onesituation. The conclusion is then that those same things will be true of the second situation. The

strength of arguments by analogy depend on how good the analogy is for the purposes of theargument.

Cogency: Just as we can evaluate valid arguments in terms of the actual truth or falsity of their

premises, we can evaluate invalid arguments. A cogentargument is a strong argument in whichall the premises are actually true in our world. This means that any argument that is either weak,

or strong with at least one false premise, is uncogent. Consider the following argument:Premise: All swans observed so far are white.

Conclusion: All swans are white.This is quite a strong argument, but, unfortunately, black swans have now been observed in

Australia. Thus the premise is false, and the argument is uncogent.

Informal Fallacy: Not all valid arguments are good arguments. Those with clearly falsepremises are bad, of course. However, there are some particular arguments that seem to have a

valid form and possibly true premises, but, upon examination of the content of the argument,clearly exemplify errors is reasoning.


31/175

5

Straw Man:This is an argument attacking another persons argument. Often, when debating apoint with another, it is wise to reconstruct the other persons argument in your own words. This

ensures that you understand the argument, and can help you evaluate it. We often do this in classwith the readings for the day; after all you cannot criticize someones argument until you

understand exactly what it is. The straw man fallacy occurs when you misrepresent the other

persons argument, or a statement made by the other person, in a way that makes it much easierto criticize his or her argument.

Argument against the person(Ad hominemfallacy): This is also an argument against theconclusion of another persons argument. The conclusion of the argument is that the other person

is wrong, but the premises of the argument, instead of giving good reasons not to believe theclaim itself, attack the other persons character, intelligence, or motivation. In general statements

about a person have nothing to do with whether that persons argument is valid, sound, strong,etc.

Appeal to the People: This is an argument in which the basis for the conclusion is the desire or

need to be accepted or valued by a larger group. Unless there are specific reasons given to thecontrary (as in scientific inquiry, for example), the fact that some group of people believe a

statement (or supposedly believe a statement), is not an indication that the statement is true.

Appeal to Ignorance: This is an argument in which the basis for believing the truth of a statementis that its falsity has not been proven. Unfortunately, the converse argument seems just as

persuasive: believing a statement is false merely because its truth has not been proven. Again,unless there are specific reasons given to the contrary, lack of proof for (or against) a statement

does not constitute proof that the statement is false (or true).

Equivocation: This fallacy is made when an argument has a valid form, but in fact two or moremeanings of a single word or phrase are used when validity depends on using just one meaning.

Begging the Question: This fallacy is also made when an argument has a valid form, but in fact

the conclusion is merely a restatement of one or all of the premises. That is, begging the questionoccurs when one is assuming (using as a premise) that which is to be proven (the conclusion).

False Dilemma: This is another fallacy made when the argument has a valid form, but one of the

premises is unfairly narrowing the possibilities to be considered. It usually occurs when someoneasserts a dilemma (either A or B, with the implicit assumption that nothing other than A or B

could be the case), but the fallacy can also occur for an assertion of three, four or more choices(as in: either A, B, or C, with the implicit assumption that nothing other than A, B, or C could be

the case).

Appeal to Unreliable Authority: Generally, an appeal to authority can make a very strong (in thetechnical sense) argument for a particular conclusion. However, this is only true when the

authority appealed to is actually a reliable source of information about the subject. This fallacyoccurs when the source cited as a reason to believe a claim is either (1) not actually a reliable

authority, (2) a reliable authority on some subjects, but not the one in question, or (3) is assertinga claim that is quite controversial even among reliable authorities in the field.


32/175

6

False Cause: This fallacy occurs when the claim is made that one possible cause of a

phenomenon is a (or the) cause of another phenomenon without evidence excluding otherpossible causes. This fallacy comes in two common forms. The first ispost hoc ergo propter hoc

(after this, therefore because of this). This fallacy occurs when A is asserted to be the cause of

B solely because A occurred before B. The second form is the slippery slope. This fallacy occurswhen A is asserted to inevitably cause B, through some sort of chain reaction, despite lack ofevidence that each link in the chain will inevitably lead to the next link.

II. Structure

The structure of you paper is very important. I should be able to follow your line of reasoning

from the first sentence to the last without getting lost, confused, or sidetracked. Wonderful ideasarent worth much if it is not clear how they all fit together.

Argument. In its most basic form, your paper should be an extended argument, the conclusion ofwhich is a statement you are asserting to be true. An extended argument is one that incorporates

one or more sub-argument into the main argument. For example, your main argument may havefour premises which lead to your conclusion. If, however, your premises are slightly

controversial, not obviously true, or not common knowledge, you should provide sub-argumentsto support your premises. Moreover, the premises which support these sub-conclusions may need

arguments of their own. Thus, an extended argument is a series of embedded arguments,ultimately leading to your main conclusion.

Thesis. The thesis is a statement that is the conclusion of the main argument of your paper.

Every paper should have a thesis. Your thesis should be clearly stated at the beginning of yourpaper in the introduction.

Introduction: The introduction to your paper has at least two purposes. First, you need to tell

your reader what your conclusion will be. This is your thesis statement. Second, you need to tellyour reader roughly how you will get to this conclusion. This is not a detailed outline of your

paper, but rather a concise summary of the steps in argument.

Body of the paper: The body of the paper comprises the argument for your thesis. Eachparagraph should be a step in the process of supporting your final claim. This process has several

parts:

(1)

A demonstration of how the truth of a set of premises leads makes the truth of theconclusion (or sub-conclusion) either necessary or probable. This step may be skipped ifthe argument or sub-argument follows a known valid form.

(2)Assertions of factual claims, obtained from a variety of sources, that are used as premisesis an argument or sub-argument.

(3)Considerations of counter-arguments. This follows the recognition that there may bepeople who would disagree with you about either (1) or (2). You should consider the

objections they might raise, and respond.


33/175

7

Conclusion. Contrary to what you may have learned in High School, your conclusion should not

be a restatement of your introduction. Rather, the conclusion is a handy place to tie up looseends. Here is where, for example, you can consider objections to your argument to which you do

not have the space or expertise to respond. This is also a good place to briefly consider the

implications of the acceptance of your conclusion. Or this may be a good place to explain whatfurther work may need to be done in this area.

III. Content

The content of your paper will mostly be dependent on the structure, for the content is the filling-

in of the outline dictated by the structure. However, the particular kinds of arguments andevidence you use will depend on the topic you are considering, and the particular claims you are

making.

Arguments. While all philosophy papers should advance an argument, there are many differenttypes of arguments that one can make. Thus, writing in philosophy can take many forms:

Advancing your own original argument

Exploring the consequences of a particular hypothesis or position Reconstructing someone elses position, theory, or argument

Evaluating someone elses argument Some combination of the above

Advancing your own original argumentis the basic task of providing a set of premises and

demonstrating how these premises lead to your conclusion. The premises can be asserted on thebasis of a number of different kinds of sources that constitute evidence that they are true.

Exploring the consequences of a particular hypothesis or positionis the task of assuming that the

hypothesis, or the set of statements comprising the position, is true, and showing whatconclusions can be drawn from them, either alone, or in addition to some other set of premises

you assert.

Reconstructing someone elses position, theory, or argument is the task of putting the otherpersons position, theory, or argument into your own words with as little excess verbiage as

possible. In so doing, you should always abide by the principles offairness and charity. Theprinciple of fairness says that you should always paraphrase someone else in a way that is as

close to his or her intentions as possible. This prevents the possibility of committing the Straw

Man fallacy. The principle of charity says that, when you may be confused about the authorsintentions, you should always interpret him or her in the best possible light. This means, for anargument for example, interpreting the premises and conclusion in such a way as to make the

argument valid instead of invalid, or strong instead of weak. This kind of reconstruction is itselfa kind of argument because you will need textual support to provide the evidence that your

reconstruction is as fair as possible.


34/175

8

Evaluating someone elses argumentconsists, generally, in showing that his or her argument isgood or bad. The goodness of an argument may depend on many things, but as a rule, sound

arguments and cogent arguments are considered good, while valid but unsound, strong butuncogent, and weak arguments are all considered bad. In any case, you need to provide your own

argument to show that the other persons argument fits some established criteria of what

constitutes a good or bad argument. In general, if you disagree with the conclusion of someoneelses argument, you have two options for evaluating it negatively: you can either show that theargument form is bad (i.e. show that it is a weak argument), or you can show that the premises

are false (or you an do both).

Evidence. One thing that differentiates philosophy papers from other kinds of papers (Englishpapers, scientific papers) is the wide variety of the kinds of evidence that are routinely used and

accepted to support various different kinds of claims: Derivations from first philosophical principles (e.g., I think, therefore I am)

Scientific or factual claims Logical principles

Appropriate personal experience

Linguistic analysis (definitions, colloquial uses of terms and phrases)

Expert testimony Textual evidence

The kinds of evidence you use to support your thesis may vary according to the assignment and

the topic, but some basic guidelines should be followed.

Relevance. The evidence you use should be directly and obviously related to the claim it issupposed to support. If it is either indirectly or not obviously related, you need to provide

additional arguments why this evidence does indeed support your conclusion. We have alreadyseen some fallacies you may commit if you use irrelevant evidence, likeAd hominem, appeal to

the people, or appeal to ignorance.

Reasonableness. The evidence you use to support your claim should be such that any reasonableperson would accept it, based not on the information you have, but on the information you have

given in your paper. This means that, in general, if your evidence is controversial or notobviously true given the information you have offered, you will have to provide additional

reasons to accept it.

Sufficiency. The evidence you use to support your claim must be sufficient to support it. That is,you must have enough relevant, reasonable evidence to persuade your reader. This means that,

for example, you may not make sweeping generalizations about what all philosophers have beenconcerned about throughout the course of history if you have only read a handful of philosophy

texts.

Definitions. As I said above, philosophers are generally very concerned with definitions, andespecially concerned with being clear and consistent. This means that if you are going to use or

consider important philosophical terms or concepts, you need to define them as clearly as


35/175

9

possible. However, a conventional dictionary is usually not up to the task of providing adequatedefinitions of technical philosophical terms. There are many reasons for this:

(1) A conventional dictionary is usually concerned with giving colloquial uses of words, nottechnical definitions that may vary by field or author (consider the difference between the

colloquial use of the word force, and the technical definition this word has in physics).

(2)

Conventional dictionaries do not generally contain philosophical phrases as singleentries, and the meaning of a phrase often cannot be captured by a conjunction of themeanings of the individual words.

(3) Philosophers writing in different settings may use words and phrases in slightly differentways. In addition, philosophers often spend good portions of articles, or entire articles, on

determining the precise definition of a term or phrase, exactly because there may be nogenerally accepted definition of it. In this case, a dictionary will be of no use at all.

Citations: Different sources may be used for different purposes, but all should be documented

clearly and correctly. Many of the kinds of evidence you will use to support a claim involvesources that you will need to document.

Scientific or factual claims unless you are stating a fact that is obviously commonknowledge and generally uncontroversial (e.g. the earth is round), you will need to cite

the sources of your scientific and/or factual claims (like how many people visit thePittsburgh Zoo every year, or what a Pandas natural habitat is).

Expert testimony if you are going to state that an expert in the field has made someclaim, you need to cite the source in which the claim was advanced, be it a scholarly

article, a newspaper story, or personal communication. Textual evidence in making a case for a particular interpretation of an authors words or

reconstruction of an authors argument, you will need to cite textual evidence to showthat you are following the principles of fairness and charity.


36/175

10

IV. General Miscellaneous Advice

The thesis should something sufficiently interesting, about which reasonable people could

differ, and which can be defended. In this paper, I will show that two plus two is four isobviously true and not worth writing about. In this paper, I will prove that Colorado

Springs has a tropical climate is impossible to prove. Dont set your sights either too highor too low.

Not everything you assume to be true is actually self-evidently true. A lot of what happens inphilosophy is learning about ones own assumptions and how to defend those, or to get rid of

the ones that are indefensible. You might think its impossible for anyone to disagree withthe statement that There are no true liberals left in American politics; even Clinton and Gore

are basically Republicans, but not everyone will be like-minded. If, halfway through, you cant figure out what the sentence youre writing has to do with the

thesis, stop and try to figure out how it links. If it doesnt at all, it probably doesnt need tobe there. If it does but you need to explain why, add the explanation.

Transitions are important; they lead the reader through your argument. Transitional words,

though, should only be used for actual transitions. Obvious problems are using words liketherefore and thus followed by a statement which has nothing to do with what camebefore it.

Texts have to be interpreted, which means you need to quote passages to support yourinterpretation. The following are not appropriate uses of text as they stand. (Nor are the

appropriate citations.)o The Bible says be nice to people.

o Kierkegaard says that Abraham is important (Kierkegaard).o Nietzsche says Kierkegaard is tiresome (Nietzsche, 20-150).

o Marx loves capitalism. As he says, it involves amazing wealth (Marx, 241).o Platos dialogue is about love (class notes).

o

Plato believes in a theory of forms (Plato Introduction, 2).o Descartes believes in dualism (www.randomhack.com/descartes/bs.html).

Evidence for a claim can be of varying types. A lot of the work in evaluating arguments rests

in deciding how to evaluate evidence.

o A reference to an authority is not necessarily the best evidence. One must consider heauthoritys credentials, and whether there is reasonable disagreement among experts.

Arguments from authority are not valid, but they can be good arguments nonetheless.o Ad hominems are logical fallacies: Marx said it, and hes stupid, therefore its false

is not a good argument. There are many other logical fallacies you should avoid as

well; you should become familiar with them.o I think is not evidence in itself. I think it is going to rain tomorrow is not a better

reason to bring out an umbrella than it is going to rain tomorrow. Generally, since itis yourpaper, I will assume that statements you assert are things you think are true.

Avoid useless generalities: In todays modern complex society, throughout time peoplehave struggled with political questions, etc. They waste space and make you sound like

someone who doesnt have enough to say. Youre also probably not qualified to say thingslike philosophers have always struggled with the question of How many philosophers

have you read?


37/175


38/175

11

Avoid making absolutist, overly general claims. Any argument whose conclusion istherefore Plato is totally wrong is probably a bad argument. Remember that most issues

worth talking about are not simple. Most issues worth talking about also arent cut and dried: the other side probably does

have something worth taking into consideration. The best reasoning is that which

understands and anticipates opposing arguments, and takes them into account. With respect to vocabulary, there are a number of points:

o Keep your writing simple, but not simplistic. Only use big words and jargon if you

must; otherwise use simple, easy-to-understand language.o Dont use a variety of different words to refer to the same idea or problem just because

you dont want to sound repetitive. You should worry more about precision and lessabout beauty.

o Philosophers often give words precise, technical meanings that are not used in thatprecise or technical way in ordinary conversation. You should know if a word is being

used in this way in anything you read; and you should only use these words in yourwriting if you fully understand them and can use them correctly.

o

In addition, you may want to give a term a precise meaning that is different from itscolloquial meaning in your paper. It is fine to do this as long as you carefully explain

the definition youre using, and use the term consistently with this definitionthroughout your paper.

o Some philosophers also have a habit of giving the same technical word a slightlydifferent meaning than others do. Thus, while there is no need to explain terms like

valid or necessary in your paper, you do need to explain what you mean byphysicalism or sense data.

Always follow the guidelines for quotations and citations. A coherent argument is difficult to sustain over many pages. I recommend having someone

else read your paper for clarity, cohesiveness, etc., before you hand it in. Proofreading your paper is not the same as running spell-check. Spell-check wont catch

common mistakes like:o Using their for there or vice versa

o Using its for its or vice versao Writing based off when the appropriate clause is based on

o Writing centered around when the appropriate clause is centered ono Mismatching the subject of the sentence with the possessives: One should do their

own work should be One should do his or her own work.o Misspelling an authors name

Do not use rhetorical questions or slang in your writing. They diminish the clarity of yourpaper by making the reader guess at your meaning.


39/175


40/175


41/175


42/175


43/175


44/175


45/175


46/175


47/175


48/175


49/175


50/175


51/175


52/175


53/175

Using Argument Diagrams 1

Using Argument Diagrams to Improve

Critical Thinking Skillsin Introductory Philosophy

Maralee Harrell

Carnegie Mellon University

Department of Philosophy

135 Baker HallPittsburgh, PA, 15213

[email protected]

(412) 268-8152


54/175


Abstract

In an experiment involving 139 students in an introductory philosophy course we tested whetherstudents were improving their ability to think critically about arguments and whether using

argument diagramming as an analysis aid contributed to this improvement. We determined that

the students did develop this skill over the course of the semester. We also determined that the

students in one section of the course gained significantly more than the students in the othersections, and that this was due almost entirely to their ability to use argument diagrams. We

conclude that learning how to construct argument diagrams significantly improves a students

ability to analyze, comprehend, and evaluate arguments.


55/175


Using Argument Diagrams to Improve Critical Thinking Skills

in Introductory Philosophy

In the introductory philosophy class at Carnegie Mellon University (80-100 What Philosophy Is)

one important learning goal is the development of general critical thinking skills. Even though

there are a few generally accepted measures of these skills (e.g. the California Critical ThinkingSkills Test and the Watson Glaser Critical Thinking Appraisal, but see also Halpern, 1989 and

Paul, Binker, Jensen, & Kreklau, 1990), there is surprisingly little research on the sophistication

of, or on effective methods for improving, the critical thinking skills of college students. Theresearch that has been done shows that the population in general has very poor skills (Perkins,

Allen, & Hafner, 1983; Kuhn, 1991; Means & Voss, 1996), and that very few courses actually

improvement these skills (Annis & Annis, 1979; Pascarella, 1989; Stenning, Cox, & Oberlander,1995).

Critical thinking involves the ability to analyze, understand, and evaluate an argument.

Our first hypothesis is that students improved on these tasks after taking the introductoryphilosophy course. However, we wanted to determine not only whether they improved, but how

much improvement could be attributed to alternative teaching methods.One candidate method is the use of argument diagrams as an aid to overall argument

comprehension, since we believe that they significantly facilitate understanding, analysis, andevaluation. An argument is a series of statements in which one is the conclusion, and the others

are premises supporting this conclusion; and an argument diagram is a visual representation of

these statements and the inferential connections between them.For example, at the end ofMeno, Plato (1976) argues through the character of Socrates

that virtue is a gift from the gods (89d-100b). While the English translations of Platos works are

among the more readable philosophical texts, it is still the case not only that the text containsmany more sentences than just the propositions that are part of the argument, but also that,

proceeding necessarily linearly, the prose obscures the inferential structure of the argument. Thusanyone who wishes to understand and evaluate the argument may reasonably be confused. If, on

the other hand, we are able to extract just the statements Plato uses to support his conclusion, and

visually represent the connections between these statements (as shown in Figure 1), the structureof the argument is immediately clear, as are the places where we may critique or applaud it.

Recent interest in argument visualization (particularly computer-supported argument

visualization) has shown that the use of software programs specifically designed to help students

construct argument diagrams can significantly improve students critical thinking abilities overthe course of a semester-long college-level course (Kirschner, et al. 2003; Twardy, 2004; van

Gelder, 2001, 2003). But, of course, one need not have computer software to construct an

argument diagram; one needs only a pencil and paper. However, to our knowledge there hasbeen no research done to determine whether it is the mere ability to construct argument

diagrams, or the aid of a computer platform and tutor (or possibly both) that is the crucial factor.

Our second hypothesis is that the crucial factor in the improvement of critical thinkingskills is the ability to construct argument diagrams. This hypothesis posits that students who

construct correct diagrams during argument analysis tasks should perform better on these tasks

than students who do not.


56/175


FIGURE 1 An argument diagram representing one of the arguments in Platos Meno.

We typically teach several sections of Carnegie Mellon Universitys introduction to

philosophy course (80-100 What Philosophy Is) each semester, with a different instructor foreach section. While the general curriculum of the course is set, each instructor is given a great

deal of freedom in executing this curriculum. For example, each section is a topics based coursein which epistemology, metaphysics, and ethics are introduced with both historical andcontemporary primary-source readings. Each instructor, however, chooses a text, the order of the

topics, and the assignments for his or her section. The students who take this course are a mix of

classes and majors from all over the University.

In the Spring of 2004, students in Section 1 were explicitly taught how to constructargument diagrams to represent a selection of text. In contrast, students in Sections 2, 3, and 4

were not explicitly taught the use of argument diagrams, but ratherif they were taught to

analyze arguments at allwere taught to use more traditional kinds of representations (e.g. listsof statements).

In this study, we test the first hypothesis by comparing the pretest and posttest scores of

all the students in 80-100 in the Spring semester of 2004. We test the second hypothesis in threeways: (1) by comparing the pretest and posttest scores of students in Section 1 to students in

Sections 2, 3, and 4, (2) by comparing the pretest and posttest scores of students who constructed

correct argument diagrams on the posttest to those students who did not, and (3) by comparing

total scores on individual questions on the posttest of students who constructed the correctargument diagrams for that question to those students who did not.

Virtue is not knowledge

Virtue is a true belief

Virtue is either knowledge or true belief

True belief is a gift from the gods

Virtue cannot betaught

Something canbe taught if andonly if it isknowledge

Virtue guidescorrect actions

Virtue is a gift from the gods

Only knowledgeand true beliefguide correctactions

There are noteachers ofvirtue

Something can betaught if and only if ithas teachers


57/175


Method

Participants

139 students (46 women, 93 men) in each of 4 sections of introductory philosophy (80-100 What

Philosophy Is) at Carnegie Mellon University in the Spring of 2004 were studied. Each section of

the course had a different instructor and teaching assistant, and the students chose their section.

There were 35 students (13 women, 22 men) in Section 1, 37 students (18 women, 19 men) inSection 2, 32 students (10 women, 22 men) in Section 3, and 35 students (5 women, 30 men) in

Section 4. The students in Section 1 were taught the use of argument diagrams to analyze the

arguments in the course reading, while the students in the other three sections were taught moretraditional methods of analyzing arguments.

Materials

Prior to the semester, the four instructors of 80-100 in the Spring of 2004 met to determine the

learning goals of this course, and designed an exam to test the students on relevant skills. The

identified skills were to be able to, when reading an argument, (i) identify the conclusion and thepremises; (ii) determine how the premises are supposed to support the conclusion; and (iii)

evaluate the argument based on the truth of the premises and how well they support theconclusion.

We used this exam as the pretest (given in Appendix A) and created a companionposttest (given in Appendix B). For each question on the pretest, there was a structurally

(nearly) identical question with different content on the posttest. The tests each consisted of 6

questions, each of which asked the student to analyze a short argument. In questions 1 and 2, thestudent was only asked to state the conclusion (thesis) of the argument. Questions 3-6 each had

five parts: (a) state the conclusion (thesis) of the argument; (b) state the premises (reasons) of the

argument; (c) indicate (via multiple choice) how the premises are related; (d) provide a visual,graphical, schematic, or outlined representation of the argument; and (e) decide whether the

argument is good or bad, and explain this decision.

Procedure

Each of the four sections of 80-100 was a Monday/Wednesday/Friday class. The pretest wasgiven to all students during the second day of class. The students in sections 2 and 3 were given

the posttest on the last day of classes, while the students in sections 1 and 4 were given the

posttest as one part of their final exam, during exam week.

Results and Discussion

Test Coding

Pre- and posttests were paired by studentsingle-test students were excluded from the sample

so that there were 139 pairs of tests in the study. Tests which did not have pairs were used for

coder-calibration, prior to the coding of the 139 pairs of tests.Two graduate students independently coded all 278 tests (139 pairs). Each pre-/posttest

pair was assigned a unique ID, and the original tests were photocopied (twice, one for each

coder) with the identifying information replaced by the ID. We had an initial grader-calibrationsession in which the author and the two coders coded several of the unpaired tests, discussed our

codes, and came to a consensus about each code. After this, each coder was given the two keys

(one for the pretest and one for the posttest) and the tests to be coded in a unique random order.


58/175


The codes assigned to each question (or part of a question, except for part (d)) were

binary: a code of 1 for a correct answer, and a code of 0 for an incorrect answer. Part (e) of eachquestion was assigned a code of correct if the student gave as reasons claims about the truth of

the premises and/or the support of premises for the conclusion. For part (d) of each question,

answers were coded according to the type of representation used: Correct argument diagram,

Incorrect or incomplete argument diagram, List, Translated into logical symbols like a proof,Venn diagram, Concept map, Schematic (e.g., P1 + P2/Conclusion (C)), Other or blank.

To determine inter-coder reliability, the Percentage Agreement (PA), Cohens Kappa ()

and Krippendorffs Alpha () were calculated for each test (given in Table 1).

TABLE 1Inter-coder Reliability: Percentage Agreement (PA),

Cohens Kappa (), and Krippendorffs Alpha () for each test

PA

Pretest .85 .68 .68Posttest .85 .55 .54

The inter-coder reliability was fairly good, however, upon closer examination it wasdetermined that one coder had systematically higher standards than the other coder on the

questions in which the assignment was open to some interpretation (questions 1 & 2, and parts

(a), (b), and (e) of questions 3-6). Specifically, on the pretest, out of 385 question-parts on which

the coders differed, 292 (75%) were cases in which Coder 1 coded the answer as correct whileCoder 2 coded the answer as incorrect; and on the posttest, out of 371 question-parts on which

the coders differed, 333 (90%) were cases in which Coder 1 coded the answer as correct while

Coder 2 coded the answer as incorrect. In light of this, the codes from each coder on thesequestions were averaged, allowing for a more nuanced scoring of each question than either coder

alone could give.

Since we were interested in how the use of argument diagramming aided the student in

answering each part of each question correctly, the code a student received for part (d) ofquestions 3-6 were preliminarily set aside, while the addition of the codes received on questions

1 and 2, as well as parts (a), (b), (c), and (e) of questions 3-6 determined the raw score a student

received on the test.

TABLE 2The variables and their descriptions recorded for each student

Variable Name Variable Description

Pre Fractional score on the pretestPost Fractional score on the posttest

A* Averaged score (or code) on the pretest for question *B* Averaged score (or code) on the posttest for question *Section Enrolled sectionSex Students sexHonors Enrollment in Honors courseGrade Final Grade in the courseYear Year in school

The primary variables of interest were the fractional pretest and posttest scores (the raw

score converted into a percentage), and the individual average scores for each question on the

pretest and the posttest. In addition, the following data was recorded for each student: which


59/175


section the student was enrolled in, the students final grade in the course, the students year in

school, the students sex, and whether the student had taken the concurrent honors courseassociated with the introductory course. Table 2 gives summary descriptions of these variables.

Average Gain from Pretest to Posttest for All Students

The first hypothesis was that the students critical thinking skills improved over the course of thesemester. This hypothesis was tested by determining whether the average gain of the students

from pretest to posttest was significantly positive. The straight gain, however, may not be fully

informative if many students had fractional scores of close to 1 on the pretest. Thus, thehypothesis was also tested by determining the standardized gain: each students gain as a fraction

of what that student could have possibly gained. The mean scores on the pretest and the posttest,

as well as the mean gain and standardized gain for the whole population of students is given inTable 3.

TABLE 3Mean fractional score (standard deviation) for the pretest and the posttest,

mean gain (standard deviation), and mean standardized gain (standard deviation)

Pre Post Gain GainSt.WholePopulation

0.59 (0.14) 0.78 (0.12) 0.19 (0.01) 0.43 (0.03)

The difference in the means of the pretest and posttest scores was significant (paired t-test;p< .001). In addition, the mean gain was significantly different from zero (1-sample t-test;

p< .001) and the mean standardized gain was significantly different from zero (1-sample t-test;p

< .001). From these results we can see that our first hypothesis is confirmed: overall the studentsdid have significant gains and standardized gains from pretest to posttest.

Comparison of Gains of Students by Section and by Argument Diagram Use

Our second hypothesis was that the students who were able to construct correct argumentdiagrams would gain the most from pretest to posttest. Since the use of argument diagrams was

only explicitly taught in Section 1, we first tested this hypothesis by determining whether the

average gain of the students in Section 1 was significantly different from the average gain of thestudents in each of the other sections. Again, though, the straight gain may not be fully

informative if the mean on the pretest was not the same for each section, and if many students

had fractional scores close to 1 on the pretest. Thus, we also tested this hypothesis using thestandardized gain. The mean scores on the pretest and the posttest, as well as the mean gain and

standardized gain for the sub-populations of students in each section is given in Table 4.



Pre Post Gain GainSt.

Section 1 0.64 (0.14) 0.85 (0.10) 0.21 (0.02) 0.51 (0.07)

Section 2 0.53 (0.16) 0.70 (0.14) 0.17 (0.03) 0.32 (0.05)

Section 3 0.58 (0.14) 0.79 (0.08) 0.21 (0.02) 0.48 (0.04)

Section 4 0.63 (0.10) 0.80 (0.09) 0.17 (0.02) 0.42 (0.05)


60/175


Since there was such variability in the scores on the pretest among the different sections,

we ran an ANCOVA on the each of the variables Post, Gain, and GainSt, with the variable Preused as the covariate. This analysis indicates that the differences in the pretest scores was

significant for predicting the posttest scores (df = 1, F= 24.36,p< .001), the gain (df = 1, F=

125.50,p< .001), and the standardized gain (df = 1, F= 29.14,p< .001). In addition, this

analysis indicates that, even accounting for differences in pretest score, the differences in theposttest scores among the sections were significant (df = 3, F= 8.71,p< .001), as were the

differences in the gains (df= 3, F= 8.71,p< .001) and the standardized gains (df= 3, F= 6.84,p

< .001).This analysis shows that a students section is a significant predictor of posttest score,

gain, and standardized gain, but it does not tell us how they are different. The hypothesis is that

the posttest score, gain and standardized gain for students in Section 1 is significantly higher thanall the other sections. Thus, we did a planned comparison of the variables Post, Gain, and GainSt

for Section 1 with the other sections combined, again using the variable Pre as a covariate. This

analysis again indicates that the differences in the pretest scores was significant for predicting theposttest scores (df = 1, F= 32.28,p< .001), the gain (df = 1, F= 107.37,p< .001), and the

standardized gain (df = 1, F= 21.42,p< .001). In addition, this analysis indicates that, evenaccounting for differences in pretest score, the differences in the posttest scores between Section

1 and the other sections were significant (df = 1, F=11.89,p= .001), as were the differences inthe gains (df= 1, F= 11.89,p= .001) and the standardized gains (df= 1, F= 8.07,p= .005),

with the average posttest score, gain, and standardized gain being higher in Section 1 than in the

other three sections.Although these differences between sections (at least with standardized gain scores)

obtained, they do not provide a direct test of whether students who (regardless of section)

constructed correct argument diagrams have better skills. The explanation is that, although thestudents in Section 1 were the only students to be explicitly taught how to construct argument

diagrams, a substantial number of students from other sections constructed correct argumentdiagrams on their posttests. In addition, a substantial number of the students in Section 1

constructed incorrect argument diagrams on their posttests. Thus, to test whether it was actually

the construction of these diagrams that contributed to the higher scores of the students in Section1, or whether is was the other teaching methods of the instructor for Section 1, we introduced a

new variable into our model.

Recall that the type of answer given on part (d) of questions 3-6 was the data recorded

from the test. From this data, a new variable was defined that indicates how many correctargument diagrams a student had constructed on the posttest. This variable is PostAD (value = 0,

1, 2, 3, 4).

The second hypothesis implies that the number of correct argument diagrams a studentconstructed on the posttest was correlated to the students pretest score, posttest score, gain and

standardized gain. Since there were very few students who constructed exactly 2 correct

argument diagrams on the posttest, and still fewer who constructed exactly 4, we grouped thestudents by whether they had constructed No correct argument diagrams (PostAD = 0), Few

correct argument diagrams (PostAD = 1 or 2), or Many correct argument diagrams (PostAD = 3

or 4) on the posttest. The results are given in Table 5.


61/175




Pre Post Gain GainSt.

No Correct 0.56 (0.16) 0.74 (0.12) 0.18 (0.02) 0.39 (0.03)

Few Correct 0.57 (0.13) 0.75 (0.12) 0.17 (0.02) 0.37 (0.04)

Many Correct 0.66 (0.13) 0.88 (0.06) 0.22 (0.02) 0.56 (0.06)

Again, since there was such variability in the scores on the pretest among the different

sections, we ran an ANCOVA on the each of the variables Post, Gain, and GainSt, with the

variable Pre used as the covariate. This analysis indicates that the differences in the pretest scoreswas significant for predicting the posttest scores (df = 1, F= 24.68,p< .001), the gain (df = 1, F

= 132.81,p< .001), and the standardized gain (df = 1, F= 30.97,p< .001). This analysis also

indicates that, even accounting for differences in pretest score, the differences among the

students who constructed 0, Few or Many correct argument diagrams on the posttest aresignificant (df= 2, F= 14.66,p< .001), as are the differences in gains (df= 2, F= 14.66,p