1

“Asking Interesting Questions” Prepared for Sage Handbook of Research Methods in Political Science and International Relations. Robert J. Franzese, Jr. and Luigi Curini, Editors. Forthcoming. ByWilliamRobertsClark1TexasA&MUniversity

Goodresearchisdrivenbyimpatiencewithbadanswerstointerestingquestions.But

wheredointerestingquestionscomefrom?Sincethisistheopeningchapterofahandbookon

researchmethods,itisimperativetopointoutatthestartthatthereisno“method”toasking

researchquestionsinthesenseofacookbookthatyoucanfollowthatwilllead,inexorably,to

scientificdiscovery.Theremaybeascientificmethodforevaluatinganswers,butthereis

certainlynoscientificmethodforaskingquestionsorgeneratinganswers.Andthereis

certainlyroomforalotofcreativityindevelopinginterestingandenlighteningresearch

designs,andseriousshortcomingsto“cookbook”approaches.2KarlPopper,(1962,2003)for

example,arguedthatsciencebeginsafterascientisthasconjecturedananswertoaquestion.

Thescientificmethod,therefore,ismore(perhapsonly)usefulinevaluatinganswersto

questions.Generatingquestionsandanswers,incontrast,isasmuchanartasitisascience.

Butthatisnottosaythattheprocessisrandomorlacksstructure.ThomasKuhn(1962)

saysepisodesofscientificdiscoverybeginwithanindividualwiththe“skill,wit,orgeniusto

recognizethatsomethinghasgonewronginwaysthatmayproveconsequential.”(p.763)But,

hehastenstoadd,“anomaliesdonotemergefromthenormalcourseofscientificresearch1TheauthorwishestothankBranislavSlanchevandLaurieClarkforthoughtfulcommentsandusefulsuggestionsonanearlierdraftofthispaper.2Thefadaround“cleveridentificationstrategies”isbutthemostrecentinstantiationofthisphenomenon.

2

untilbothinstrumentsandconceptshavedevelopedsufficientlytomaketheiremergencelikely

andtomaketheanomalywhichresultsrecognizableasaviolationofexpectation.”(p.763)

Intheparlanceofsocialmedia,scientificdiscoverybeginswitha“WTF”moment.

Scientificdiscoverybeginswhenascholarobservessomethingcontrarytoexpectationsand

recognizesthatthisanomalousobservation“mayproveconsequential.”Notethatthe

motivatingfactmaybeanobservationabouttheworld,butitmayalsobeaboutwhatothers

havesaidabouttheworld.3

Butnotjustanysurprisewilldo.Anyonewhohaseverparentedayoungchildisfamiliar

withthequestions,bornoutofwonder,suchasthosethatourchildrenaskedmypartnerand

me:“whyistheskyblue?,”“wheredoesthesungo(attheendoftheday),”?or,“ifmybrain

controlsmybody,whydoIhavetogotothedoctortofindoutwhat’swrongwithmewhenI

amsick”?Answerstoallofthesequestions(assumingtheyareconsistentwithwhatscientists

currentlybelieve)arediscoveriesfortheinquirerbecausetheychangewhattheyknow,but

theydonotleadtoscientificdiscoveriesunlesstheychangewhatweknow.Thefieldsof

optics,astronomy,andneurosciencehavetheirrespectiveanswerstothequestionsabove

(althoughthelastquestionisprobablylesssettledthantheothertwo).

So,questionsoftenbeginwithsurprise,butgoodresearchquestionsbeginwithwell-

informedsurprise.Ifyoualonearesurprisedbyanobservation,theanswertoyour“WTF

3InthewordsofBranislavSlantchev,(personalcommunication)“theoreticinnovationdoesnothavetobeginwithanempiricalobservationbutwithapotentialflawinthelogic,inconsistencyoftheassumptions,oraninsightaboutageneralclaim(e.g.,theimpossibilityresults)”.

3

moment”islikelytobepersonallyrewarding.Ifmostwell-informedobserversaresurprisedby

anobservation,thenananswerislikelytobesocially,and,therefore,scientificallyvaluable.4

Butsometimes,scienceproceedswhenanindividualrecognizesthattheanswers

embodiedinwhat“weknow”aboutasubjectarenotverygood.Forexample,formillennia

“we”knewthattheanswertothequestion“wheredoesthesungo”tobesomethinglike“the

suncirclesastationaryearth,soatacertainpointeachdayitleavesoursightwhileshinningon

theotherhalfoftheplanetonlytoreturnthenextmorning.”Eventually,however,scientists

with“theskill,wit,orgenius”torecognizethemountinganomaliescreatedbymodelsbasedon

ageocentricviewoftheuniversecametotheconclusionthatabetteranswerwasneeded.At

firstthesebetteranswerscameinattemptstomodifytheheliocentricviewwithelaborate

patchesmeanttoexplainawayanomalousobservations.Inaddition,to“skill,wit,andgenius”

itrequiredagreatdealofcouragetochallengetheexistingviewinamorefundamental

fashion.

So,goodquestionscomefromknowingwhat“we”know.Buttheyalsocomefrom

thinkingdeeplyaboutwhatweknowandbeingsufficientlyunsatisfiedwithbadanswersto

taketheriskofthinkingdifferentlyaboutaproblem.Aswithallthearts,goodscienceseems

tocomefromindividualsandgroupsthatengageinacertainkindofpractice.Iwouldliketo

beginthisessaybycommentingonwhatIseeasacommonstructureofmanygreat

contributionstopoliticalscienceandinternationalrelations.Specifically,Iwillputforwardalist

4Itisfashionableinmanytopgraduatepoliticalscienceprogramsforfacultytosaythat“substantivecourses”areawasteoftimeandenterprisingstudentsshouldhaveanalmostsingle-mindedfocusonmethodstraining.Itisalsocommonplaceforprofessorstocomplainthattheirstudentsarenotadeptatidentifyinginterestingquestions.Isuspectthatthesephenomenaarenotunrelated.

4

offivequestionsthat,whenansweredwell,arelikelytoproduceworkthatasksandanswers

interestingandimportantquestionsandgivesusareasontobeconfidentinthoseanswers.In

thesecondhalfoftheessayIwillruminateonthekindofpracticethatIexpecttoleadtogood

questionaskingandgoodanswergiving.

FiveQuestions

WhenIwasingraduateschool,oneofmyprofessors,D.MichaelShafer,taughtme

howtoread.Hedidsobyencouragingmetoemployatemplatehecreatedsostudentscould

recordthekeypartsofwhattheyread:“Whatisthedependentvariable?”“Whatarethe

independentvariables?“Whatisthelogicthattiesthemtogether?Etc.”Ifoundthis

enormouslyhelpfulingettingthroughtheridiculousamountofreadingrequiredinmy

graduateclasses.WhenIbeganteachingIsharedthislistwithmystudentsandovertheyearsI

haverefineditforvariousreasons.Ihavecometobelievethatthislistofquestionsisuseful

notjustinfocusingourreadingefforts,butalsoinourresearchefforts.Ifyouaskwhatthe

author’sanswertoeachofthefollowingquestionsis,youwillhaveagoodsummaryofmost

articlesorbooksinourdiscipline.5Ifyouaskwhethertheauthorhasagoodanswertoeachof

thequestions,youwillhaveagoodcritiqueofthepaperinquestion.Andifyouareimpatient

withanybadanswersprovidedbytheauthor,anddevelopbetterones,youwillbeonyourway

tomakingyourowncontributiontotheliterature.Consequently,Ihavecometobelievethat

thesequestionscanalsoserveasanexcellentguidewhendesigningaresearchproject.Ifyou

havegoodanswerstothesefivequestions(andatleastoneoftheseanswersisan5Thequestionswouldhavetobeadaptedtoservethispurposeforliteraturereviews,methodspapers,andpurelytheoreticalpapers.

5

improvementoverexistingwork),youwillhaveagoodpaper,dissertation,orbook.These

questionsalsocorrespondtotheorganizationofthemodalpaperinourdiscipline:

“Introduction”,“LiteratureReview”,“Theory”,“ResearchDesign”,and“Findings”.

Itisimportanttoaddthatresearchquestionsneednotbegeneratedbyreading.They

canjustaseasily,andperhapsmoreprofoundly,beprovokedbyourinteractionand

observationofthesocialworld.Wemightobservebehaviorandsay“whydoesthathappen”?

Itisgoodpracticetoofferone’stentativeanswertosuchaquestionunencumberedby“the

literature.”Butitisimprudenttospendverymuchtimeonsuchactivitybeforeevaluating

existinganswerstoyourquestion.

Question1:WhatdoIwishtoexplain?(TheIntroduction)

FollowingKuhn’sdescriptionofscientificrevolutions,mostgoodworkbeginswitha

puzzlingobservation.Beginningwithobservationisimportantbecausegoodreaderswouldlike

tobeconvincedthatthephenomenonyouareexplainingactuallyoccurs(thoughitis

frequentlyfruitfultoengageinthoughtexperimentsaboutthingsthathavenotoccurred).

Thisstepisbynomeanstrivialandconsiderablemethodologicalsophisticationmaybe

necessarytoaccuratelydescribetherealworldeventsor,betterstill-patternsofevents-which

youwishtoexplain.

SamuelHuntington’sclassicPoliticalOrderinChangingSocieties(1968)seekstoexplain

therisingpoliticalinstabilityheobservedaroundtheworld.Asevidenceofthisrising

instability,onpagefourofthis462pagebook,theauthorpresentsU.S.DepartmentofDefense

datashowingthatthenumberofnationsaroundtheworldexperiencingmilitaryconflictsof

6

varioustypesrosealmostmonotonicallyfrom34in1958to57in1965(Table1.1).Thisisa

dramaticincrease:inlessthanadecadethenumberofconflictsnearlydoubled!Theproblem,

however,isthat,asaresultofdecolonization,thenumberofindependentcountriesinthe

worldalsogrewrapidlyduringthisperiod.IfonetakesHuntington’snumbersanddividesthem

bythenumberofindependentcountriesineachyear(asameasureoftheopportunityfor

militaryconflict),therelativefrequencyofmilitaryconflictactuallydeclinedoverthisperiod.

Sincemilitaryconflictwasjustoneproxyforpoliticalinstability,itisentirelypossiblethat

politicalinstabilityactuallyincreasedduringtheobservedperiod.Butifyoubelievethat

therelativefrequencyofconflictisabetterindicatorofpoliticalinstabilitythantheraw

frequency,youwouldbejustifiedinwonderingifthephenomenonexplainedinthesubsequent

fourhundredorsopagesactuallyoccurred.

Thefirstorderofbusiness,therefore,indemonstratingthatsomethingthatmayprove

consequentialhashappened;istodemonstratethatthatthinghashappened.Thiscrucialtask

7

isoftenbestaccomplishedwiththepresentationofclearlypresented,wellthoughtout,

descriptiveevidence.Whilethisoftenrequiresafairamountofmethodologicalskill,

sometimesitsimplyrequiresnumeracy–which,unfortunately,isofteninshortsupply.

Effectivelypresentingevidenceforone’sexplanandumis,perhaps,bestdescribedinthe

breach.Forexample,youcanreadnewspaperheadlinesonalmostadailybasisthatpurportto

capturesomeimportantchangeintheworldthatis,infact,notsupportedbythetextofthe

accompanyingarticle.Wouldthatitwerethecasethatthesemistakeswererareinacademic

work.

Onecommonmistakeistomakeaclaimaboutinter-temporalchangeinavariableby

citingonlycurrentvaluesofthatvariable.“Tenure-trackjobsaredisappearing”readsthetitle

ofanarticle,butthearticlemakesnoreferencetothenumberofsuchjobsthatwereavailable

inthepast.Howdoweknowthatchangehasoccurred?Arelatedissuethatrequiresabit

moremethodologicalskilltoavoidistopointoutadifferencebetweenthevaluesofafew

recentvaluesofavariablefromprecedingvaluesandclaimthattheyareevidenceofanew

trendwithoutcomparingthenewobservationswithalongenoughtrendofdatatodetermine

whethertheyrepresentameaningfuldeviationfromthetrendor,asisoftenthecase,just

typicalvariationwithinthetrend.

Anothercommonerroriswhatmightbecalled“thedenominatorproblem”–thefailure

tochooseadenominatorthatwouldtransformthedataintoavariableappropriateforthe

conceptualcomparisonrelevanttothediscussionathand.Wealreadysawanexampleofthis

whenHuntingtonconfusedatrendintherawfrequencyofavariableforatrendintherelative

frequencyofthedata,whichIarguedwouldhavebeenmoreappropriate.Butitisalsopossible

8

thattherawnumberiswhatmostinterestsus–inwhichcaseweshouldnotbedistractedby

anapparentlyrelatedratio.Toreturntothe“disappearingtenuretrackjobs”problemwe

oftenhearaboutinthepopularpress,intherareinstanceswhereinter-temporaldatais

presentedinanattempttoestablishthistrend,thequantitypresentedistypicallytheratioof

tenuretrackjobstothetotalnumbercollegeteachingjobs.Thisisproblematicbecauseitis

entirelypossiblefortheshareoftenuretrackjobstobedecliningwhenthenumberoftenure

trackjobsisincreasing(ashasbeenthecaseintheUnitedStatesfordecades).Anditis

probablythelatternumberthatisofinteresttomostreaders(forexample,currentdoctoral

studentshopingtoforecastfuturedemandforpeoplewiththecredentialstheyareworking

hardtoobtain).

Question2:Whydoesitneedtobetoexplained?(TheLiteratureReview)

Havingexplainedthatthisthinghasoccurred,itisimportantforauthorstodemonstrate

thata)thisthingviolatesexpectationsinsomeway(i.e.“somethinghasgonewrong”)andb)

thatthisviolationmay“proveconsequential.”Inotherwords,inthewordsofMilesDavis,“so

what?”

Onceagain,itmightbeeasiertosaywhatoneshouldnotdo.Ionceattendedapractice

jobtalkwhereasmart,hard-workingand,subsequently,verysuccessfulscholar,whenpressed

tosaywhathewastryingtoexplain,saidthathewastryingtoexplainwhyaparticularvariable

varies.BeinglesssupportivethanIshouldhavebeen,Iasked,“doyouhaveatheorythatleads

ustoexpectthisvariabletobeaconstant?”Variablesvary.Itisevenintheirname.

9

Observingthatvariation,therefore,hardlyconstitutesasurprise.Soifvariationinavariable

doesnotconstituteaviolationofexpectations,whatdoes?

Asacomparativepoliticsscholar,itpainsmetosaythatIhaveattendedmanyseminar

talksoverthelastfewdecades,mostgivenbysuccessfulandinfluentialseniorscholars,where

theworkinprogressismotivatedbyanassertionthatissomevariantofthefollowing“puzzle”:

theoryQclaimsthathighlevelsofvariableXshouldcauseYtohappen,butincountryiattimet,Xwasveryhigh,andYdidnotoccur.

Theproblemwiththis“puzzle”isthatoncethemisunderstandingitisbasedonisclearedup,it

isnolongerapuzzle.Themisunderstandingisthis:withveryfewexceptions(Icannotthinkof

one)theempiricalimplicationsofsocialscientifictheoriesarebesttreatedasprobabilistic

(Lieberson1991).Whetheronetracesthereasonstotheintrinsicallyprobabilisticnatureofall

humanbehaviorderivingfromhumanagency,thelimitationsofourunderstanding,thefact

thatmost(all?)socialphenomenahavemultiple,context-dependentcauses,orthepossibility

ofclassificationerror(didYoccurordiditnot?wasXreallyhighorlow?andcomparedto

what?)itisbesttothinkofourhypothesesasprobabilistic.WhichmeansthemosttheoryQ

canclaimisthat“highlevelsofvariableXshouldmakeYmorelikelytohappen.”

Consequently,thefactthatYdidnotoccurincountryiattimet,despitethefactthatXwas

veryhighisnot,atleasttomyear,particularlypuzzling.Unlikelyeventsareexpectedtohappen

occasionally.Consequently,onecannotreasonablycallaprobabilisticconjectureintoquestion

withasinglenullcase.Doingsoislikebeingpuzzledaboutone’sunclewholivedtoaripeold

agedespitebeingaheavysmoker.Thisisnotpuzzlingbecausethebestscientificevidenceis

thatsmokingincreasesthelikelihoodofcancer,notthatitalwaysleadstocancer.Incontrast,it

10

wouldbesurprisingtofindanentiresub-sampleofthepopulationthatappearstobeimmune

tothedeleteriouseffectsofsmoking,or,thataftercontrollingforincomeoreducation(orany

otherpotentialconfound),smokersarenotmorepronetocancerthannon-smokers.Insum,

sinceourtheoriestypicallyjustifyexpectationsaboutpatternsofdata,ittakesobservations

aboutpatternsofdata,notdiscreetdatapoints,toviolatethoseexpectations.

Whilerecognizingapatterninthedataisoftennecessaryforgeneratingsurprise,itisbyno

meanssufficient.GoingbacktothemanycomparativepoliticsseminarsIhaveattended,be

waryofthescholarwhoselectsasmallsampleofobservationsanddemonstratesthatawidely

corroboratedempiricalregularity,suchastheincumbencyadvantage,thedemocraticpeace,

Gamson’sLaw,Duverger’sLaw,ortheresourcecurse,“doesn’thold”inthatsub-sample.Why?

Becausesocialbehaviorisprobabilistic,soevenhighlypredictiveempiricalmodelsyield

predictionswithnon-zeroerrors.Asaresult,onecanalwaysfindasubsampleofdatawhere

thebroaderpatterndoesnothold.Takeany“footballshaped”scatterplot,suchasthefamous

scatterplotshowinFigure1.1.6Onecanselectoutasub-sampleofcases,suchasthoseinthe

ellipse,tosuggestthattheregressionlineisflatorevennegativeeventhoughthereisclearlya

positiverelationshipinthesampleonthewhole.

6ThescatterplotisbasedondatafromanexamplefromPearsonandLee’s(1903)examinationofthecorrelationbetweentheadultheightsoffathersandsons.

11

Figure1.1RelationshipBetweentheHeightofFathersandSons(DataSource:Freedman,Pisani,andPurves(2007)addedrandomnoisetodatafromPearsonandLee(1903)whoonlyhaddatatonearestinch,)http://myweb.uiowa.edu/pbreheny/data/pearson.html

12

RecallthatIsaid,“bewary”ofascholarwhomotivatestheirstudywithasub-sampleof

casesthatappeartoruncontrarytoawell-corroboratedsetofexpectations.ButIwouldnot

encourageyoutodismisssuchascholar.Itis,forexample,entirelyappropriatetoshowthat

thereareboundaryconditionsoneventhemostwell-corroboratedempiricalregularities.But

themereexistenceofsuchasub-sampledoesnotconstituteapuzzleuntilonecanconvince

thereaderthatthesub-sampleconstitutesacomprehensiblecategoryandisnotjusttheresult

offelicitous(fromthestandpointoftheauthorseekingsomethingtowriteabout)case

selection.Further,ifonedoestakeastheirprojectthetaskofexplainingwhyawell-

corroboratedregularitydoesnotapplytoaparticularsub-sample,itisincumbentuponthemto

developanexplanationforwhythesub-sampleisdifferentthatyieldspredictionsotherthan

thefactthatthesub-sampleisdifferent.Otherwise,theyareengagedinbothpost-hocandad-

hocreasoning.

Yetanotherproblemcanarisewhenonegeneratestheirresearchprojectbygazingata

scatterplot.ManywilllookatalikeFigure1.1afterestimatingaregressionlineandbe

disturbedthatsomanyobservationsfallfarfromtheregressionline.Itisokaytowantthe

modeltofitthedatawell,butgiventheprobabilistic,multi-causalnatureofourhypotheses,it

isnotpuzzlingthatsomeobservationsfallfarfromtheregressionline.Myfatherwassixfeet

tall,whileI,ahem,amnot.Thatisnotsurprisingbecauseotherfactorsenterintoheightat

adulthoodotherthanmygeneticinheritancefrommyfather–dietandcontributionsfrommy

mother’sgeneticmake-upcometomind.Beingpuzzledinthiswayisaslightlymore

sophisticatedversionofthe“ifXishighincountryiattimet,whydowenotobserveY”

problem.Bothmethodsarefrequentlyusedtojustifytheclaimthat“existingexplanationsare

13

incomplete.”Theproblemisthatanyexplanationtheauthorcomesupwithislikelytobe

susceptibletothesamecriticism.

Iwanttobeclear,thereisnothingwrongwithbeingunsatisfiedwithexplanationsthat

donotfitthedatawell.However,iftheonlyresultofpointingoutobservationsthatfalloffthe

regressionlineisanewmodelthatmarginallyincreasesmeasuresofgoodnessoffit,donotbe

surprisedifreadersfailtoseethisas“consequential.”Ceterisparibus,papersthatare

motivatedbytheidentificationofunclear,misleading,orincorrectunderstandingsinthe

existingliteraturearemoreconsequentialthanthosethatpointtomerely“incomplete”

understandingsbecausetheformercausesustorevise(thatisto“lookatagain”)ratherthan

merelysupplementourcurrentunderstanding.

Sofar,wehavebeenseekingtoidentifyviolationsofexpectationsthatare

consequentialforourunderstandingoftheworld,butonemightalsoplaceapriorityon

consequencesthataremorepractical.Onewayofaskingthe“sowhat”questionistoask,“if

youweresuccessfulinexplainingyouranomalousobservation,howwouldtheworldbe

different?”Unlessoneisentirelynaïve,thisisaverytoughquestiontoanswer.Butsincemost

ofusbecamepoliticalscientistsandinternationalrelationsscholarsbecausewewantedto

maketheworldabetterplace,itisstillworthwhile.Onereasontothinkaboutthe“normative”

implicationsofthequestionsweaskisthatanevenpassingfamiliaritywiththeliteraturein

politicalscienceandinternationalrelationsisenoughtounearthaseeminglyendlesssupplyof

unclear,misleading,orincorrectunderstandings.Inlightofthis,itisnotunreasonabletotryto

tacklefirstthosethataretiedtoissueswecaredeeplyabout.

14

NobellaureateRobertLucassaid“onceyoustartthinkingabouteconomicgrowth,itis

hardtothinkaboutanythingelse.”Isuspectthatisbecauseitisnothardtoseetherealworld,

sticktoyourribs,consequencesofeconomicgrowth.Likewise,immigration,politicalviolence,

economicinequality,governmentcorruption,racialandethnicdiscrimination,financial

instability,authoritarianism,genderbias,illiteracy,failingschools,orahostofotherpolicy

issuesareofinterestbecauseoftheirimpactonmattersofjusticeandhumanwell-being.

Explainingobservationsthatviolateourexpectationscanbequiteconsequentialwhendoingso

shedslightontheseandothersocialproblems.

Marx’slastandmostfamousthesesonFeuerbachisthat“thephilosophershaveonly

interpretedtheworld,invariousways.Thepoint,howeveristochangeit”anditisinteresting

thatitisetchedonhistombdespitehavingneverbeenpublishedwhilehewasalive.8It

capturesthefrustrationofmanyscholarswhowouldliketo“makeadifference.”Itcertainly

capturedmyromanticheartwhenIfirstreaditasayoungman(notmuchyoungerthanMarx

waswhenhewroteit)atthestartofgraduateschool.ButIwasnotingraduateschoollong

beforeIrealizedthecomplexityof“interpreting”theworldandthedangersthatcouldresultif

onesoughttochangetheworldwithouthavinginterpreteditcorrectly.Understandingthe

worldisaprerequisiteforchangingitinaresponsiblemanner.

Whileitisdesirable,perhapsevennoble,bridgingthegapbetweenstudyingtheworld

thewayitisandusingthisinformationtoimprovesocialconditionsisdifficult-particularly

whenpeople,and,therefore,politicsareinvolved.Oneproblemisthatifsocialillshave

politicalrootsevenaccurateexplanationsoftheircausesarelikelytobeinsufficientfor

8Marx(1888).

15

mitigatingthem.Onereasonforthisisthefactthatthehallmarkofpoliticsisconflictingvalues.

Explainingtoprisonersconfrontedwithpleadealsthatrewardthemforincriminatingeach

otherthattheycollectivelybenefitbykeepingmumwillnotsolvetheprisoner’sdilemma

becausetheywillstillhaveindividualincentivestoratontheirco-conspirators.9

So,whileunderstandingtheworldmaybeanecessaryconditionfor(responsibly)

changingit,itisnotlikelytobesufficient.And,conversely,changingtheworldcanmakeitalot

hardertounderstand.Oneofthethingsthatmakessocialsciencedifficultisthattheentities

westudycanreadwhatwewriteandchangetheirbehaviorinwaysthatmakeourmodelsless

predictivelyaccurate.10

SomethinglikethismayhavebeenatworkinthewritingsofMarx.Thephrase

“workersofalllands,unite!”alsoappearsonMarx’stomb.Incontrasttohistheseson

Feuerbach,thisphrasewaspublishedduringhislife-time.Threeyearsafterbemoaningthe

irrelevanceofpriorphilosophersMarxandEnglesclosedoneofthemostinfluentialpolitical

pamphletseverwrittenwithit.11Inan1890appendixtoTheCommunistManifestoEngels

admitsthatfewheededthecallin1848butsuggestsmanyeventuallydidsoovertime,

includingthosewhowereorganizinginsupportoftheeight-hourworkday1890.Itisnot

unreasonabletosuggestthatMarx’sanalysisofaninternallogictocapitalism(thatthe

9Incontrast,iftheonlyproblemisaco-ordinationproblemthenthemeredisseminationofinformationislikelytobesufficient.Butsuchproblemsareaboutaspoliticalasgettingdriverstostayontheirsideoftheroad.10Thoughsometimesthisworksintheoppositiedirection.Forexample,experimentshaveshownthatstudentswhotakeeconomicsclassesbehavemuchlesscooperatively,and,therefore,moreinlinewiththemodelslearnedinthesecourses.11TheCommunistManifesto(MarxandEngels,1996)hadlittleimmediateimpactonembryonicsocialistmovements,butitslongruninfluenceisundeniable.

16

inexorableimmiserationoftheproletariatwouldleadtorevolution)helpedfueltheformation

oflaborunionsandthecreationofsocialprogramsthatimprovedthematerialconditionsof

workers.Butindoingso,thismadethemlessrevolutionary–therebyreducingtheprobability

oftherevolutionhepredicted.

Anotherexampleofhowitishardtohavebothinfluenceintherealworldand

predictiveaccuracycomesfromtherecentliteratureon“thehappinesscurve”–therobust

empiricalregularitythatreportedlifesatisfactiontendstodeclinewhenpeopleareintheir

fortiesandriseconsistentlystartingintheirearlyfifties(Rauch2018).Oneexplanationforthis

empiricalregularityisthatbecausehumanpsychologyisbiasedtowardsoverlyoptimistic

forecasts,youngpeopleover-estimatehowmuchtheirliveswillimproveintheirthirtiesand

forties.Thisresultsindisappointmentduringtheirmiddleyearsevenifindividuals’liveshave

improvedconsiderably,butnotasmuchastheyexpected.Thisdisappointmentalsoleads

peopletoupdatetheirexpectationsandmakegrimforecastsforthefuture.Consequently,

whenlifeintheirfifties,sixties,andbeyondturnsouttobenotasbadasexpected,theyreport

highlevelsoflifesatisfaction.Ifthisprocessistrulyatwork,peoplewhoreadthisliterature

mightbeinclinedtomakemorerealisticpredictionsaboutfuturelifesatisfaction.Iftheydidso

inlargenumbers,the“happinesscurve”coulddisappear.

NoticethattotheextentthatMarxchangedhistoryitmayhavebeeninwaysthat

frustratedbothhispredictiveaccuracyandhissocialdesires(forrevolution)butifhappiness

researchersturnouttohavethesamedegreeofimpactonsocietytheymightbeperfectly

willingtotradepredictiveaccuracyfortangibleimprovementsinpeople’slifesatisfaction.

17

Insum,wewouldliketoanswerquestionsthat,whenanswered,wouldprove

consequential.Theseconsequencescanbeeitherforthewaywethinkabouttheworld,orthe

waypeoplebehave.While,allelseequal,wewouldlikeourresearchtoleadtoimprovements

inhumanwell-being,thestrategicnatureofpoliticsmeansthatevenwhenweprovidegood

answerstoquestionsthatareimportanttousitmaynotleaddirectlytoimprovementsinsocial

outcomes.Thatisnottosuggestweshouldstoptrying.

Question3:Whatistheexplanation?(Theory)

Agoodexplanationwilltakeanobservationthatissufficientlysurprisingthatitjustifies

yourstudy,andturnitintosomethingthat,inretrospect,shouldhavebeenexpectedallalong.

InwhatremainsoneofthefewbooksIknowofthatattemptstoteachpeoplehowtoexplain

things,theauthorsofAnIntroductiontoModelsintheSocialSciences(LaveandMarch,1975)

describeexplanationasaprocessinwhichoneimaginesapriorworldsuchthat,ifitexisted,

thesurprisingfact(s)wouldhavebeenexpected.Technically,anysetofstatementsthat

logicallyimplytheoccurrenceoftheanomalousobservationconstituteanexplanation.But

goodexplanationshaveadditionalattributes,andwewouldliketoproducethebest

explanation.Asatisfyingexplanationwillgivethereaderanunderstandingoftheprocessor

mechanismthatislikelytoproducethepreviouslyanomalousobservation.Readerswantto

knowhowsurprisingeventscameabout,andexplanationsshouldtellthem.Good

explanationsareefficient–theratioofthingstheyexplain(implications)tothingstheyrequire

youtobelieve(assumptions)ishigh.

18

Thereisanoptimaldegreeofnoveltytoanexplanation.Anexplanationshouldbe

interesting,yetsound.By“interesting”Imeanthatanexplanationshouldcauseustoseethe

worldinanewway.By“sound”Imeananexplanationshouldfitinwithotherthingsweknow

abouttheworld.Anexplanationthatcausesustoseeeverythinginanewwayislikelytobe

wrong.Anexplanationthatdoesnotrequireustochangeourmindatallisprobablyjusta

corollaryofthingswealreadyknew(and,byextension,ourmotivatingpuzzlemustnothave

beenmuchofapuzzle).

Finally,explanationsmustbelogicallyconsistent.Ihavehadempiricallyminded

politicalscientistsandinternationalrelationsscholarstellmethatformaltheoryisnot

importantbecausetheyaresophisticatedenoughtolivewiththeoriesthatcontain

contradictions.Thisisnonsense.Itcanbeshownwithelementarylogicthatanythingfollowsa

contradiction.Consequently,ifyourtheorycontainsacontradiction,anythingcanbesaidto

followfromit.Asaresult,acontradictorytheoryrulesnothingoutand,therefore,noamount

ofempiricalinformationwillbesufficienttofalsifyit.Sincepotentialfalsificationisthe

hallmarkofscience,atheorythatcontainsacontradiction,therefore,isnotascientific

theory.12

Onewaytoincreasethelikelihoodthatyourexplanationislogicallyconsistentistotry

tocaptureitwithaformalmodel.Formalmodelsallowustodemonstratethatour

explanation’sconclusionsfollowfromitsassumptions-mostimportantly,thatourpreviously

puzzlingobservationisnotsurprisinginlightoftheworldthatourexplanationposits.Also,by

12InthepossiblyapocryphalwordsoftheoreticalphysicistWolfgangPauli,itis“notevenwrong.”

19

makingtheassumptionsofourexplanationexplicitwearemorelikelytonoticeifthey

contradicteachother.

Whilethesebenefitsofformalizationareundeniable,itdoesnotfollowthatevery

explanationshouldbeformalized.Itypicallyencouragemystudentstofirstarticulatetheir

explanationsasastorythatrevealsaprocessthatproducesthepreviouslyunexpected

observation.Formalizationisonlynecessarywhenonehearssuchastoryandasks,“whywould

peopledothat?”or,equivalently,“Thatdoesn’tsoundlikeanequilibrium,”or,“isn’ttherea

tensionbetweenthispartofthestoryandthatpartofthestory?”Whenoneisconfronted

withsuchquestions,agoodformalmodelcanoftenprovideanswers.Thus,Itellmystudentsto

learnhowtowritedownformalmodelsnotbecausetheywillalwaysneedone,butlikefire

insurance,theyarealwaysatriskofneedingone.

Anotherreasontobeginwithaninformalstatementofone’stheoryistoavoidthetrap

ofthinkingthatagametheoreticmodelwillgenerateatheoryforus.Formalmodelshelpus

interrogatecertainaspectsofourtheory,theydonotproducethetheoryforus.Wemust

beginwithsometheoreticalintuitionaboutwhatexplainsthephenomenoninquestionbefore

wecanbegintomodeltheprocess.

Question4:Iftheexplanationistrue,whatelseshouldweobserve?(ResearchDesign)

Ifyouofferaviewofatheoreticalworldthathasthepreviouslypuzzlingobservationas

oneofitsimplications,youhaveofferedanexplanation.Andwhiletherearevariouswaysto

evaluatethatexplanation,tobescientific,youranswertoyouroriginalquestionmustprovide

ananswertothefollowingquestion:“ifyourexplanationiscorrect,whatelseoughttobe

20

true?”Goodscientificexplanationsprovidelotsofanswerstothisquestion.Ifyour

explanationonlyimpliesthefactsthatyousetouttoexplain,thenthereisnowayto

empiricallyevaluateyouranswer.Youcannotusethefactthatdemocraciesseldomfighteach

other,orthefactthatthereisalotofcorruptioninpresidentialdemocraciestoevaluateyour

explanationofthesethingsbecauseitwasthosefactsthatledyoutodevelopyourexplanation

inthefirstplace.

Thispartoftheresearchprocessisastumblingblockformanyresearcherswhenthey

areattractedtoasubject,ratherthanaquestion.IoncehadastudentwhovisitedBraziland

wasshockedbythelevelofcorruptioninthegovernmentthereanddevelopedanexplanation

thatpointedtoaspectsofthelargedistrictmagnitudeproportionalrepresentationelectoral

systemasacause.ThestudentwassurprisedwhenIsaidIthoughttheargumenthadmerits,

butthatreturningtoBraziltocollectdatawasnotapromisingavenueforevaluatingthe

argument:wealreadynewthatBrazilfittheargument!Perhapsdataoncorruptionlevelsin

countrieswithdifferentelectorallaws(suchastheUnitedStates)wouldbemoreuseful.The

student,however,respondedthathedidnotwanttostudycorruptioninothercountries,after

allhewasinterestedinBrazil!

AsimilarproblemisfoundinaveryfamousbookbyThedaSkocpol,StatesandSocial

Revolutions(1978).Init,theauthorwishestoexplaintheoccurrenceofsocialrevolutionsand

shearguedthathersubjectdictatedherempiricalstrategy.Givenherdefinition,thereareonly

fivehistoricalcasesofsocialrevolution.Shearguedthatasaconsequenceofthisfact,

structuredfocusedcomparison(specifically,Mill’sMethodofAgreement)wastheonlypossible

methodforevaluatingherexplanation.Thatisnottrue.

21

Thechiefproblemhereisthatifanexplanationforasetofrareeventsonlyhas

implicationsaboutthoserareevents,theauthordoesnothaveadataproblem,theyhavea

theoryproblem.Ifanexplanationforglobalwarmingonlypredictsthegeneralriseinthe

temperaturethatmotivatedtheexplanation,thenitisnotaveryusefulexplanation.

Cosmologistshaveofferedexplanationsforthecreationoftheuniverse,buttheydonotchoose

theirmethodologyforevaluatingtheirexplanationsbasedonthefactthattheobjectoftheir

studyonlyhappedonce.Instead,theyask,“ifmyexplanationforthisuniqueeventiscorrect,

whatelseoughttobetrue?”Theythenthinkabouthowbesttocarefullyobservethe

implicationsoftheirargument.

Thegoalofempiricalresearch,therefore,shouldbetoexamineasmanyimplicationsof

one’sexplanationaspossible.Becausemany,manyscholarsrestricttheirattentiontothe

empiricalpuzzlethatmotivatedtheirstudytobeginwith,manyimportantpaperscanbe

writtenbysimplyaskingofexistingexplanations,“ifthisargumentistrue,whatelseoughtwe

observe?”

Onereasonwhyscholarsoftenrestricttheirattentiontothedatathatgeneratedthe

questionisthatitcanoftentakeconsiderablecreativitytothinkabouttheimplicationsofan

explanation.Thereisnocookbook-likeapproachthatcanbeappliedthatwillautomatically

revealtothescholarthatseeminglyunrelatedeventsmightbeinstantiationsofasinglesocial

process.ButonepracticeLaveandMarchrecommendistotrytoseeyouranswertoa

particularquestionasrelatedtoamoregeneralprocess.

Forexample,inhercriticalreviewofSkocpol’sbook,BarbaraGeddes(2003)suggests

thatoneelementofSkocopol’sexplanationofraresocialrevolutionshadimplicationsforthe

22

occurrenceofpeasantrevolts.Geddessuggestthatastatisticalmodelexaminingtheconditions

underwhichpeasantrevoltsdoanddonotoccurwould,therefore,beusefulinevaluatingthe

empiricalrelevanceofSkocpol’sexplanationofsocialrevolutions.

Notice,thatwhenweask“whatelseoughttobetrue”weseparatethequestionof

“whatistheauthorsexplanandum?”from“whatistheauthor’s“dependentvariable?”The

explanandumisastatementofwhattheauthordevelopsatheorytoexplain.The“dependent

variable”istheendogenousvariableinamodeltestingoneormoreoftheimplicationsofthe

author’stheory.Therearetimeswhenthesemightbethesame,butthereisnoreasonto

assumetheywillbe.Infact,whentheyare,weshouldwonderiftheauthorisengagedinpost-

hocreasoning–“havetheyobservedthedependentvariableanditscovariatesandconstructed

acausalstoryafterthefact?”Doingsowouldconstitutea“test”ofthetheoryonlytothe

extentthatthelion’sshareoftheobservationscouldbethoughttohavebeenappreciably

differentfromthosethatwereobservedbeforethetheory’sformulation.Conversely,atheory

thatproducesalotofnovelimplicationshelpsassuagethereader’ssuspicionthattheauthoris

merelyengagedinacurve-fittingexercise.

Insum,itistypicallymorehelpfultothinkofempiricalworkastestingtheimplications

ofatheory,ratherthantestingthetheorydirectly.Onereasonthisistrueisthattestingthe

theorydirectlycaneasilydescendintomoreorlesscomplicatedversionofcurve-fittingand

post-hocreasoning.Instead,spendtimethinkingabouttheimplicationsofyourexplanationfor

observationsotherthanthosethatmotivatedyourquestioninthefirstplace.Themorevaried

thoseimplicationsthebetter,becauseitisonlythoseobservationsthataremadeafterthe

constructionofyourtheorythatruntheriskofbeingfalseandthereforeactuallyconstitutean

23

empiricalcheckonyourexplanation.Andremember:ifyourtheoryonlyhasimplicationsfora

setofeventstoosmalltousestandardinferentialtoolstoevaluate,youdonothaveadata

problem-youhaveatheoryproblem.

Question5:Doweobservetheimplicationsofourexplanation?(Findings)

Determiningifevidenceisconsistentwithone’stheoreticalexpectationsistheprimary

focusofresearchmethodologyand,so,isthecentralfocusoftheremainderofthisvolume.

HereIwillmerelystressthefollowing:many,manystudiespresent,oftenindizzyingdetail,

reamsofinformationthatiseitherirrelevanttoorinconsistentwiththeoreticalexpectations.

Typically,however,itispresentedinamannerthatsuggeststhatthisinformationconfirmsthe

author’sexpectations.Distinguishingwhenthisisthecaseisalargepartofwhatismeantby

learningtoreadcritically.

AsIsaid,allofthecollectivewisdomofresearchmethodologistsisrelevantfor

becomingacriticalreaderandproducerofknowledgebutIwillfocusononeadmonition:

presentclearestimatesofthequantitiesofinterestaswellasastatementaboutthedegreeof

confidenceonehasinthoseestimates.13Thereareafewwaysinwhichthisadmonitionis

frequentlyviolated,andIwouldliketobrieflydrawyourattentiontothem.

AtleastinthesocialscientificpapersIread,explanationstypicallyproduceclaimsabout

theassociationbetweenvariables.Evenwhenoneisengagedinwhatlookslikeadescriptive

exercise,likeHuntingon’sattempttodemonstraterisingpoliticalinstability,oneisengagedin

demonstratingthatvaraiblesarerelatedtoeachotherinaparticularway.Ifonewantsto

13King,Keohane,andVerba(1994).

24

demonstratethataphenomenonischangingovertime,onemustlookattherelationship

betweenthatvariableandtime.Ifonewantstodemonstratethataparticularbehavioror

attitudeismoreprevalentinsomeplacesoramongsomegroups,onemustlookatthe

relationshipbetweenthatvariableandgroupmembershiporspatiallocation.Consequently,

mostofourempiricalclaimsareabouttherelationshipbetweenvariables.Inalinearmodel

wethinkofthisquantityofinterestasaslopecoefficient,soIwillusethatterminologyhere,

thoughtheterm“derivative”mightbeevenmoreappropriate.

Acommonwayinwhichscholarsbecomedistractedfrompresentingthequantityof

interestisbypresentingsomethingotherthananestimateofaslope,whenthatisthequantity

theyareconcernedwith.Forexample,ithasbecomecommonforscholarstoplotthe

predictedprobabilitiesfromalogitmodelonthey-axiswithsomevariableofinterestonthex-

axiswhenthequantitiyofinterestistheassociationbetweenachangeinthatpredicted

probabilityandameaningfulchangeinsomevariableofinterest.Theproblemwithdoingsois

thatitrequiresthereadertoinfertheslopeofthatrelationshipfromthepicture.Whileitis

truethatslopesarenotconstantinnon-linearmodelssuchaslogit,and,thereforethequantity

ofinterestdoesnotreducetoasinglenumber,itwouldbebettertoplotthemarginaleffectof

thevariableofinterestacrossameaningfulsetofvaluesofthatvariableofinterest.14Adding

confidenceintervalsaroundthepredictedprobabilitydoesnothelpbecausethattellsthe

readerifthepredictedprobabilityissignificantlydifferentfromzero,whichistypicallynotthe

hypothesisbeingtested.

14Inthelanguageofcalculus:ifthequantityofinterestisdy/dx,thenplotdy/dxagainstx,notyagainstx.Theformertellsthereaderwhattheyneedtoknow.Thelattermakesthereadertrytoinferwhattheyneedtoknowfromthepicture.

25

Forexample,Hellwig,Ringsmuth,andFreeman(2008)presentthegraphsinFigure1.2

asevidenceinthatthepropensityforcitizenstobelievegovernmentshavelittleroomto

maneuverpolicyinaglobalizedeconomy.Eachpanelplotsthepredictedprobability(and90%

confidenceintervals)thatasurveyrespondentsaidtheydidnotbelievetheU.S.government

retainsthe“roomtomaneuver”policyagainsttherespondent’spartisanship.Theauthors

interprettheapparentdifferencebetweentheslopeoftheplotsinthelefthandpanelfromthe

righthandpanelasevidencethatpartisanshiphasaneffectonrespondentbeliefsamong

respondentswithCollegeDegrees(panela)butnotwithHighSchoolDegreesorLess(panelb)

andamongrespondentsabovetheageoffifty-nine(panelc)butnotbelowtheageofforty

(paneld).Butwhatisthebasisofthisconclusion?Theslopesontherightclearlylooktobe

closetozeroand,incomparison,theslopesontheleftappeartobepositive.Butweare

offeredneitheranestimateoftheslopesforanydegreeofpartisanship,noranestimateofour

uncertaintyaboutthatestimate.Wecantrytocalculatetheslopeatdifferentpointsontheline

byestimatingthe“riseoverrun”andwecankindofcomparethatestimatewiththe

uncertaintyimpliedbytheerrorbars,butwhymakethereaderconstructat-testfromthe

pictureratherthanpresentthatinformationforthereaderbyplottingmarginaleffectswith

theirassociatedconfidenceintervals?Neitherdotheauthorsprovideanyevidencewhether

theslopesintheleft-handpanelsaredifferentfromtheslopesintherighthandpanels.Asa

consequence,thesepictures,andoneslikethemthatappearfrequentlyintheliterature,

providealmostnoquantitativeevidenceaboutthequantityofinterest(underwhatconditions,

ifany,achangeinpartisanshipassociatedwithachangeincitizenbeliefsaboutthe

government’s“roomtomaneuver”).

26

Figure1.2PartisanshipandBeliefsabout‘RoomtoManeuver:TheConditionaleffects

ofKnowledgeandAge.SourceHellwig,Ringsmuth,andFreeman(2008,Figure2,p.875.)

27

Anothercommonwayofobscuringthequantityofinterestisinpresenting“marginal

effects”thatarenotmarginal.Itiscommonplaceforauthorstosaythingslike“togainsome

substantiveunderstandingoftheseresults,InotethataonestandarddeviationchangeinXis

associatedwitha0.056changeinY.”Theproblemwiththisisthatthereisnothingtypicalor

representativeaboutastandarddeviation–indataapproximatinganormaldistributionabout

two-thirdsofallobservationswillbelessthanastandarddeviationawayfromthemean.Asa

consequence,achangeofastandarddeviationinthevariableofinterestisnotaparticularly

meaningfulcounterfactualtoconsider.Thisisparticularlytruewherethispracticeismost

frequentlyfound–wheninterpretingtheresultsofanon-linearmodel.Underthis

circumstance,themarginaleffectofavariableisextremelysensitivetowhereitisbeing

evaluated.Theslopedescribedbya“marginaleffect”thesizeofastandarddeviationislikely

tobeveryfarfromtheslopeofanyestimatedmarginaleffectwithinthisinterval.Another

reasonwhythisisnotaparticularlyusefulcounterfactualcomparisonisthatmarginaleffects

areinterpretedunderaceterisparibusclausewhereotherfactorsareheldconstant–

somethingwhichisnotlikelytobeapproximatedintherealworldwhenthevariableofinterest

experiencesanunusuallylargechangethesizeofastandarddeviation.15

Anothercommonwayscholarspresentinformationthatisnotthequantityofinterestis

whentheyhaveahypothesisthatisconditionalinnatureandeitherpresentresultsfroman

unconditionalmodel,or,equallycommon,estimateaconditionalmodelbutgoontointerpret

someofitsresultsasiftheywereunconditional.16

15SeeKingandZeng(2006)on“TheDangersofExtremeCounterfactuals.”16SeeBrambor,ClarkandGolder(2006)orKamandFranzese(2007)forafullerdiscussion.

28

Summary

Myclaim,uptothispoint,isthatapaper,book,ordissertationthathasgoodanswers

tothefivequestionsabovewillbeausefulpaper,book,ordissertation.Itdoesnotfollowthat

apaper,book,ordissertationmusthaveaninnovativeanswertoallfiveofthosequestions.

Progresscanbemadeaslongasoneoftheanswersisbetterthanexistinganswersandnone

areworse.

Whichquestionsare“mostimportant”and,therefore,whichonesshouldbethefocus

ofyoureffortstoinnovate?Itishardtosay.ThoughIbelievethatitisprobablynotbesttotry

toexplainsomethingthatnoonehasexplainedbefore.Thisisanimportantpoint.Ihavehad

manygraduatestudentsinformmegloomilythatsomeonehasbeatenthemtotheir

“question.”Mystandardreactionistosay,“well,Idoubttheyhavecomeupwiththedefinitive

answer,sowhatareyouworriedabout?”Sinceanyquestionworthaskingislikelytobe

difficulttoanswer,itishighlyunlikelythatanotherscholarislikelyto“beatyoutothepunch”

andstatethe“lastword”onasubject.Indeed,ifyouareaskingaquestionthatnooneelse

hasasked,itshouldgiveyoupause.Maybeitisnotaveryinterestingquestion:ormaybethere

issomethingaboutaskingthequestioninthatwaythatledotherscholarstobelieveproductive

answerswerenotforthcoming.Thatsaid,themerefactthatothersmartpeoplehaveasked

thequestiondoesnotmeanitisagreatideaforyoutotrytoanswerit.

Graduatestudentsaretoldthattheyneedtomakeanoriginalcontributionwhichleads

themtobelievethattheymustaskaquestionthathasneverbeenasked,oratleast,never

beenansweredbefore.Thatisnottrue.Rather,an“originalcontribution”requiresonlythat

thestudentprovideabetteranswertoatleastoneofthequestionsmentionedabove.So,ifa

29

studentattheprospectusstageisgoingtoattempttoofferanovelexplanation,thenpartof

theiranswertoquestion2shouldcontainastatementaboutwhattheybringtothetablethat

mightallowthemtomakeprogresswhereothershavefailed.Whattheoreticalinsight,

methodologicaladvantage,orhistoricalknowledgeputstheauthorinapositionto

simultaneouslyrecognizethat“thingshavegonewrong”withexistingexplanationsandoffera

solutionthatpushesthefieldinapromisingdirection?

Since“theoreticalinnovation”isoftenthoughttobethemostprizedcontributiona

politicalscientistcanmake,scholarsoftenbelievethatagoodpapershouldofferanovel

explanation.Ibelievethiscomes,inpart,fromphysicsenvycombinedwiththenotionthat

theoreticalphysicistshaveahigherstatusthanexperimentalists.Ibelievetheideathatevery

importantcontributionmustcontainatheoreticalinnovationhasgreatlyhamperedthe

progressofourdiscipline.Howistheaccumulationofknowledgepossibleifeverytimea

scholarputspentopapertheyhavetoofferanewexplanation?Givenfrequentlyimperfect

researchdesignsandflawedempiricalmethods,Ioftenthinktheoppositeistrue.Wemightbe

temptedtodeclareamoratoriumonthedevelopmentofnewexplanationsuntilthediscipline

hasreachedconsensusaboutempiricaltestsoftheimplicationsofexistingexplanations.Asmy

critiqueofHuntingtonsuggests,ifwedonotgetatleastsomeoftheempiricsright,howdowe

evenknowifourobservationsviolatecurrenttheoreticalexpectationsenoughtowarrantnew

explanations?Onereasontoresistsuchatemptationisthatnewtheoriesdomorethan

explainanomalies.Foroneexample,theyalsoaddressconceptualandlogicalproblemswith

existingexplanations.

30

Practicesthatencouragegoodquestionasking

FollowingKuhn’slineofreasoningabove,itisworthaskingwhatislikelytopromote

“theskill,wit,andgenius”capableofrecognizingwhenthingshave“gonewronginwaysthat

mayproveconsequential.”OfKuhn’sthreedesiderata,“skill”seemstheleastconstrainedby

naturalabilityand,therefore,mostresponsivetotheenvironmentswecreate.Whileartistic

creationinvolvesmanyaspects,adegreeofcraftsmanshipistypicallyinvolvedand

craftsmanshipisderivedlargelyfrompractice.Extensivetrainingingametheoryandstatistics

isnowcommonplaceinmostgraduate(andsomeundergraduate)programsinpoliticalscience

andinternationalrelationsandthisiswhatistypicallythoughtofwhenscholarsevaluatethe

“skills”ofjobapplicants.Theseskillsareimportantbecausewithoutthem,scholarsmightask

questionsbasedonfaultyreasoningbasedonformalorinformalfallaciessuchastheecological

fallacy,adhominemattacks,hastygeneralization,confusingcorrelationwithcausation,

ignoringstrategyinducedselectioneffects,andfailingtorecognizethepresenceofconfounds.

Butwhilemethodstrainingisextremelyhelpful,itisnotsufficienttoproducescholars

whoaskandanswerinterestingquestions.Theproblemsetstypicallyassignedinquantitative

methodsandformaltheoryclassesdohelpbuildtheskillsnecessarytoexecutesophisticated

research.Justasplayingscalesandarpeggiosbuildthetechniquesnecessarytoexecute

sophisticatedmusic.Butthereismoretotrainingamusicianthanplayingscalesandarpeggios

becauseasimportantasscalesandarpeggiosare,theyarenotmusic.Ihaveheardmusicians

criticizedforhavingsufficienttechniquethatthey“knowhowtosaythingsontheir

instruments,buttheydonotseemtohaveanythingtosay.”Theanalogouscriticismis

frequentlyleveledatnewlytrainedpoliticalscientistsandinternationalrelationsscholars.

31

So,whatistobedone?Toplaygoodmusic,studentshavetolistentogoodmusicand

theyhavetohavealotofexperiencemakinggoodmusic.Mostgraduateprogramsprovide

studentswiththeequivalentoflisteningtomusic.WhenIwasanewlymintedPh.D.Iheard

BruceBuenodeMesquitagivealectureattheHooverSummerPrograminGameTheoryand

InternationalPoliticsatStanfordUniversity.Hebuiltagametheoreticmodelbasedonthe

assumptionsofhegemonicstabilitytheory–seeminglyontheflybasedoncommentsshouted

outbymyclassmates.Ihadanepiphany.Ofcourse,ifdevelopingsocialscientific

explanationsisanart,thenitmustbetaughtastheartsaretaught!Iwaswatchingthemaster

attheeasel–engagedintheverycraftIwastryingtolearn.Itsuddenlyoccurredtomethat

muchofmygraduatetrainingamountedtotheequivalentofsittinginaroomlisteningto

recordingsofmusic,thenwhenitwastimetowritemydissertationitwasasifadoorhadbeen

flungopen,IwashandedaninstrumentIhadneverplayed(Iimaginedacello)andpushedout

ontoastagewhereIwasexpectedtoperform.Mostgraduateprogramsinpoliticalscience

teachpeopletheequivalentofplayingscalesinmethodsclassesandmusichistoryor

appreciationinsubstantiveclassesandarelefttofigureoutontheirownhowtoputthis

togethertomakemusic.

Themissingpieceinmostofourgraduateeducationiswhatmusicianscall“etudes.”

Theseareexercisesdesignedtobemusic-like(sostudentscanbegintothinkabout

interpretationandexpression)butareartificiallydesignedtoallowforadegreeofrepetitionof

particulartechniques(articulation,vibrato,dexterity)thatallowsthoseskillsnecessaryfor

musicalexpressiontoseepintothestudent’smusclememory.Manydoctoralprograms

emphasizethatstudentsshouldwritepublishablepapers,butIbelievethatsuccessisunlikelyif

32

thisisattemptedbeforestudentshaveengagedinmanyrepeatedattemptstoexplainthingsor

thinkaboutwhatobservationsareimpliedbytheirexplanations.Studentsneedtopractice

askingandansweringthefivequestionsoutlinedaboveandwritingasinglepaperineach

seminardoesnotgivethemthe“reps”todevelopmusclememory.Virtuallynoskillworthyof

thenamecanbedevelopedafteradozenorsoattempts.

Consequently,Ihavearguedthatproblemsetsin“substantiveclasses”canhelp

studentsbecomeproficientataskingandansweringthequestionsthatwillmakeforinnovative

research.Ananalogytothevisualartsmightbeuseful.Whenstudentsarelearningtodraw,

theyarenothandedablanksheetofpaperandtoldto“thinkofsomethinginterestingtodraw,

thatnooneelsehasdrawn.”Rather,abowloffruit,orperhapsawoodenmodelofahuman

figureisplacedonatable.Then,everyoneintheclassdrawsthesamethingafterreceiving

instructionfromtheinstructorabouthowtodoso.Incontrast,manypoliticalscience

departmentsdotheequivalentofhandingtheirstudentsablanksheetofpaperandaretoldto

“drawsomethinginteresting.”Problemsetsinsubstantiveclassescanbetheequivalentofa

bowloffruit.Theinstructorcanassignstudentstoaquestionrelatedtoaparticularresearch

area.“ExplainwhyXoccursunderZcircumstances.”“IfPexplainsY,whatelseoughtwe

observe?”“WhyisQaninterestingquestion?”“DoesFigure2countasconfirmingor

disconfirmingevidenceforhypothesis2,andwhy?”

Studentsneedalotofexperience“makingmusic”beforethey“havesomethingtosay.”

Iftheanalogytotheartsdoesnotresonatewithyou,considerthefollowing.Politicalscience

andinternationalrelationscantakealessonfromtheso-called“benchsciences”where

studentsworkonmanyprojectsasmembersoflargeteamsbeforetheyaretaskedwiththe

33

responsibilityofdecidingonthetopicofthegroup’snextproject.Experienceandrepetition

helpsstudentslearnwhatworksandwhatdoesnot.

Whilegraduatepedagogyisimportantforstimulatingcreativequestionaskingand

answering,thebroaderclimateandculturewecreateinourdepartmentsandresearchcenters

isequallyimportant.Inparticular,itisextremelyimportanttocreateanenvironmentwhereit

issafetoplaywithideasandchallengeorthodoxy.Ioncehadacolleaguewho,whilewalking

downthehallreadapassagefromabookthathethoughtwasincorrectandloudlydeclared

theauthoran“idiot.”Creativityandrisktakingisnotencouragedbyaculturethatsuggests

thatonlystupidpeoplesaystupidthings.Instead,itisimportanttocreatetheideathatthe

smartestamongusarecapableoferrorandthatthereisabigdifferencebetweensaying

somethingthatstupidandbeingstupid.Tothatend,Ithinkitisextremelyimportantforsenior

scholarstobetransparentabouttheerrorstheyhavemade.Youngscholarsneedtolearnthat

ifthey’vemadeamistake,theyareinverygoodcompanyandiftherequirementforadmission

wasnevermakingamistakethebuildingwouldbeempty.

Whileacultureofsupportforindividualrisktakingisvitaltoanyscientificorartistic

community,thereisanoptimaldegreeofindividualismbehindscientificdiscovery.Ifyoudon’t

readwhateverybodyelsereadsandfailtotrainlikeeveryoneelsetrains,youwillasknaïve

questionsthattherestofyourcommunityknowstheanswersto.Butifyouonlyreadwhat

everyreads,andonlytrainlikeeveryoneelsetrains,youareunlikelytoexperiencethat

momentwhenyouseesomethingthathasgonewrongthatnooneelsesees.

JazzbassistScottLaFarostartedplayingthebassin1954whenhewas19yearsoldand

inthefewshortyearsbeforehewaskilledinatragiccaraccidentin1961,hecompletely

34

changedtheworld’sconceptionofwhatcouldbeaccomplishedonadoublebassandwhatrole

theinstrumentcouldplayinapianotrio.Priortoplayingthebasshehadplayedtheclarinet

andsaxophoneforyearsandmanyhaveattributedhisphenomenaltechnicalprowesstothe

factthathepracticedthebassbyplayingetudescomposedfortheclarinetbyHyacintheKlosĕ

inthe19thcentury(LaFaro-Fernandez,2009).ThelessonLaFarotaughttheworld,inadditionto

thegeneralbenefitsofinter-disciplinarily,was“ifyouwanttosoundlikeeveryoneelse,

practicelikeeveryoneelse;butifyouwanttosoundlikeno-oneelse,practicelikenoone

else.”17

Justasthereisanoptimaldegreeofindividualitythatislikelytoproducescholarswith

theskill,wit,andgeniustodeterminewhensomethinghasgonewronginwaysthatmayprove

consequential,communitiesthatstriketherightbalancebetweenconformityanddiversityare

likelytoencouragethehabitsthatleadtoscientificbreakthrough.

Ontheonehand,itisimportantforascientificcommunitytoshareacommitmentto

thegrowthanddisseminationofknowledgeandacommonunderstandingofthelogicof

inferenceandthestandardsofevidence.Withoutthissharedunderstanding,criticismislikely

tofallondeafears.Butontheotherhanditisimportantforacommunitytobeasdiverseand

eclecticaspossible.Peoplefromdifferentcultural,class,linguistic,andreligiousbackgrounds

arelikelytoseethesocialworlddifferentlybecausetheyarelikelytohavehaddifferent

experiences.Thesedifferentexperiencesarelikelytoleadtodiversemoral,political,andsocial

intuitionsthatleadthemtoraisequestionsthatamorehomogeneousgroupmightnot(Page,

2007).17Atthesametime,nearlyeveryinnovativejazzmusicianlearnedtheircraftbymemorizingperformancesofmusiciansthatcamebeforethem.

35

Inaddition,diversegroupsarelesslikelytofallpreytowhatIcall“strategic

confirmationbias.”Confirmationbiasoccurswhenanindividualembracesanideauncritically

becauseitconformstotheirpriorbeliefs.Whenconfirmationbiasisatwork,peopleareless

likelytoscrutinizetheresearchpracticesthatproducedtheclaiminquestion.Theyareless

likelytolookforconfounds,toaskaboutthedetailsofdatacollection,ortothinkcritically

abouteitherthemicro-foundationsormoralimplicationsofaclaimbecausetheresultsconfirm

whattheyhavelongsuspectedabouttheworld.

Butstrategicconfirmationbiasoccurswhenanindividualisabletoovercomefirst-order

confirmationbiasandthinkcriticallyabouttheclaimbeingmade,butisdeterredfromvoicing

thecriticismbecausetheybelieveothersarerefrainingfromcriticismasaresultof

confirmationbias.Undersuchcircumstances,criticallyengagingtheclaiminpublicmightsignal

toothersthatthecriticdoesnotsharetheirbeliefsonthematter.

Strategicconfirmationbiasismostlikelytobeaproblemincommunitieswhere

“everybody”sharesparticularbeliefs.Insuchanenvironment,thinkingcriticallyaboutaresult

thatconfirmsthecommunity’sbeliefscouldresultinostracism,orattheveryleast,fewer

dinnerinvitations.Acommunitycomprisedofindividualsfromdiverseeducational,class,

religious,andideologicalbackgroundsislesslikelytoproducethekindofmonolithicviewsthat

encouragestrategicconfirmationbias.Individualsaremorelikelytosaysomethingwhenthey

seesomethingwrongthatmayproveconsequentialbecausethesetoftakenforgranted

sharedbeliefsislikelytobesmaller.Diversityismostlikelytobehelpfulinthisregardwhen

themultipledimensionsofidentityarerelativelyuncorrelated.Ifgender,race,orideologyare

heavilycorrelated,thendissentononedimensioncanbeseenasdefectiononanother.Thus,

36

inidealcircumstancescommunitieswouldhaveasmuchwithingroupdiversityasbetween

groupdiversity.18Ofcourse,diversityhastobesufficientlydevelopedtogiveindividuals

confidencethatspeakingupundersuchcircumstanceswillnotsimplyconfirmthatoneisan

“outsider.”Ifacommunitypromulgatesthenormthatinamultidimensionalspaceweareall,

ononedimensionoranother,outsiders,thecostofrevealingthatone“thinksdifferently”

aboutsomethingislikelytobelesscostly.Thedauntingthingaboutstrategicconfirmationbias

isthatitismostlylikelytooccuraroundissuesscholarsfeelpassionateabout.Asaresult,

thereisadangerthataresearchcommunitywillbeleastscientificaboutthemattersthatit

caresmostdeeplyaboutandmostscientificaboutmattersitsparticipantsviewaslargely

inconsequential.

CONCLUSION

Goodscientistsaskinterestingquestionsandareunsatisfied,evenimpatient,withbad

answers.Ihavearguedthatmostworkinpoliticalscienceandinternationalrelationscanbe

understoodthroughthelensoffivequestionsandthatcontributionscanbemadetothe

literaturebyimprovingonaresearchcommunity’sanswertoanyofthefivequestions.

Sincecomingupwithbetteranswerstoquestionsisasmuchart,asitisscience,Ihave

arguedthatthebestwaytotraingoodsocialscientistsistolearnfromthewayartistsare

trained.Musicalandvisualartistslearntheircraftsthroughstructuredrepetitivepractice.

Theimplicationofthisinsightforthesocialsciencesisthatscholarsshouldbegivenmaterials

toworkwiththatallowthemtoengageinthedailypracticeofaskingandansweringthefive18Theconnectionbetween“intersectionality”andcross-cuttingcleavagesshouldbeexploredfurther.

37

questionsoutlinedinthefirstsectionofthepaper.Ihavesuggestedthatthebestwayto

encouragethisisthroughtheuseofproblemsetsinoursubstantivecourses.Ihavealsohinted

thattherearegreatbenefitstointer-disciplinarity.Bybringhabits,techniques,andinsights

thatarenormalinonedisciplinetoasettingwheretheyarerare,individualsaremorelikelyto

recognizewhensomethinghas“gonewronginwaysthatmayproveconsequential.”Finally,I

havearguedthatdiversecommunitiesaremorelikelytoproducegoodquestionaskers,inpart

becausetheyarelesslikelytofallpreytostrategicconfirmationbias.

38

References

Brambor, Thomas, William Roberts Clark and Matt Golder. 2006. “Understanding Interaction Models:

Improving Empirical Analysis” Political Analysis. 14(1): 63-82.

Cindy D. Kam and Franzese, Robert J. Jr. 2007. Modeling and Interpreting Interactive Hypotheses in

Regression Analysis. (Ann Arbor: University of Michigan Press).

Friedman, David, Robert Pisani and Roger Purves. 2007. (New York: W.W. Norton).

Geddes, Barbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in

Comparative Politics.

Hellwig, Timoth T, Eve M. Ringsmuth, and John R. Freeman. 2008. “The America Public and the Room

to Maneuver: Responsibility Attributions and Policy Efficacy in an Era of Globalization.” International

Studies Quarterly 52(4).

Huntington, Samuel. 1968. Political Order in Changing Societies. (New Haven: Yale University Press).

King, Gary, Robert O. Keohane, and Sidney Verba. 1994. (Princeton: Princeton University Press).

King, Gary and Langche Zeng. 2005. “The Dangers of Extreme Counterfactuals,” Political Analysis

14:131-159.

Kuhn, Thomas S. 1962. “Historical Structure of Scientific Discovery,” Science 136(3518):760-764.

Lafaro-Fernandez, Helene. 2009. Jade Visions: The Life and Music of Scott LaFaro. (Denton, Tx:

University of North Texas Press)

Lave, Charles A. and James G. March. 1975. (New York: Harper and Row).

Lieberson. Stanley. 1991. “Small N’s and Big Conclusions: An Examination of the Reasoning in

Comparative Studies Based on a Small Number of Cases.” Social Forces 70: 307-320.

39

Marx, Karl. 1888. “Theses on Feuerbach” in Friedrich Engels, editor, Ludwig Feuerbach and the End

of Classical German Philosophy.

Marx, Karl and Friedrich Engels. 1996. The Communist Manifesto. (London: Pluto Press).

Page, Scott. E. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools,

and Societies. (Princeton: Princeton University Press).

Pearson, K. and Lee, A. (1903). On the laws of inheritance in man: I. Inheritance of physical characters. Biometika, 2(4), 357-462.

Popper, Sir Karl. 2003 [1959]. The Logic of Scientific Discovery. New York: Routledge.

Popper, Sir Karl. 1962. Conjectures and Refutations: The Growth of Scientific Knowledge. New York:

Basic Books.

Rauch, Jonathan. 2018. The Happiness Curve: Why Life Gets Better After 50. (New York: St. Martin’s

Press).

Skocpol, Theda. 1979. States and Social Revolutions: A Comparative Analysis of France, Russia, &

China. New York: Cambridge University Press.