Edward J. Hackett Edward J. Hackett May 2012May 2012

http://upload.wikimedia.org/wikipedia/commons/5/52/Henry_Oldenburg.jpg

Boyle: science as avocation

Weber: science a vocation

Albert: “it’s a job.”

NSF invests in the best ideas from the most capable people,

determined by competitive merit review.

The intellectual merit of the proposed activity Creativity and originality and transformative potential Potential to advancing knowledge and understanding

within and across fields Conceptualization and organization Qualifications of investigators Access to resources

The broader impacts of the proposed activity Discovery while promoting teaching, training and

learning Participation of underrepresented groups Enhancement of infrastructure for research and

education Dissemination of results to enhance scientific and

technological understanding Benefits to society

The context of peer reviewAn institutional analysis of peer review

What does it do?Intended and unintended consequences

What values or principles guide or result?Values compete: ambivalence

… Let’s set the stage with some alternatives to peer review….

Legislators may allocate fundsEarmarking and Pork Barrelling

+ Democratic+ Legitimate+ Distributional fairness- “Political”- Inexpert- Culturally corrosive About $2.25B earmarked in FY08

Strong Manager (DARPA)+ Flexible and responsive+ Expert0 Assumes clear objectives and standards0 Requires outcome accountability0 May not work for all aims or fields - Projects end, programs sustain fields- May not scale up (DARPA $3B NIH $31B)- Must accept failure and cut losses

Formula funding$$= Who writes the formula, in what terms?$$ to states or institutions or departments?

Then merit review? Another formula?

Will it encourage creativity and responsiveness?

How to start and complete careers?Gaming and unintended outcomes

There are alternatives and hybridsNSF made its choice at “birth,” has

adapted over time, and remains committed

Peer review is a transducer that operates across organizational boundaries to convert one form of energy into another.

Peer review is:

That is, for allocating scarce resources, of ¥¥ or $$, pages, attention, honor

At NIH reviewers are asked to evaluate the science, the whole science, and nothing but the science of a proposal

Substantive advice to the author

Strategic advice to program (journal) for wise allocations of resources

Cumulatively, shapes the research area

Stores energy, challenging new ideas to prove that they are original & sound

Imparts momentum to research that helps researchers “stay the course” through bumpy patches

Ideas circulate during review

preparing acceptance accelerating uptakeavoiding (some) duplicationreducing prematurity?

Distinguishes science from other government activities

Insulates decisions from politics & fads.

Symbolic power (talisman)

Broader impacts criteria at NIH and NSF—which may be given differential weight in different decisions

Program offices balance portfolios (gender, ethnicity, youth, geography, college/university type [HBCU, undergrad, EPSCOR])

Proposals and manuscripts acquire legitimacy and credibility by passing through peer review.

Reciprocally, the peer review process acquires legitimacy and credibility through its workings and its outcomes.

Purposes reflect diverse social values.

Values are durable and shared standards of goodness, truth, beauty

“Ambivalence” = values in tension.Criticisms of peer review overlook the depth and inconsistency of these values

Openness-SecrecyTrust-SkepticismEfficacy-EfficiencySensitivity-SelectivityTransformative-InertialAutonomous-AccountableMeritocratic-FairRigorous-ResponsiveReliable-Valid

Referring to S’s discussion, M mentioned a paper he did on birds. They had a problem because they did not look at enough decimal places [measure with enough precision]. Maybe this is S’s problem?

S talks about a small mammal she studied and rejects M’s suggestion. M asks if she is differentiating between 10 and 11 grams; says it could be a matter of [too few] significant

figures. S rejects M’s proposal, says more about heritability among small mammals, then makes a connection to

heritability of traits in giraffes. … M then says it might be geographic. S [holding up her pen] says, “I’m going to stab you with this!” M says he is just doing his job.

S continues to discuss the specifics of this paper and a related analysis, reasserting the claim that body size is not inherited among small mammals.

A asks about error in the calculation. S says it doesn’t matter. A pushes the issue, saying there might be error in sampling. M notes that standard errors go up as body size increases …. S replies that they have tried this….look in the appendix [to the paper] and see very clearly that what he is

suggesting is not the case. M says there might be another statistical artifact…. S says, “We’re focusing on the wrong thing…look at this.” … … B says loudly, “But you can’t … [do that, claim that]! This is the argument that I had with [a name]!” He

slams his fist. “This doesn’t have any basis!” J says, “… You need to look at the phylogenetics of the group.” S says defensively, “It depends on what you are looking at.” J says, also defensively, “That’s fine! I gave you all this data.” S replies that she didn’t know where J had got this data, so she didn’t use it.” J says that he got the data from the ____ project. B says, “He did the calculations wrong!”

Peer review is integral to the research process, not a filter or screen or gate that precedes or follows research.

Arguments are construed and constructed—interpreted for present purposes and built for transport

Articulation work, after Joan Fujimura

Policies and procedures for peer review are a score without an orchestra:Depends on a community and cultureBoundary process=diverse actorsWhere transformative research and societal

impacts are created (not merely found)Science agencies as learning organizations

Build an experimental platform & use it!

Scrutiny, accountability (e.g., ARRA)Declining awe & authority (Coburn

Report)Multi-Inter-Cross-Trans-disciplinesTighter coupling to social purposesCitizen participationPressure ($$) beyond resolving power

(VPR)International harmonization & culture