Scienti–c Research Measurespro1.unibz.it/projects/research_evaluation/FRITTELLI.pdf · Workshop on Research Evaluation Scienti–c Research Measures Marco Frittelli, Univ. Milano

Scientic Research MeasuresJoint with Loriano Mancini and Ilaria Peri

Marco Frittelli

Workshop on Research EvaluationMay 10, 2013 Bolzano, Italy

Workshop on Research Evaluation () Scientic Research Measures Marco Frittelli, Univ. Milano 1 / 43

Outline

On bibliometric indices

From Risk Measures to Research Measure

On Scientic Research Measures (SRM)

The dual approach

Empirical results

Idea: We allocate the citation records of a scientist in merit familiesdetermined by performance functions and we measure the research qualityas a quasi-concave "research (risk) measure"


On the valuation of the scientic research

Scope of the valuation of the scientic research:

Provide a updated picture of the existing research activity, in order toallocate nancial resources in relation to the scientic quality andscientic production.

Increase the quality of the scientic research (of the structure)

Method:

Systematic (yearly) use of bibliometric indices

Peer review (on a multiple years base) that may control, harmonize,and tune the outcome based on bibliometric indices

The output is the classication of authors (and structures) into fewmerit classes of homogeneous research quality (not a ne ranking).

Problem: Determine a good bibliometric index in accordance with boththe above mentioned aims.Workshop on Research Evaluation () Scientic Research Measures Marco Frittelli, Univ. Milano 3 / 43

Scientic Research Measure (SRM)

We propose a family of Scientic Research Measures that are

exible to t peculiarities of di¤erent:

areasages

Calibrated to the scientic communityCoherent,

as they share the same structural properties (axiomatic approach)as they take into consideration the whole citation record.

Inclusive, as they comprehends several popular indices.


On the valuation of scientic research

The methodologies for the valuation can be divided into two categories:

content valuation, based on:

internal judgments committee;external reviews of peer panels;

context valuation, based on:

bibliometrics (i.e. statistics derived from citation data);characteristics of the Journals associated to the publications.


Content evaluation (peer review): Pros and Cons

Pros:

e¤ective assessment of the quality of the research;

Cons:

expensive, in term of time and people involved: It can not be usedsystematically.subjective, since the result depends on the referees: do they operateproperly, are they competent and reliable ?Problem with the choice of the referees.non-uniformity of the judgment, as each evaluator has a personalscaling preferences leading to di¤erent ranking (specially in di¤erentareas).very problematic, when the number of assessments is large (200.000research products are presently under evaluation by the VQR2004-2010)


Context evaluation: bibliometric indices

Pros:

Easily accessible, from the online databases (Google Scholar, ISI Web,MathSciNet, ...);Not expensive: can be used systematically, especially if tested - everyn years - with peer review.Can be implemented even with a large number of research productsQuick to computeObjective



Cons:

Subjective interpretation of citations, (it can be more subjectivethan the judgment of experts - see Citation Statistics Report of theInternational Mathematical Union (2008).

The new metric must be validated against other (possibly non metric)criterion already validated.It has been pointed out that citation metrics are extremely correlatedwith peer reviews.

Improper comparison of papers belonging to di¤erent elds, since thesize of the scientic communities are di¤erent.

The SRM will only be used to rank each author inside his scienticcommunity (e.g.: top 10% - top 30% - average...). This however,allows the comparison among di¤erent areas.In the Italian system this correspond to rank all researchers (in an ageinterval, or seniority) in the same scientic eld (SSD).Not absolute values, but relative - to elds - values



Cons:

Improper comparison of papers having di¤erent ages.

Our SRM may be calibrated to di¤erent ages (as well as di¤erentareas).

Di¤erent databases provide di¤erent citations.

Many areas (naturally) share the same database.The outcome of the scientic measure is in relative terms: the rankingof one author is compared with the ranking of all researchers in thesame area (hence using the same database).Di¤erent databases (google scholar, MathSciNet,...) provide di¤erentnumbers (in terms of citation of each paper) but maintain - more orless - the overall ranking.



Cons:Co-authors

It is possible to normalize the citation numbers per each single author.For some elds (where papers have typically many co-authors) this maybe problematic.

Incorrect citations attributed to an author and self citationsBoth problems can be easily addressed by the systematic use of AuthorCodes (a code that identify the author).

A single number is insu¢ cient for the evaluation of a complexfeature, such as scientic research.

We agree: It is necessary to nd multiple metrics (including time-basedmetrics). We propose one of them.This argument should not lead to abandon the search of appropriatemultiple metrics.

Quality of the scientic research can not be reduced to citationsAgree: it is only one component that however should be properlyquantied.



Cons:

Negative credit: citations may be attributed not as reward citations(to give credit to the work of the cited author) but as negative credit(or rhetorical creditdue to the prestige of the cited author).

True. Many are the motivations of citations and they varies amongauthors: they do not always reects reward, but certainly a largepercentage of citations are credit ones. Indeed:The fact that citation based statistics often agree with other validatedform of valuation (peer review) suggest that, to some degree, thesemetrics indeed reects the impact of the authors research.The periodical peer review valuation should point out the macroscopicexceptions to reward citations (papers mostly cited for their fallacy).

Disincentive for young researcher to study subjects more innovativebut less popular

True, even though this could be compensated by the consideration thatinnovative paper (in a new eld) typically receive many citations.



Cons

Negative Implications: The use of citation based metrics will increasethe number of citations (and improper ones).

The abuse of citations is comparable with intentional misjudgment byreferee: unfortunately this is always possible.When citations number are high (in the order of hundreds) it is di¢ cultto modify the citation records with self or friendly citations.It is not completely unfair that a strong scientic group (capable toproduce a large number of published papers) receives additional credit(due to potential additional citations from the group).


Conclusion on the use of citations

Citation based indices can be used as one component of thevaluation of the quality of the scientic research.

Going upstream, to be against the increased use of these indices, maybe not benecial. I prefer to work for an approach that could befruitful in the end.

We propose bibliometric index in accordance with both the scopesoutlined in the introduction.


From Risk Measures to Research Measures

(Ω,F ,P) is a probability space.L(Ω) is a linear subspace of L0(Ω,F ,P), the space of all P.a.s.nite random variables.

L∞ L(Ω) L0.

Each random variable x 2 L(Ω) represents the payo¤ of a nancialposition and

ρ : L(Ω)! R

measures its risk.


Monetary Risk Measure

Denition

A map ρ : L(Ω)! R is called a monetary risk measure on L(Ω) if it hasthe following properties:

(CA) Cash additivity :8 x 2 L(Ω) and 8 c 2 R ρ(x + c) = ρ(x) c .

(M) Monotonicity : ρ(x) ρ(y) 8 x , y 2 L(Ω) such that x y .(G ) Grounded property : ρ(0) = 0.


Acceptance set

Denition

Let ρ : L(Ω)! R be a monetary risk measure. The set of acceptablepositions is dened as

Aρ := fx 2 L(Ω) : ρ(x) 0g := A0.

Remark

ρ : L(Ω)! R satises cash additivity (CA) if and only if there exists a setA L(Ω) :

ρ(x) = inffa 2 R : a+ x 2 Ag.


Coherent and Convex Risk Measures

A convex risk measure is a monetary risk measure which satises:

(CO) Convexity: ρ(λx + (1 λ)y) λρ(x) + (1 λ)ρ(y),λ 2 [0, 1], x , y 2 L(Ω)

Notice that (CO) (and ρ(0) = 0) implies:

ρ(λx) λρ(x) for all λ 2 [0, 1]ρ(λx) λρ(x) for all λ 1.

A coherent risk measure is a convex risk measure that satises:

(SA) Subadditivity : ρ(x + y) ρ(x) + ρ(y) 8 x , y 2 L(Ω).(PH) Positive homogeneity: ρ(λx) = λρ(x) 8 λ 0.


Robust representation of monetary risk measures

Coherent Risk Measures (Artzner, Delbaen, Eber, Heath (1997))

ρ(X ) = supQ2P 0

EQ [X ]

whereP 0 P := fQ << P, Q probabilityg

Convex Risk Measures (Follmer-Schied (2002) and F.-Rosazza (2002))

ρ(X ) = supQ2P

fEQ [X ] α(Q)g

where α is the penalty function α : P ! [0,∞].


Diversication = Quasiconvexity

Let λ 2 R, 0 λ 1The convexity of ρ : LF ! R implies

ρ(λX + (1 λ)Y ) λρ(X ) + (1 λ)ρ(Y ) ρ(X ) _ ρ(Y ).

Quasiconvexity alone:

ρ(λX + (1 λ)Y ) ρ(X ) _ ρ(Y )

allows to control the risk of a diversied position.

As pointed out in Cerreia-Vioglio, Maccheroni, Marinacci Montrucchio[CVMMM09], the principle that diversication should not increase therisk has the mathematical counterpart in QCO, not in convexity .


Acceptability Indices (Cherny - Madan (2009) )

DenitionA map Φ : LF ! [0,∞] satisfying Quasiconcavity, MON ("), ScaleInvariance and the Fatou property is called Acceptability Index.

Quoting (Cherny-Madan (2009)):

For a risk measure, all the positions are split in two classes:acceptable and not acceptable.

In contrast, for an acceptability index we have a whole continuum ofdegrees of acceptability dened by the system (Am), m 2 R, and theindex measures the degree of acceptability of a trade.


Acceptability Indices and associated acceptability family

Let (Am), m 2 R, be a collection of subsets Am LF such that1 Am is convex, for any m2 Am # with respect to m3 Am is monotone (Y X 2 Am ) Y 2 Am), for any m

ThenΦ(X ) := sup fm 2 R jX 2 Amg

is MON (") and Quasi-Concave.Viceversa, to any MON (") and Quasi-Concave map Φ : LF ! R we mayassociate the acceptance set at level m:

AmΦ := fX 2 LF jΦ(X ) mg

and (AmΦ) is an acceptability family.


Risk Measures - Scientic Research Measures

Structural approach

Diversication (Quasi-convexity, Quasi-concavity)

Acceptance sets

Calibration to the nancial market - calibration to the citation records

Dual approaches


On our Scientic Research Measure

We associate to each author a vector of citations X , where

each component of X represents the number of citation of the n thpublication,

the components of X are ranked in decreasing order

Notice: We will evaluate each single author based on his/her citationrecord. The valuation of the structures (Dept. Colleges...) will be obtainedby the aggregation of the outcomes of all the members of the structures.


Example of Author citations


Scientic Research Measures

Let X be the set of all authors X ;

We consider a family of performance curves F := ffqgq2R monotoneincreasing over q and the associated family of performance setsAF := fAqgq2R:

Aq := fX 2 X j X fqg

that are convex and monotone;

We can construct the index

φF(X ) := sup fq 2 R j X 2 Aqg

that is a quasi-concave and increasing monotone map.


Example of one Scientic Research Measure: the h index

h(X ) = sup fq 2 R : X fqg = 4


Three popular examples of IndicesThat are particular cases of our Scientic Research Measure

1 Max # of Cit: The number of citations of the publication withlargest number of citations.

2 h index: The largest integer number h satisfying: X has hpublications, each one having at least h citations.

3 Max # Pub: The number of cited publications (the number ofpublications that received at least one citation).


Examples of families of performance curves for the threeindices

Max # Cit h Index Max # Pub


The SRM of an Author


On the SRM

The index φF depends on the family of performance curvesF := ffqgq2R, which depends on the scientic area and the age(seniority) under consideration.

However, all indices φF share the same structural properties.Additional properties of the index φF can be built in fromcorresponding properties of the family of performance curves F, or ofthe family of acceptance sets AF: for example

φF(X + 1N ) φF(X ) + 1

Continuity properties.


Calibration to scientic areas

Each scientic area should determine their own family F := ffqg byusing existing data:

from a large group of researchers in the same area,from a set of well known established scientic expertresearchers inthe same area.

This family will then reect the characteristics of the citation recordsof that particular scientic community.

Keep in mind that this SRM is used only in relative terms (tocompare the author quality with respect to the other researchers inthe same area).


Calibration to age

Similarly, in each area, it could be possible to determine n (two/three) families of performance curves Fi :=

f iq, i = 1, ...n, that

correspond to di¤erent ages (seniority in research), each determiningn indices φiF that could be used to compute the scientic researchquality of authors of the same seniority.

A more advanced aspect is to calibrate the time evolution of theperformance curves, in order to capture also the scientic productivity.


Citation curves of several authors in Math Finance area


Continues...


Final remarks on calibration

It is evident the importance of the balance between:

the number of citations of the group of highest cited publications andthe total number of publications that received some citations.

This balance depends on the scientic areas and is captured by thefamily of performance curves F := ffqgThe shape of the performance curves ffqg is evident form this data

The exact functional formulation is the calibration problemmentioned from the beginning

The performance curves of the popular h index seems quiteinappropriate, in this case as well as in most cases.

Other bibliometric indices (the popular h-index as well as many othercomplementary indices) do not take into consideration the wholecitation record, Instead, we are comparing the whole citationrecord with the family of citation curves.


Comparison between the h index and our index

M5YZXD0E


On the dual representation of the SRM

TheoremEach (upper semicontinuous) SRM φ : L∞ ! R can be represented as

φ(X ) = infQP

P(Q,EQ (X ))

where P : P R ! R is dened by

P(Q,EQ (X )) := sup fβ 2 R j EQ (X ) γ(Q, β)g

and γ : P R ! R is given by:

γ(Q, β) := inf fEQ (X ) j φ(X ) βg , β 2 R.


Question: Which is the duality in this model ?

The set of all possible events is the set of all Journals (of a givenscientic eld)

The primal space is formed by the vectors (random variables) ofcitations of all authors

The dual space is formed by all the "Arrow Debreu Price" of eachJournal, i.e. all the possible values of each Journal, (for example,these values could be the Impact factors).


Alternative dual approach to SRM

How do we build a SRM ?

Fix a family Q of probabilities Q and a family γ(Q, β), β 2 R.

Q represents the values attributed to the journals

EQ (X ) is the average citation of author X weighted under Q

γ(Q, β) the citations average (under Q) for index β

A specic Q could attribute more value (importance) to a set of topjournals: However, a priori there will be no consensus about this selection:

Hence a robust approach, i.e. the selection of a reasonable set Q ofprobabilities Q.


Building a SRM via duality

For a xed probability Q,γ(Q, β) is the smallest citation Qaverage of authors with indexgreater than β.

For xed Q and X ,P(Q,EQ (X )) is the maximum level β such that the citation averageEQ (X ) is greater than γ(Q, β):

P(Q,EQ (X )) := sup fβ 2 R j EQ (X ) γ(Q, β)g

To dene the SRM φ, we take the worst case approach:

φ(X ) := infQ2Q

P(Q,EQ (X )),

that is the dual representation of a quasi concave monotone map.


Empirical results

We select 173 full professors a¢ liated to 11 main US businessSchools.

We extract their citations from Google Scholar: the dataset is createdon Sept. 6, 2012.

We use the following SRM to rank the authors

Φ(X ) = supnq 2 R j X (x) q

xc, x 2 [1, p]

owhere the parameter c > 0 is estimated from the data.


CitationsFull name Faculty Avg Med Std

Berkeley Haas School of Business 8 90 21 179Chicago Chicago Booth 17 279 68 562Columbia Columbia Business School 34 121 40 237Cornell Johnson 8 72 23 142Harvard Harvard Business School 15 152 45 283NYU Stern School of Business 26 110 27 285Penn. Wharton, University of Pennsylvania 21 138 48 267Princeton Bendheim Center for Finance 13 127 37 268Stanford Graduate School of Business 11 131 52 226UCLA Anderson School of Management 9 145 39 281Yale Yale School of Management 11 161 55 333

Total 173

Table: Faculty = number of full professors in each Finance Dept.


Estimate t-statistic R2

log(q) c log(q) cBerkeley 8.5 1.8 20.8 14.5 86.5Chicago 10.2 2.0 19.7 13.0 82.7Columbia 8.8 1.7 22.9 14.1 84.3Cornell 8.1 1.6 31.3 21.1 87.3Harvard 8.7 1.9 17.5 11.0 83.3NYU 8.9 1.7 21.3 14.0 83.8Penn. 8.6 1.7 29.1 15.5 84.0Princeton 9.2 1.6 35.9 20.4 87.8Stanford 8.3 1.8 14.0 8.5 78.4UCLA 9.3 1.8 17.0 11.1 80.8Yale 9.0 1.8 20.6 13.4 83.1

Table: Estimates of hyperbolic functions. The hyperbolic function isfq(x) = q/xc is tted in the log-log space of citations and number ofpublications. For each authors citation curve, the functionlog(fq(x)) = log(q) c log(x) is tted using least square regression.


φ h h2 hα w A R gAvg Med Avg Med Avg Med Avg Med Avg Med Avg Med Avg Med Avg Med

Berkeley 653 642 27 25 10 10 35 32 47 41 185 160 69 65 56 44Chicago 1766 1440 41 38 16 15 48 42 66 60 479 331 135 117 83 85Columbia 1262 704 38 30 12 11 46 36 63 50 229 186 88 79 73 57Cornell 980 904 32 29 10 11 43 36 57 51 171 146 73 71 69 56Harvard 660 501 28 23 11 10 33 26 45 34 247 239 79 68 48 44NYU 1315 815 33 31 12 12 40 38 57 52 238 186 85 81 71 67Penn. 1021 616 31 25 11 10 39 32 52 42 235 174 79 64 58 52Princeton 2050 929 51 49 14 13 67 62 91 82 296 278 119 94 106 82Stanford 742 401 23 19 11 9 28 20 37 28 219 218 70 61 43 31UCLA 1491 1724 37 34 13 15 45 40 63 53 286 320 100 113 76 61Yale 862 615 30 29 12 11 36 33 49 44 274 244 85 84 59 54Research measures. For each index, Avg and Med are average and medianof the index of all authors in a given university. The dataset of citationscovers 173 authors and is from Google Scholar on September 6, 2012.


Epilogo

At present Im involved in the assessment of the Italian researchquality as a Panel member of the (ANVUR), the:National Agency for the Valuation of Universities and ResearchInstitutes (VQR2004-2010).

My panel will evaluate more than 15.000 research articles (3 for eachone of the 5,000 scientists in the eld of statistics and appliedmathematics to social science, economics and nance) and ...

... we really need a robust method for the valuation.

Thank You


Epilogo




Thank You


Epilogo




Thank You


Epilogo




Thank You


Documents

Scienti–c Research Measurespro1.unibz.it/projects/research_evaluation/FRITTELLI.pdf · Workshop on Research Evaluation Scienti–c Research Measures Marco Frittelli, Univ. Milano