Upload
others
View
34
Download
0
Embed Size (px)
Citation preview
Multiparametric characterization of scientometricperformance profiles assisted by neural networks: a studyof Mexican higher education institutions
Elio Atenogenes Villasenor1 • Ricardo Arencibia-Jorge2 •
Humberto Carrillo-Calvet3
Received: 13 April 2016 / Published online: 11 November 2016� Akademiai Kiado, Budapest, Hungary 2016
Abstract Development of accurate systems to assess academic research performance is an
essential topic in national science agendas around the world. Providing quantitative ele-
ments such as scientometric rankings and indicators have contributed to measure prestige
and excellence of universities, but more sophisticated computational tools are seldom
exploited. We compare the evolution of Mexican scientific production in Scopus and the
Web of Science as well as Mexico’s scientific productivity in relation to the growth of the
National Researchers System of Mexico is analyzed. As a main analysis tool we introduce
an artificial intelligence procedure based on self-organizing neural networks. The neural
network technique proves to be a worthy scientometric data mining and visualization tool
which automatically carries out multiparametric scientometric characterizations of the
production profiles of the 50 most productive Mexican Higher Education Institutions (in
Scopus database). With this procedure we automatically identify and visually depict
clusters of institutions that share similar bibliometric profiles in bidimensional maps. Four
perspectives were represented in scientometric maps: productivity, impact, expected vis-
ibility and excellence. Since each cluster of institutions represents a bibliometric pattern of
institutional performance, the neural network helps locate various bibliometric profiles of
academic production, and the identification of groups of institutions which have similar
patterns of performance. Also, scientometric maps allow for the identification of atypical
& Humberto [email protected]
Elio Atenogenes [email protected]
Ricardo [email protected]
1 Center of Research and Innovation in Information and Communication Technologies INFOTEC,Circuito Tecnopolo Sur 112, Mexico, Aguascalientes, Mexico
2 Empresa de Tecnologıas de la Informacion, La Habana, Cuba
3 Laboratory of Nonlinear Dynamics, Faculty of Sciences and Center of Complexity Sciences,National Autonomous University of Mexico, Mexico City, D.F., Mexico
123
Scientometrics (2017) 110:77–104DOI 10.1007/s11192-016-2166-0
behaviors (outliers) which are difficult to identify with classical tools, since they outstand
not because of a disparate value in just one variable, but due to an uncommon combination
of a set of indicators values.
Keywords Bibliometric rankings � Higher education � Institutional academic assessment �Scientometric indicators � Self-organized neural networks � Scientometric data mining �Mexico
Mathematics Subject Classification 68T10 � 62H30 � 91C20
JEL C630 � I230
Introduction
Higher education plays an important role in the domestic systems of science, technology
and innovation. According to the theoretical framework of the Triple Helix model (Ley-
desdorff and Meyer 2007), the articulated work of academy, industry and government is
fundamental for the development of knowledge-based economies. The global generation of
new institutional and social structures for the production, transfer and application of
knowledge is considered a strategic action in agendas of national science policies around
the world. Measures for these kind of linkages are still in growth, but pressures on research
systems to achieve such goal have been presented in the evaluation initiatives developed by
industrialized and developing nations (Krishna et al. 2002; Kurzydlowski 2003; Statzner
and Resh 2010; Weingart 2005; Zell 2005).
The adoption of policies to improve the efficiency and effectiveness of national higher
education systems and the implementation of more accurate systems for academic research
performance evaluation are essential steps for the development of a functional triple helix.
However, there are other aspects that affect collaboration between universities, industries
and government such as research budget, university localization, radicalness of research,
degree of risk-taking culture and researcher’s publication strategies (Belkhodja and Landry
2007). In this competitive environment, the bibliometric based (and other types of)
rankings of universities developed during the last years emerge as important tools for
benchmarking Higher Education Institutions (HEIs), with the aim of identifying ‘‘excel-
lence’’ in universities and among researchers (Bengoetxea and Buela-Casal 2013).
The introduction of a full scale of bibliometric rankings has become a ‘‘hot topic’’ in the
field of quantitative studies of science and technology, with literature in favor and against. In
the midst of it all, it has been recognized that the politicized use of rankings is an unavoidable
competitive tool, urgently demanded by science administrators (Weingart 2005).
Various authors have deeply criticized methodologies that use bibliometric tools,
because of misuse of their results for purposes they were not designed for, or for proving
incapable to cover essential missions of universities such as teaching quality, knowledge
transfer, international orientation and regional engagement (Billaut et al. 2010; Ioannidis
et al. 2007; Zhao et al. 2009). On the other hand, some authors remark the effectiveness of
the bibliometric approach (Abramo et al. 2013; Aguillo et al. 2010; Belter 2013; Huang
2012), and the correlation of excellence measured by peer-reviewed versus bibliometric
methods (Allen and Heath 2013; Mryglod et al. 2013). Other authors analyze the
78 Scientometrics (2017) 110:77–104
123
possibility to obtain more accurate rankings, based on improved indicators (Bengoetxea
and Buela-Casal 2013; Benito and Romera 2011; Torres-Salinas et al. 2011).
In this context, the appearance of new tools to measure academic research performance
is unstoppable. Every day, global initiatives using advanced bibliometric methods are
launched. Projects like SciVal (USA), the CTWS Leiden Ranking (The Netherlands), and
the SCImago Institutions Rankings (Spain), offer some of the most relevant rankings, due
to their valuable set of advanced citation-based indicators. However, some countries have
developed their own individual efforts to characterize and evaluate universities and
research institutions (Allen and Heath 2013; Vanclay and Bornmann 2012).
In Latin America, isolated efforts are focused on analyzing productivity and impact in
their scholarly systems. Experiences in Brazil, Argentina, Chile, Cuba and Venezuela have
been reported (Arencibia-Jorge and de Moya-Anegon 2010; Caputo et al. 2012; Helene and
Ribeiro 2011; Krauskopf et al. 2007; Miguel et al. 2010), but there is still a lack of research
on the effectiveness of science systems in Latin America and their contribution to the
research and development process in higher education institutions.
Mexico is not the exception. Some papers have analyzed quality of the Mexican pro-
duction in different research fields such as Physics and Astronomy, Agriculture, Nanos-
ciences and Health Sciences, among others (Collazo-Reyes et al. 2008, 2010; Lena 1997;
Licea de Arenas et al. 2002; Macias-Chapula et al. 2007; Sierra-Flores and Barnard 2009;
Sierra-Flores et al. 2009). Other papers have developed approaches on the Mexican higher
education system (Arvanitis et al. 1996; Licea de Arenas et al. 2000; Luna-Morales 2012).
The National University and some centralized federal agencies have developed biblio-
metric research performance evaluations at macro level. Three of the most prominent
initiatives include: the project Atlas of Mexican Science developed by the National Council
of Science and Technology (CONACyT 2012); the Comparative Study of Mexican
Universities developed by the National Autonomous University of Mexico (UNAM 2012);
and a bibliometric study developed by the SCImago Research Group for the Foro Con-
sultivo Cientıfico y Tecnologico of Mexico (FCCyT 2011).
For the Latin American context (including the Mexican case), three global projects of
rankings offer useful analytical tools: the Web Ranking of World Universities by the
Cybermetric Lab (Spain), which measures web visibility (Aguillo et al. 2005), the SCI-
mago Journal & Country Rank (SJCR), and the SCImago Institutions Rankings (SIR).
These analytical products provide a novel set of indicators that combine the traditional
output-impact approach with indicators of specialization, leadership and research excel-
lence (Gomez-Nunez et al. 2011). The Ibero American Ranking of Universities, one of the
information products of SIR, offers a scientometric view of Latin American institutions
with at least one paper published in a journal indexed by Scopus (SCImago Research
Group 2013). Decision makers and researchers can refer to each edition of this ranking,
published since 2009, to analyze the scientific production of Ibero American higher edu-
cation institutions.
The current work complements previous studies and provides the following new elements
for the analysis of Mexican scientific production: (1) the increase of members in the National
Researchers System (NRS, SNI in Spanish) and its relation to the rise of Mexican output
indexed by WoS and Scopus; (2) the evolution of national and institutional productivity and
the visibility of published papers; (3) the multiparametric characterization of universities
bibliometric performance through the application of Self-Organizing Maps based on neural
networks (Kohonen 2001) and advanced scientometric indicators. The aim is to analyse the
NRS contribution to the scientific performance of Higher Education, with emphasis in the
study of productivity and impact of universities, from a scientometric perspective.
Scientometrics (2017) 110:77–104 79
123
The paper is organized as follows. In the section ‘‘Materials and Methods’’ we present the
battery of indicators that will be used, and show a diagram depicting the devised procedure.
In section ‘‘Results and Discussion’’, we begin analyzing the evolution of the national
production and its impact, both in WoS and Scopus; and then present a bi-parametric
analysis of the Mexican HEIs’ performance indicators at a mezzo level. The rest of the
section is devoted to application and discussion of the results of the multiparametric char-
acterization of the HEIs’ performance profiles, assisted by SOM neural networks. We end
with a ‘‘Conclusions’’ section, where we summarize the results and contribution of our study.
Materials and methods
Sources and time windows
Our study focuses in two time slices. First, we identified and compared the total scientific
production of Mexico, in Scopus and the Web of Science (WoS), during a sixteen-year
period from 1996 to 2011. In the second part of our study, we analyzed the scientific
production in Scopus of the Mexican HEIs, during a five-year period from 2007 to 2011.
With the aid of artificial intelligence technology, we carried out a multiparametric bib-
liometric performance comparison of the 50 most productive Mexican HEIs, and were able
to represent their various production profiles in topographical clusters. To carry out the
analysis, we developed various scientometric indicators (described below), and used two
scientometric tools developed for Scopus by the SCImago Research Group (Spain): the
SCImago Journal and Country Rank (SJCR), and the SCImago Institutions Rankings (SIR).
All data were retrieved in May, 2013.
Production indicators
The annual number of documents produced in the country (ANdoc) is a standard pro-
duction indicator. However, we differentiate this indicator taking into account the pro-
duction of documents that are indexed in Scopus and Web of Science (Table 1, section a)
and define ANdoc Scopus and ANdoc WoS.
National research system indicators
The NRS constitutes the main resource of scientific production in Mexico. Hence, the
output volume of the nation is expected to grow in direct proportion to the number of NRS
scientists. Thus we define the indicator Nnrs as the number of NRS members. We eval-
uated Nnrs annually, and for each HEI, we define the indicator INnrs as the Institution’s
number of NRS scientists had per year, on average, during the period 2007–2011.
Nnrs was obtained from the project Atlas of Mexican Science, developed by CONACyT
(2012). INnrs values were obtained from the portal of the National Council of Science and
Technology (CONACyT 2013).
Country’s productivity indicators
Besides considering production indicators, we analyze the evolution from 1996 to 2011 of
the average number of scientific articles produced in the country each year, per each NRS
80 Scientometrics (2017) 110:77–104
123
member. This perspective leads to the concept, and indicators, of the scientific production
efficiency of the country, that we call the National Scientific Productivity: NSP WoS and NSP
Scopus (Table 1, section b). These indicators calculate the productivity restricted to each of
these databases, relative to the NRS. However, it is important to remark that these numbers
represent only estimations of the real quantities, because there is a residual number of sci-
entific articles produced in Mexico by researchers that do not belong to the NRS.
Performance profiles of the Mexican HEIs
Descending at the meso level, we identify and compare, with the neural network, the
scientometric performance profiles of the most productive Mexican HEIs bibliometric,
during the period 2007–2011. For this we consider 50 HEIs with production rates greater
than or equal to 30 articles per year (150 articles during this period), in journals indexed by
Scopus.
The analysis of each HEI was carried out considering four perspectives of scientific
production: output (volume), productivity (i.e., efficiency in the production), expected
Table 1 The summary of indicators used in this study
(a) Production indicators
ANdoc Annual number of documents produced by the country
ANdoc WoS Annual number of documents produced in journals that are indexed in the WoS
ANdocScopus
Annual number of documents produced in journals that are indexed in Scopus
Nnrs Number of National Research System (NRS) members
(b) Country’s productivity indicators
NSP National Scientific Productivity = ANdoc/Nnrs
NSP WoS National Scientific Productivity in WoS = ANdoc WoS/Nnrs
NSP Scopus National Scientific Productivity in Scopus = ANdoc Scopus/Nnrs
(c) Indicators of institutional productivity
AINdoc Annual Institutional Production = Average number of Scopus documents the institutionproduced per year, during the period 2007–2011
INnrs Institution’s number of NRS scientists had per year, on average, during the period2007–2011
ISP Institutional Scientific Productivity = AINdoc/INnrs
IPR Institutional Productivity Rate = [(AINdoc/Average ANdoc Scopus, during 2007–2011/(INnrs/Average Nnrs of Mexico during 2007–2011)]
(d) Indicators of expected and real visibility
%Q1 Measured by the percentage of the total amount of documents published in the first quartileof any Scopus’ subject category, according to the SCImago Journal Rank indicator
%Exc Excellence, measured by the percentage of the total number of documents (produced byeither the country or an institution) included in the 10% of the most cited articles of eachscientific field
RI Relative Impact = (Average number of citations that articles of a country, published in theyear (y), and indexed in Scopus, have accumulated during the time interval T)/(Averagenumber of citations that articles, published by the whole world in the year (y) and indexedin Scopus, have accumulated during T)
NI Normalized impact = Average value during the period 2007–2011 of the item oriented fieldnormalized citation score average
Scientometrics (2017) 110:77–104 81
123
visibility (estimated from the SCImago Journal Rank indicator), research impact (in terms
of the citation rates), and excellence (accounting highly cited documents). We define these
indicators in the following paragraphs and summarize them in Table 1, sections c and d.
Indicators of institutional productivity
To estimate annual institutional productivity of each HEI, relative to the NRS and Scopus,
during the period 2007–2011, we introduce concept of Institutional Scientific Productivity
(ISP). It represents the number of papers produced by each HEI per NRS member, in the
Scopus database. For instance, if ISP = 1 for an institution, that means that the institution
produces one research article per each national researcher on average. To calculate the ISP
indicator, we used information provided by both, the SCImago’s SIR and the CONACyT
of Mexico.
Equivalently, we could also assess the institutional scientific productivity from a dif-
ferent perspective, introducing the concept of Institutional Productivity Rate as the fol-
lowing percentages ratio: IPR = (Percentage of the National production, produced by the
institution)/(Percentage of the Nnrs of the country, belonging to the institution).
Thus, the IPR indicates the degree of productivity of an institution, relative to the
general productivity degree of the country to which it belongs. If an institution has an
IPR = 2, it means that the institution has a productivity level which doubles that of the
country.
There is a strong correlation between the two institutional productivity indicators (ISP
and IPR) that we have just defined. In fact they are linearly dependent, since for each
institution:
IPR ¼ ISP/NSP�;
where NSP* = (Average ANdoc Scopus, during 2007–2011)/(Average Nnrs during
2007–2011) = 0.86.
One might interpret NSP* as the ‘‘Average value of NSP, during 2007–2011’’, but this
interpretation would not be correct in general: we must notice that the constant NSP*,
being a ratio of averages (RoA), is not equal to the average National Productivity Rate
during the period 2007–2011, because the latter is not a RoA but an average of ratios
(AoR). However, these two quantities are strongly related: it is well known that a RoA can
be seen as a weighed AoR (Waltman et al. 2011); furthermore, it has been proved that the
regression line of a cloud of points of the form (AoRi, RoAi) is the identity line (Egghe
2012).
Since ISP and IPR have different values and different interpretations, they provide two
complementary views of data. For instance, if we consider the reference value (1) of the
two indicators as a threshold for ‘‘acceptable productivity efficiency’’, it results that only
36 of the 50 most productive HEIs of Mexico during 2007–2011 have an ISP larger than 1.
However, during this period 43 HEIs have an IPR over 1.
Indicators of expected and real visibility
As an indicator of expected visibility we consider %Q1, measuring the percentage of the
total amount of documents published in the first quartile of any Scopus’ subject category,
according to the SCImago Journal Rank indicator (Miguel et al. 2011).
82 Scientometrics (2017) 110:77–104
123
To estimate the actual visibility, we consider other three indicators of the SJCR: RI, NI
and %Exc.
For a given time interval T (in our case 1996–2011), and for each year (y) included in
the time interval T, the Scopus-based relative impact, RI, was calculated relativizing the
number of citation accumulated by the country by the number of citations accumulated by
the whole world. Authors used various names for this indicator in bibliometric literature:
‘‘impact normalized by world average’’ (Noyons 2005), or ‘‘impact relative to world’’, in
the glossary of inCitesTM (Thomsom Reuters 2015). For simplicity, we just call it ‘‘rel-
ative impact.’’
From the SJCR we obtained data to calculate and analyze the evolution of RI for
Mexico (Fig. 3) and, to analyze the RI of institutions, we used another impact indicator
(IN), obtained from SIR. This measure is based in the concept of ‘‘item oriented field
normalized citation score average’’ proposed by the Karolinska Institutet: ‘‘…it is calcu-
lated by normalizing each individual publication’s citation rate against an average citation
rate for articles in the same subject area, the same type and of the same age, and finally the
average of all the normalized citation values is calculated’’ (Rehn and Kronman 2008).
This indicator is useful to quantitatively compare the citation volumes in different areas of
knowledge.
As an indicator of excellence we use %Exc, measured by the percentage of the total
number of documents (produced by either the country or an institution) included in the
10% of the most cited articles of each scientific field (Bornmann et al. 2012).
Artificial intelligence method
In this paper we assume that Mexican HEIs have different performance profiles that can be
characterized from a scientometric perspective. This assumption places the hard problem
of automatically carrying out a multiparemetric characterization of scientometric perfor-
mance profiles of the 50 most productive HEIs of Mexico (in terms of the battery of
scientometric indicators described above). To deal with this problem we devised a data
mining method based on the SOM family of neural networks (Teuvo Kohonen 2013). The
method constitutes a novel tool for computational multiparametric analysis, presented here
as a scientometric data mining application graphically described in Fig. 1.
There have been other successful applications of SOM neural networks to scientometric
studies. Pioneer applications of SOM in scientometrics studies include the analysis of
keywords co-occurrence matrix in biomedical domains (Sotolongo-Aguilar et al.
2001, 2002), and for science and technology mapping (Polanco et al. 2001; Moya-Anegon
et al. 2006). More recent applications include the analysis of a biomedical domain evo-
lution (Guzman et al. 2010), and the improvement of SOM based mapping of science
(Skupin et al. 2013). In general, the problem of science mapping has been the main
application of the SOM neural network in the scientometric context. As far as we know,
there have been no applications of SOM based multiparametric analysis and characteri-
zation of the academic institutions performance profiles using scientometric indicators.
Our method was implemented in a software system called LabSOM, developed by the
Laboratory of Nonlinear Dynamics at the Faculty of Sciences of the National Autonomous
University of Mexico (UNAM) and the Company ‘‘Tecnologıas Inteligentes y Modelacion
de Sistemas’’. With this tool, the analysis and interpretation of the multidimensional data is
automated and facilitated through friendly visualization.
Typically, the SOM neural network is modelled as a two dimensional hexagonal grid.
Each hexagon represents an artificial neuron and, at the same time, a location where data
Scientometrics (2017) 110:77–104 83
123
points can be mapped. The application of SOM neural networks develops a nonlinear
projection of data into the neural network. The final (self-organized) map is the result of
the neural network iterative training process, by which the network learns to project similar
patterns into close locations (hexagons) in the map. The projection provides a represen-
tation of the distribution of the data set in the multidimensional space in a 2D cartography
that allows for visualization of the data. Thus, two HEIs can be searched in the resulting
map, and their distance can be analyzed as distance in the multidimensional space. The
main assumption to make inferences from this application is that the similarity between
HEIs’ performances can be estimated by calculating the ‘‘scientometric distance’’ among
their multidimensional representations.
One of the main issues to be faced in this investigation is the search for atypical
performance profiles. We call the institutions whose set of scientometric indicators exhibit
an atypical pattern outliers. It is very simple to identify atypical values of a single indi-
cator. A more difficult task is to identify atypical combinations of multiple indicators. We
show that the neural method is suited to automatically carry out this complex task.
To identify outliers, it is convenient to use a neural network with more neurons than
data points, which is contrary to what is recommended in a typical application of SOM
neural networks. To get the SOM maps exposed in this paper we used a neural network
with 2100 (30 9 70) neurons, trained after 500,000 computational iterations. The ratio
between length and width of this rectangular neural net is approximately the ratio between
the two main eigenvalues of the covariance matrix of the data set. This proportion has also
been successfully used in other contexts by other authors (Ultsch and Morchen 2005).
The clusters map obtained represents the set of performance patterns: it visually exhibits
the outliers and groups of institutions that share the various scientometric performance
styles identified by the neural network. The optimal number of clusters was calculated by
the software system LabSOM, with an algorithm that uses the Dunn index (Dunn 1973).
In order to obtain satisfactory visualizations which reveal non-trivial information about
the different patterns of performance, by means of the so called ‘‘Component Planes’’
Fig. 1 This diagram graphically explains the devised data mining procedure and the use of the SOMnetwork to process ISP, NI, %Q1 and %Exc data
84 Scientometrics (2017) 110:77–104
123
(Vesanto 1999), we had to apply an original normalization technique for data, based on the
logarithmic transformation of an appropriate piecewise linear scaling. The explanation of
this scaling, and other technical aspects of this methodology, will be detailed in further
publications.
Results and discussion
From 1996 to 2011 Mexico’s scientific production has increased in WoS and Scopus, with
average rates of change of 560 and 695 articles per year, respectively. Thus, the 2011
production indicator in Scopus tripled with respect to 1996, and in WoS, although in
smaller scale, it had a similar behavior.
Consequently, during the five-year period 2007–2011, in Elsevier database the country
had 67,116 registers, representing 46.28% of the whole output during the sixteen-year
period 1996–2011 (144,997). The output in Thomson Reuters’ databases during the same
period was 141,067 records, of which 62,087 (44%) were produced during 2007–2011.
The national researchers system
The growth of the scientific production in Scopus correlates to the growth of the total
amount of NRS members (Nnrs) which also tripled from 1996 to 2011 (Fig. 2), going from
5969 in 1996, to 17,568 in 2011, with an average annual growth rate of 7.5%. This is five
times greater than the average annual growth rate of the Mexican population (1.4%) during
the same period, passing from 97.2 million of people in 1996 to 119.4 in 2011. Therefore,
the number of NRS members per inhabitant grew from 61.4 Nnrs per million of people in
1996 to 147.1 in 2011. This significant growth in the number of Mexican scientists was due
to a sustained human resources development program of the national council of science
and technology (CONACYT). As a result of this effort, the annual scientific production of
the country in WoS has doubled, passing from 4.8 scientific articles published per million
Fig. 2 Parallel growth of the total number of NRS members and the Mexican scientific output in Web ofScience and Scopus, 1996–2011. The lower curve plots the difference Ndoc Scopus—Ndoc WoS
Scientometrics (2017) 110:77–104 85
123
people in 1996 to 11 in 2011. Outstandingly, the output in Scopus tripled during this
period, passing from 4 to 12 articles per million of people. However, there is still a long
road to go for Mexico if we compare the value of this indicator in the year 2011 with Brazil
(27), China (28), USA (185) or Finland (300).
Rates of growth
During the eleven years that go from 1996 to 2006, the average growth rate of NRS
members was 7.3%, and it grew to 7.7% during the five-year period 2007–2011. However,
growth rates of scientific production processes have a different behavior. In Scopus
database, from 1996 to 2006 there was a production average growth rate of 10.17%;
meanwhile during 2007–2011, the average went down to 5.34%, which means a decrement
of 47.40% among these two time windows. Similarly, from 1996 to 2006, we observe an
average growth rate of 8.80% in the WoS database, while in the period 2007–2011 the
average is 5.48%, signifying a decrement of 32.20%.
This situation should not be worrisome. In this scenario one should not expect rates of
growth to be increasing or constant, since in the latter case this would imply growth at an
exponential rate, and in the first case it would imply that production would grow
unbounded in finite time. Furthermore, it is well known that production rates of growth
might decrease in inverse proportion to the production indicator (number of scientific
articles being published per year), while this production indicator grows at a linear rate
(with a constant slope).
Parallel growth of Mexican scientific productivity and the nationalresearchers system
The number of Mexican papers produced per year in both databases is close to the number
of NRS members of the country. However, in Fig. 2 the curve of the number of NRS
members is above Scopus and WoS curves during the whole period, which can be assumed
as evidence that the country published less than one document per each NRS scientist in
high visible journals. The validity of this statement is accentuated by the fact that there are
contributions to the total national scientific production coming from a minority of Mexican
researchers that do not belong to the NRS. But we have to carefully interpret this data for
assessment purposes. In the first place, scientific papers are not the only valid product for
national researchers, since books, patents, utility models, etc. are also an important part of
their production. Secondly, we can statistically conclude that each NRS member con-
tributes to the national production with less than one document per year during the studied
period, but we cannot conclude that each NRS member produces less than one scientific
paper per year, because due to scientific collaboration, a considerable amount of docu-
ments are coauthored by more than one NRS scientist.
Quality versus quantity
In Fig. 2 we see that the production of scientific articles and the total number of NRS
members have both increased constantly during the period 1996–2011. However, different
trends in their rates of change have negatively impacted the national scientific productivity
indicators (NSP Scopus and NSP WoS) in the latest part of this period (Fig. 3). The
86 Scientometrics (2017) 110:77–104
123
decrement of values for these indicators might be interpreted as a loss of efficiency in
scientific production.
The NSP in WoS oscillated around a mean value of 0.85 during 1996–2007, but from
thereon, it has decreased to a minimum value of 0.74. During this period, the NSP in
Scopus increased from its minimum value (0.74) to a peak value of 0.95 in 2006, and
thereon oscillated around a mean value of 0.86 during the last five years. The evolution of
the two national scientific productivity indicators (NSP Scopus and NSP WoS) is compared
in Fig. 3 with the evolution of the relative impact (RI) of publications during 2007–2011.
We recall that, providing a comparison among the average number of citations per paper of
Mexico and the world, a relative impact of value 1 means that the number of citations that
Mexico has received—on average per article—is the same than the average number of
citations that the whole world has received per article in Scopus; larger values of the
relative impact would mean that Mexico receives more citations per paper, in Scopus, than
the world average.
In Fig. 3 the NSP indicators show a growing trend during the first part of the 1996–2011
period, but this increasing trend stops in the later part of the period. In fact, the NSP WoS
indicator decreases from 2003, reaching its minimum value in 2011.
Contrasting with this decrement of NSP WoS, the relative impact of the Mexican
scientific production exhibited an average growth trend of 5.53% during the period
2003–2011 (Fig. 4), approaching the value 1.07 from below with an evolution pattern
similar to that of Japan from 2003 on, and having higher growth rates during the period
2010–2011 than the five most productive countries: Mexico 10.3%, UK 6.2%, Germany
5.2%, Japan 3.0%, USA 2.0% and China 1.6%.
The noticeable increment that the relative impact of the Mexican production has had
since the year 2006 on, must be interpreted with care because the values of the relative
impact are a function of time, and recent values of relative impact could undergo large
variations (Bornmann and Leydesdorff 2013). Therefore, it would be expected that the
Fig. 3 Behavior of national scientific productivity indicators and the relative impact of Mexican scientificproduction in Scopus (1996–2011)
Scientometrics (2017) 110:77–104 87
123
strong rate of growth trend (18.88%) of the relative impact of Mexico during the interval
2010–2011 may be subject of adjustments in the years to come.
The 50 most productive higher education institutions of Mexico
Since most of the Mexican productive scientists are affiliated to a HEI, the Higher Edu-
cation System is the most important research sector of Mexico; from a total of 357 HEIs
registered in the SIR in 2011, the 50 most productive have the 80.21% of the NRS
members.
In what follows, descending at the meso level, and using the battery of previously
defined scientometric indicators, we carry out a comparative analysis of the dynamics of
scientific production in the 50 Mexican HEIs with greatest output in Scopus in the period
2007–2011. For this comparison, we first display scatter plots of various indicator pairs and
finally use a methodology based on artificial neural networks, that we develop to auto-
matically identify the main scientometric performance profiles within this group of HEIs,
taking into account all the indicators ensemble.
According to the Ibero American Ranking of Universities 2013 (SCImago Research
Group 2013) all of the 50 selected HEIs have produced more than 150 documents, covered
by Scopus, during this five-year period (Table 2).
Institutional productivity rates
The Universidad Nacional Autonoma de Mexico (UNAM) outstands as having the largest
number of NRS members, and the largest volume of scientific production during the
studied period, producing 19,349 articles, which represent the 28.83% of the national
output, and concentrating 21.78% of the Mexican NRS members to score an Institutional
Productivity Rate of 1.32. Its annual average production in this period was 3870 articles.
Being an outlier by its production score, we exclude it from some of the graphs of this
Fig. 4 Comparison of Mexico‘s relative impact evolution with the top 5 world’s most productive countries(SCImago Country Ranking)
88 Scientometrics (2017) 110:77–104
123
Table
2T
he
mo
stp
rod
uct
ive
Mex
ican
HE
Isin
Sco
pu
sd
uri
ng
the
per
iod
20
07
–2
01
1
Hig
her
educa
tion
inst
ituti
on
Ndoc
Ndoc%
Nnrs
Nnrs
%A
Ndoc
ISP
IPR
NI
Exc%
Q1
%
Un
iv.
Nac
.A
uto
n.
de
Mex
ico
(UN
AM
)19,349
28.83
3320
21.78
3869.8
1.1
71
.32
0.79
7.31
45.46
Cen
t.d
eIn
ves
t.y
de
Est
.A
van
z.(C
INV
ES
TA
V)
7072
10.54
621.6
4.08
1414.4
2.28
2.58
1.03
9.68
42.05
Inst
.P
oli
tec.
Nac
.(I
PN
)5581
8.32
668.2
4.38
1116.2
1.67
1.9
0.6
35
.27
30.69
Un
iv.
Au
ton
.M
etro
po
lita
na
(UA
M)
3934
5.86
816.6
5.36
786.8
0.9
61
.09
0.6
86.54
34.88
Un
iv.
de
Gu
adal
ajar
a(U
dG
)2097
3.12
579
3.8
419.4
0.7
20
.82
0.5
33
.56
32.81
Un
iv.
Au
ton
.d
eN
uev
oL
eon
(UA
NL
)1884
2.81
348
2.28
376.8
1.0
81
.23
0.6
25
.62
29
.62
Ben
em.
Un
iv.
Au
ton
.d
eP
ueb
la(B
UA
P)
1876
2.8
331.8
2.18
375.2
1.1
31
.29
1.07
8.54
34.38
Inst
.T
ecn
ol.
yd
eE
st.
Su
p.
de
Mo
nte
rrey
(IT
ES
M)
1649
2.46
236.4
1.55
329.8
1.4
1.5
90.82
8.36
29
.23
Un
iv.
de
Gu
anaj
uat
o(U
G)
1579
2.35
217.8
1.43
315.8
1.45
1.64
0.77
7.78
37.68
Un
iv.
Mic
h.
de
San
Nic
ola
sd
eH
idal
go
(UM
SN
H)
1488
2.22
262.8
1.72
297.6
1.1
31
.29
0.88
7.46
33.74
Un
iv.
Au
ton
.d
eS
anL
uis
Po
tosi
(UA
SL
P)
1482
2.21
21
3.6
1.4
296.4
1.3
91
.58
1.09
9.64
42.91
Un
iv.
Au
ton
.d
elE
stad
od
eM
ore
los
(UA
EM
)1
30
21
.94
20
7.2
1.3
62
60
.41
.26
1.4
30.79
6.38
42.63
Un
iv.
Au
ton
.d
eB
aja
Cal
ifo
rnia
(UA
BC
)1
19
81
.78
19
1.4
1.2
62
39
.61
.25
1.4
20
.59
4.8
92
5.3
8
Cole
gio
de
Post
gra
duad
os
(CO
LP
OS
T)
1130
1.6
8226.4
1.49
22
61
1.1
30
.34
2.2
316.81
Un
iv.
Au
ton
.d
elE
stad
od
eM
exic
o(U
AE
ME
X)
10
75
1.6
238.8
1.57
21
50
.91
.02
0.5
64
.89
27
.44
Univ
.V
erac
ruza
na
(UV
ER
)880
1.3
1233.4
1.53
17
60
.75
0.8
60
.56
4.6
134.32
Un
iv.
Au
ton
.d
eY
uca
tan
(UA
Y)
82
11
.22
12
8.2
0.8
41
64
.21
.28
1.4
50.78
8.16
34.47
Un
iv.
de
So
no
ra(U
SO
N)
80
11
.19
17
9.2
1.1
81
60
.20
.89
1.0
10
.53
3.9
636.2
Un
iv.
Au
ton
.d
elE
stad
od
eH
idal
go
(UA
EH
)7
47
1.1
11
51
.40
.99
14
9.4
0.9
91
.12
0.4
62
.79
27
.44
Un
iv.
Au
ton
.d
eQ
uer
etar
o(U
AQ
)6
76
1.0
11
18
.80
.78
13
5.2
1.1
41
.30.71
6.41
33.73
Univ
.Ib
eroam
eric
ana
(UIB
ER
)578
0.8
694.6
0.6
2115.6
1.2
21.3
92.42
24.4
59.69
Un
iv.
de
Co
lim
a(U
CO
L)
53
80
.81
08
.20
.71
10
7.6
0.9
91
.13
0.6
34
.64
40.89
Un
iv.
Au
ton
.C
hap
ing
o(U
AC
HA
P)
52
10
.78
10
8.2
0.7
11
04
.20
.96
1.1
0.3
62
.04
15.93
Un
iv.
de
las
Am
eric
asP
ueb
la(U
AM
ER
P)
52
10
.78
54
.80
.36
10
4.2
1.9
2.17
0.78
7.95
27
.45
Scientometrics (2017) 110:77–104 89
123
Table
2co
nti
nued
Hig
her
educa
tion
inst
ituti
on
Ndoc
Ndoc%
Nnrs
Nnrs
%A
Ndoc
ISP
IPR
NI
Exc%
Q1
%
Un
iv.
Au
ton
.d
eZ
acat
ecas
(UA
ZA
C)
44
20
.66
11
0.4
0.7
28
8.4
0.8
0.9
10
.56
4.5
29
.41
Un
iv.
Au
ton
.d
eS
inal
oa
(UA
SIN
)4
29
0.6
41
12
.20
.74
85
.80
.76
0.8
71.53
12.5
40.79
Univ
.A
uto
n.
de
Tam
auli
pas
(UA
TA
M)
391
0.5
862.8
0.4
178.2
1.2
51.4
10.5
55.7
23.02
Inst
.T
ecn
ol.
de
Tij
uan
a(I
TT
IJ)
37
90
.56
16
.80
.11
75
.84.51
5.08
0.74
9.73
12.66
Un
iv.
Au
ton
.d
eC
iud
adJu
arez
(UA
CJ)
33
80
.56
6.6
0.4
46
7.6
1.0
21
.14
0.6
96.64
28
.7
Univ
.A
uto
n.
de
Aguas
cali
ente
s(U
AA
GU
)318
0.4
751.2
0.3
463.6
1.2
41.4
0.4
3.3
523.58
Inst
.T
ecn
ol.
de
Cel
aya
(IT
CE
L)
31
70
.47
35
0.2
36
3.4
1.81
2.05
0.9
10.3
35.33
Un
iv.
Au
ton
.d
eC
hih
uah
ua
(UA
CH
IHU
)2
91
0.4
34
60
.35
8.2
1.2
71
.43
0.5
25
.93
26
.12
Un
iv.
Juar
ezA
uto
n.
de
Tab
asco
(UJA
TA
B)
27
90
.42
47
.40
.31
55
.81
.18
1.3
50
.38
1.5
720.79
Un
iv.
Au
ton
.A
gra
ria
An
ton
ioN
arro
(UA
AA
N)
27
80
.41
44
0.2
95
5.6
1.2
61
.42
0.4
4.4
322.3
Un
iv.
Juar
ezd
elE
stad
od
eD
ura
ng
o(U
JED
)2
53
0.3
83
10
.25
0.6
1.63
1.87
0.6
74
.96
23.72
Inst
.T
ecn
ol.
Au
ton
.d
eM
exic
o(I
TA
M)
25
00
.37
65
.60
.43
50
0.7
60
.86
0.6
54
.83
34.4
Cen
t.N
ac.
de
Inv
est.
yD
esar
r.T
ecn
ol.
(CN
IDT
)2
44
0.3
62
1.6
0.1
44
8.8
2.26
2.54
0.77
5.9
617.62
Univ
.A
uto
n.
de
Tla
xca
la(U
TL
AX
)243
0.3
647.2
0.3
148.6
1.0
31.1
61.11
10.5
39.09
Un
iv.
Au
ton
.d
eC
oah
uil
a(U
AC
OA
H)
21
70
.32
47
0.3
14
3.4
0.9
21
.04
0.6
65
.21
28
.57
Univ
.A
uto
n.
de
Cam
pec
he
(UA
CA
MP
)214
0.3
233.2
0.2
242.8
1.2
91.4
70.6
77.73
37.85
Un
iv.
Au
ton
.d
eB
aja
Cal
ifo
rnia
Su
r(U
AB
CS
)2
05
0.3
12
80
.18
41
1.46
1.69
0.5
72
.56
31.71
Univ
.A
uto
n.
de
Guer
rero
(UA
GU
E)
205
0.3
132.4
0.2
141
1.2
71.4
61.14
13.7
31.71
Un
iv.
Au
ton
.d
ela
Ciu
dad
de
Mex
ico
(UA
CM
)1
99
0.3
59
.60
.39
39
.80
.67
0.7
70
.59
3.1
441.71
Un
iv.
Pan
amer
ican
a(U
PA
N)
19
80
.33
70
.24
39
.61
.07
1.2
40
.47
4.9
421.72
Inst
.T
ecn
ol.
de
Mo
reli
a(I
TM
OR
)1
83
0.2
71
40
.09
36
.62.61
2.94
0.4
21
.08
18
.03
Un
iv.
Tec
no
l.d
ela
Mix
teca
(UT
MIX
)1
80
0.2
71
9.6
0.1
33
61.84
2.1
0.6
95
.52
13
.89
Inst
.T
ecn
ol.
de
To
luca
(IT
TO
L)
16
60
.25
6.6
0.0
43
3.2
5.03
5.77
0.3
92
.78
22
.29
Inst
.T
ecn
ol.
de
Ver
acru
z(I
TV
ER
)1
59
0.2
41
80
.12
31
.81.77
2.03
0.4
11
.31
15
.72
90 Scientometrics (2017) 110:77–104
123
Table
2co
nti
nued
Hig
her
educa
tion
inst
ituti
on
Ndoc
Ndoc%
Nnrs
Nnrs
%A
Ndoc
ISP
IPR
NI
Exc%
Q1
%
Univ
.A
uto
n.
de
Nay
arit
(UA
NA
Y)
159
0.2
420.6
0.1
431.8
1.54
1.78
0.6
32
.88
29
.56
Un
iv.
Au
ton
.d
eC
hia
pas
(UA
CH
IA)
15
40
.23
37
.60
.25
30
.80
.82
0.9
30
.38
2.7
42
6.6
2
Av
erag
es1
30
91
.951
21
5.4
1.4
13
26
1.9
1.4
1.5
90
.70
6.1
53
0.2
5
Nnrs
and
AN
doc
can
be
frac
tional
num
ber
sbec
ause
thes
ear
eav
erag
eval
ues
duri
ng
the
per
iod
2007–2011
Bold
val
ues
indic
ate
the
val
ues
above
the
aver
age
Scientometrics (2017) 110:77–104 91
123
paper (e.g., Fig. 5) in order to gain a better picture of the behavior of the other 49
institutions.
There are only two other institutions that had an annual average production greater than
1000 articles: the Centro de Investigaciones y Estudios Avanzados (CINVESTAV; 1414
articles), and the Instituto Politecnico Nacional (IPN; 1116 articles). These institutions
concentrated 10.5 and 8.3% of the Mexican scientific production, with 4.08 and 4.38% of
the total national researchers, obtaining institutional productivity rates of 2.28 and 1.67,
respectively. With an IPR of 1.09, the Universidad Autonoma Metropolitana (UAM) is the
second institution with more NRS members (5.16%), and the last one with more than 5%
of the country’s scientific output.
Institutional scientific productivity
The NSP indicator has an average value of 0.86 during the period 2007–2011, and the top
50 most productive HEIs average an ISP value of 1.4. So these 50 institutions have a
relevant contribution to the national scientific productivity. There were 14 HEIs that had
ISP values higher than 1.4, among them: Instituto Tecnologico de Veracruz (ITVER; 1.77),
Instituto Politecnico Nacional (IPN; 1.67), Universidad Juarez del Estado de Durango
(UJED; 1.63), Universidad Autonoma de Nayarit (UANAY; 1.54), Universidad Autonoma
de Baja California Sur (UABCS; 1.46), and Universidad de Guanajuato (UG; 1.45).
Thirteen HEIs in the selected sample score values of ISP smaller than one, and 37
produced more articles per year (visible in Scopus) than their number of national
researchers. Seven institutions have an ISP below the NSP indicator value (0.86):
Universidad Autonoma de Chiapas (UACHIA; 0.82), Universidad Autonoma de la Ciudad
de Mexico (UACM; 0.67), Instituto Tecnologico Autonomo de Mexico (ITAM; 0.76),
Universidad Autonoma de Sinaloa (UASIN; 0.76), Universidad Autonoma de Zacatecas
(UAZAC; 0.8), Universidad Veracruzana (UV; 0.75), and Universidad de Guadalajara
(UdG; 0.72).
Fig. 5 Productivity of the most prolific Mexican universities during the period 2007–2011. UNAM is notdepicted in the picture because it’s an outlier with 3320 National Researchers System members(ISP = 1.17)
92 Scientometrics (2017) 110:77–104
123
On the other hand, some institutions stand out by producing more than two papers per
NRS researcher annually (in Scopus journals), therefore obtaining an ISP[ 2. The most
conspicuous outliers with respect to this indicator are Instituto Tecnologico de Toluca
(ITTOL; 5.03) and Instituto Tecnologico de Tijuana (ITTIJ; 4.51). We will exclude them
from some of the graphics (e.g., Fig. 7) to gain better understanding of the institutions that
have more typical ISP values. There are other HEIs with large ISP values: Instituto Tec-
nologico de Morelia (ITMOR; 2.61), Centro de Investigacion y Estudios Avanzados
(CINVESTAV: 2.28), and Centro Nacional de Investigaciones y Desarrollo Tecnologico
(CNIDT; 2.26).
Other institutions with high productivity levels (ISP[ 1.8) are: Universidad de las
Americas de Puebla (UAMERP; 1.9), the Universidad Tecnologica de la Mixteca
(UTMIX; 1.84) and the Instituto Tecnologico de Celaya (UTCEL; 1.81).
The lowest values of the ISP indicator were obtained by Universidad Autonoma de la
Ciudad de Mexico (UACM; 0.67), Universidad de Guadalajara (UdG; 0.72), Universidad
Veracruzana (UVER; 0.75), Instituto Tecnologico Autonomo de Mexico (ITAM; 0.76), and
Universidad Autonoma de Sinaloa (UASIN; 0.76). In Fig. 5 we can see that, with the
exception of CINVESTAV, IPN, ITESM and UG, the highest values of ISP were obtained
by HEIs with a small number of national researchers (Nnrs\ 100).
Visibility, impact and excellence of Mexican Universities
Considering the inverse relation of productivity and impact observed at the macro level, we
would expect that the production efficiency of each institution (number of papers per NRS
member) might not be positively correlated with the production’s efficacy (production’s
impact, estimated in terms of the number of citations). In what follows, to enrich our
analysis, and to focus in this type of questions, we will consider a set of citation based
indicators proposed by the SIR.
For the most productive Mexican institutions, Table 2 shows the normalized impact
(NI) of its scientific production (compared to the world‘s) as well as %Q1, the percentage
of articles published in the most visible Scopus journals (Scopus quartile 1, according to
the SJCR), and %Exc, the percentage of papers that belong to the set of the 10% most cited
articles of each subject category of Scopus (the Productions Excellence Core). The other
indicators in Table 2 are: Total output (Ndoc), percentage of the total amount of docu-
ments published by the country (Ndoc%), average annual number of NRS members (Nnrs),
percentage of the total amount of national researchers (Nnrs%), institutional scientific
productivity (ISP) and institutional productivity rate (IPR).
In Table 2 we see that seven Mexican HEIs in the 50 institutions sample had a higher
normalized impact than the world during the period 2007–2011, and nine HEIs have %Q1
larger than the sample’s average (30.25%). It is noticeable that all the 50 institutions of the
sample have positive values of the excellence indicator (%Exc), meaning that they had at
least one paper in the Scopus Excellence Core.
The Universidad Iberoamericana (UIBER) does not stand out in this group of insti-
tutions for its productivity: with an ISP = 1.22 it is below the mean value of the 50 HEIs
(ISP = 1.4). However, it is the most outstanding outlier in this group of HEIs, achieving
extremely atypical values in the three citations-based indicators: UIBER published 59.7%
of their articles in highly visible journals, obtaining a normalized impact of 2.42 (its papers
obtained 142% more cites than the world’s average). Remarkably, 24.4% of its scientific
output was located in the ‘‘Excellence core’’, which is one of the best scores in Latin
America, comparable to the scores of some prestigious universities of USA and UK
Scientometrics (2017) 110:77–104 93
123
(Harvard University 28.69%, Stanford University 27.29%, and University of Cambridge
24.56%). The number of authors in UIBER publications also has very extreme values: 45%
of its articles have more than 500 authors and accumulate 80% of the total amount of
citations of the period 2007–2011. These data might indicate that this institution has an
Fig. 6 Normalized impact versus Excellence in Mexican HEIs (2007–2011). Universidad Iberoamericanais not depicted in the picture because it’s an outlier with the highest value (24.41%) of its production in the
10% most cited at the world level, and a NI value of 2.42. For these two indicators R2 ¼ 0:9:
Fig. 7 Excellence versus Institutional Scientific Productivity in Mexican HEIs (2007–2011). TheUniversidad Iberoamericana is not depicted in the picture because it’s an outlier with an extreme valueof %Exc = 24.41% (ISP = 1.22)
94 Scientometrics (2017) 110:77–104
123
important level of participation in large international research projects. As we did with
other outliers, we excluded the UIBER from some of the graphs (Figs. 5, 6, 7).
The mean value of the normalized impact was around 0.7 in the studied sample. There
are six HEIs besides UIBER, that have attained a NI[ 1: Universidad Autonoma de
Sinaloa (UASIN; 1.53), Universidad Autonoma de Guerrero (UAGUE; 1.14), Universidad
Autonoma de Tlaxcala (UTLAX; 1.11), Universidad Autonoma de San Luis Potosı
(UASLP; 1.09), Benemerita Universidad Autonoma de Puebla (BUAP; 1.07), and Centro
de Investigacion y de Estudios Avanzados (CINVESTAV; 1.03). The institutions Colegio
de Postgraduados (COLPOST; 0.34) and Universidad Autonoma de Chapingo (UACHAP;
0.36) showed the lowest values. For worldwide comparison purposes, it is convenient to
consider the institutions that belong to the 2011 top normalized impact ranking: Mas-
sachusetts Institute of Technology (29.86), The Rockefeller University (29.18), Harvard
University (28.28), and London Business School (27.01).
The excellence indicator (%Exc) showed a mean value of 6.2% in the 50 HEIs sample,
and 20 institutions had values of this indicator above the mean value. UAGUE (13.7%),
UASIN (12.5%) and UTLAX (10.5%) were in the leaders group. The Instituto Tecnologico
de Celaya (ITCEL; 10.3%) was the last institution with more than 10% of their papers in
the excellence core. All the 50 elements of the sample have at least one article in the
excellence core and the lowest values of the excellence indicator correspond to the Instituto
Tecnologico de Morelia (ITMOR; 1.1%) and the Instituto Tecnologico de Veracruz
(ITVER; 1.3%). At the world level in the top ranking of the excellence indicator we have
the following values: London Business School (70.33), The Rockefeller University
(61.64), Massachusetts Institute of Technology (57.87), and Harvard University 56.84.
With 45.5% of its papers published in 1st quartile-journals of Scopus, UNAM was
second in publishing in the most visible journals—after UIBER (60%). The Universidad
Autonoma de San Luis Potosı (UASLP; 42.9%), Universidad Autonoma del Estado de
Morelos (UAEM; 42.6%), CINVESTAV (42%) and Universidad Autonoma de la Ciudad
de Mexico (UACM; 41.7%), were also among the leaders of this indicator which has a
mean value of 30.2%. However, some institutions registered values under 20%.
Remarkably, the ITTIJ, which has the fifth place of excellence in the country, with
%Q1 = 12.7% has the lowest percentage of the 50 studied universities.
Biparametric analysis
In Figs. 5 and 6 we display data of these 50 institutions from two different perspectives:
first, we detect institutions which have the most efficacious bibliometric performance by
plotting the normalized impact and excellence (Fig. 6); secondly, we compare efficiency
versus effectivity, contrasting productivity with excellence (Fig. 7). Figure 8 complements
the biparametric analysis comparing excellence (%Exc) versus expected visibility (%Q1).
In all these figures we exclude UIBER in order to obtain a better picture of the most typical
behavior. This analysis covers the period 2007–2011.
Figure 6 exhibits the group of Mexican institutions with best bibliometric performance
from an efficacy-based perspective: it plots indicators which do not take into account
volume or efficiency of production but its impact, estimated with publication’s visibility
revealed by the citation process. The fact that the institutions arrange around a straight line
in the graph (with correlation coefficient of 0.9), proves that there is a considerable degree
of correlation among these two indicators: the values of NI increase in direct proportion to
the values of %Exc. Five institutions stand out with values of the NI above threshold
(UASIN, UTLAX, UASLP, BUAP and CINVESTAV), and 16 institutions lie in the graph
Scientometrics (2017) 110:77–104 95
123
above the mean values of these two indicators: UIBER, UASIN, UAGUE, UTLAX,
UASLP, UNAM, BUAP, CINVESTAV, ITCEL, ITTIJ, ITESM, UAMERP, UMSNH,
UAEM, UAQ y UAY.
A different situation is observed in Fig. 7, where the relation of Excellence (%Exc) and
ISP indicators is exhibited. There is no positive correlation among these indicators (cor-
relation coefficient R2 ¼ 0:0002Þ and the institutions distribute widely in the Cartesian
space. The highest values of %Exc were attained by UIBER, UAGUE, UASIN, UATLAX;
and all of them have relatively low productivity values (below the mean). The ITTIJ is a
notable exception. Above the mean values, and with an important degree of balance among
these indicators we find CINVESTAV, ITCEL, UAMERP and UG. On the other hand,
although with relatively small score in excellence, ITTOL and ITMOR outstand with very
high values of the productivity indicator. It looks as if less quantity would be the price of
quality, and vice versa.
Contrary to what one might expect, the scatter plot of %Exc versus %Q1 in Fig. 8
exhibits very low degree of correlation among the indicators (correlation coefficient R2 ¼0:3755Þ: However, an important number of institutions (15) score above the mean values
of both indicators. ITTIJ shows the most atypical behavior: the lowest score in %Q1 (12.6)
with high score in %Exc (9.7). Nevertheless, for the rest of the institutions, the emptiness
of the lower right quadrant (below %Q1 = 30.25 and to the right of %Exc = 6.15) sug-
gests that the achievement of excellence is strongly linked to the publication in high
visibility journals. Differently, the upper left quadrant populated by 8 institutions suggests
that, with significant probability, a poor achievement of excellence can occur in spite of
publishing in high impact journals or equivalently: ‘‘Low excellence is not the result of the
publication in low impact journals’’.
Some of the issues revealed by the biparametric analysis are better understood from a
multiparametric perspective. In the next section an artificial intelligence approach, based
on self-organized mapping techniques, will serve to obtain a multiparametric characteri-
zation of the most productive Mexican universities.
Fig. 8 Excellence versus expected visibility in Mexican HEIs (2007–2011). For these two indicators
R2 ¼ 0:3755:
96 Scientometrics (2017) 110:77–104
123
Scientometric performance profiles of the Mexican Universities
Now we proceed to carry out the multiparametric scientometric performance analysis of
the 50 most productive Mexican HEIs, according to four indicators: ISP, %Q1, %Exc and
NI. Instead of obtaining four different rankings for each of these indicators, or analyzing
the dataset from the perspective of two indicators by means of scatter plots, here we face
the problem of comparing the performance of these institutions taking into account the four
scientometric indicators simultaneously.
For this purpose, we first represent each university as a point in a Euclidean four-
dimensional space, whose coordinates are given by values of the four indicators. We then
use a neural network to deal automatically with the comparison and visualization problem.
Our visualization technique throws five maps (cartographies) which exhibit the perfor-
mance comparison and clustering done by the neural network. These maps are drawn over
a hexagonal grid, where each hexagon represents one neuron. In this case we used a
70 9 20 neurons grid (Fig. 9).
Once the neural network is trained by means of an adaptive and unsupervised proce-
dure, each institution is plotted in the map applying a nonlinear projection from the 4D
space into the neural grid. Since we are using a similarity measure that considers the four
scientometric indicators, institutions projected close to each other over the grid share
similar bibliometric characteristics. The projection clusters the institutions on the map in
such a way that those having the most similar profiles are plotted inside the same cluster.
Since each cluster of institutions represents a bibliometric pattern of institutional
Fig. 9 Self-organized clusters map of Mexican universities obtained by the multiparametric analysis
Scientometrics (2017) 110:77–104 97
123
performance, this technology allows us to visually locate various bibliometric profiles of
academic production, and to identify groups of institutions that have similar patterns of
performance.
In Fig. 9 we observe five maps, the first one displays the clusters map of the 50
institutions. This map exhibits various regions (clusters) distinguished by colors, and the
group of universities that belong to each region. Universities in each cluster have similar
scientometric performance patterns, according to the four selected indicators. Atypical
behaviors (outliers) are represented by regions to which only one university belongs. Such
are the cases of UIBER, UASIN, UA GUE, ITTIJ, ITTOL, ITMOR, and UACM. The rest
of the universities share their production profile with other institutions of the group.
The other four maps in Fig. 9 display another type of visualization provided by the
neural network. They are called component maps. There is one map for each of the four
indicators that compose the quantitative bibliometric information considered for each
university. These maps are fundamental for the results interpretation. By means of a
chromatic scale they show the map’s distribution of indicator’s values. In each component
map the maximum value of the indicator corresponds to the darker red, the mean value is
associated to the yellow tone, and the minimum value to the lighter green. Intense yellow is
associated with values close to the mean of each indicator. Thus, component maps allow us
to read the meaning of the data distribution on the neural grid.
In the cluster visualization in Fig. 9, we can see that the neural grid is divided in thirteen
regions (clusters). As we have pointed out before, seven of these regions (O1,…,O7) are
characterized by the fact that there is only one institution (in this group of 50 universities)
that is mapped into them; the other six regions (C1,…,C6) contain more than one institution.
For the results interpretation, one must observe that the upper zone of the map labeled
%Q1 in Fig. 9, corresponds to the highest values (darker red) of the %Q1 indicator and, the
values of this indicator continuously decrease as we go down in the map. For example, the
institutions that belong to the regions O1, O2, O3, O7, C1, C2, and the upper part of the
region C3, have the highest percentages of articles published in first quartile journals. We
call this zone the High Expected Visibility Zone.
Similarly, in the map labeled %Exc, the upper-left zone corresponds to the highest
values of the %Exc indicator, and the same is true for the map labeled with the NI
indicator—a fact that was expected given the high degree of correlation among these two
indicators. We will call this zone the Efficacy Zone. The institutions that have the highest
scores in %Exc and NI are located in this zone of the map. This zone intersects with the
High Expected Visibility Zone and %Exc and %NI values decrease continuously as we
advance diagonally towards the lower-right corner of the map.
The lower-right corner of all the component maps is a relatively low value zone (below
average values) for all four indicators (ISP, NI %Exc, and %Q1). Most of the institutions in
Cluster C5 are characterized by low efficiency, low expected visibility, and low effectivity
profile.
Efficacy zone
Institutions that belong to this zone might have a good deal of high impact publications or
might be participating in some large international research projects that produce articles
(with hundreds of authors) which accumulate an extraordinary amount of citations. The
elite strip of the Efficacy Zone is composed by the institutions UIBER, UASIN, UAGUE
(O1, O2, O3), and the group of institutions that compose the cluster C1 = {UASLP,
CINVESTAV, UTLAX, BUAP, ITCEL}. Also in the Efficacy Zone, but with lower scores
98 Scientometrics (2017) 110:77–104
123
of NI and %Exc, are eight institutions belonging to cluster C2 = {UNAM, UAEM, UG,
UCOL, UMSNH, UAM, UAQ}. All the institutions in the Efficacy Zone are characterized
by high efficacy (impact of their outputs), but efficiency (ISP and IPR) mostly under the
mean values of the 50 most productive Mexican HEIs (this is observed in Scatter Plots
Figs. 5, 6). CINVESTAV has the most balanced profile within the 50 institutions sample;
and with scores well above averages in the four indicators considered, it is an exemplary
exception to this rule (observe the isolated red spot over CINVESTAV in the ISP Com-
ponent Map). ITCEL is the other institution of the group that has a well balanced profile.
Institutions that belong to the region C2 are characterized by their high percentage of
papers in first quartile (%Q1), efficacy parameters close to the mean value from above, and
efficiency scores (ISP, IPR) below the average.
As if efficiency were the price of efficacy, the three outliers (O1,O2,O3) of the Efficacy
Zone, having the highest %Exc and NI scores, do not outstand in their ISP scores. In fact,
UIBER, which conspicuously stands out in this group because of its extreme values of
%Exc and NI, does not have high efficiency indicator values (ISP and IPR). Also UASIN
and UAGUE share this high efficacy-low efficiency profile. UASIN is extreme in this
respect, because its ISP score is very close to the minimum value for all the universities,
but it is within the top three in %Exc and NI. The neural net differentiates these two
outliers, UASIN and UAGUE, in two clusters (O2, O3) due to their %Exc and NI disparity
(Fig. 6: NI larger for UASIN and %Exc larger for UAGUE).
Efficiency zone
In the ISP component map, we observe that the lower-left corner, the Efficiency Zone, is
where institutions with the highest ISP scores are located. Three outliers O4, O5, O6, part
of the region C6 (CNIDT), and the region where CINVESTAV is located, constitute the
elite group of the Efficacy Zone. ITTOL and ITTIJ are the leaders in this indicator for the
whole group of 50 institutions, followed by ITMOR and CNIDT. After them, we find
UTMIX and ITVER, with lower efficiency level but still above average. It is important to
remark that the average ISP value (1.4) for the 50 institutions is high with respect to the
same value for the rest of the country: the NSP in Scopus for the 2007–2011 period is
approximately 0.86. So ISP = 1, might be considered low with respect to the average
value (1.4), but it is relatively high with respect to the whole country.
High expected visibility zone
The upper part of the %Q1 component map concentrates institutions that have highest
percentage of publications in first quartile journals. As might be expected, the Efficacy
Zone is a subset of the Expected Visibility Zone, meaning that institutions with high
impact scores also have high scores of %Q1, but this is not a rule without exception. An
interesting exceptional case is that of ITTIJ (O4): while having the lower %Q1 value
among the 50 HEI, it scores high in %Exc, and has the second place in productivity (ISP).
Because of this unique profile ITTIJ does not fit in another cluster.
High expected visibility and low efficacy and low efficiency zone
There is a triangular zone in the upper-right part of the map that contains various insti-
tutions with relatively high values of %Q1 but low or just below average value of ISP,
Scientometrics (2017) 110:77–104 99
123
%Exc and NI indicators. This triangle contains the lower part of cluster C2, the upper part
of cluster C3, and cluster O7. Among the institutions with this bibliometric profile we find:
UACM, UAM, UAQ, USON, ITAM, and UVER. We remark that the ISP values for these
institutions are low, with respect to its average value (1.4), but for some of them this value
is still high relatively to the NSP.
The existence of this High Expected Visibility & Low Efficacy Zone confirms the well
known fact that publication in high impact journals does not guarantee high impact.
Furthermore, there is an uncommon situation in which high impact is reached without
publishing in high impact journals. Such counterexample is shown in the %Exc component
map. In the lower-left part of this map, we can see a region (red) with high %Exc score that
corresponds to the exceptional ITTIJ, for which the %Q1 indicator attains its minimum
value.
Conclusions
The artificial intelligence procedure that we devised has been successfully applied here for
scientometric data mining. Particularly, it was useful to carry out the multifactorial analysis
and scientometric characterization of the most productive Mexican universities. It allowed
us to automatically identify various institutional scientometric performance profiles, the
universities that fit in them, as well as bibliometric outliers that stand out with peculiar
profiles. Furthermore, it provided clear and useful data visualization resources that resulted
to be an excellent complement to the traditional scatterplot and correlation analysis. From
all of this we conclude that it is worth to consider the use of this analysis and visualization
procedure for science and technology data mining.
The non-linear projection capabilities offered by the SOM neural network and the novel
data normalization technique based on the logarithmic transformation of an appropriate
piecewise linear scaling, not only cluster the data but also offer different visual repre-
sentations which allow for further inferences about the data-set. As illustrated in the paper,
the traditional scatterplot and correlation analysis is useful only to observe global corre-
lations—outliers are identified only in the case where there is an extreme value. The
visualizations obtained are powerful tools to discover non-trivial outliers that otherwise
would be very difficult to discover.
It follows from our analysis that there is not a consistent correlation among productivity
(efficiency of the production) and effectivity (understood as the production impact) for the
Mexican HEIs (e.g., it is not the case that the more productive institutions are producing
the highest impact research). This phenomenon would have to be investigated in greater
depth in order to relate quantity and quality of the Mexican HEIs scientific production. A
negative correlation among quality and quantity was also observed at the country level: in
the more recent period a decrement in the number of scientific articles per national
researcher coexisted with a growing trend of the impact, estimated with the RI indicator.
However, this conclusion must be taken with caution, given the intrinsic volatility of this
indicator.
It was very important that the battery of selected scientometric indicators not only
considered production volume, but also provided an efficiency and quality estimation of
HEIs research output. Since there are great differences in size among the 50 most pro-
ductive Mexican HEIs, we introduced new size independent indicators that try to balance
these differences. For this, the use of NRS data was fundamental.
100 Scientometrics (2017) 110:77–104
123
Our approach has shown the prominence, not only of the larger institutions of the
country, but also some intermediate and, remarkably, small ones with outstanding scores in
scientometric indicators. This is a positive indication of the existence of excellent
researchers in these institutions, which is commendable and noteworthy. Notwithstanding,
for assessment purposes, or for the comparison of the larger with the smaller universities,
this fact has to be interpreted with care: statistical analysis, like the one we have applied
here, takes in account averages, and it is not the same to average the work of thousands of
researchers (e.g., UNAM case) than to average over just a few dozens of them. In big
universities the large numbers law minimizes the possibility of statistical irregularities.
Also, large universities, unlike relatively smaller technological institutes, tend to be the-
matically generalistic with large groups of researchers in areas of social science, arts and
humanities, which do not have the same publication patterns of engineers, mathematicians
or natural science researchers.
Scientometric characterizations are useful, but further studies are necessary for the
assessment of institution’s performance, to take into account other aspects of scientific
production. For instance, our current research remarks the importance of taking the number
of authors in publications into consideration, which should be analyzed in future studies. It
is noteworthy that most of the highest scores in the %Exc and NI indicators do not
correspond to the larger Mexican HEIs, but to relatively small ones, and a similar situation
is present in relation with the ISP indicator. Indicator %Q1 also displays unexpected
behavior. We have found that, with some exceptions, this indicator does not correlate
positively with the productivity (ISP indicator) nor with the impact indicators.
We close our conclusions pointing out, from a country perspective, that a comparative
analysis during the period 1996–2011 of the Mexican Scientific production in WoS and
Scopus reveals a sustained growth in both databases, which is kept with the growth of the
National Research System of Mexico. The evolution of the Mexican scientific production
in WoS and Scopus is very similar from 1996 to 2005, but from there on Scopus production
exhibits a higher level of growth.
Acknowledgements This research was partially supported by the Proyecto CITMA-CONACyT (B330.166)and the Empresa de Tecnologıas Inteligentes y Modelacion de Sistemas S.A. de C.V. The authorsacknowledge the collaboration of Jose Luis Jimenez Andrade (UNAM, Mexico), and of Dr. Felix de MoyaAnegon (CSIC, Spain) for the data support given from SCImago Institutions Rankings.
References
Abramo, G., Cicero, T., & D’Angelo, C. A. (2013). National peer-review research assessment exercises forthe hard sciences can be a complete waste of money: The Italian case. Scientometrics, 95(1), 311–324.
Aguillo, I. F., Bar-Ilan, J., Levene, M., & Ortega, J. L. (2010). Comparing university rankings. Sciento-metrics, 85(1), 243–256.
Aguillo, I. F., Granadino, B., & Llamas, G. (2005). Web positioning of the university system in LatinAmerica. Interciencia, 30(12), 735–738.
Allen, N., & Heath, O. (2013). Reputations and research quality in british political science: The importanceof journal and publisher rankings in the 2008 RAE. British Journal of Politics & InternationalRelations, 15(1), 147–162.
Arencibia-Jorge, R., & de Moya-Anegon, F. (2010). Challenges in the study of Cuban scientific output.Scientometrics, 83(3), 723–737.
Arvanitis, R., Russell, J. M., & Rosas, A. M. (1996). Experiences with the national citation reports databasefor measuring national performance: The case of Mexico. Scientometrics, 35(2), 247–255.
Scientometrics (2017) 110:77–104 101
123
Belkhodja, O., & Landry, R. (2007). The Triple-helix collaboration: Why do researchers collaborate withindustry and the government? What are the factors that influence the perceived barriers? Sciento-metrics, 70(2), 301–332.
Belter, C. W. (2013). A bibliometric analysis of NOAA’s Office of Ocean Exploration and Research.Scientometrics, 95(2), 629–644.
Bengoetxea, E., & Buela-Casal, G. (2013). The new multidimensional and user-driven higher educationranking concept of the European Union. International Journal of Clinical and Health Psychology,13(1), 67–73.
Benito, M., & Romera, R. (2011). Improving quality assessment of composite indicators in universityrankings: A case study of French and German universities of excellence. Scientometrics, 89(1),153–176.
Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? Sciento-metrics, 84(1), 237–263.
Bornmann, L., & Leydesdorff, L. (2013). Macro-indicators of citation impacts of six prolific countries:InCites data and the statistical significance of trends. PLoS ONE, 8(2), e56768.
Bornmann, L., Moya-Anegon, F., & Leydesdorff, L. (2012). The new excellence indicator in the WorldReport of the SCImago Institutions Rankings 2011. Journal of Informetrics, 6(2), 333–335.
Caputo, C., Requena, J., & Vargas, D. (2012). Life sciences research in Venezuela. Scientometrics, 90(3),781–805.
Collazo-Reyes, F., Luna-Morales, M. E., Russell, J. M., & Perez-Angon, M. A. (2008). Publication andcitation patterns of Latin American and Caribbean journals in the SCI and SSCI from 1995 to 2004.Scientometrics, 75(1), 145–161.
Collazo-Reyes, F., Luna-Morales, M. E., Russell, J. M., & Perez-Angon, M. A. (2010). Enriching knowl-edge production patterns of Mexican physics in particles and fields. Scientometrics, 85(3), 791–802.
CONACyT. (2012). Atlas de la Ciencia Mexicana. Mexico D.F: Academia Mexicana de Ciencias.CONACyT. (2013). Resultados Anteriores del Sistema Nacionales de Investigadores (SNI). http://conacyt.
gob.mx/index.php/resultados-anteriores.Dunn, J. C. (1973). A fuzzy relative of the ISODATA process and its use in detecting compact well-
separated clusters. Journal of Cybernetics, 3(3), 32–57.Egghe, L. (2012). Averages of ratios compared to ratios of averages: Mathematical results. Journal of
Informetrics, 6(2), 307–317.Foro Consultivo Cientıfico y Tecnologico. (2011). Ranking de Produccion Cientıfica Mexicana. http://www.
foroconsultivo.org.mx/libros_editados/ranking_por_institucion_2011.pdf.Gomez-Nunez, A. J., Vargas-Quesada, B., de Moya-Anegon, F., & Glanzel, W. (2011). Improving SCImago
Journal and Country Rank (SJR) subject classification through reference analysis. Scientometrics,89(3), 741–758.
Guzman, M. V., Carrillo, H., Jimenez, J. L., & Villasenor, E. A. (2010). Bioinformetric studies on TBvaccines research. In: N. M. Nor, A. Acosta & M. E. Sarmiento (eds.), The art and science ofTuberculosis vaccine development (pp. 425–441). Oxford: Oxford University Press.
Helene, A. F., & Ribeiro, P. L. (2011). Brazilian scientific production, financial support, establishedinvestigators and doctoral graduates. Scientometrics, 89(2), 677–686.
Huang, M. H. (2012). Exploring the h-index at the institutional level: A practical application in worlduniversity rankings. Online Information Review, 36(4), 534–547.
Ioannidis, J. P., Patsopoulos, N. A., Kavvoura, F. K., Tatsioni, A., Evangelou, E., Kouri, I., et al. (2007).International ranking systems for universities and institutions: A critical appraisal. BMC Medicine,5(30), 1.
Kohonen, T. (2001). Self-organizing maps (3rd ed.). Berlin: Springer.Kohonen, T. (2013). Essentials of the self-organizing map. Neural Networks, 37, 52–65.Krauskopf, M., Krauskopf, E., & Mendez, B. (2007). Low awareness of the link between science and
innovation affects public policies in developing countries: The Chilean case. Scientometrics, 72(1),93–103.
Krishna, D., Mohan, S. R., Murthy, B. S. N., & Rao, A. R. (2002). Performance evaluation of publicresearch institutes using principal component analysis. Journal of Scientific & Industrial Research,61(11), 940–947.
Kurzydlowski, K. J. (2003). Materials research in Poland. In W. Lojkowski & J. R. Blizzard (Eds.),Interfacial effects and novel properties of nanomaterials (pp. 349–356). Zurich-Uetikon: Trans TechPublications Ltd.
Lena, M. (1997). Scientific productivity in environmental psychology in Mexico—A bibliometric analysis.Environment and Behavior, 29(2), 169–197.
102 Scientometrics (2017) 110:77–104
123
Leydesdorff, L., & Meyer, M. (2007). The scientometrics of a Triple Helix of university–industry–gov-ernment relations (introduction to the topical issue). Scientometrics, 70(2), 207–222.
Licea de Arenas, J., Castanos-Lomnitz, H., & Arenas-Licea, J. (2002). Significant Mexican research in thehealth sciences: A bibliometric analysis. Scientometrics, 53(1), 39–48.
Licea de Arenas, J., Valles, J., & Arenas, M. (2000). Educational research in Mexico: Socio-demographicand visibility issues. Educational Research, 42(1), 85–90.
Luna-Morales, M. E. (2012). International scientific collaboration and recognition of Mexican science from1980 to 2004. Investigacion Bibliotecologica, 26(57), 103–129.
Macias-Chapula, C. A., Mendoza-Guerrero, J. A., Rodea-Castro, I. P., & Gutierrez-Carrasco, A. (2007).Institutional health research collaboration in Mexico: A Bibliometric study. In: Proceedings of the 11thinternational conference of the international society for scientometrics and informetrics (pp. 894–895).Madrid: ISSI.
Miguel, S., Chinchilla-Rodrıguez, Z., & Moya-Anegon, F. (2011). Open access and Scopus: A newapproach to scientific from the standpoint of access. Journal of the American Society for InformationScience and Technology, 62(6), 1130–1145.
Miguel, S., Moya-Anegon, F., & Herrero-Solana, V. (2010). The impact of the socio-economic crisis of2001 on the scientific system of Argentina from the scientometric perspective. Scientometrics, 85(2),495–507.
Moya-Anegon, F., Herrero-Solana, V., & Jimenez-Contreras, E. (2006). A connectionist and multivariateapproach to science maps: The SOM, clustering and MDS applied to library and information scienceresearch. Journal of Information Science, 32(1), 63–77.
Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2013). Absolute and specific measures of researchgroup excellence. Scientometrics, 95(1), 115–127.
Noyons, C. M. (2005). Science maps within a science policy context. In: H. F. Moed, W. Glanzel & U.Schmoch (eds.), Handbook of quantitative science and technology research (pp. 237–255). Springer:Netherlands.
Polanco, X., Francois, C., & Lamirel, J. C. (2001). Using artificial neural networks for mapping of scienceand technology: A multi-self-organizing-maps approach. Scientometrics, 51(1), 267–292.
Rehn, C., & Kronman, U. (2008). Bibliometric handbook for Karolinska Institutet. Estocolmo: KarolinskaInstitutet University Library.
SCImago Research Group. (2013). SIR Iber 2013. http://www.scimagoir.com/pdf/SIR%20Iber%202013.pdf.
Sierra-Flores, M. M., & Barnard, J. M. R. (2009). The most productive research groups of the NationalAutonomous University of Mexico (UNAM) in the area of physics, 1990–1999. Investigacion Bib-liotecologica, 23(48), 127–155.
Sierra-Flores, M. M., Guzman, M. V., Raga, A. C., & Perez, I. (2009). The productivity of Mexicanastronomers in the field of outflows from young stars. Scientometrics, 81(3), 765–777.
Skupin, A., Biberstine, J. R., & Borner, K. (2013). Visualizing the topical structure of the medical sciences:A self-organizing map approach. PLoS ONE, 8(3), e58779.
Sotolongo-Aguilar, G., Guzman-Sanchez, M. V., & Carrillo-Calvet, H. (2002). ViBlioSOM: Visualizacionde informacion bibliometrica mediante el mapeo autoorganizado. Revista Espanola de DocumentacionCientıfica, 25(4), 477–484.
Sotolongo-Aguilar, G., Guzman-Sanchez, M. V., Saavedra-Fernandez, O., Carrillo-Calvet, H. A. (2001).Mining informetric data with self-organizing maps. In Proceedings of the 8th international society forscientometrics and informetrics (pp. 665–673). Sydney: BIRG.
Statzner, B., & Resh, V. H. (2010). Negative changes in the scientific publication process in ecology:potential causes and consequences. Freshwater Biology, 55(12), 2639–2653.
Thomson Reuters. (2015). http://incites.isiknowledge.com/common/help/h_glossary.html.Torres-Salinas, D., Moreno-Torres, J. G., Delgado-Lopez-Cozar, E., & Herrera, F. (2011). A methodology
for institution-field ranking based on a bidimensional analysis: The IFQ(2)A index. Scientometrics,88(3), 771–786.
Ultsch, A., & Morchen, F. (2005). ESOM-maps: Tools for clustering, visualization, and classification withemergent SOM. Technical report. 46. Department of Mathematics and Computer Science, Universityof Marburg, Germany.
UNAM. (2012). Estudio Comparativo de Universidades Mexicanas. http://www.ecum.unam.mx/.Vanclay, J. K., & Bornmann, L. (2012). Metrics to evaluate research performance in academic institutions:
A critique of ERA 2010 as applied in forestry and the indirect H-2 index as a possible alternative.Scientometrics, 91(3), 751–771.
Vesanto, J. (1999). SOM-based data visualization methods. Intelligent Data Analysis, 3(2), 111–126.
Scientometrics (2017) 110:77–104 103
123
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011). Towards a newcrown indicator: An empirical analysis. Scientometrics, 87(3), 467–481.
Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scien-tometrics, 62(1), 117–131.
Zell, D. (2005). Pressure for relevancy at top-tier business schools. Journal of Management Inquiry, 14(3),271–274.
Zhao, S. X., Zheng, L. C., & Cai, Z. C. (2009). Evaluation of science and technology innovation ability ofcolleges and universities in China. In International conference on management of tech-nology,pp. 220–225.
104 Scientometrics (2017) 110:77–104
123