Research Counts, Not The Journal

‘If there is one thing every bibliometrician agrees, is that you should never use the journal impact factor (JIF) to evaluate research performance for an article or an individual – that is a mortal sin’. Few sentences could define so precisely the uses and misuses of the Journal Impact Factor (JIF) better than Anthony van Raan’s. This manuscript presents a critical overview on the international use, by governments and institutions, of the JIF and/or journal indexing information for individual research quality assessment. Interviews given by Nobel Laureates speaking on this matter are partially illustrated in this work. Furthermore, the authors propose complementary and alternative versions of the journal impact factor, respectively named Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal – never of a paper or an author. The idea behind impact factors is not useless, it has just been misused.


Introduction
The Journal Impact Factor (JIF), today calculated by Clarivate Analytics (previously intellectual property of Thomson Reuters), was originally proposed by Irving H. Sher and Eugene Garfield in 1963 [1] as a tool to help librarians identify journals worth purchasing; not as a measure of the research quality in an article. The JIF is computed in each annual issue of the Journal Citation Reports (JCR, an annual publication by Clarivate Analytics), and represents the average number of citations per article in that year, concerning only papers published in that journal in the last two years: on five years (instead of two) prior to the JCR year. Citing Clarivate Analytics [2]: "the JIF is a journal-level metric, thus it does not apply to individual or subgroups of papers published in the journal, nor to authors of papers, research groups or institutions." Even so, it´s well-known in the scientific community that the JIF has increasingly become a means of judging the scientific quality of a journal and of everything that is published in it [3,4].
According to Pudovkin [5], and confirmed by the authors, there is extensive literature criticizing the JIF (mis)uses. The main argument against the use of JIF for characterization of individual papers and scientists is a lack of strong correlation between its value and the citedness of the individual papers published in the journal. There is no guarantee that an article published in a high IF journal will attract profuse citations [6]. It is known [7] that only a fraction of the papers published in a high IF journal receive most citations contributing to the JIF, regularly known in scientometrics as the 'invitation's paradox' [8]. By the end of the last century, Seglen [9] reported that 15 and 50 per cent of the most cited papers account for 50 and 90 percent of the citations used to compute the JIF, respectively. Moreover, (i) information abounds about classic highly-cited papers that were initially rejected by high IF journals [10,11], and (ii) data from several lines of evidence suggest that the reliability of scientific experiments decreases with increasing JIF [12].
This manuscript presents a critical overview on the use of the journal impact factor and/or journal indexing information for the scientific performance assessment of a researcher. The authors present a research-based opinion article, which addresses a serious problem that, in our opinion, remains fully topical until it is solved. Particularly, Figures 1-7 partially illustrate interviews given by Nobel Laureates on this topic. Lastly, complementary and alternative versions of the JIF are proposed, respectively called Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal -never of a paper or an author. The idea behind impact factors is not useless; it has just been misused. The authors believe that in the ever-evolving landscape of science, it is necessary to keep the discussion going and to keep searching for improvements.

The JIF and indexing information as individual research quality measures
Deans, sponsors, government agencies and employment panels use the JIF as a convenient, yet flawed, performance measure [13]. According to Pudovkin's 30-year experience [5] as a member of research staff recruitment committees at the institute of Marine Biology and the Russian Academy of Sciences, members in those committees typically just look at the journal titles. In many Universities in non-English-speaking countries, only papers published in WoS-indexed journals count towards merit [14]. The authors of this article oppose these types of judgement and support the opinions shared by many scientists as presented in the next section. In what follows, several examples are given of countries and institutions supporting these types of (mis)evaluation.
Instituto Superior Técnico, one of the most renowned schools of engineering and technology in Portugal (the only Portuguese member of the CLUSTER -Consortium Linking Universities of Science and Technology for Education and Research [15]), part of the University of Lisbon, considers the journal impact factor quartiles published in WoS and/or Scopus databases to compute a quantitative measure of the scientific performance of its faculty staff (as described in chapter II, article 17 of UL [16]) -papers from journals in the 1st (Q1), 2 nd or 3 rd quartiles are given six, four or two times higher classification than the Q4 counterpart, respectively. Instituto Superior Técnico is also known, at least in some departments, for disregarding or giving much less credit to articles not published in Q1/Q2 WoS-(e.g., UL 2017b, d [17, 18]) or Scopus-(e.g., [19]) indexed journals when assessing candidates for faculty positions in public tenders.
Ostravská univerzita v Ostravě [20] shows an example in the Czech Republic of a job application requiring a list of publications in WoS-indexed journals and the number of citations provided by that database, meaning that what matters the most is not the content of the publications, but where they are indexed. Furthermore, the inbound citations 'from journals' not indexed in WoS are of no value for that evaluation committee.
In Brazil, the evaluation of graduate programs relies heavily on JIFs. Journals and their papers are classified within seven categories with decreasing IF ranges (A1, A2, B1, B2, B3, B4, C), known as the 'Qualis System' [22]. In 2001, the Brazilian ministry of science and technology created a program to affiliate ten promising young doctorates to research centers. In the same year, the Brazilian national research council (CNPq) opened a two-year research fellowship call. The selection criterion for both programs was primarily based on the (i) number of publications and (ii) JIF [23]. After interviewing some postgraduate students about their concerns in academia, one typical statement was 'The adviser doesn´t care about my thesis so much. He believes that a thesis is the consequence of good work, and good work means papers published in good journals' [23].
Chinese policies offering financial reward based on WoS-indexed publications began in earnest in the 1990s, aiming to increase production and international visibility. In some Chinese and Swedish institutions, PhD students should publish at least two articles 'with an average IF' of four to get their degree [4].
In India, recruitment, awards, fellowships and promotions are determined by IF [21]. For the evaluation and ranking of researchers in Ecuador [24], their publications are categorized as follows: (i) "Level 1": publications indexed by WoS or Scopus, Q1 or Q2 in JCR or Scimago Journal Ranking (SJR), (ii) "Level 2": publications indexed by WoS or Scopus, Q3 or Q4 in JCR or SJR, (iii) "Level 3": relevant works and indexed articles not covered by Levels 1 and 2. For faculty hiring [25] peer-reviewed papers count for 2 points per paper, and a maximum of 4 points, and, additionally, indexed papers count for 2 points per paper and a maximum of 10 points. For the overall ranking of universities in Ecuador, similar metrics are used. A group of 74 Ecuadorian academics already criticized this practice [26], bringing up the argument (amongst others) that only 242 Latin American journals (out of more than 5000) are indexed in WoS and Scopus, and those typically rank low. Moreover, the currently used metrics result in low ratings for researchers and institutions in the humanities and social science fields.

Scientific community against the way research quality is assessed
The 2012 San Francisco Declaration on Research Assessment (DORA) [27] was outlined by a group of journal editors and publishers at a meeting of the American Society for Cell Biology (ASCB) and its aim is to promote a world in which the content of a research article is not assessed by the IF of the journal where it appears. Up until now (September 26 th , 2018) the declaration gathered 580 and 12779 online institutional and individual signatures, respectively. Momentum is being built, particularly in the UK, where recently all seven research councils announced their support [28].  Electronic copy available at: https://ssrn.com/abstract=3368683 In another example [34] concerning the Molecular Pathology journal and the Journal of Clinical Pathology, which have a combined JIF, the total number of citations to their papers in the year 2000 was no less than 7144 (making them rank 7 th among all pathology journals for the total number of citations). However, only 930 of those more than 7144 citations were to papers published in 1998-1999, meaning that no more than 13 per cent of citations was used to compute the 2000's JIF. Considering the former, aiming to balance the way research performance is measured, and trying to keep the current Impact Factor definitions useful, a new metric is suggested. The 'Complementary Impact Factor (CIF)' is proposed as the exact complement of the current JIF or 5-year IF. Therefore, it accounts only for citations to articles more than two or five years old, respectively. It is calculated as shown in Eq. (2) as the complement for the 2-year JIF and in Eq. (3) as the complement for the 5-year JIF.  [40,56].
As an alternative to CIF and the current IF metrics, the authors would also like to propose a single novel impact factor named 'Timeless Impact Factor (TIF)', where the differences for the current IFs are that it accounts for citations (i) to all publications ever published in the journal, (ii) received until the end of the year preceding the calculation, and (iii) from any source (e.g., WoS, SCOPUS, others) -not just those from WoS-indexed journals (a minute proportion of the citing ones [3]). Moreover, it is proposed that each citation is weighted as function of its source, as proposed by Abambres and Arab [57]-1.0 for WoS, 0.8 for SCOPUS, and 0.5 for others. The expression of the TIF then reads: where (i) Np is the total number of papers published in the journal before year n, (ii) ni is the number of papers citing paper i before year n, and (iii) wij is the weighted citation from citing article j to cited article i (1.0, 0.8 or 0.5 if the citing article is indexed in WoS, SCOPUS, or other databases, respectively). When the dispersion on citations needs to be studied, the metric using the Gini coefficient as proposed in [58] can be analyzed.   As a final remark, the authors suggest the withdrawal of all self-citations (citations from journal articles to the journal in which they are published) for the calculation of any type of impact factor. Not that it is necessarily unethical, but because it helps reducing the extent of 'fabricated IFs' [60]. It is believed that the previously proposed metrics could make the concept of impact factor far more useful, as an average measure of the journal quality.

Final Remarks
This paper highlighted several viewpoints about the widespread misuse of the journal impact factor and indexing databases in the assessment of individual performance of scientists. The authors seriously disagree with this form of evaluation, as already did several Nobel Laureates who publicly spoke out against it. Since the concept inherent to impact factors is not useless, two novel metrics (Complementary and Timeless Impact Factors) are proposed in this work, aiming to contribute to a more sustainable academia.
Author Contributions: Conceptualization and first-draft writing: M.A.; All other contributions: equally distributed among all authors (AS's contribution was given before April 1 st , 2018).

Conflicts of Interest:
The authors declare no conflict of interest.