Going strong - RASP-UK

 It’s lovely to start the day with a little good news, better still to start the week. This morning I welcomed the week with the notification that our 2020 publication from the RASP-UK programme has hit over 100 citations [1]. But what does this really mean? Well, certainly, it is one of my highest cited papers from the last 40 years of research. In recent years, bibliometric indicators have increasingly been applied in the context of research evaluation as well as research policy more generally. Examples include the use of citation indicators in evaluation of the scientific performance of research groups, departments, and institutions [2]; evaluation of research proposals [3]; and allocation of research funding [4]. Citation measures are also core indicators in several university rankings, such as the Leiden ranking and Academic Ranking of World Universities (ARWU) [5].

 Citations are the standard means by which authors acknowledge the source of their methods, ideas and findings. Sixty years ago, Eugene Garfield published the Science Citation Index (SCI), the first systematic effort to track citations in the scientific literature. Indicators or metrics are applied for a variety of purposes and have permeated many aspects of the research system and, in the biological science at least, having over 100 citations for a scientific article is a significant achievement and can be interpreted as an indicator of the quality, impact, and value of your research. Particularly as ours is not a ‘methods’ paper, which by becoming standard applications can dominate the literature. The most cited work in history, for example, is a 1951 paper describing an assay to determine the amount of protein in a solution [6]. It has now gathered more than 305,000 citations — a recognition that always puzzled its lead author, the late US biochemist Oliver Lowry. “Although I really know it is not a great paper … I secretly get a kick out of the response,” he wrote in 1977 [7].

 The suggestion is that the higher your number of citations the higher its relevance, i.e., that work is addressing a question or topic of substantial interest to the scientific community, and it may have filled an important gap in knowledge or introduced new insights, methodologies, or applications. A high citation count indicates that many researchers find your work credible and useful as a foundation or reference for their own studies. It also suggests that your findings have been widely recognised and integrated into subsequent research. As a contributor to the work you would hope that it also indicates that it contributes meaningfully to the development of the field, whether through theoretical advancement, experimental results, or practical applications, possibly even helping to shape new directions in research, policies, or industry practices.

 Articles with high citation counts often have broader implications, influencing not only their primary field but also related disciplines. It may signal interdisciplinarity, making the findings applicable in multiple contexts. Citation counts are riddled with other confounding factors. The volume of citations has increased, for example — yet older papers have had more time to accrue citations. A high citation count over time indicates sustained relevance, demonstrating that the research is not a short-term trend but has lasting importance. Our citations have continued to build since publication in 2020.

 There is no evidence that citations reflect other key dimensions of research quality. In fact, evidence points to the contrary [8], citation-based indicators don’t provide nuanced or robust measures of quality when used in isolation and are little more than peer validation, suggesting that the scientific community considers the methodology, analysis, and conclusions robust and trustworthy. Certainly, our score of 100 is far below that of Lowry et al [7]. We don’t even come close to scores for some ‘failed’ research. The most-cited retracted paper is a 2002 Nature paper that claimed adult stem cells from bone marrow could become almost any cell type in the body [9]. The paper was retracted 22 years after its publication due to questions about the legitimacy of the data. The publishing journal concluded (eventually) that they no longer had confidence in the reliability of the data or that the conclusion that the stem cells could engraft in bone marrow was supported.

 In summary, an article with over 100 citations typically (hopefully) reflects high scientific quality and significant value to the academic and/or professional community. However, in some fields, like asthma, articles can accrue more citations than in niche or theoretical fields. In the end, a comprehensive assessment of your work’s impact should consider other factors such as real-world applications, influence on policies, and feedback from peers. We have yet to uncover the true legacy of the RASP-UK’s research efforts on the treatment of severe asthma, but we are still contributing to the discussions.

References

1. Heaney L, et al. (2020) Composite type-2 biomarker strategy versus a symptom–risk-based algorithm to adjust corticosteroid dose in patients with severe asthma: a multicentre, single-blind, parallel group, randomised controlled trial. The Lancet Respiratory Medicine, Volume 9, Issue 1, 57–68. DOI: 10.1016/S2213-2600(20)30397-0
2. Moed H F. (2005) Citation analysis in research evaluation. Dordrecht, The Netherlands: Springer.
3. Cabezas-Clavijo A. et al (2013) Reviewers’ ratings and bibliometric indicators: Hand in hand when assessing over research proposals? PLoS ONE, 8(6), e68258.
4. Carlsson H. (2009) Allocation of research funds using bibliometric indicators—Asset and challenge to Swedish higher education sector. InfoTrend, 64(4), 82-88.
5. Piro FN, Sivertsen G. (2016) How can differences in international university rankings be explained? Scientometrics, 109, 2263-2278.
6. Lowry OH, et al. (1951) Protein measurement with the Folin phenol reagent. J Biol Chem 193, 265–275.
7. Van Noorden R, et al. The top 100 papers. Nature explores the most-cited research of all time. Nature 29 October 2014
8. Aksnes DW. (2019) Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. Sage Open doi.org/10.1177/2158244019829575