Step 6: Research evaluation
When research is to be evaluated, quantitative methods have historically been important. Primarily, great importance has been placed on bibliometric methods, that is, measuring publication volumes and the number of citations for the publications. In recent years, however, such methods have been debated, which has led to an ongoing reform of the evaluation system that includes an upgrade of peer review as a method for analysis, and a more cautious use of quantitative methods. Familiarize yourself with the evaluation model that applies to the funder you are applying for funds from, to understand how different aspects are valued.
Coara
Several major research funders have signed the so-called CoARA agreement. CoARA stands for Coalition for Advancing Research Assessment, an initiative that more than 300 organizations from 40 countries collaborated on and which was launched in 2022. The ambition with CoARA is to reform research evaluation, so that more research activities become meritorious, and to increase transparency and fairness in research evaluation. Among other things, CoARA states that activities such as teaching, leadership and supervision should be given greater recognition than before, and when assessing individual research projects, activities such as early and open sharing of research data or various contributions to societal benefit should be rewarding. Overall, CoARA includes ten overarching principles that should guide the organizations that have signed.
Bibliometrics
CoARA emphasizes the importance of responsible use of bibliometrics, and wants to tone down the importance of bibliometric measures such as Journal Impact Factor and h-index. Since such and other bibliometric measures are still used, a brief introduction to what bibliometrics is about is given below.
Bibliometric analyses aim to provide a basis for understanding how research is produced, organized, and how collaboration around research takes place. Analyses of publications can be done at the individual level, by research groups, higher education institutions, or countries. For the most part, citations are considered as a quality measure and are thus an important measure in the analyses. Citations show how much impact a particular publication has had. The starting point is that the more citations a publication has received, the higher quality it is considered to have. Citation analyses can also show which journals, organizations and countries have a great impact in different research fields.
For you as a researcher, the visibility of your research can increase if it is included in databases used for bibliometric analyses. Databases often used for bibliometric analyses are Web of Science and Scopus.
Journal Impact Factor
Journal Impact Factor has been a commonly used measure to rank the popularity of journals. The measure gives an overall picture of how often a journal's articles are cited. For journals, it has become important to try to get as high an impact factor as possible to increase the journal's prestige and to attract researchers. Therefore, the measure has also been misused in a variety of ways. A journal's official impact factor is the one linked to the Web of Science database, but other databases can use similar measures.
H-index
Research funders sometimes ask applicants to submit their h-index. The h-index measures an author's productivity and citation impact. The h-index is based on the author's most cited publications, which are compared to the number of citations the author has received on other publications. The h-index will be different in different databases, as the content in them can vary. If you need to state your H-index in an application, therefore remember to also state which database has been used.
Altmetrics
The term altmetrics is a combination of the words "alternative" and "metrics". Altmetrics is a way to measure the impact of research in a different way, or as an addition to bibliometrics. Although altmetrics is primarily intended for journal articles, it is also used to measure the impact of researchers, journals, books, datasets, presentations, videos, program code, web pages and much more.
Altmetrics can be used to measure the impact a journal article has had in the world outside academia, for example by calculating how many downloads it has received or mentions in social and news media.
At present, altmetrics is not used by the government or Karlstad University to distribute research funds. For researchers, however, it may be interesting to follow how the publications circulate within and outside academia. For that reason, our publication database DiVA uses altmetrics from the company Altmetrics to show how much impact the research gets in social media.