One of the largest reproducibility studies to date in economics and political science has been published in Nature. The project was led by the Institute for Replication (I4R) and examined 110 articles from leading journals with strict requirements for open sharing of data and code.
– The study provides a more nuanced picture than the common narrative that published research either stands or falls, says Carl Bonander, who together with Niklas Jakobsson, both at Karlstad Business School, took part in the study.
A central principle of science is that results should be verifiable by other researchers. The study therefore investigated two key questions:
- Can published results be reproduced using the original code and data?
- Do the results hold up when reasonable alternative analytical choices are made?
The study covers quantitative articles published in 2022–2023 in leading journals in economics and political science.
Main findings of the study
Computational reproducibility
More than 85 percent of the reviewed results could be reproduced using the original code and original data. At the same time, roughly one in four studies contained non-trivial coding errors.
Robustness
When researchers applied reasonable alternative analytical choices to the same data, 72 percent of the results that were originally statistically significant remained statistically significant in the same direction.
Effect sizes
The median effect size in the re-analyses was very close to the original results—about 99 percent of the original effect. This suggests that differences often concerned statistical precision and significance rather than large changes in the point estimates themselves.
Experience matters
The most experienced replication teams tended to find lower robustness. However, the researchers found no clear relationship between robustness and the authors’ background or data availability.
An important limitation
This is likely a best-case scenario. The study focuses on top-tier journals with strong requirements for data and code sharing, which means that the results probably represent an upper bound for reproducibility in the social sciences as a whole.
Contribution from the School of Business at Karlstad University
Among the more than 350 co-authors are Niklas Jakobsson, Professor of Economics, and Carl Bonander, Associate Professor of Statistics, both at the School of Business at Karlstad University. They contributed to the project by reviewing two of the 110 studies.
Effects of an age-specific lockdown in Turkey (Altindag et al. 2022)
The original study used a regression discontinuity design to examine how a lockdown targeted at individuals aged 65 and older in Turkey affected mobility and mental health during the COVID-19 pandemic. Jakobsson, Bonander, and Gabriella Chauca Strand successfully reproduced the original results and tested robustness by, among other things, removing clustering of standard errors, excluding control variables, and using alternative bandwidth choices. The mobility results were stable, while the effects on mental health were more sensitive to methodological choices.
The carbon tax and emissions in Sweden (Andersson 2019)
The original study used the synthetic control method to estimate that Sweden’s carbon tax reduced emissions by approximately 11 percent during the period 1990–2005. Jakobsson, Bonander, and Naimi Johansson reproduced the original results and tested 14 alternative specifications. The point estimates were relatively stable (median −0.28 tons of CO₂ per capita, compared with the original −0.29), but statistical significance varied depending on model choice.
In both cases, the original results were largely confirmed, but the robustness analyses also showed that some findings are more sensitive to methodological choices than is apparent from the published articles. These are precisely the kinds of nuances that the larger Nature project brings to light.
Niklas Jakobsson and Carl Bonander have an ongoing collaboration with the Institute for Replication and also participate in other studies within the framework of I4R’s activities.
– This project shows that reproducibility cannot be taken for granted, not even in the best journals, says Niklas Jakobsson. That independent researchers systematically review published results is crucial for strengthening the credibility of social science research.
– In our two reviews, we were largely able to confirm the original results, but we also saw that some findings were more sensitive to methodological choices than the published articles suggested. These are exactly the kinds of nuances that need to be highlighted. Our collaboration with the Institute for Replication continues, and we look forward to contributing to further reviews.
Why is this important?
– The study provides a more nuanced picture than the common narrative that published research either holds or collapses, says Carl Bonander. Much can be reproduced technically, but that does not mean that all results are stable under closer analytical scrutiny. At the same time, the findings do not suggest that effect sizes generally collapse upon re-analysis; rather, the main picture is that uncertainty often increases and fewer results remain statistically significant.
– The study therefore underscores the value of open data, shared code, and independent replication studies, Bonander adds. These practices strengthen both the ability to detect errors and the foundations for more reliable social science research.
More than 350 researchers from around the world participated in the study.
