Replicating a study doesn't just put the robustness of scientific findings to the test – researchers also learn something for their own work along the way. In Replication Games, researchers replicate papers together in teams, so everyone benefits directly from one another's experience and expertise. Lenka Fiala from the Institute for Replication (I4R) presented the format at the Coffee Lecture on Open Science Education – and made clear that it can also be a useful tool for teaching Open Science in economics.
On 8 September 2026, the seventh event in the “Coffee Lecture on Open Science Education” series took place. Lenka Fiala, Research Scientist at the Institute for Replication (University of Ottawa) and affiliated with Tilburg University, presented the Replication Games organised by the I4R, framing them as both a teaching tool and a piece of research infrastructure.
To establish a common conceptual framework, Fiala first distinguished between different types of reproducibility: Computational Reproducibility means that the same findings can be generated using the original data and code. Robustness Reproducibility likewise draws on the original data, but applies different, equally plausible analytical approaches. Direct Replicability, by contrast, occurs when the same research question is examined using new data and either the same method or a slightly modified one, and arrives at the same conclusion.
Taken together, the three concepts address different questions, and it's only together that they give a complete picture of how robust a finding really is. Testing as many findings as possible in this sense is essential, Fiala argued, so that researchers know which results they can safely build on. Errors aren't always caught, since Peer Review does not uncover every type of error or questionable analytical choice.
This is where Replication Games – in which teams of typically three to six researchers reproduce a paper – can make a real difference: even on a small scale, they catch incorrect results early and so save valuable resources. Beyond this substantive effect, Replication Games can also shape research culture more broadly. While replications and comments have long been seen as adversarial, the collaborative format helps make dialogue and independent verification a normal part of doing research. According to Fiala, some original authors are genuinely grateful when someone considers their paper important enough to spend time reproducing – or replicating – it.
Another advantage, particularly relevant for researchers themselves, is the learning effect it has on their own work. After signing up, participants prepare by reading the assigned paper and working through its replication package. The Replication Games run by I4R take the form of one-day hackathons, during which teams test various types of reproducibility for their paper and receive feedback and guidance from I4R along the way. The resulting report first goes to the original authors in confidence, before the final version is published on the I4R website together with their response. Individual reports can ultimately feed into joint meta-papers with, in some cases, hundreds of co-authors – papers that are particularly robust precisely because so many researchers have been involved and scrutinised the results multiple times over.
Fiala devoted a section of her talk specifically to (open) data in the context of replication games. The advantages of Open Data are clear: they enable anyone to verify research findings. This increases the chances of identifying errors. Because the data used are archived, verification also remains possible long after publication – which is precisely what makes large-scale replication projects feasible in the first place. Caution is needed, though, when data contain personal information: publication isn't always possible in that case. To avoid sensitive information being shared by accident, Fiala recommends that researchers use the PII Checker.
Checking papers based on restricted-access data is more demanding, but at least as important. For one thing, fewer people are able to carry out this kind of scrutiny, since Open Code alone, without access to the underlying data, is often not enough to spot errors. For another, many analyses using restricted-access data – administrative data, for example – feed directly into policy decisions, so the stakes involved are often high. Replication Games can offer a way forward here too: with the CBS Replication Games, Fiala presented a new initiative, so far carried out in the Netherlands, that adapts the format for papers based on restricted-access administrative data.
Finally, Fiala placed the current state of reproducibility and replicability in economics in context, drawing on a new study by Brodeur et al. (2026) (link to PDF). The study carried out original analyses and robustness checks on 110 articles published in leading economics and political science journals with mandatory data- and code-sharing policies. The trend is clearly positive: between 2014 and 2023, the share of empirical papers with a replication package rose from 60 to over 90 per cent. The 85 per cent reproducibility rate, as well as the share of papers sharing their analysis code, are also encouraging. Fiala likewise sees a positive trend in the sharing of so-called cleaning code, used to clean raw data. When it comes to sharing the raw and analysis data themselves, however, there's still room for improvement.
Replication Games could be one way of pushing the shift towards Open Science in economics further. When it comes to one's own teaching, it's worth first looking at existing formats such as I4R's: organising one's own games involves no small amount of effort and cost, whereas joining one of the many events already taking place is refreshingly straightforward. That fits neatly with Fiala's underlying idea: reproducibility is ultimately a collective action problem – and collaboration that draws on participants' comparative advantages is how we solve it.
Tip: You can read more about replication research in our article ‘R2: A new home for replication research’.