I am happy to announce that Robbie van Aert, Jelte Wicherts, and I received seed funding from the Herbert Simon Research Institute for our project to screen COVID-19 preprints for statistical inconsistencies.
Inconsistencies can distort conclusions, but even if inconsistencies are small, they negatively affect the reproducibility of a paper (i.e., where did a number come from?). Statistical reproducibility is a basic requirement for any scientific paper.
We plan to check a random sample of COVID-19 preprints from medRxiv and bioRxiv for several types of statistical inconsistencies. E.g., does a percentage match the accompanying fraction? Do the TP/TN/FP/FN rates match the reported sensitivity of a test?
We have 3 main objectives:
- Post short reports with detected statistical inconsistencies underneath the preprint
- Assess the prevalence of statistical inconsistencies in COVID-19 preprints
- Compare the inconsistency-rate in COVID-19 preprints with the inconsistency-rate in similar preprints on other topics
We hypothesize that high time pressure may have led to a higher prevalence of statistical inconsistencies in COVID-19 preprints as opposed to preprints on less time sensitive issues.
We thank our colleagues at the Meta-Research Center for their feedback and help in developing the coding protocol.
See the full proposal here.