Category Archives: statcheck
New Preprint: Using Statcheck in Peer Review May Reduce Errors
We investigated whether statistical reporting inconsistencies could be avoided if journals implement the tool statcheck in the peer review process.
In a preregistered study covering over 7000 articles, we compared the inconsistency rates between two journals that implemented statcheck in their peer review process (Psychological Science and Journal of Experimental and Social Psychology) with two matched control journals (Journal of Experimental Psychology: General and Journal of Personality and Social Psychology, respectively), before and after statcheck was implemented.
Preregistered multilevel logistic regression analyses showed that the decrease in both inconsistencies and decision inconsistencies around p = .05 is considerably steeper in statcheck journals than in control journals, offering support for the notion that statcheck can be a useful tool for journals to avoid statistical reporting inconsistencies in published articles.
You can find the full preprint here: https://psyarxiv.com/bxau9
Statcheck Version 1.4.0 on CRAN!
Happy to announce that the latest version of statcheck, version 1.4.0, is now officially published on CRAN. Some new features include:
- better stats finding
- completely refactored internals
- cleaner variable names
You can find a list of all updates here: https://github.com/MicheleNuijten/statcheck/blob/master/NEWS.md
Young eScientist Award
In December 2020, Willem Sleegers and I were awarded the Young eScientist Award from the Netherlands eScience Center for our proposal to improve statcheck’s searching algorithm. Today marks the start of our collaboration with the eScience Center and we are very excited to get started!
In this project, we plan to extend statcheck’s search algorithm with natural language processing algorithms, in order to recognize more statistics than just the ones reported perfectly in APA style (a current restriction). We hope that this extension will expand statcheck’s functionality beyond psychology, so that statistical errors in, e.g., biomedical and economics papers can also be detected and corrected.
More information about the award can be found here.
METAxDATA Meeting at QUEST, Berlin
Last month, the QUEST center in Berlin organized the first METAxDATA meeting on building automated screening tools for data-driven meta-research. On the first night of the meeting, 13 researchers gave lightning talks about their tools. The clip below features my <2 minute lightning talk about statcheck.
All lightning talks were recorded and can be found here.
My dissertation is finished!
The cover: my own desk, feat. SIPS, BITSS, and COS. Cover design by Niels Bongers.
The contents: statcheck, data sharing, meta-analysis, power, bias, and more.
You can find the full thesis here: https://psyarxiv.com/qtk7e
Reporting Inconsistencies in X-Phi
In a new paper, we ran statcheck on a bunch of experimental philosophy papers. Inconsistency rates are lower than in psychology, and evidential value seems high. Good news for the philosophers! See the full paper here.
Science Insider: statcheck’s Accuracy is High
In the latest Science Insider written by Dalmeet Singh Chawla I argue that statcheck does exactly what it’s supposed to do: check the consistency of APA reported NHST results.
Read the entire piece here.
New Preprint: statcheck’s Validity is High
In our new preprint we investigated the validity of statcheck. Our main conclusions were:
- statcheck’s sensitivity, specificity, and overall accuracy are very high. The specific numbers depended on several choices & assumptions, but ranged from:
- sensitivity: 85.3% – 100%
- specificity: 96.0% – 100%
- accuracy: 96.2% – 99.9%
- The prevalence of statistical corrections (e.g., Bonferroni, or Greenhouse-Geisser) seems to be higher than we initially estimated
- But: the presence of these corrections doesn’t explain the high prevalence of reporting inconsistencies in psychology
We conclude that statcheck’s validity is high enough to recommend it as a tool in peer review, self-checks, or meta-research.
Paper Accepted @ Collabra: Psychology
Our paper “Journal data sharing policies and statistical reporting inconsistencies in psychology” has been accepted for publication in the open access journal Collabra: Psychology!
The updated (accepted) pre-print can be found on PsyArXiv: https://psyarxiv.com/sgbta.