Category Archives: statcheck

pexels-photo-20279300.jpeg

NWO SSH-XS Grant: Gaming Automated Peer Review

Posted on by

I am very happy that my project about “gaming” automated peer review tools has been awarded an NWO XS grant!

We know that peer review is increasingly overburdened, so could we use automated tools to help out? … maybe, but one of the things we still don’t know much about is the risk of “gaming” these tools.

A big upside of automation is that it is efficient. Some tasks, like spotting if certain elements are present in a manuscript seem particularly suited for automated checks. BUT: what if authors can bypass these checks by just superficially changing some wording but not fixing the actual problem? What if, for example, instead of actually doing a power analysis, they work in the word “power” and the tool counts it as “check passed”?

In my study, I will look at the risk of gaming in a variety of tools, check if traces of gaming are detectable, and ultimately, formulate advice for journals and other interested parties which (types of) tools and checks are “safe” to include in automated peer review.

Thanks very much NWO and the peer reviewers for this vote of confidence, and also a wholehearted congratulations to my TSB colleagues who also received this grant!

For more information about the awarded grants, see https://www.nwo.nl/en/researchprogrammes/open-competition-ssh/granted-projects.

Full abstract:

Fixing the Flag but Not the Problem: Can Authors “Game” Screening Tools in Automated Peer Review?

Peer review is under strain: too few reviewers assess too many papers, and methodological problems are routinely missed. Automated Screening Tools (ASTs) – software that checks manuscripts for quality problems – may offer a solution, but a risk is strategic gaming: authors making minimal changes to pass automated checks without fixing the underlying problem. This study investigates how authors could game ASTs, whether gaming leaves detectable traces, and whether risk differs across tool types. An expert workshop, pilot study, and pre-registered experiment with ~240 researchers produces the first empirical evidence on AST gaming, with actionable guidance for journals.

New Preprint: Using Statcheck in Peer Review May Reduce Errors

Posted on by

We investigated whether statistical reporting inconsistencies could be avoided if journals implement the tool statcheck in the peer review process.

In a preregistered study covering over 7000 articles, we compared the inconsistency rates between two journals that implemented statcheck in their peer review process (Psychological Science and Journal of Experimental and Social Psychology) with two matched control journals (Journal of Experimental Psychology: General and Journal of Personality and Social Psychology, respectively), before and after statcheck was implemented.

Preregistered multilevel logistic regression analyses showed that the decrease in both inconsistencies and decision inconsistencies around p = .05 is considerably steeper in statcheck journals than in control journals, offering support for the notion that statcheck can be a useful tool for journals to avoid statistical reporting inconsistencies in published articles.

You can find the full preprint here: https://psyarxiv.com/bxau9

Young eScientist Award

Posted on by

In December 2020, Willem Sleegers and I were awarded the Young eScientist Award from the Netherlands eScience Center for our proposal to improve statcheck’s searching algorithm. Today marks the start of our collaboration with the eScience Center and we are very excited to get started!

In this project, we plan to extend statcheck’s search algorithm with natural language processing algorithms, in order to recognize more statistics than just the ones reported perfectly in APA style (a current restriction). We hope that this extension will expand statcheck’s functionality beyond psychology, so that statistical errors in, e.g., biomedical and economics papers can also be detected and corrected.

More information about the award can be found here.

Image result for escience center netherlands

METAxDATA Meeting at QUEST, Berlin

Posted on by

Last month, the QUEST center in Berlin organized the first METAxDATA meeting on building automated screening tools for data-driven meta-research. On the first night of the meeting, 13 researchers gave lightning talks about their tools. The clip below features my <2 minute lightning talk about statcheck.

All lightning talks were recorded and can be found here.

New Preprint: statcheck’s Validity is High

Posted on by

In our new preprint we investigated the validity of statcheck. Our main conclusions were:

  • statcheck’s sensitivity, specificity, and overall accuracy are very high. The specific numbers depended on several choices & assumptions, but ranged from:
    • sensitivity: 85.3% – 100%
    • specificity: 96.0% – 100%
    • accuracy: 96.2% – 99.9%
  • The prevalence of statistical corrections (e.g., Bonferroni, or Greenhouse-Geisser) seems to be higher than we initially estimated
  • But: the presence of these corrections doesn’t explain the high prevalence of reporting inconsistencies in psychology

We conclude that statcheck’s validity is high enough to recommend it as a tool in peer review, self-checks, or meta-research.

statcheck-01