My dissertation is finished!
The cover: my own desk, feat. SIPS, BITSS, and COS. Cover design by Niels Bongers.
The contents: statcheck, data sharing, meta-analysis, power, bias, and more.
You can find the full thesis here:
Our new meta-meta-analysis on intelligence research is now online as a preprint at https://psyarxiv.com/ytsvw.
We analyzed 131 meta-analyses in intelligence research to investigate effect sizes, power, and patterns of bias. We find a typical effect of r = .26 and a median sample size of 60.
The median power seems low (see figure below), and we find evidence for small study effects, possibly indicating overestimated effects. We don’t find evidence for a US effect, decline or early-extremes effect, or citation bias.
Comments are very welcome and can be posted on the PubPeer page https://pubpeer.com/publications/9F209A983618EFF9EBED07FDC7A7AC.
In our new preprint we investigated the validity of statcheck. Our main conclusions were:
- statcheck’s sensitivity, specificity, and overall accuracy are very high. The specific numbers depended on several choices & assumptions, but ranged from:
- sensitivity: 85.3% – 100%
- specificity: 96.0% – 100%
- accuracy: 96.2% – 99.9%
- The prevalence of statistical corrections (e.g., Bonferroni, or Greenhouse-Geisser) seems to be higher than we initially estimated
- But: the presence of these corrections doesn’t explain the high prevalence of reporting inconsistencies in psychology
We conclude that statcheck’s validity is high enough to recommend it as a tool in peer review, self-checks, or meta-research.
Our paper “Journal data sharing policies and statistical reporting inconsistencies in psychology” has been accepted for publication in the open access journal Collabra: Psychology!
The updated (accepted) pre-print can be found on PsyArXiv: https://psyarxiv.com/sgbta.
We just published the preprint of our new study “Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology” at https://osf.io/preprints/psyarxiv/sgbta.
In this paper, we ran three independent studies to investigate if data sharing is related to fewer statistical inconsistencies in a paper. Overall, we found no relationship between data sharing and reporting inconsistencies. However, we did find that journal policies on data sharing are extremely effective in promoting data sharing (see the Figure below).
We argue that open data is essential in improving the quality of psychological science, and we discuss ways to detect and reduce reporting inconsistencies in the literature.
I’m very excited to announce that the statcheck web app created by Sean Rife and sponsored by Rackspace is online and ready for use! You can find the app at http://statcheck.io.
Below you can find a short interview (Dutch with English subs) in which I explain what the app does.
The new statcheck 1.2.2* is now on CRAN!
- Improved the regular expressions to avoid that statcheck wrongly recognizes weird statistics with subscripts as chi-squares
- You can now choose whether to count “p = .000” as incorrect (this was default in the previous version)
- The statcheck plot function now renders a plot in APA style (thanks to John Sakaluk for writing this code!)
- Give pop-up window to choose a file when there is no file specified in “checkPDF()” or “checkHTML()”
For the full list of adaptations, see the History page on GitHub.
Besides the new updated package, I also created a detailed manual with instructions for installation and use of statcheck, including many examples and explanation of the output. You can find the manual on RPubs here.
* For the people who actually know what this numbering stands for: you may have noticed that the previous version on CRAN was version 1.0.2, so this seems like a weird step. It is. It’s because at first I had no idea what these numbers stood for (MAJOR.MINOR.PATCH), so I was just adding numbers at random. Actually the previous version should have been 1.1.x, which means that I’m now at 1.2.x. The last two PATCHES were because I messed up the R CMD check and had to fix some last minute things 🙂
There are too many statistical inconsistencies in published papers, and unfortunately they show a systematic bias towards reporting statistical significance.
Statistical reporting errors are not the only problem we are currently facing in science but at least it seems like one that is relatively easy to solve. I believe journal editors can play an important role in achieving change in the system, in order to slowly but steadily decrease statistical errors and improve scientific practice.
Nuijten, M.B. (2016). Preventing statistical errors in scientific journals. European Science Editing, 42, 1, 8-10.
You can find the post-print here.
In this paper we use the automated procedure “statcheck” to extract over 250.000 p-values from 30.000 psychology articles and check whether they are consistent.
We find that half of the articles contain at least one inconsistency, and 1 in 8 articles contains a gross inconsistency that affects the statistical conclusion. The prevalence of inconsistencies seems to be stable over time.
The article is Open Access and available here: http://link.springer.com/article/10.3758/s13428-015-0664-2