How Silent Disco Is Going to Save Science 

& other insights I brought home from the Open Science Retreat

“I wanted to create an event that I actually wanted to go to.” Those were the opening words of Heidi Seibold at the start of the Open Science Retreat.

It was Sunday afternoon, and around 50 people from a wide variety of backgrounds had just arrived in the dunes of Schoorl, The Netherlands. For five days, we would stay on the grounds of the Mennonite Church, surrounded by woods and dunes, to talk about Open Science. 

But not just to talk, Heidi continued to explain. In her dream event, there would be a balance of three parts: conversations, actually getting work done, but also recharging. The goal was to leave with more energy than you came with.

I was very intrigued by this proposal, because most conferences I’ve gone to have been mainly about networking and promoting your own work. A handful of experiences were also about actually getting work done, but I can’t say I ever went home from a conference (even the fun ones!) recharged – more like the opposite.

Not your standard conference

It became clear pretty much immediately that this would not be your standard conference. Within an hour of arriving, we were already split up in teams for “Old Dutch Games” (including pin the tail on the donkey, an obstacle course, and tug-of-war). We were also notified that if you were assigned to Sleeping House 1 (me), your bathroom was in another building. Outside. Oh, and also, we would be taking turns in doing the dishes and sweeping the floors.

Interestingly, Mel Imming, the main organizer of this edition of the event, later confided to me that they could’ve easily afforded the package deal where the location would take care of the dishes. She deliberately went for the do-it-yourself option. And it was the smartest choice ever. 

Imagine the standard interaction at your run-of-the-mill conference. You aimlessly wander through a hallway, end up at a random standing table with people you’ve never seen before. Every conversation starts with: “So… what is your research about?” After 20 identical conversations, nothing sticks anymore.

Now imagine you’re with a handful of people, figuring out how the industrial dishwasher works and drying forks together, while finding out how they ended up at this event and what their particular ideas are about Open Science (or what type of side jobs they’ve done as a teenager).

I learned that making connections the second way is much easier, more memorable, and more fun. And also that drying forks takes a very long time.

The three magic ingredients for high productivity

If you think that the whole event was kumbaya and nothing got done – think again. By the end of the three mornings scheduled for group work, six main ideas were transformed into viable and concrete sets of materials, guidelines, reports, and even a short film.

I think this high productivity at the Open Science Retreat was due to three main things that don’t often happen elsewhere: you were 1) locked1 in a room with 2) a small but highly diverse team, and 3) a very tight deadline until the plenary show-and-tell. 

As a researcher, this is usually not what my work looks like. Collaboration typically takes the form of having periodic 1-hour meetings and sending a lot of commented drafts back and forth. Multiple such collaborations usually co-exist, which means that the action points in each meeting or draft end up at the bottom of an ever-growing and unfocused to-do list. In contrast, being in the same room with all relevant people and having a dedicated timeslot of three hours to work (rather than discuss-and-put-on-list) gave an incredible productivity and motivation boost.

The second magic ingredient for this high productivity was the diversity of the teams. Participants were diverse in age, career stage, field, job type (we had scientists, funders, open science coordinators, people who left academia to found a startup, information specialists, data stewards, and more), which in turn meant a wide diversity in perspectives and skills. 

To give you an idea of what this diversity meant in practice: in my group, we thought about ways to hold journals accountable when Data Availability Statements (DAS) don’t match actual data availability (DAS gap2). Even though I proposed this topic, my ideas on how to achieve this were quite limited (see Fig. 1). However, a group of people with expertise in data storage and citation best practices, knowledge of recent funder reports on the topic, statistical know-how, programming skills, a network in the editing world, and on-point project management skills, however, managed to crunch the numbers on the DAS gap per journal, write an email template to contact and confront journals, compile recommendations for avoiding DAS gaps, make a guide for people wanting to flag a DAS gap, and a full report describing our activities.3 

I can guarantee you that even if I had an army of clones, we’d still be staring angrily at a DAS gap with nothing to show for it.

Figure 1. My initial ideas for our group work on holding journals accountable for void data sharing statements. Credits to our group’s meme-master — you know who you are.

Finally, having just nine hours to start, execute, and wrap-up our project helped tremendously with focus. It forced us to build in repeated moments of reflection where we collectively reconsidered whether the road we were on was still viable within the timeframe or whether we needed to change direction. In my own work, the lack of any real deadline except my retirement4 easily distracts me into pursuing all kinds of irrelevant side quests and losing sight of the main goals. Apparently, all I need is nine hours and no escape route.

The North Sea, a Disney duet, and the future of Open Science

When we arrived, large printed schedules were waiting for us next to the registration desk. And on them was… nothing. It was up to us to fill in the time slots with activities we deemed useful, interesting, fun, or all of the above. Mornings were for group work, but you could also propose a new project, have a coffee with someone, take that work call you couldn’t get out of, read a book, go for a run, or – if you really couldn’t resist – present something.

This flexibility in the schedule meant that you could match your energy to the type of activity. Were you in the mood to brainstorm about a complete overhaul of the peer review system (always), or for a mentoring conversation with someone who chose a different career path? Did you want to take a break alone, play the meme-your-open-science card game, or curate your own playlist, strap on a unicorn headband, and lose yourself at the silent disco?5

Beside the fact that I had SO much fun, there is also a more serious point to all these activities. It’s easier to connect and work together if you’ve already seen another side of a person. It’s easier to reach out afterwards if you’ve had an open-hearted conversation in the dunes about what it meant for work-life balance to become a parent, if you’ve collectively dipped in a VERY cold North Sea, or sung a heartfelt rendition of a Disney duet during karaoke. 

I strongly believe that this type of community building is key in inspiring any type of change in a rigid system. Most people invested in Open Science are in it for the greater good. They work hard – frequently in their own time – to create a system that benefits all, but ironically not always the individual. To prevent such people from burning out, becoming disillusioned, or being weeded out by a system that favors p-hacked Nature publications, we need to create a community where these people feel they belong, where they can work together, and where they can recharge. 

I’m sure I don’t only speak for myself when I say that I went home inspired, recharged, and a little bit sad that it was over. 

Photo credit: Jazelle Maira Carillo
  1. Ok, not literally. ↩︎
  2. Read this in a German accent. ↩︎
  3. Interested in this project? Stay tuned! The Report and accompanying materials will be published online soon on Research Equals: https://researchequals.com/outputs/69f4fd1f-baee-4c58-8f8f-501035b7fbcd 
    ↩︎
  4. And I’ve heard there is no such thing if you’re a professor… ↩︎
  5. It turned out that I was in the mood to do EVERYTHING, which meant that I went from a three hour jam session, to karaoke, to silent disco and then got up the next morning at 6:45AM to be in time for the daily morning yoga sessions.
    ↩︎

METAxDATA Meeting at QUEST, Berlin

Last month, the QUEST center in Berlin organized the first METAxDATA meeting on building automated screening tools for data-driven meta-research. On the first night of the meeting, 13 researchers gave lightning talks about their tools. The clip below features my <2 minute lightning talk about statcheck.

All lightning talks were recorded and can be found here.

Guest at “KennisCafé: De Foute Avond”

Yesterday, I was part of the discussion panel at the “KennisCafé: De Foute Avond”. The KennisCafé is a monthly night at de Balie, Amsterdam, where scientists and other experts discuss a certain topic for a laymen’s audience. This night, the theme was: problems and mistakes in science.

Together with panel members Lotty Hooft (director Cochrane Netherlands), Lex Bouter (professor in scientific integrity and methodology), and Paul Iske (“Chief Failure Officer”), I discussed topics such as statistical mistakes, problems with replication, and possible directions for solutions.

The livestream (in Dutch) can be found here:

The KennisCafé is a production of NEMO Science Museum, KNAW, de Volkskrant, and de Balie. More information can be found here.

Tilburg University Dissertation Prize

Yesterday I was awarded the Tilburg University Dissertation Prize. It is a great honor, but I’m especially grateful because this as a sign that Tilburg University thinks it is good to be critical about the current scientific system, and that open science is an important step forward.

I would like to thank my advisors and collaborators, without whom this dissertation would not exist.

My full dissertation, “Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science”, can be downloaded here.

dissertationprize

Open Software for Open Science

At the Solid Science Workshop in Bordeaux (September 6-7, 2018), I gave a workshop about free software to facilitate solid research practices. During this workshop, we collaboratively worked on a list of resources/software/tools that can be used to improve different stages of the research process.

Check out the list, share it with colleagues, or add your own resources to it here: https://bit.ly/opensciencesoftware.

The slides of the workshop can be found here: https://osf.io/s8wpz/.

empcycle

Dr. Nuijten

Wednesday May 30, 2018, I successfully defended my PhD thesis, which means that I can now finally call myself Dr. Nuijten!

85

I thank my promotors Jelte Wicherts and Marcel van Assen for all their advice over the last 5 years, and my committee – Chris Chambers, Eric-Jan Wagenmakers, Rolf Zwaan, and Marjan Bakker – for their interesting (and fun!) questions.

My full thesis “Research on research: A meta-scientific study of problems and solutions in psychological science” can be found here.

99.jpg

Talking Text & Data Mining at the European Commission

“The right to read is the right to mine”. That was the motto of yesterday’s meeting at the European Commission, where we discussed how new European copyright laws would affect text and data mining (TDM) research.

The new proposal would seriously impede the use of TDM for businesses; effectively they would not have the right to mine content they already have legal access to, which is of course very strange.

The proposal does include an exemption for non-commercial research organizations – which includes universities, and with that, my work – but this is still not sufficient. For one, it would prevent scientists to commercialize any breakthroughs based on TDM research. On top of that, an increasing number of scientists seaks collaboration with businesses (for example, to increase the chances of getting a Horizon 2020 grant).

For updates on this legislation, and more info on the TDM restrictions, see the website, including an open letter, of the European Alliance for Research Excellence (EARE).

Join our seminar “Improving Scientific Practice: Dealing with the Human Factors”, September 11, 2014, Amsterdam.

September 2014

This seminar will take a positive approach and will focus on practical solutions to dealing with human factors in the scientific enterprise. Highly recognized scientists from various research areas who are known for their active involvement in and contributions to the improvement of scientific practice will share their expertise and offer feasible ways to advance the way we do science.

Keynote addresses: John Ioannidis and Melissa Anderson

For more information and registration: www. human-factors-in-science.com

logo_groot

Edit