Do you have 50.000 annotators?

*Written by eScience Center Fellow, *Daniela Gawehns

Photo by Shutterstock

During ICT.open 2023, I organized a workshop for 40 computer scientists on reproducibility in computer science. This is a short summary of the main results and how I experienced the workshop as a moderator.

The workshop was part of my fellowship project at the eScience Center to find out how people in the computer sciences understand reproducibility. The idea behind the project is that communities of researchers differ in how they understand the term reproducibility. Before providing tools, guidance or incentives on how to make research outputs more reproducible, we need to figure out what people understand as reproducible research.

What happened?**

First, participants were invited to think in small groups about four ways to make their research less (sic!) reproducible (while still making a career in science). Afterward, the groups moved to another flip chart to expand on their predecessor’s ideas. Again, the prompt was to make their work as least reproducible as possible.

Participants were then invited to walk through the room, look at the posters, and pick a few ideas that they found most impactful in their own subfield of Computer Science. They labeled those ideas with post-it notes. During the last exercise, participants discussed in larger groups four different aspects of non-reproducible science: “Use overcomplicated theory”, “Use proprietary data/software”, “Require expensive tech” and “Do not share code/data”.

Thinking about the opposite of what you set out to achieve was counterintuitive for some participants, while others said “This is easy, I just describe all of my frustrations. Research that I encounter in my daily work is in large parts non-reproducible”. I didn’t give a definition of what Reproducibility means or how the term is defined for the purpose of the workshop at the beginning. This was intentional, as I was hoping to gather as many ideas as possible, without influencing people’s understanding of reproducible research. As a researcher with a background in the social sciences who has discussed reproducibility of research mainly with open science enthusiasts from the social sciences, I found it important to keep as much of my previous understanding of reproducibility outside the workshop room.

This workshop provided a glimpse into what computer scientists feel is important in achieving reproducibility and how this might differ from other areas of research. The use of expensive computing power or annotations is not often mentioned when I talk to psychologists, for example. I was surprised how often participants mentioned overly complicated theory as a barrier to reproducibility. Using version control to keep track of changes in code was only mentioned in passing. It didn’t seem to be on participants’ minds, while this is something I keep on hearing about in my own Open Science bubble.

What did I learn?

• Giving participants a lot of freedom by keeping definitions to a minimum allows participants to share their understanding. It can also lead to frustrations from people who like to set a framework for themselves before starting creative work. Inviting everyone to participate and share their ideas is a key ingredient to running a workshop where all participants are happy with the outcomes.
• Asking people to describe their field of research leads to answers that are difficult to use and interpret. Answers like “Human Centered Interaction/ Data and Visual Analytics“ make it hard to categorize responses into just one subcategory (in this case Human Computer Interaction or Data Science). Another question is how to split Data Science, AI and ML. Especially if people describe their work as “AI/ML” or other combinations.
• There was a lot more input we didn’t have time to explore. For the second part of the workshop, we picked four topics to further focus on. Other barriers to reproducibility that were mentioned by participants, but couldn’t be discussed further include: “Poorly conceptualized Desk Research”, “System Design without clear Problem description”, “Don’t explain the Evaluation, Say “well-established procedures” “, “50.000 Annotators Needed”

What’s next?

This workshop, as well as a few interviews with Computer Scientists at Leiden University, informed a survey that will be sent out in September 2023 to computer scientists working in the Netherlands. Recruitment for that survey will happen via the research schools SIKS and ASCI, social media and word of mouth. Please let me know if you know of a good way to reach a diverse group of survey respondents from the computer sciences.

The proceedings of the ACM REP conference are a good starting point for anyone interested in the topic of reproducible computer science. You can find a collection of materials in several community calls on the topic of reproducibility in Computer Science here.

A transcript of the flip charts created during the workshop can be found here.