● Recognising the importance of software in research: Research Software Engineers (RSEs), a UK example — Study is a case study that discusses the current challenges faced by RSEs and policy conclusions to further help support RSEs and to contribute to the progress of open science in Europe.
● UK’s research and innovation infrastructure: opportunities to grow our capability highlights the importance of software by recognising software and skills as one of the six computational and e-infrastructure themes.
Community **considers work on the importance of research software communities in ensuring best practice in software development.
● Computational Research Software: Challenges and Community Organizations Working for Culture Change identifies the importance of sustained community in the development of high-quality software, and introduces efforts by grassroots organisations and projects to improve software quality, productivity, and sustainability. These endeavors ensure the integrity of research results and enable more effective collaboration.
● Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained provides an overview of the grass-roots organisations and projects that have evolved to address growing technical and social challenges in research software productivity, quality, reproducibility, and sustainability. This article then discusses opportunities to leverage their synergistic activities while nurturing work toward emerging software ecosystems.
**Education and training **identifies work that considers issues round skills, training, career paths and reward structures.
● How do scientists develop scientific software? An external replication considers how scientists acquire software engineering knowledge to suggest improvements in this process.
● Software Use in Astronomy: an Informal Survey finds that all participants use software in their research, and identifies the ten most popular tools. 90% of participants write at least some of their own software. while only 8% of them report that they have received substantial training in software development.
● Unmet needs for analyzing biological big data: A survey of 704 NSF principal investigators highlights the importance of skills in software development for data analysis, with 90% of respondents indicating they are currently or will soon be analyzing large data sets.
**Research breakthroughs **include papers on significant research accomplishments that acknowledge reliance on research software tools:
● Case Study: First Photograph of a Black Hole, Enabled by NumFOCUS Tools.
● Pegasus powers LIGO gravitational wave detection analysis.
**Software **has been applied as a tag to resources that mention particular pieces of software, including when this is part of a broader focus in the resource. This category includes a sample of the tens of thousands of papers that rely on research software and that collectively build knowledge in a field, for example:
● Analysis of Human Sequence Data Reveals Two Pulses of Archaic Denisovan Admixture.
● Challenges in funding and developing genomic software: roots and remedies.
● Researchers find bug in Python script may have affected hundreds of studies.
● Securing the future of research computing in the biosciences.
● These 7 CERN Spinoffs Show The Project Isn’t Just Theoretical.
Other useful approaches
This analysis has been useful in elucidating some of the ways in which the value of software can be demonstrated. However, there are also other approaches that could be useful. For example, methods to evaluate economic value are providing valuable statistics for research data, but comparable examples for research software are rare. The recent European Union publication, Cost-benefit analysis for FAIR research data, finds that the overall cost to the European economy of not having Findable, Accessible, Interoperable and Reusable (FAIR) research data is €10.2bn per year in Europe. A 2014 Australian study by Houghton and Gruen similarly demonstrates the economic value of data, estimating the value of Australian public research data at over $1.9 billion a year. One of the few economic valuations of research software is a 2017 analysis by Sweeny et al. of the return on investment generated by three Australian virtual laboratories, which provide access to research software and data for researchers, was at least double the investment for every measure. This indicated that the services had a significant economic and user impact — by one measure the value of one virtual laboratory was over 100 times the cost of investment. It would be useful if more studies were undertaken to demonstrate the economic benefits of research software using different methodologies.
Conclusion
This summary of evidence of the importance of research software to research outcomes illustrates increasing recognition of this fact. This summary could also be useful to encourage the community to consider where additional work could be useful (such as expanding existing surveys in specific countries and disciplines to get a more global scope), and to inspire the recording of more of this information (which could include striking examples that convey the impact of failing to understanding the costs and responsibilities of thoughtful software management).
We encourage readers to submit additional resources to the ReSA resources list, which is publicly available:
● Add it directly to the ReSA Zotero group library (requires Zotero account).
● Submit an issue in GitHub (requires GitHub account).
● Email it directly to ReSA.
Authors: Michelle Barker,* Daniel S. Katz*,** Alejandra Gonzalez-Beltran*the
