eScience Center Blog

Making 3D Real: How 3DLab Is Bringing Scientific Data to Life

Thijs van Lankveld — Thu, 25 Jun 2026 00:00:00 GMT

Imagine trying to understand the structural integrity of a Roman ruin, or the complex vascular network of a biological organ, by looking at a flat map or static 2D photographs. You might get the general idea, but the depth, the context, and the reality are lost in translation.

We live in a three-dimensional world, yet so much of our scientific data remains trapped on two-dimensional screens. For researchers in fields ranging from digital humanities to environmental science, this is not just an aesthetic issue. It’s a barrier to discovery.

Luckily, there is research software to overcome this barrier and get more insight by intuitive and photo-realistic 3D visualization. However, this software comes with another barrier: ease of use. This is where 3DLab can help you: to easily access this powerful technology.

The Challenge: Capturing the World Around You

There is an active field of computer vision research trying to tackle how we explore new places. In simultaneous localization and mapping (SLAM), a camera with location sensors (e.g. GPS and IMU) has to both accurately track its location while simultaneously creating a digital representation of the surroundings.

In recent years, SLAM has seen a huge boom in interest with the application of novel machine learning techniques. Novel scene representations like neural radiance fields have made the resulting digital models photo-realistic. In turn, 3D Gaussian Splatting has made it possible to create such photo-realistic models in under an hour and render them at high frame-rates.

The Approach: Gaussian Splatting

Your browser does not support the video tag.

Source: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

If traditional 3D models (meshes) are like origami sculptures, Gaussian Splatting is like 3D impressionist painting. It represents a scene using millions of fuzzy 3D ellipses (“splats”). These splats can overlap and blend, allowing for incredibly realistic rendering of complex scenes, like the fuzzy texture of moss on a stone or the reflective sheen of polished marble, that traditional methods struggle to capture.

However, Gaussian Splatting is complex to set up and use. It requires very specific software dependencies and careful manual workflows. 3DLab focuses on removing these hurdles, allowing researchers to input raw data and receive a high-fidelity 3D visualization without needing a second PhD in computer science.

Key features of the 3DLab approach include:

Accessibility: ease of installation through detailed step-by-step installation instructions.
Automation: scripts that handle the heavy lifting of pre-processing your data and training the 3D models.
Organization: guided by a single configuration file that organizes your data and parameter settings.

Impact: From Artifacts to Algorithms

Your browser does not support the video tag.

Source: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

While the project has applications in medicine and engineering, its potential in the Social Sciences and Humanities (SH) is particularly exciting.

Consider the field of art history. Capturing and preserving historical sites is a meticulous process. By using the tools developed in 3DLab, researchers can create “digital twins” of heritage sites with unprecedented photo-realism. This preserves the site digitally for remote or future study. It also allows an immersive way to present this history to the public.

The impact of 3DLab goes beyond just pretty pictures. It aligns with the FAIR principles (Findable, Accessible, Interoperable, Reusable). By providing open-source automation tools, the project ensures that 3D visualizations are not just one-off artistic projects, but reproducible scientific outputs.

Building a Community of Visualizers

3DLab is not developed in a vacuum. It represents a strategic effort to embed 3D expertise into the Dutch research ecosystem. The project aims to leverage the vibrant NL-RSE (Research Software Engineers in the Netherlands) community, sharing knowledge and code to ensure these tools survive beyond the initial funding cycle.

The project also highlights the collaborative nature of modern science. It sits at the intersection of computer vision, software engineering, and domain-specific research. By making the code open source on GitHub, the eScience Center invites contributions from developers and researchers worldwide, fostering a culture of shared innovation.

Looking Ahead

The work on 3DLab is just the beginning. As techniques like Gaussian Splatting mature, we gaze at the branching possibilities, like integrating these visualizations into virtual reality (VR) and augmented reality (AR) environments. Imagine a medical student walking inside a simulation of a patient’s heart, or a historian walking through a digital reconstruction of 17th-century Amsterdam.

By lowering the barrier to entry, 3DLab is ensuring that the future of science isn’t just data-driven but it’s immersive, interactive, and accessible to all.

Get Involved

Are you a researcher struggling to visualize high-dimensional data? Or an RSE interested in the bleeding edge of computer graphics?

Explore the Code: Check out the 3D Gaussian Splatting automation repository on GitHub.
Read the Report: Dive deeper into the technical details in the 3D Lab project report.
Connect: Reach out to Thijs van Lankveld or the eScience Center team to discuss how 3D visualization can transform your research.

Research Software Engineers in the Age of GenAI: Same Value, Changing Practice

eScience Editorial Team — Thu, 28 May 2026 00:00:00 GMT

Stephan Druskat, Michelle Barker, Ian Cosden, Cunliang Geng, Robert Haines, Daniel S. Katz, Joseph Shingleton, Ben van Werkhoven

This blog post is being cross-posted by the Research Software Alliance (ReSA), Better Scientific Software (BSSw), the European Virtual Institute for Research Software Excellence(EVERSE) and the Software Sustainability Institute (SSI).

DOI: 10.5281/zenodo.20320179

Research software and its creators have long played a critical role in the advancement of research worldwide. This role is changing in the age of “generative AI” (GenAI), but both the software and the people remain of key importance. Understanding these changes is essential in enabling Research Software Engineers (RSEs) to continue contributing the same high value to the research process and its outputs.

Before GenAI, the RSE movement had learned to clearly articulate the value proposition of embedding expert software engineering in research to its stakeholders. This blog post highlights how RSEs use GenAI to increase their capacity in both software engineering and research, and visualize this evolution. While GenAI is changing — perhaps considerably — how RSEs work in practice, their value and the value of their work for research remains steady and likely to increase.

Generative AI, specifically Large Language Models (LLMs) and coding agents, is rapidly changing software engineering. The practice of software engineering is shifting away from writing lines of code to specifying requirements, implementations, designs and verification strategies in higher-level abstractions, including natural language specifications, that are then enacted by an AI coding agent. This lowers the barrier to entry for producing code and reduces the development time for new solutions.

Generative AI is not only impacting software development in industry, but is understandably also seeing rapid adoption in academia to accelerate software development processes. Its application enables researchers to produce their own research software, including those who would not have done so previously. Software solutions, once beyond the reach of researchers lacking programming and software engineering knowledge, are now accessible with these new GenAI tools. If researchers can write more — and better — code, themselves, it begs the question: where do RSEs fit in this new paradigm?

In March 2026, 36 members of the research software community came together to discuss this at a workshop entitled “Research Software Engineering in the Age of Generative AI: Building a Community Vision”, held in Edinburgh, UK. The Research Software Alliance co-led this event and was supported to undertake this work as part of a Schmidt Sciences grant. Whilst the impact of GenAI on research is being discussed in many forums and publications, this workshop provided an opportunity for a range of early adopters, innovators, and RSE leads to come together to discuss how the ecosystem was changing, and to chart constructive ways to shape and support this change.

The workshop included multiple rounds of focused working groups, one of which discussed the value of research software engineering in the age of GenAI. This blog post is an outcome of that working group. Its authors brought together their experience as leaders of different types of RSE initiatives across the ecosystem to provide an accessible way to show both how the roles of research software personnel are evolving in line with GenAI, and how these roles remain key to enabling research impact.

There are costs, concerns and risks associated with the creation and use of AI models generally [1], [2], and in a multitude of more focused areas, including skills [3], [4], [5], [6], community culture and motivation [7], [8], as well as equity and ethics [9], [10], [11]. However, while many of these risks also apply to the use of GenAI in coding tasks, there are also many immediate benefits for coding [12], [13], [14], [15]. AI tools are accelerating routine coding tasks, such as creating working scripts for data analysis, generating boilerplate code, tests and documentation, all from high-level natural language prompts. Coding agents represent a more autonomous evolution of prompt-based coding, enabling systems built around LLMs to access and operate on existing codebases to handle more complex tasks, such as refactoring or porting. They can even be used to build an entire codebase from scratch, given sufficient information such as specifications or implementation plans.

As software has become one of the primary research tools across all disciplines, a significant majority of scientific results rely on the quality of that software. RSEs emerged as a result: by combining professional software engineering expertise with an intimate understanding of the research discipline, partnerships between RSEs and researchers allowed software to evolve along with research, with RSEs guiding the productive application of methods and technologies and ensuring that software meets the reliability and reproducibility standards that are expected from scientific instruments. This unique position of RSEs bridging software engineering and research was often depicted in diagrams similar to Figure 1, which shows that RSEs vary in their skills and knowledge of both software engineering and research activities. It also shows RSE-adjacent roles (such as software engineer, researcher, and researcher who codes) to provide context, illustrating that RSEs have more software engineering experience than researchers, while also possessing a deeper understanding of research practices than software engineers. Diagrams like these have been very important tools in achieving (pre-GenAI) understanding of the value provided by RSEs, to the extent that RSEs are supported by national initiatives such as those in the UK, the Netherlands and Germany [16], [17], [18].

Figure 1: RSEs bring both software engineering and research expertise.With the advent of GenAI, two questions arise: Is this partnership between RSEs and researchers still needed and still valuable? And, how do these roles change? In the age of GenAI, both researchers and RSEs have access to LLMs and coding agents to accelerate their work. Researchers are able to prototype and explore ideas faster and more freely than before. Meanwhile, RSEs are able to use these GenAI tools to greater effect and impact, due to their expertise in software engineering, particularly in requirements engineering and specification, software architecture, testing and verification, validation, performance, and long-term sustainability. This enables them to frame problems more precisely, to evaluate tradeoffs and verify internal correctness and consistency more effectively, and to better address longstanding challenges in research software. Moreover, their continued and frequent practice with these tools compounds their advantages, allowing them to better anticipate failure modes, enforce quality standards, and integrate tooling into complex, evolving research environments.

RSEs who maintain GenAI literacy and the ability to critically evaluate complex software stacks are especially indispensable in view of the risks associated with the increasing adoption of GenAI in research software development. In particular, when researchers use agentic coding tools without sufficient expertise and experience to write increasingly complex software, over-reliance on these tools risks generating a significant disconnect between the coder and the code. This disconnect can lead to the introduction of unintended errors and inaccuracies, and could pose serious risk to the robustness of related scientific outputs and research results. In this situation, RSEs are well positioned to provide the guidance and oversight that is necessary to ensure the safe adoption of AI in research software engineering.

Taken together, the value of an RSE remains significant, as the RSE role evolves to integrate expert knowledge in the application of GenAI for software engineering in research contexts. This viewpoint is similarly articulated in the Position Statement on Generative AI in the RSE Workplace[19], which is led by US-RSE and the Alliance for Data Science and AI. The Edinburgh workshop built upon this position statement, and it is also being shared in other conversations[20], [21]. As such, the value of RSEs in relation to researchers and other stakeholders will remain unchanged as their role as a research software professional evolves to also integrate expert knowledge in the application of GenAI in research contexts. This blog post seeks to move one step further in clearly articulating that value in a way that is easy to understand.

Overall, the value of the RSE role shifts upward as AI enhances it and amplifies its impact. While the practice may change and shift focus to higher levels of abstraction, including specification and verification, the value proposition remains the same**. This is depicted in Figure 2, which shows how the use of GenAI enables the value of RSEs to increase along both the software engineering and research activity axes. It is critical for the research community to understand and acknowledge this increased potential for research impact, and to ensure that going forward, RSE expertise is appropriately supported, funded and integrated into research ecosystems.

Figure 2: AI can enable RSEs to increase their capacity for both research and software engineering, irrespective of specializations.Figure 3 highlights the unique position of RSEs in the research ecosystem. RSEs and other roles in the research sector (software engineers, researchers who code, and researchers) all evolve in response to leveraging GenAI tools, which affects all groups along the software engineering capacity axis to varying degrees. Researchers who have not previously done any programming can now generate their own code, but for all non-RSE roles these shifts largely reinforce strengths within their established areas of expertise. For RSEs, however, this evolution plays to their dual focus on both software engineering and research, so that the effective value of RSEs and their work is amplified, as GenAI enables them to provide (even) more and better software to support (even) more and better research.

Figure 3: How various roles are evolving in response to GenAI.Acknowledgements

MB and DSK were supported to undertake this work as part of Schmidt Sciences grant G-25–69965.

Author biographies

Michelle Barker is Director of the Research Software Alliance and has extensive expertise in open science, the research software community, digital skills, and digital research infrastructure. As a sociologist, Michelle is passionate about building collaborative partnerships to achieve system change. She is a former Director of the Australian Research Data Commons, where she led the national research software infrastructure investment program.

Ian Cosden is the Senior Director of Research Software Engineering at Princeton University. He leads a team of research software engineers who complement multiple traditional academic research groups by offering embedded, long-term software development expertise. He spearheaded the formation of the RSE group in 2016 and has grown the group from an initial size of two FTEs, to a total of 37 FTEs as of May 2026. Ian is the current and founding chair of the steering committee for the United States Research Software Engineer (US-RSE) Association. Additionally, he is the principal investigator for Innovative Training Enabled by a Research Software Engineering Community of Trainers (INTERSECT), an NSF-funded project to develop and deliver RSE-led training events for current researchers interested in careers in research software engineering.

Stephan Druskat is a (Research) Software Engineering Researcher based in Berlin. His research focuses on requirements, constraints, policies and practices of software engineering in academic research contexts, such as software publication, citation and sustainability, as well as software metadata and software supply chains. He is a Fellow of the Software Sustainability Institute, co-founder of the Society for Research Software in Germany (de-RSE), and lead of the Citation File Format project.

Robert Haines is Director of Research IT and an Honorary Lecturer at the University of Manchester, and a Fellow of the Software Sustainability Institute. He is one of the originators of the term “Research Software Engineer”, served for six years as an elected representative of the UK RSE Association, chaired the First Conference of Research Software Engineers in 2016, and was a founding trustee of the Society of Research Software Engineering. Robert’s research interests include software engineering, software sustainability, software use in open and reproducible research, software citation and credit, and career paths for software engineers and data scientists.

Daniel S. Katz is Chief Scientist at the National Center for Supercomputing Applications, Research Professor in the Siebel School of Computing and Data Science, and Research Professor in the School of Information Sciences at the University of Illinois Urbana-Champaign. He works at the triple point of research software, people, and policy. He is a co-founder and current Associate Editor-in-Chief of the Journal of Open Source Software, co-founder of the US Research Software Engineer Association (US-RSE), and co-founder and steering committee chair of the Research Software Alliance (ReSA).

Joseph Shingleton is a Research Software Engineer at the University of Glasgow. His research explores how artificial intelligence can be safely and effectively applied to tasks in geospatial data science and research software engineering. He is the recipient of a UKRI Metascience AI Fellowship, focusing on how generative AI is shaping RSE practice and its consequences for the scientific process. He is also a Software Sustainability Institute Fellow and leads the SSI study group on responsible AI in RSE.

Ben van Werkhoven is assistant professor at Leiden University, where he heads the Accelerated Computing research group, focusing on making high-performance computing more energy-efficient and sustainable. He holds a PhD from VU Amsterdam on GPU-accelerated scientific computing (2014) and spent nearly a decade at the Netherlands eScience Center before joining Leiden in 2023. With over 15 years of GPU programming and optimization experience, Ben has worked on scientific applications ranging from microscopy and climate modeling to radio astronomy. He is also a co-founder of the Netherlands Research Software Engineers community (NL-RSE), advocating for the recognition of research software engineering as a professional discipline. At Leiden, he teaches High Performance Computing and Multiprocessor Programming, and serves as principal investigator on several research projects. He participates in major European and national consortia, including CORTEX and ESiWACE, advancing large-scale computing infrastructure for science.

References**

[1] M. Hosseini, S. P. J. M. Horbach, K. L. Holmes, and T. Ross-Hellauer, ‘Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities’, May 24, 2024, SocArXiv*. doi: 10.31235/osf.io/zns7g.

[2] G. O’Brien, ‘Threats to scientific software from over-reliance on AI code assistants’, Nat Comput Sci, vol. 5, no. 9, pp. 701–703, Jul. 2025, doi: 10.1038/s43588–025–00845–2.

[3] S. Andleeb, B. Kantorski, and J. C. Carver, ‘ChatGPT in Introductory Programming: Counterbalanced Evaluation of Code Quality, Conceptual Learning, and Student Perceptions’, 2025, arXiv. doi: 10.48550/ARXIV.2510.00946.

[4] G. O’Brien, A. Parker, N. Eisty, and J. Carver, ‘A survey of generative AI adoption and perceived productivity among scientists who program’, Apr. 09, 2026, arXiv: arXiv:2512.19644. doi: 10.48550/arXiv.2512.19644.

[5] SAPEA, ‘Successful and timely uptake of artificial intelligence in science in the EU: evidence review report’, SAPEA, Apr. 2024. doi: 10.5281/ZENODO.10977676.

[6] M.-A. Storey, ‘From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI’, Apr. 06, 2026, arXiv: arXiv:2603.22106. doi: 10.48550/arXiv.2603.22106.

[7] R. Choudhuri, D. Liu, I. Steinmacher, M. Gerosa, and A. Sarma, ‘How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering’, in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, Lisbon Portugal: ACM, Apr. 2024, pp. 1–13. doi: 10.1145/3597503.3639201.

[8] M. Koren, G. Békés, J. Hinz, and A. Lohmann, ‘Vibe Coding Kills Open Source’, Jan. 21, 2026, arXiv: arXiv:2601.15494. doi: 10.48550/arXiv.2601.15494.

[9] S. A. Besser, E. A. Jensen, and D. S. Katz, ‘How generative AI is shaping research software development and maintenance at a research-intensive university’, Open Res Europe, vol. 6, p. 56, Feb. 2026, doi: 10.12688/openreseurope.22009.1.

[10] D. R. Newman-Griffis, ‘AI thinking and the enterprise of science’, Jun. 25, 2023, Open Science Framework. doi: 10.31219/osf.io/wb83e.

[11] D. Salah, ‘Invisible Risks, Visible Code: A Vision for Understanding Ethical Debt in AI-Based Coding’, in 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Honolulu, HI, USA: IEEE, Oct. 2025, pp. 442–446. doi: 10.1109/ESEM64174.2025.00074.

[12] E. W. Bridgeford et al., ‘Ten Simple Rules for AI-Assisted Coding in Science’, Oct. 31, 2025, arXiv: arXiv:2510.22254. doi: 10.48550/arXiv.2510.22254.

[13] L. C. McInnes et al., ‘Report of the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science’, 2025, arXiv. doi: 10.48550/ARXIV.2510.03413.

[14] L. Nordling, ‘How ChatGPT is transforming the postdoc experience’, Nature, vol. 622, no. 7983, pp. 655–657, Oct. 2023, doi: 10.1038/d41586–023–03235–8.

[15] G. O’Brien, ‘How Scientists Use Large Language Models to Program’, in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Yokohama Japan: ACM, Apr. 2025, pp. 1–16. doi: 10.1145/3706598.3713668.

[16] M. Barker et al., ‘Software and skills for research computing in the UK’, Zenodo, Jan. 2024. doi: 10.5281/ZENODO.10473186.

[17] D. Kempf et al., ‘Establishing central Research Software Engineering units in German research institutions’, 2025, [Online]. Available: https://de-rse.org/2023_paper-RSE-groups/paper.pdf

[18] LCRDM, ‘Professionalizing the role of Research Software Engineers in the Netherlands’, Zenodo, Mar. 2025. doi: 10.5281/ZENODO.15019998.

[19] ADSA and US-RSE, ‘Position Statement on Generative AI in the RSE Workplace’. Accessed: Apr. 28, 2026. [Online]. Available: https://github.com/Academic-Data-Science-Alliance/rse-ai-position-statement/blob/main/RSE-AI-Final_Statement.md

[20] S. Gesing, ‘RSEs 2035: Surviving or Thriving in the Age of AI’, in 2025 IEEE International Conference on eScience (eScience), Chicago, IL, USA: IEEE, Sep. 2025, pp. 381–382. doi: 10.1109/eScience65000.2025.00081.

[21] P. Leather, D. Silver, and S. Frezza, ‘Software engineering role archetypes’, presented at the IEEE SWEBOK Summit 2026, Apr. 2026. [Online]. Available: https://conf.researchr.org/details/icse-2026/swebok-2026/5/Talk-Software-engineering-role-archetypes-mapping-knowledge-skills-and-competenci

Co-Designing a Roadmap for Training on Research Software Templates and Code Auditing

eScience Editorial Team — Fri, 15 May 2026 00:00:00 GMT

Training materials are most effective when they are created hand-in-hand with the communities who will actually use them.

Photo of the eventBy Manuel G. Garcia and Serkan Gigin.

Training materials are most effective when they are created with the communities who will actually use them. Co-design brings together lived experience, local knowledge, and technical expertise, ensuring that learning resources are relevant, accessible, and grounded in real needs. During the Research Software Training Day 2026, we engaged with research software trainers from across the Netherlands to explore how training materials for Meta-Template and Code Auditor should be developed and delivered.

This the story of our experience and the lessons we learned along the way:

Meta-Template** is a tool for aligning software best practices across teams, groups, and organisations through software templates. These templates provide structure and boilerplate code for software projects, offering several key advantages for research software development: they reduce the time and barriers involved in setting up a new project, encourage the adoption of best practices such as the FAIR principles for software, and promote standardisation and consistency across projects within a research team, group, or organisation.

Code Auditor is a conformity checking tool that provides insights into the structure and content of a research software project. When applied to a code base, such as a code repository, Code Auditor can identify missing or conflicting best practices, suggest solutions to the issues it detects, and generate conformity reports in both machine- and human-friendly formats. It can also be used to verify compliance with software management plans and, when integrated with software templating tools, enables automated corrections.

These two tools complement each other. Meta-Template facilitates the adoption of software best practices from the very inception of a research software project, while Code Auditor helps monitor compliance with those practices at key points throughout the software lifecycle.

To promote the adoption of these tools, we need training materials, such as tutorials, that introduce them to potential users, demonstrate their value, and attract collaborators to the open-source projects that maintain them. With this goal in mind, we consulted the Research Software Training community about which directions and approaches to follow.

Fact Sheet** Participants: 16 Experience as trainners: beginners (1 year) to proficient (> 5 years) Location: Utrecht Duration: 1.5 hours

Directions for Training Tutorials

Training tutorials can be developed in several directions. During our engagement with software trainers, we focused on understanding their preferences and opinions regarding two key questions:

How independent should training materials be from other training materials and initiatives?** Independent materials might be easier to develop and maintain, but risk being too niche to reach a broad pool of potential users. On the other hand, integrating materials with existing training programmes could provide good leverage to reach a wider audience, though it requires more coordination and effort.
Should training materials focus on the tools themselves, or on the research software development topics they relate to? Tool-focused materials would have a narrower scope but highlight the capabilities of the tools in detail, while topic-focused materials would cover broader software topics, such as publishing research software, and introduce the tools as mechanisms to ease the workload.

Independence vs Integration

Most trainers agreed that training materials for Meta-Template and Code Auditor should be developed as part of existing training programmes, taking advantage of established initiatives and institutional structures. However, some arguments in favour of independent training highlighted that standalone materials could be shorter and use-case oriented, requiring less time investment from potential users. A few others acknowledged these arguments but felt that choosing a single direction might not be necessary, since both approaches offer advantages and drawbacks.

Despite the differences in opinion, the discussion concluded with trainers agreeing that a good starting point would be to explore integrating materials into existing and related training initiatives.

Tool Focus vs Topic Focus

A large majority of trainers agreed that training materials should be topic-focused. The strongest argument in favour was that topic-focused materials would emphasise the value of the tools within a particular use case or research domain, demonstrating how useful they can be in a specific context and therefore serving as a better strategy to attract users. The argument for tool-focused materials, on the other hand, emphasised that such materials could be shorter and concentrate on demonstrating the how-to of each tool. Those who favoured neither direction raised arguments similar to those in the discussion about independence vs integration.

Audiences, Formats, and Collaborations

In the final part of the workshop, trainers proposed ideas on three topics:

Which audiences should the training materials target?
Which formats would be most suitable?
How can we attract collaborators to help develop materials?

The most favoured ideas pointed towards targeting PhD candidates, MSc students, data support staff, and data repository managers, through on-site training sessions such as workshops. However, a wide range of ideas were proposed for each aspect — see Table 1.

Regarding collaboration, the preferred approach was to organise on-site events, such as content-a-thons, where trainers work together on developing materials.

Table 1. Ideas for addressing topics related to audiences, formats and collaborations for developing training materials for Meta-Template and Code Auditor.*

Key Takeaways

The main takeaways from the workshop were as follows:

Work towards integrating training materials for the tools into existing initiatives, such as data and software training programmes, to leverage the opportunities they offer for reaching a wider audience.
Topic-focused materials offer a strategic advantage in demonstrating the value of new tools such as Meta-Template and Code Auditor, which in turn helps attract new users.
Prioritise PhD candidates, MSc students, and data/software support staff as target audiences, delivering training through on-site formats such as hands-on workshops.
Trainers prefer collective efforts during on-site events, such as content-a-thons, for contributing to the development of training materials.

Contact

If you would like to learn more about Meta-Template** and Code Auditor, or explore how to use them in your own training activities, we would love to hear from you. We also warmly invite contributions from the community — whether by improving the existing materials, sharing feedback, or helping co-develop new training resources together.

Dr. Manuel Garcia Alvarez Research Software Engineer TU Delft DCC m.g.garciaalvarez@tudelft.nl

Dr. Serkan Girgin Associate Professor University of Twente, Faculty ITC s.girgin@utwente.nl

The project “Best Practices for Sustainable Software” with file number ICT.TDCC.002.001 of the research programme NWO Implementation Plan for ICT Infrastructure for TDCC Bottleneck Projects is financed by the Dutch Research Council (NWO).

Provenance in the age of AI: Why we need AI Declarations for more transparency

Faruk Diblen — Thu, 23 Apr 2026 00:00:00 GMT

In research, if our methodology is not transparent, our results are not verifiable. Using AI to help generate code or synthesize data is dramatically affecting transparency, sustainability and maintenance. But how can we make the AI involvement more transparent and traceable?

This image was generated using Google Gemini 3.1. See the declaration file at the end of this blog post for more details.Before I started as a Research Software Engineer, I worked in a physical lab where the culture of provenance was absolute. You documented everything: the materials, the hardware, the exact version of the software you used to process the data, and so forth. Without that trail, the results were meaningless. You could not trust them, and you certainly could not reproduce them.

Today, my “lab” is entirely digital, but the need for that same precision remains.

Over the last two years, a new variable has entered our work. We are using Generative AI to suggest code refactors, autonomous agents to synthesize datasets, and generative models to draft technical reports. AI is no longer a separate tool we are experimenting with; it is already becoming a standard part of our daily routine.

Today, when I download a dataset or review a colleague’s pull request, I have no way to know where the human logic ends and the model’s output begins. This ambiguity is a problem for scientific reliability. If we cannot trace the logic of an algorithm or the origin of a dataset, we lose reproducibility. If we cannot identify which parts of a codebase were generated by a model, we are building up a “maintenance debt” that will eventually come due. And if we ignore the compute cost of these models, we are ignoring the sustainability of our field.

We built the AI Declaration Format**** (https://ai-declaration.org)** to address this. Our goal is to make structured documentation of AI usage a standard, routine part of professional work.

The foundation: A machine-readable file format

The AI Declaration Format is the foundation: a machine-readable, general-purpose file format designed to describe AI involvement in any digital artifact. It does not matter if you are publishing software, a research paper, a dataset, or media; you can use the same structured approach to say how AI was involved.

We designed the schema to be modular, so you can focus on the parts that matter for your project:

Maintainability and Security: You can track which modules were AI-assisted and document security reviews. Knowing which blocks of code were generated helps future maintainers understand the intent behind them.
Environmental Impact: You can record compute hours and estimated carbon footprints. This turns “sustainability” into a measurable metric.
Explainability and Governance: For systems that interact with users, you can document technical decision logic or interpretability methods. You can also name responsible officers and link to internal ethics reviews.
Regulations (not just the EU Act): With the EU AI Act, the US Executive Order, and other global frameworks, transparency is becoming a legal requirement. The AI Declaration Format lets you map your project to risk categories and link to the technical documentation these laws demand.

Why another format?

A common question is why we need a new format when standards like SPDX 3.0 (AI Profile) and CycloneDX (ML-BOM) already exist. The answer lies in the scope and the audience:

SPDX and CycloneDX are “Bill of Materials” (SBoM) standards. They are designed for legal teams and security auditors to track every sub-component and vulnerability in a supply chain. They are powerful but often complex for an individual researcher or a small dev team.
CodeMeta and CITATION.cff are useful for general project metadata and academic citations. However, they lack the specialized fields needed for AI transparency — such as risk classification, model versions, and human oversight.
The AI Declaration Format is a declaration format, not a full inventory. It is designed to be human-writable and project-focused. It answers the specific question of “How did the humans on this project use AI?”.

The AI Declaration Format sits alongside these standards: it is the missing piece of the metadata puzzle that captures the usage and methodology of Generative AI.

How do I get started?

A standard is only useful if it is easy to adopt. We have built an ecosystem of tools and automation around the AI Declaration Format to ensure it fits into your existing workflow. We are also working on improving and extending this tooling, including GitHub/GitLab Actions and workflows to automate validation and updating as part of your CI/CD pipelines:

The Web App**: If you want a guided interface, we built a web app specifically to create these declaration files. It runs entirely in your browser, so your project data never hits our servers. You can generate an aidecl.yaml, validate it, or look at examples from other fields.
The CLI: For those of us who live in the terminal, the aidecl tool lets you initialize and validate declarations as part of your git workflow. It runs semantic checks to make sure your claims actually make sense—like flagging if you claim no AI was used but then list three models in your inventory.

Scientific and Technical Standards

In research, we document our hardware and our data sources because that is how we maintain the standard of our work. This is the bedrock of Open Science: if our methodology is not transparent, our results are not verifiable. Using AI to help generate code or synthesize data is simply another part of that methodology. It belongs in the formal record.

Adding an aidecl.yaml to your repository shows a commitment to open science principles. It is a way to make sure that the things we are building today will be MORE understandable, maintainable, and reproducible in future.

Join the effort

We are in the early stages of this, and we need a community to make it a standard.

Adopt: Add an aidecl.yaml to your next project. It takes about five minutes.
Endorse: If your lab or organization values AI transparency, we need early adopters to help build credibility.
Contribute: The schema and tools are open-source. Help us improve the validation rules or the web interface.
Feedback: We need to know how this works (or does not) for your specific field — whether that is genomics, legal tech, or creative media.

We are all figuring this out as we go. As the technology changes and we get more feedback, the file format and the tools will evolve to keep up.

Project page: ai-declaration.org
GitHub organization: github.com/ai-declaration

AI Declaration (aidecl.yaml) for this post

schema_version: “1.0.0” project: name: “Provenance in the age of LLMs: Why we need AI Declarations” version: “1.0.0” content_type: “document” license: “CC BY 4.0” ai_usage: used: true summary: “Structured and refined by Gemini (LLM) based on human-provided themes, constraints, and source documentation.” level: “moderate” activities: - content_drafting - content_editing tools: - name: “Gemini” vendor: “Google” type: “assistant” model: “gemini-3.1” purpose: - content_refining - structural_organization - image_generation ai_proportion: qualitative: mostly_human method: self_reported governance: responsible_officer: “Faruk Diblen” ethics_review_status: “not_applicable” environmental: compute_hours: 0.05 estimation_method: self_estimated security: review_performed: true review_type: - human_in_the_loop declaration: date: “2026-04-07” declared_by: “Faruk Diblen” organization: “Resoft Labs”

Acknowledgements

Thanks to [Flavio Hafner], [Carlos Martinez-Ortiz] and [Elena Ranguelova] for their feedback on the text and [Pablo Rodríguez-Sánchez] and [Alexander Hadjiivanov] for their editorial support.

FOSDEM 2026: Observations and key takeaways

eScience Editorial Team — Thu, 09 Apr 2026 00:00:00 GMT

If you missed FOSDEM 2026 — the largest conference dedicated to open-source software — you can still absorb some of the experience as shared by NLeSC RSEs Flavio Hafner*, Ole Mussmann and Faruk Diblen.*

Flavio Hafner on FOSDEM 2026: Security, LLMs, and Software Performance

This is a version of the original post by Flavio adapted for Medium.

My highlights from this year’s FOSDEM are in the areas of LLM/security/open source, in machine learning/software performance, and in databases/search.

Upcoming features in git

Patrick Steinhardt from GitLab, and git contributor, presented about some upcoming changes, planned for the major 3.0 release towards mid-2026.

Under the hood, git changes the hashing algorithm from SHA-1 to SHA-256

This is because SHA-1 is not secure — a paper from around 2017 showed that it does not create unique hashes.
While git itself does not rely on uniqueness of hashes, the ecosystem implicitly does — for instance, by pinning software dependencies to git hashes.
At the same time, large parts of the ecosystem (GitHub for instance) currently do not support SHA-256 hashes.
By moving the default in git, the contributors want to solve the chicken-and-egg problem of “no-one wanting to use the feature because no-one supports it” and vice versa.

New command: git history for easier rewriting of history

In git, rewriting history through an interactive rebase is cumbersome and takes several steps.
Another limitation is that it leads to orphaned branches because other branches depending on the changed commits are not updated. For instance, this makes is tedious to use workflows with stacked branches.
The new git history command, inspired by other version control systems such as Jujutsu and Mercurial, provides some functionality that makes such workflows easier. For instance, git history reword <commit> allows to amend the commit message of a specific commit; git history split <commit> allows to split a specific commit.
The new commands also rebase other branches that depend on respective commits.

Open Source, LLMs, and security

This was a major topic in this year’s conference and featured in two keynotes: Michael Leenaars (talk) from NLnet, and Daniel Stenberg, founder and lead developer of cURL (talk).

Both speakers highlighted that LLMs can help malicious actors find and exploit vulnerabilities in open-source code, and thus open-source will become more vulnerable to supply-chain attacks.
Stenberg further detailed how LLMs bring out the worst and best at the same time. On one hand, the cURL project is bombarded by AI-generated security reports. This has led cURL to stop their bug bounty program. On the other hand, they use LLMs selectively to find security issues and review code.

Several talks in the security devroom addressed the same problem.

Updated governance model for open-weight LLMs

A talk about LLMs and cyber risks argued for an updated governance model for open-source (and maybe open-weight) LLMs.
While closed LLMs are easily controllable, safeguards in open models can easily be fine-tuned away.
Therefore, open LLMs cannot be regulated like an API, and closed LLMs may even have an advantage because they are easier to regulate — a “mitigation gap”.
The proposed solution is to define fine-tuned models as “substantial modification”, and shift the liability burden from the issuer of the original model to the fine-tuner.

Auditing and securing supply chains

The Open Source Technology Improvement Fund (OSTIF) presented their work on providing security audits to open-source software (talk).
AboutCode presented their tool for detecting LLM-generated code (talk). If I understood correctly, the tool finds parts of a codebase that have been regurgitated from another source, and can pinpoint to the source. One challenge was that LLMs often create similar control flow, but different variable names from the original. In their scancode.io tool, they solve this problem with code stemming, a method also used by treesitter.

Data, Search and LLMs

Vector search

In a RAG pipeline, the hard part is data engineering: one has to understand the data and the context (talk). For instance, the chunking strategy is crucial for the retrieved the results.
Weaviate presented and demonstrated multi-vector retrieval. This is particularly useful for search on PDFs that include images. Their tool implements the MUVERA algorithm (paper).

Speeding up LLM inference

The vLLM project explained how they speed up LLM inference with quantization and speculative decoding.
Quantization compresses the network weights into buckets. This leads to a smaller footprint in memory and to faster transfer of the weights from the GPU’s high-bandwidth memory to the SRAM and Tensor cores that do the matrix multiplications.
Their benchmarks show that the ideal strategy (4-bit integer quantization vs. 8-bit integer quantization vs. no quantization) depends on the number of queries per second.
For speculative decoding, one trains a light-weight “speculator” model that generates tokens at inference time, and the main model approves or rejects the generated tokens.

Machine learning, performance, and observability

Performance engineering

Two talks discussed best practices for performance engineering: to reliably capture performance regressions, benchmarks should be repeatable and representative, and setting them up for this requires some thought. Challenges include isolating the benchmark environments and avoiding too many false positives.
The first talk outlined a statistical testing approach based on increasing the signal-to-noise ratio and deciding when to reject the null hypothesis of no performance regression. Further, running benchmarks in the cloud poses challenges, and they recommend avoiding virtualized environments. The slides of the talk are here.
The second talk focused on change point detection. I also liked the idea of using canaries to track the performance of the benchmark infrastructure itself — you want to know when the problem is with the infrastructure and when it is with your software.
Both talks argued for continuous performance monitoring and presented some tools for this.nyrkioprovides CI runners for change point detection; the runners are not free but according to the provider, they are of better quality than other runners (such as github-action-benchmark). hyperfine is a command-line benchmarking tool.

Performance monitoring of deep learning workloads on HPC

Two talks addressed performance monitoring for deep learning workloads on HPC systems.

The first talk highlighted the shortcomings of nvidia-smi compared to dcgmi. In short, the former only tells us whether the GPUs are busy, but not how efficiently they are being used (tensor cores, streaming multiprocessors, DRAM). One example was that using 32-bit and 16-bit precision on a H100 shows the same utilization on nvidia-smi, even though FP32 is less efficient because H100s do not support tensor core computations with this precision.
The same talk also suggested that running dcgmi incurs no overhead because it is reading data that is already being tracked. I found useful docs from SURF here, and they are more cautious, mentioning that dcgmi may slightly slow down your code. I guess one has to test and see it for themselves. The NVIDIA docs for dcgmi are here.
The second talk presented an open-source observability dashboard for deep learning on HPC. It covers hardware, workload, and model health (such as gradient tracking). The tool was just released during the conference. I’m curious to see how it evolves and how it compares to other tools.

Ole Mussmann on FOSDEM 2026: Nix, International OSS, Accessibility and Collaboration

Opening

A quote that hit hard:

If we lose our democracies, Open Source is irrelevant and goes away.

Nothing to add here.

Nix and NixOS

nix for Determinism

Bruce Gain discussed using nix for “deterministic distributed-system benchmarking”. Without special care, library versions and kernel versions will drift over time. The low-hanging fruit docker is not helping here. It solves packaging, not reproducibility, for two reasons:

docker uses the host’s kernel, and
it is really hard to make a container deterministic. apt-get update is not reproducible, any unpinned library will drift over time. You could distribute the images instead of Dockerfiles, but those are huge and not meant to be changed (only appended).

nix solves those issues by treating hashing and pinning every input of a project. Infrastructure is treated as a pure function. For identical inputs, the output must be identical as well.

Software Bill of Materials (SBOM) Tools for nix

What’s inside your software package? Which dependencies do you use, and what’s their dependencies? This surprisingly hard question is relevant for license compliance (did you obey the licenses of all libraries that you used?) as well as cybersecurity. If there’s a vulnerable package somewhere in the stack, you’d like to know, right? There’s a few tools that can help you out:

Determinate Systems seem to have their own tool as well, but it does not seem to be public (yet?).

Thanks for the heads up from Tristan “TheComputerGuy” Ross.

Sphinx Documentation for nix Code

Sphinx was originally created as a documentation tool for Python code, but it has since become a generic tool targeting all sorts of projects. [1] With Rémi “minijackson” project [sphinxcontrib-nixdomain](https://github.com/minijackson/sphinxcontrib-nixdomain) (rolls right off the tongue, doesn’t it?), one can document nix options, modules, functions, package sets…

This is the first time I see a structured approach to documenting a nix project. Well done!

International Open Source

Open Source in China

Open Source looks different in China, says Richard Lin. There are several factors at play:

FLOSS is seen as a market capture tool in three easy steps:
Turn standards into de facto rules,
Rules into monopoly, and
Monopoly into profits.

Going global is not expansion, it’s survival.

The market in China is dry, so branching out is a necessity.

FLOSS is a declaration, not procurement. The goal is to nurture industries that will enlarge future tax revenue, not buying a working product. The process for companies is:

Self-declare a directional FLOSS project,
Build it,
Pass inspection, and
Receive cash.

Software development, even FLOSS, is a cathedral, not a bazaar.

Development is top-down.
Pressure from the FLOSS community is slowly changing that.

You might notice a lack of FLOSS culture. Instead of “software wants to be free”, this looks very market-driven. And yet, even through this lens, this flavor of open source looks better to me than closed source. It will be interesting to watch how FLOSS develops in China, and how the different viewpoints will evolve.

Accessibility

I arrived late to Mike Gifford’s talk “Accessible Sovereignty: Why the Four Freedoms Depend on Inclusion”, so I can’t say much about the actual content. What did impress me was that he had a live-transcription of his voice underneath the slides. That is not only terrific for the hard-of-hearing, but also a very comforting addition for everyone else.

Lessons at the eScience Center often use web slides made with [reveal.js](https://revealjs.com/). Modern browsers have a Web Speech API which can be used for speech recognition. Do you see where this is going…? Would it not be great to have a plugin for reveal.js presentations that display speech-to-text below the slides? Any volunteers to build this? Anyone?

Ok, fine. I’ll do it myself.

P.S.: Here it is: https://github.com/OleMussmann/RevealSubtitles

Collaboration

Thierry de Pauw makes an argument that pull requests are useful for open-source work, but are more of a hindrance in the corporate world. They were coaching a novice team of developers and, to make things simple, they introduced trunk-based development with Non-Blocking Continuous Code Reviews. This means:

Everything happens on the main branch, there are no other branches.
Changes are (automatically) tested before deployment.
Every morning, developers review some commits.
In the end, every commit will be reviewed, after* being deployed(!).
Fixes are applied, if needed, though this is rare.

Woah.

I can see this working for a very specific developer situation while having absolutely rock-solid tests. Distributed development? Difficult. Working on features, while keeping the main branch stable? Impossible. Troubleshooting bugs, bisecting a commit history? Tough.

This is probably not a good fit for research-software-engineering. But I have to say, kudos for trying something weird and making it work. And as a side effect, I realize that I have to be more flexible with my assumptions about how to develop software. That alone is already worthwhile.

https://documentation.help/Sphinx/index.html

Faruk Diblen on FOSDEM 2026: Sovereignty, Burnout, and the AI Reality Check

I hope that by providing this short summary, I can convince you to participate in the next FOSDEM. It remains the biggest and most awesome open-source conference in Europe — and it is still free. It is so large that it reminds me of certain free open-source software projects: it is full of a specific kind of chaos. This is not due to the organization, but rather the scale of the conference and the range of topics and talks happening simultaneously. I must give kudos to the organizers for doing an amazing job.

Reflections on AI and the Community

As I do every year, I was expecting amazing talks from people who are very passionate about open source and “geeky” topics. However, this time I also expected to hear more about how AI is supporting software development. Instead, most talks focused on how AI is dangerous and toxic for open source.

I think I partly agree with that sentiment, but it may still be useful for certain tasks as a supporting tool. There were many interesting talks and discussions regarding the cultural, ethical, legal, and technological effects of recent developments in AI.

The infamous xkcd comic about infrastructure was also compulsory for all speakers to show. Perhaps the organizers were checking presentations beforehand, and if you did not have this comic, they were not allowing you to present. Jokes aside, I also agree now that this comic represents the current reality; it is perhaps even too modest.

Networking and the Research Track

One of the other fun parts of FOSDEM is the chance to meet and have conversations with great minds, contributors, and initiators of very impactful open-source projects. Like previous years, I gathered new ideas and identified new potential collaboration opportunities. I should not skip the “fun stands”; you can talk to the amazing groups of people who made your favorite Linux distribution or who are working on drone development.

As a researcher, I also had a chance to follow some of the Open Research track. Although the research community was under-represented, it had a great variety of interesting topics, some of which are relevant to my own work. My positive experience has made me think about submitting a proposal for next year. For the other talks I followed or found interesting but could not attend in person, please see the Highlights** section below.

After leaving the conference, I had three things in my mind: sovereignty now, sovereignty in the near future, and sovereignty in the far future. I also deeply felt the messages of “AI is killing Open Source,” “Even if AI gets better, we, the developers, will be needed,” and “Everything will collapse if we do not support open source.” These messages were sometimes subliminal and sometimes mentioned openly.

Keynotes

FOSDEM 2026 — FOSS in times of war, scarcity and (adversarial) AI: Given by Michiel Leenaards, our neighbor next door. I was expecting him to talk about how NLnet supports open-source software, but he focused on the threats caused by geopolitics. Given the recent political changes in many countries, I think it was a wisely chosen topic that forces us to think about upcoming challenges.
FOSDEM 2026 — Free as in Burned Out: Who Really Pays for Open Source?: Marga Manterola, a long-time Debian developer, talked about why the “donations and sponsorships” model is failing maintainers. She listed common funding sources including a sort of “Open Source tax.” This talk may help us find solutions for software maintenance and sustainability funds.
Open Source Security in spite of AI: This was the most fun keynote, given by Daniel Stenberg. He told the story of the cURL project and his fight with issues and pull requests created by LLMs. Eventually, they decided to stop their Bug Bounty program. You can download the talk by using the curl command below:** curl https://ftp.belnet.be/mirror/FOSDEM/video/2026/janson/B7YKQ7-oss-in-spite-of-ai.av1.webm --output curl_keynote.webm

Highlights

(The talks I liked the most)

RRP: Reproducible Research Platform for FAIR Open Research**: Andreas Cuny showcased the RRP: a heavyweight solution combining openBIS (RDMS), Git, Docker, and Kubernetes. It allows users to mount datasets into Jupyter/VS Code environments instantly. It is a bit complex to set up, but powerful for institutions. I had a very pleasant conversation with Andreas regarding potential collaborations, such as adding actual FAIR software checks (via howfairis) to their platform.
Who Funds the Egg? Cracking the FOSS Funding Paradox: Deborah Udoh highlighted the “chicken and egg” problem: you need impact to get funding, but you need funding to create impact. The “Pre-seeds” concept — non-monetary investments like visibility and mentorship — seems like a viable bridge for early-stage projects.
Multimodal support in llama.cpp — Achievements and Future Directions: Huge updates for local AI. libmtmd finally standardizes multimodal support. They showed llama-mtmd-cli doing low-latency OCR and experimental audio input with Ultravox.
It’s Time to Audit Open Source: Success Stories with OSTIF: OSTIF has now facilitated over 100 security audits for projects like git, cURL, and Kubernetes. The data shows that independent code review works.
Supply chain security meets AI: Detecting AI-generated code: Philippe Ombredanne introduced a new tool for “fuzzy matching” code snippets to detect AI-generated content. With bots writing billions of lines of code, exact hashing does not work any more.
The AI Shockwave in Open Source Communities: How AI Is Reshaping the Foundations of Open Source Communities: A worrying trend where newbie questions are dropping because people ask ChatGPT instead. This breaks the “contributor pipeline.”
Burnout in Open Source: A Structural Problem We Can Fix Together: Miranda Heath identified “hyper-responsibility” and “toxic entitlement” as key burnout drivers. We need structural changes in how we value maintenance work.
Introducing Jupyter Book 2: Next-generation Tools for Creating Computational Narratives: A complete rewrite based on the MyST (Markedly Structured Text) engine. It produces semantic, machine-readable content that is much easier to cross-reference.
ParticleOS, from Fedora to Feast: Stirring Traditional Distros into Immutable Delights: Luca Boccassi showed how systemd and mkosi are changing OS building, creating a fully immutable, signed, and verifiable image.
What do we mean when we say Sovereign AI?: Katharine Jarmul explained that true “Sovereign AI” is not just about where the model is hosted, but having the full training data and code available to inspect.

Other Notable Tracks and Talks

Research & Science

Main Track & Community

AI & Machine Learning

Security, Databases & Infrastructure

Booths and Extra Notes

After hearing about sovereignty everywhere, I had lengthy chats with the folks from GitLab, NextCloud, LibreOffice, Forgejo, and Codeberg to learn about open-source office solutions and infrastructure. I also talked to someone from Germany who explained the government migration to Linux (see: Yet another European government is ditching Microsoft for Linux).

I visited the Tor stand, where you could get fancy stickers if you donated to help them fight censorship. The Free Software Foundation Europe (FSFE) is also worth following closely as their goals are related to our own. Finally, I learned about funding.json, a format to declare financial needs of projects in a machine-readable format.

Bring Your Own Backend

eScience Editorial Team — Thu, 26 Mar 2026 00:00:00 GMT

A pattern for sustainable research software collaboration

By Stefan Verhoeven.

Image source https://www.pnnl.gov/news-media/new-nationwide-modeling-points-widespread-racial-disparities-urban-heat-stress

As a research software engineer (RSE) working at the Netherlands eScience Center, I work on projects together with domain scientists at universities and research institutes.

For example, currently I am working on the Urban-M4 project, where we are trying to improve a weather model by telling it the properties of buildings in a city. As part of this project, I built a web-based application called streetscapes-explorer to create, review and edit segmented street view images. My engineering colleagues and project partners worked on the machine learning models and data processing pipelines that powered the backend of the web application.

But as the project neared its end, a familiar worry crept in: what happens after I leave?

The sustainability problem

The project partners are first and foremost domain scientists. They have the expertise and time to maintain a Python backend — updating dependencies, fixing bugs, evolving their data processing pipelines. But maintaining a highly interactive frontend written in TypeScript, with its constantly shifting JavaScript ecosystem, build tools, and UI frameworks, is a different story. That’s not where their expertise or time should go.

A traditional monolithic application — with frontend and backend tightly coupled and deployed together — would force them to maintain both. They’d need to keep the server running, dependencies updated, security patches applied, and yes, understand that frontend code when something inevitably breaks. Given their constraints, this is a recipe for software rot and eventual abandonment.

I needed a different approach. One where the frontend could essentially “freeze” after I left, requiring zero maintenance, while my partners could continue evolving their backend independently.

The solution: Bring Your Own Backend

The key insight was simple: decouple where the frontend is hosted from where the backend runs**.

Instead of deploying the frontend and backend together on some server that my partners would need to maintain, I separated them completely:

Frontend: As a single page web application using React framework. Hosted as static files on GitHub Pages — zero maintenance, free hosting, always available
Backend: Runs on my Users or partners’ own machines, under their full control

The magic that connects them? A URL query parameter.

The main idea, in a nutshell

The Magic: URL-based backend configuration

Here’s the core pattern. The frontend reads the backend URL from the query string:

export function App() {** const backend = new URLSearchParams(location.search).get(‘backend’); if (!backend) { return <BackendForm />; } return <StreetscapesExplorer backend={backend} />; }When someone visits the frontend without specifying a backend, they see a helpful form with instructions:

When the frontend is unable to connect to the specified backend, it shows an error message with troubleshooting tips:

The workflow becomes simply:

A user starts their backend locally
A user visits the frontend URL with ?backend=http://localhost:5000
The frontend connects to their local backend
All data stays on their machine

No rebuilds. No redeployments. No complex configuration. Just a URL parameter.

Keeping it simple for project partners

For this pattern to work after I leave, the backend needs to be dead simple to run.

from dataclasses import dataclass, field from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware

app = FastAPI()Web browsers do not like the frontend and backend to be at different URLs. For this in the backend we need to enable CORS like so:

app.add_middleware( CORSMiddleware, allow_origins=[""], allow_methods=[""], )Then anyone can simply write a decorated and type annotated Python function to define an API endpoint:

@dataclass class FilterParams(Bbox): ratings: list[int] = Field(default=[])

@dataclass class Image: id: str url: str lat: float lon: float @app.get(“/images”) async def fetch_images(filter: Annotated[FilterParams, Query()]) -> list[Image] # … implementationAnd to make it even more user-friendly, the backend prints a clickable link to the frontend on startup:

def main() -> None: url = “https://urban-m4.github.io/Urban-M5/?backend=http://localhost:8000” print(f”Waiting for the streetscapes-explorer to start…”) print(f”Goto {url}”) print(“(Press CTRL+C to quit)”) uvicorn.run(app, host=“0.0.0.0”, port=8000)

if name == “main”: main()If you’re feeling more helpful you can use webbrowser.open(url) to open the URL automatically in the user’s default browser.

The contract: OpenAPI as the bridge

The frontend and backend need to agree on an API contract. Rather than maintaining documentation that inevitably drifts from reality, I used OpenAPI as the single source of truth.

FastAPI automatically generates an OpenAPI specification from the Python code. I then use OpenAPI TypeScript to generate TypeScript types:

openapi-typescript http://localhost:5000/openapi.json -o ./src/lib/streetscapes-api.tsThe frontend then makes type-safe API calls:

import createClient from ‘openapi-fetch’; import type { paths } from ’./lib/streetscapes-api’;

Try it out yourself

I made a minimal example of this pattern with a simple TODO application. To use follow the instructions to run the backend locally.

The source code is available at byob-todo-frontend and byob-todo-backend. In the frontend I used SolidJS and bun to keep the frontend light. In the backend I used uv and inline dependencies to keep the backend easy to run.

When to use BYOB

This pattern isn’t for every project, but it shines when:

Different expertise levels**: Frontend specialists work with domain experts who prefer other languages
Data sensitivity: Users need to keep data on their own machines
Limited resources: No budget for ongoing server maintenance
Collaborative research: Multiple groups might want to run their own backends

The “Bring Your Own Backend” pattern turned what could have been abandoned software into a sustainable tool that my partners can use and evolve long after our collaboration ended. Sometimes the best code you write is the code others don’t have to maintain.

This post was originally published at the author’s GitHub account.

Give Me Data, Yesterday

eScience Editorial Team — Wed, 18 Mar 2026 00:00:00 GMT

I want this data, and I want it yesterday!This post is adapted, with the author’s permission and collaboration, from this post in his personal blog.

This is a quest. A quest for the right tool for the right job, for usability, and performance. We will torture databases, run tens of thousands of benchmarks and conjure charts, numbers, and tables. Because science. 🧑‍🔬

But first, some background.

The patient is the package search program [nps](https://github.com/OleMussmann/nps/), which provides information about all packages the [nix](https://nixos.org/) package manager has to offer. What’s the patient’s complaint? nps needs a better database and a better way to update its data.

Erm, nix? Never heard of it? It’s kinda special. Unlike apt, pacman, or their siblings, it is declarative, reproducible and reliable. To top it off, it has a gigantic library. If you don’t know what that means, no worries! Here’s a primer.

`nix`? What’s That?

How do you set up your operating system? You install, for example, Ubuntu. Then you install a bunch of packages. You configure them. Then you set the wallpaper, the color scheme, the fonts, and make yourself at home. And if you set up a new system, you start again from square one. This whole process is procedural, you follow one step after another. If you re-do all your steps again, you get the same result; and any changes to the recipe will result in a different outcome.

With nix, you describe the end state you want to have, and nix will build and set up the system like you describe. Reliably. Reproducibly. nix is declarative.

Worry not, you can still apply tweaks and modifications. The desired end state including all customizations can be described in a single configuration file (keep it simple), or a set of files (make it modular). Let’s call this configuration file a “flake”. ❄️ Throw git at this flake file and now you can jump between different versions on demand.

If you share your flake with someone else, they can re-create the exact environment on their system. This is great for collaborative development! Runs on your computer? Runs everywhere. You can run other people’s software without installing, and others can run yours. For even more portability, you can create Docker containers, virtual machines, or even a bootable ISO from your project.

Sounds like magic? It comes close.

Current gotchas include a lack of broad adoption, a steep learning curve for the nix language, as well as not great (but improving) documentation. And, well, a distinct lack of a native, fast, offline program for searching packages.

Paper Cuts

nix’s usability has, let’s say, still room for improvement. Prime example: how to find out which packages one can install. What do these packages do exactly, and what version do I get? The recommended way is – drumroll – a website: search.nixos.org/packages. It’s fast, comprehensive, and detailed. It works well.

But it’s a website.

When I’m pondering about installing packages, I’m not staring at a browser. I’m in the terminal. I want to have the equivalent of apt search or pacman -Ss¹.

There are some wonderful CLI search programs out there², but they hook into the ElasticSearch database powering the backend of search.nixos.org/packages. This means that you need to be online to get any package information, and (minor quibble, I know) it takes a bit for the information to appear on your screen.

`nps`

I built the Nix Package Search [nps](https://github.com/OleMussmann/nps/) program to do my part to make nix more user-friendly. nps stores the package information locally, searches this cache, and presents the result – color-coded and sanely sorted. It works offline. It’s fast. It’s beautiful. Yeah, yeah, I know, I’m biased. Try it for yourself, though! Remember, you can run it without installing.

Lately, kevb1973 asked me to provide more package information in the (dense) output that nps provides. It’s a great idea and would increase the utility that nps provides! Unfortunately, this is not something I can just bolt on to the existing program. Let’s have a look at the moving parts of nps to understand why.

Architecture

Ok, architecture is a big word. Let me run by you how nps currently works:

Run nix search nixpkgs ^ – get all information for “all” packages.
Create a text file with package names, versions, and descriptions and store them locally.
grep through the text file (yes, it’s that low-tech) to find matches.
Sort matches nicely into exact hits (searching for “avahi” matches package avahi), direct hits (finds packages starting with “avahi”, e.g. avahi-compat) and indirect hits (“avahi” appears anywhere in the package name or description).
Create columns for better readability.
Add a lick of paint to color-code the different types of matches.
Print the result.

The issue: the data is stored as one row per package. Adding more information to each package row will make it harder for grep to find the packages you’re looking for. This looks like a job for a “proper” database now. But which one? There’s plenty of cool tech out there, but which would be the “best” one for this use case?

Database Candidate Boundary Conditions

Alright, before we dive in, let’s define a few corner stones.

Data Source

Remember the nix search command from above? It does not actually find “all” packages, https://search.nixos.org finds significantly more³. Why? No idea. Also, nix search supplies only the package name, version, and description. No other information. Not great. Another aspect that’s missing is that packages might provide a differently named executable: you install the package neovim and use the executable nvim; some packages even provide multiple different executables. nix search can’t find those.

The data needs to come from a better source.

The ElasticSearch backend is a good choice, but it’s meant to be queried, not downloaded. This means we have to scrape it in its entirety first.

Setting up a project that does the scraping and regularly builds a cache that is usable by (a future version of) nps is a task that goes beyond the scope of this post. Let’s assume for now that we have the data available, and we can squish (very scientific term here) it into any shape or form that we need for querying.

Substring Matching

Subwhat? A substring is a part of a string: “crow” is a substring of “miCROWave”. It can be anywhere in a string, beginning, middle, or end. When searching for “nvim” I want to find nvim (duh!), as well as nvimpager “use neovim as pager”, and gnvim “GUI for neovim, without any web bloat”. As most databases are specialized in retrieving “exact” matches, substring matching might take some convincing.

Query Time

I want future nps to stay fast. I’m happy with the responsiveness of the current implementation. To get a feeling for this, using nps to search for the package nps takes about 6ms on my desktop – so let’s take that as a baseline. If the pure data retrieval of the database candidates takes longer than current-nps, then that’s a red flag, since sorting, coloring, etc. will take some extra time on top of that.

Quick Cold Start**

Related to “Query Time”: many “serious” databases are designed to run as a service; that makes querying fast, but you need something running in the background all the time. We don’t want that. Instead, we have a single program that starts up, queries a database, prints stuff, and then quits. This excludes many of the usual suspects like MySQL and its colleagues.

Data Amount

The current data amount is about 8.5 Mb of package information and metadata for one nixpkgs channel. The future, more detailed one (also including a long description, project URL and location within the nixpkgs repo) would be 26 Mb of plain text. For three NixOS channels (“unstable”, “current stable”, “previous stable”), this would be about 80 Mb. While compression can bring this down a bit, some databases need to index their database to be able to search quickly; this will increase the data amount again. How much? That depends on the database used.

The increased data amount is defensible when taking into account that one could regularly download the incremental changes instead of the whole database for refreshing the cache.

Yes, this is pretty hand-wavy for now. We’ll get more concrete about this later.

Memory Usage

A bit less crucial than the above, I’d like to use as little RAM as possible during the search. nps is a guest on your machine and it should use only as much of your resources as needed.

Many databases (as well as the good ol’ grep) don’t load the whole data into memory before searching, they “crawl” through the data and drop the parts they have already used. I’d set that as a “nice to have”.

Database Candidates

First, let’s discuss the candidates and their structural strengths and weaknesses. We get to the hard numbers below, in the chapter “Gimme Numbers!”.

`grep` a Text File 🦖

Why change a working system? The old dinosaur grep is fast. Really fast. We have the “GNU” flavor that runs on the command line, then there’s a Rust implementation called [ripgrep](https://github.com/BurntSushi/ripgrep), and lastly the Rust crate called [grep](https://docs.rs/grep/latest/grep/), which is – despite the name – closer to ripgrep than to grep.

Let’s dial it up a notch. I’d like to see how just calling grep or its cousin ripgrep from the command line (equivalent to step 3. from “Architecture”; we will label all those tests with the “CLI” suffix) compares to calling it from Rust and splitting the results (necessity for the “new” architecture). ripgrep likes to work with streaming data, but we can’t do that if we want to form columns later. So what’s the overhead of the extra data mangling?

That brings us to the following list:

GNU grep (called from CLI, results not split)
ripgrep (called from CLI, results not split)
GNU grep (called from Rust)
ripgrep (called from Rust)
Rust crate grep (called from Rust)

Benefits

Simple
Single file “database”
Easy and good compression
Trivial substring matching

Drawbacks

Extra finagling to only search certain parts of a line, yet still returning the whole row of data; in practice this means extra lines of code plus possibly a few extra gray hairs
Incremental data updates are difficult

SQLite 🪶

The gold standard for “simple” databases. Scales less well for serious applications than MySQL (or MariaDB or whatever), but should be “good enough” for a few hundred thousand packages.

We use trigram (N-gram with N=3)⁴search to achieve fast substring matching. This makes it, well, fast. But it also inflates the database size and ignores any search terms shorter than 3 characters. One could check for search string length and use a different matching algorithm for shorter terms, but is that worth the effort?

Benefits

Single file database
Battle-tested and stable
“Proper” database queries, the results are well-structured
Incremental updates are trivial

Trade-offs

How compressible is an already compressed database?
Needs indexing for fast substring matches, balancing database size vs performance

Drawbacks

Compression per row, not on the whole file

Tantivy 🐎

I sparred with ChatGPT about which database to use, and it was very enthusiastic about Tantivy — we will see later if that was a good idea or not. It is designed for a quick cold-start and fast data retrieval, sounds like a match to me! Tantivy stores data in several files and has a meta.json file to figure out what’s what.

A quirk: technically, you can’t delete already stored data. Instead, you add a “delete” file that ignores entries and you can delete storage files that eventually become obsolete. Still, incremental updates should not be too hard this way. Every once in a while one would need to re-download the whole thing once the local index grew too large with all those “delete” files.

Same as with SQLite, we use trigram matching for speed, with the same benefits and drawbacks.

Benefits

Pretty much designed for this purpose
Incremental updates are easy
“Proper” database queries, the results are well-structured

Trade-offs

The database is a folder with multiple files
Needs indexing for fast substring matches, balancing database size vs performance

Drawbacks

Incremental updates do need a strategy, with an occasional fresh download of the whole database
The many files might take some time to read, benchmarks will tell if that’s an issue
Compression per row, not on the whole file

DuckDB 🦆

Completing our zoo is the new duckling on the block. DuckDB sports rad analytics which might be handy for retrieving already correctly sorted search results.

However, DuckDB can’t do N-grams. Yet. Sad quack.

What can we do instead?

Benchmark the slower, standard way of case-insensitive substring matching, called ILIKE.
Benchmark the slower, other way of case-insensitive substring matching with regular expressions: regex.
Who needs built-in N-gram indexing when Gemini can build one (let’s call it trigram) for you? We will soon find out if that’s a good idea.

Benefits

Ducks are cool
Single file database
Supports incremental updates
“Proper” database queries, the results are well-structured

Trade-offs

Would the analytics work for proper sorting?

Drawbacks

No built-in N-gram indexing for fast substring matching, uh oh!
Compression per row, not on the whole file

Benchmarking Boundary Conditions

We will be testing for a lot of things at once, here’s a break-down.

Parameters

We are searching for four different search terms with increasing number of matches. This should give us enough data points to make out a trend.

“nps” (21 matches)
“nvim” (1301 matches)
“python” (21123 matches)
“e” (131567 matches, almost the whole database)

Data Types

To be able to compare database performances with the current implementation of nps we start with a “minimal” database, containing the package name, version, and short description. This mirrors the current implementation of nps and allows for comparisons.

The entire “minimal” data for the abcde package look like this:

abcde 2.9.3 Command-line audio CD ripperThen we try a “detailed” database, containing:

package name
executable names (which might be different than the package name)
version
description (short)
description (long)
homepage URL
and the URL to the package in the nixpkgs repository.

Formatted, the information for the same package could look like this:

abcde [abcde-musicbrainz-tool, abcde, cddb-tool] 2.9.3

Command-line audio CD ripper abcde is a front-end command-line utility (actually, a shell script) that grabs tracks off a CD, encodes them to Ogg/Vorbis, MP3, FLAC, Ogg/Speex and/or MPP/MP+ (Musepack) format, and tags them, all in one go. Project URL: http://abcde.einval.com/wiki/ Nixpkgs URL: https://github.com/NixOS/nixpkgs/pkgs/by-name/ab/abcde/package.nix:83Off-topic: this project’s website gives me heavy 90’s vibes, woah.

`hyperfine`, Engage

If you counted correctly, you should have four search terms, two data types, and eleven search approaches. We are skipping some combinations, e.g. “detailed search” is not supported on the O.G. nps, and Tantivy and DuckDB can’t find anything shorter than 3 characters. In the end we have more than 80 distinctly different benchmarks. Some are run from Rust, some from the command line. To be able to compare them, we benchmark them with [hyperfine](https://github.com/rmlmcfadden/hyperfine). Bonus: on top of min-, max-, mean-, and median-runtimes we also get the memory usage of the runs. Nice!

This is what a typical run on the command line looks like:

$ hyperfine ‘nps neovim’ Benchmark 1: nps neovim Time (mean ± σ): 6.0 ms ± 0.4 ms [User: 1.5 ms, System: 4.4 ms] Range (min … max): 5.0 ms … 7.3 ms 351 runsEven better, we can also get the results in JSON format. Unfortunately, this also means that it’s written to file.

Sigh. So we have to read in that file to get the data we need. On the plus side, the JSON detour makes sure we don’t have any unforeseen hick-ups with data parsing, which could have happened if we wanted to retrieve the runtimes via regex from the output shown above.

The JSON data looks like this:

{ “results”: [ { “command”: “nps neovim”, “mean”: 0.006715249454725274, “stddev”: 0.0026061538013605248, “median”: 0.0065737821799999995, “user”: 0.001504577142857143, “system”: 0.005087126153846152, “min”: 0.00505421018, “max”: 0.03085960318, “times”: [ 0.03085960318, 0.00686827018, … ], “memory_usage_byte”: [ 12791808, 12791808, … ], “exit_codes”: [ 0, 0, … ] } ] }For benchmarking, we are using hyperfine the following way:

hyperfine —shell=none —warmup=5 —export-json [filename] ‘COMMAND’Break-down of the moving parts:

--shell=none tells hyperfine to, well, not start any shell. This shaves a few milliseconds off the benchmark score and allows for easier runtime comparisons.
--warmup=5 tells hyperfine to run COMMAND 5 times before actually measuring the runtime. This makes sure that all files that are read are properly cached first.
--export-json [filename] tells hyperfine to write out the data in JSON format.

Gimme Numbers

Now to the fun part. Buckle up, it’s finally benchmark time! With repetitions, we have >40.000 data retrieval runs.

Below are all the tests we run against the above parameters and data types. To be able to compare apples to apples, these are the steps each approach conducts, unless noted otherwise:

Read in the data,
Find package matches in the whole row (minimal databases) , or in the fields “package name”, “executable names”, and “package description (short)” (detailed databases),
Split the matched data if necessary, so we have access to the “fields” of the data row,
Collect the split data lines into an array, so we could sort and color them later, and
Print all results to screen.

Disclaimer: Apart from *nps*, all search approaches are mostly vibe-coded with “Gemini 3 Pro Preview”. I wanted to iterate quickly to test as many approaches as needed. I did my best to avoid the common pitfalls, but it’s always possible that I missed something. To be fair, the same warning would apply if I coded it by hand.*

This code will not end up in production.

Once I identify the correct approach, I will re-write it from scratch and work it into the *nps* implementation. This way I make sure I am responsible for and understand every line of it.

Comparing Speed against Current `nps`

We start out to compare the new contenders against the existing approach. For this, we query the smaller, “minimal” database.

`nps`

Providing a baseline, we try the current nps program. It should be slower than the other search implementations, since it provides the full pipeline instead of the four steps mentioned above; this includes forming proper columns and coloring the output.

We also throw the Rust crate grep into the ring; this is what nps currently uses internally. Usually, Rust grep likes to work with a data stream; this makes it really fast. For the benchmark, however, we are deliberately sabotaging its performance by collecting the output into a vector before printing. This is needed for the future nps implementation, where we will sort the results and color them appropriately. Without hamstringing Rust grep, the benchmark comparisons would not be entirely fair.

Regarding the data: often the *y*-axis (the vertical one) will be logarithmic. Each tick on the *y*-axis will denote “10x more than the previous one”. This makes sure we can comfortably see a difference between 2 ms, 2.5 ms and 2000 ms in the same plot.

Now we have something to aim for. If the pure search approach is slower than nps – which includes creating columns and coloring the output – then we shout “boo!”. Is it faster than the current data retrieval “Rust grep”? Extra brownie points.

GNU `grep`

“Rust grep” is already there. Now we compare it to “GNU grep” both from the command line and from within Rust. We keep the above run times for nps and Rust grep as a gray band for comparison.

The base speed is decent, but remarkably, the old “GNU grep (CLI)” becomes faster with shorter search terms. Afterwards there’s some extra work to do, namely splitting the lines into fields, negating this speed benefit somewhat. This run is labeled “GNU grep (CLI, rs)”.

`ripgrep`

Will the Rust flavor of grep fare any different?

Yes. Yes it does. Here we start faster than the “GNU grep (CLI)” above, but take a bit more time the more matches we find, labeled “ripgrep (CLI)”. Including line splitting, we’re competitive compared to nps. This is marked as “ripgrep (CLI, rs)”.

`Tantivy`

Next, we put ChatGPT’s favorite through the wringer. Note that we are dropping the search for e, since Tantivy cannot find it with the trigram matching in place.

While reasonably fast for few matches, it becomes pretty slow for increasing hits. Sad pony. After whining to ChatGPT about it, it suggests that it could have to do something with the number of files to be read. Doubtful, since it can be fast for few matches. I’m not sure why it can’t keep up.

`DuckDB`

We are probing three ducklings, one that uses ILIKE, one that uses regex, and the self-made trigram indexing.

Ok, this is bad. Really bad. Remember that we have a logarithmic y-axis? The bump you see in the plot is the jump from 88 ms for searching for “nvim” to 2.3 s(!) for searching for “python”. For the letter “e” it probably uses the same fallback search that the other two versions have, since anything shorter than three letters not in the trigram index.

Apparently Gemini cannot successfully hallucinate a feature which would take a small team of software engineers weeks to create, tune, polish, and test.

Moving on. The other approaches are not disastrous, but they are not great either. Since DuckDB has to crawl through every row of data to find substring matches, that’s not too surprising. If we were searching for exact words only, that would look much different. But alas, we’re not.

`SQLite`

How about the most widely deployed SQL database engine in the world?Fun fact, SQLite is open-source, but maybe not in a way you’d expect: instead of having a license attached, the code is entirely in the Public Domain.

Nice. For few-result queries, SQLite is really fast. Approaching thousands of results, it’s on par with Rust grep and only takes a moderate performance hit beyond that.

Summary

Here we condense the above results in a single plot and aim for the realistic worst-case scenario of searching for “python”. The results are not spanning orders of magnitude, so the y-axis is back to linear.

Comparing Speed for Future `nps`

To cut an entirely too long story slightly shorter, we get pretty much comparable results for querying the “detailed” database.

Let’s condense the above results in a single, easy-to-interpret table.

So far Rust grep and ripgrep are leading the pack, with SQLite and GNU grep hard on their virtual heels.

Database Size

That one is easy, we ask the disc usage tool du.

du —apparent-size -h *Break-down of the arguments:

--apparent-size - show the size of the files as they would be without file-system compression, and
-h - human-readable output, e.g. M for megabytes instead of just bytes.

On top of the plain size (as it would appear on a user’s machine), we also compress the databases with gzip to see how large a typical download would be.

$ du —apparent-size -h * 63M detailed.sqlite 28M detailed.sqlite.tar.gz 15M detailed_duckdb.db 6.3M detailed_duckdb.db.tar.gz 169M detailed_duckdb_trigram.db 48M detailed_duckdb_trigram.db.tar.gz 30M detailed_tantivy_index 21M detailed_tantivy_index.tar.gz 46M minimal.sqlite 23M minimal.sqlite.tar.gz 4.8M minimal_duckdb.db 2.9M minimal_duckdb.db.tar.gz 148M minimal_duckdb_trigram.db 42M minimal_duckdb_trigram.db.tar.gz 23M minimal_tantivy_index 15M minimal_tantivy_index.tar.gz 25M package_list_detailed.txt 4.7M package_list_detailed.txt.tar.gz 8.5M package_list_minimal.txt 2.3M package_list_minimal.txt.tar.gzIn the same discussion we also need to have a look at what updating the database with fresh package information would look like.

For plain text files, we would either download the whole thing from scratch, or supply diff information, so the existing file could be “patched” with the updates. This would work on a per-row basis which is unfortunate, considering that most of the time you would have an updated version number. To make things worse, the diff would need to indicate the old data and the new data. Not great.

Tantivy cannot directly change data, but works with “delete” files that ignore entries. Once a data fragment is ignored in its entirety, it can be deleted. In practice this would mean that the local database size would slowly grow with “delete” files, until a whole, fresh database would be downloaded. The same per-row gotcha from plain text files applies here as well.

DuckDB and SQLite not only support incremental updates, they also store the data as fields. This allows the updates to be tiny, containing just the information that has changed. This does require some data-dance of “downloadable full databases” and “incremental update files” as well as logic to combine those, but this is solvable.

Summarized in a table:

RAM usage

For both “minimal” and “detailed” searches:

This is excitingly unexciting. Ignoring the poor performance of DuckDB with bad trigram indexing, the worst contender only uses a smidgen more than 30 Mb of RAM. This is fine for all but embedded systems, which would be outside the target group.

Conclusion

Tallied up in one neat overview:

The family of greps were surprisingly fast. They did a splendid job up till now, but do not scale too well with additional information. Larger datasets also introduce the challenge of download size for updating package information.

Tantivy gallops quickly for few-result queries, but struggles a bit with large result sets. It’s unclear to me why that would be. Do let me know if you either know why, or point out where I made performance mistakes. Incremental updates are possible, but complicated.

The paddling (this is the actual collective noun!) of ducks did not fare too well, no thanks to the missing N-gram indexing. It’s probably plenty fast for retrieving exact words, but that’s not what we’re after here. B̶u̶i̶l̶d̶i̶n̶g̶ ̶t̶h̶e̶ ̶i̶n̶d̶e̶x̶ ̶o̶u̶r̶s̶e̶l̶v̶e̶s̶ Gemini vibing the index leads to the worst performance in all metrics. Shocker.

SQLite truly shines here. Speed is great, even with many matches. The only challenge is the increased database size due to indexing, but this can be solved by both extra compression for downloads as well as tiny diff files for updating existing databases.

Phew. That was longer than planned, but we finally have a candidate for the future nps implementation.

Award Ceremony

Due to outstanding scores in the “update” category, as well as excellent speed scores, we are handing the “cup of brrrrrr” to SQLite.

🪶🏆👌

The question of the data source remains unsolved. For the short term, I will scrape and provide the package data myself. Maybe in the future, nixpkgs data could be offered as a SQLite database, or at least as a JSON file?

But that, dear reader, is a challenge for another day.

They have their flaws as well. The packages that I want to check out are usually somewhere in the middle of a wall of text. Now I have to search the results of my previous search. Yay.
https://github.com/peterldowns/nix-search-cli and https://github.com/nix-community/nh?tab=readme-ov-file#nh-search
About 27%, or 132,108 vs 103,640 — as of January 2026.
This form of indexing stores strings of N letters pointing to the word they are extracted from: “duc”, “uck”, and “cks” would point to the word “ducks”, for N=3.

Making LLMs Work for Science: How AI-Powered Storytelling Helps Researchers Communicate Their Software

Jesse Gonzalez — Thu, 26 Feb 2026 00:00:00 GMT

Large language models get a lot of attention in research for what they might replace. We wanted to explore what they can enable: better, faster scientific communication that actually reaches the right audiences.

Welcome page on the storytelling application.

Authors: Jesse González, Pablo Rodríguez, Kody Moodley*
Application:* https://nlesc.github.io/data-storytelling

There’s an ongoing conversation in the research community about large language models. Most of it focuses on risks: hallucinations, bias, the worry that generated text might replace genuine scholarly work. Those concerns are valid and worth taking seriously.

But there’s another side to this conversation that doesn’t get enough attention. LLMs can be genuinely useful tools when they’re pointed at the right problems, with the right constraints and the right human oversight. Scientific communication is one of those problems.

At the Netherlands eScience Center, we build research software across four scientific domains: Environment & Sustainability, Life Sciences, Natural Sciences & Engineering and Social Sciences & Humanities. This means hundreds of projects, each of them representing months or years of collaborative work with research teams across the country and internationally, all catalogued in the Research Software Directory. The software is solid, sometimes groundbreaking. But communicating that work to different audiences (the public, academic peers, funders, workshop participants) takes time that most research teams simply don’t have. There’s a persistent gap between the research software we produce and the stories we tell about it.

We built a tool to close that gap. And in doing so, we learned some things about what it actually looks like when LLMs serve science instead of the other way around.

An interactive journey through four research domains

The first thing you see when you open the Data Storytelling app is a 3D particle system. Not a dashboard. Not a table. A cloud of two thousand glowing dots that shift and pulse as you scroll down the page.

The idea was simple: before you read a single word about a piece of software, you should already have a sense of the research domain it belongs to. Visuals create that context faster than text ever could.

Floating projects retrieved via the Research Software Directory

The entire application is one long scroll. As you move down, you pass through the domains, each with its own custom 3D scene built in Three.js. Each 3D scene is peppered with floating cards. Actual projects from the Research Software Directory, fetched live from the API. You hover over one, it glows and scales up. You click it and boom, a modal with the full project details. Description, DOI, metadata, links.

And tucked inside that modal? A tab labelled “ Generate Story.” That’s where the real magic happens.

The communication bottleneck in research software

Anyone in research communications will recognise this situation.

You’ve got a brilliant piece of software. It does something novel with satellite imagery or protein folding or natural language processing for historical Dutch manuscripts. You need to write about it. But write about it for whom?

The version for the general public needs to be warm, accessible, maybe open with a compelling question like “What if we could predict floods before they happen?” The version for an academic journal needs an abstract, proper methodology discussion, and citations formatted just so. The internal review for your board needs FTE estimates, risk assessments, and honest talk about technical debt. And your programme director just wants a one-pager they can scan in three minutes between meetings.

Same project. Four different pieces of writing. And you need to do this for a hundred projects.

This is where good research software becomes invisible. Not because the work isn’t worth communicating, but because the communication itself is a bottleneck.

The software ships, the README gets written, maybe a tweet goes out, and that’s it. Months of collaborative work, reduced to a paragraph.

This is also precisely the kind of problem where LLMs can be a real asset to researchers. Not as a replacement for human judgment, but as a drafting tool that handles the structural heavy lifting: tone calibration, audience adaptation, first-pass content generation. The key is being intentional about how you use them. Not “throw a description at a chatbot and hope for the best”, but structured, context-rich prompts designed for specific communication goals.

Generating stories feeding the model with extra information

The generator pulls related software from the same research domain via the RSD API, without any extra effort from the user. So if you’re generating a story about a climate modelling tool, it already knows about the other environmental software in the ecosystem and can reference them.

Project & Software Information

You can also upload documents. PDFs, text files, markdown. Research proposals work great for this. The app extracts the text (up to 10,000 characters worth), folds it into the prompt, and the output goes from generic to grounded in specifics.

For custom projects (things not in the RSD) there’s an “Own Project” button in the top navigation. Pop in a title, description, some reference URLs, upload whatever context you’ve got, pick your audience, and off you go. Same six templates, same quality, no RSD dependency.

Six audiences, six different stories

The story generator ships with six prompt templates. I want to be specific about what these are, because “prompt template” sounds trivial and these are anything but.

Generating the story given the selected audience.

Each one is 500-plus words of careful instructions, closer to a creative brief than a prompt. They specify structure, tone, section order, word count targets, what to emphasise, what to skip. Here’s what each does:

Communications (roughly 800 to 1,200 words). Tells the model to write like a science journalist.

Academic (1,500 to 2,000 words). Formal. Structured. Abstract up front, methodology section, validation results, related work comparison, citation placeholders in `[Author, Year]` format.

Internal Review (1,000 to 1,500 words). This is the honest one. The prompt tells the model to be “candid about challenges and risks” and to include resource analysis with FTE estimates.

One Pager (under 300 words). Concise to the point of ruthlessness. What it does, who it’s for, key benefits, quick stats, how to get started.

Course Material (2,000 to 3,000 words). We added this one not long ago and I’m kind of excited about it. It generates educational content you could hand to someone: learning objectives, a technology deep dive, step-by-step tutorials, three tiers of exercises (beginner, intermediate, advanced), and self-assessment questions. Give it to a workshop organiser and they’ve got a starting point for a full training session.

Blog Post (800 to 1,200 words). This one was modelled directly on the eScience Center’s own Medium blog. We analysed dozens of our published posts to capture the voice: community-focused, accessible, story-driven with a problem-solution structure. The idea is that a researcher or RSE could generate a first draft and hand it to communications with minimal rework.

The difference between a good and a mediocre generated story almost never comes down to the model. It comes down to the prompt. Two thousand five hundred words of structured instructions. That’s the actual product. The Gemini API call is almost the easy part.

What this taught us about LLMs as tools for science

Building and using this tool surfaced a few insights about how LLMs can genuinely serve researchers, rather than just generate noise.

Nobody needs another text generator.

Researchers need a tool that adapts to the audience and understands the difference between writing for a review committee and writing for the general public. The real value is in the gap between generate some text about this software and generate an internal review with risk assessment and FTE estimates in the right tone, because once you add that kind of specificity to prompts, LLMs stop being gimmicks and start being useful.

Another big win is generating multiple perspectives on the same project. If you create a communications piece, an academic paper, a review document, a one-pager, a course module, and a blog post from the same source, each lens reveals something different, like overlooked methodological contributions, the true core value in twenty words, what is actually teachable, and how the work connects to the wider research community, so the LLM becomes a thinking tool, not just a writing tool.

And honestly, context matters more than model sophistication. Running the same prompt on Gemini Flash versus Gemini Pro changes little, but enriching the input with a research proposal, team info, and related software packages changes everything, which suggests institutions will get more value by investing in structured metadata and rich context than by chasing the newest model.

LLMs as collaborators, not replacements

Let me be clear about what this tool does and doesn’t do.

It doesn’t replace the science communicator who knows that a particular project has a funny origin story, or that the lead developer gave a brilliant conference talk last month, or that “ digital bridges ” works better for this audience than “computational pipelines.” Those things require human judgment, institutional memory, and taste. No model provides that.

That’s the pattern we think works for LLMs in science more broadly. Not autonomous generation. Not replacing experts. Instead: structured tools that handle well-defined tasks with rich context and human oversight. Tools that make researchers more effective at the parts of their job that aren’t their core expertise but still matter, like communicating their work to the people who need to hear about it.

The conversation about LLMs in research is often framed as a threat. We think it’s more productive to ask: where can these tools genuinely help, and what does it take to use them responsibly? For me, scientific communication turned out to be a great answer.

A hundred projects, six audience types, one tool. The stories were always there. We just needed a better way to start telling them.

Under the bonnet tech stack (for the curious ones)

A few technical choices that I think are interesting, even if they’re invisible to most users:

No backend. The Gemini API gets called straight from the browser. No server, no proxy, no infrastructure to babysit. You deploy it as a static site and walk away. The tradeoff? Users bring their own API key, stored in localStorage, never sent anywhere except to Google. For a research-oriented audience that already juggles API keys for half a dozen services, this felt like a reasonable ask.

Svelte 5 runes everywhere. The whole app runs on Svelte 5’s new reactivity system: ` $state\`, \`$ derived`, `$effect`. When AI-generated text streams in, only the story display component re-renders. Not the 3D scene. Not the navigation. Not the settings panel. For something that’s running WebGL animations, parsing SSE streams, and managing modal state all at once, that granularity needs to be optional.

Scroll velocity drives the 3D. This is the detail I’m most proud of. The scroll store doesn’t just track how far down the page you are. It measures how fast you’re scrolling and in which direction. That velocity feeds into the Three.js scenes. Scroll fast and the camera pulls back, particles scatter wider, everything feels like it’s accelerating with you. Scroll slow and things settle into a gentle rotation. It’s the kind of thing you won’t notice unless you’re looking for it, which is the point. The best interactions are the ones you feel rather than see.

Prompt engineering treated as product design. I keep coming back to this because I think it’s the least obvious and most important decision we made. The prompt templates weren’t written by a developer in ten minutes. They were iterated on, tested against real projects, revised, tested again. The “Internal Review” template went through five or six drafts before it produced output a programme manager would find useful on a consistent basis. Treating prompts with the same rigour you’d give a product spec? That’s where the real value lives.

— -

The Data Storytelling application is open source at https://github.com/NLeSC/data-storytelling. Built at the Netherlands eScience Center, and the Google Gemini API. We’d love to hear how other research organisations approach this challenge. Contributions and conversations welcome.

Enabling best practices for sustainable software in the Natural & Engineering sciences (SS-NES)

eScience Editorial Team — Wed, 11 Feb 2026 00:00:00 GMT

Enabling best practices for sustainable software in the Natural & Engineering sciences (SS-NES)

Authors: Roel Janssen, Jason Maassen and Carlos Martinez-Ortiz

With the available constellation of data and code repositories, it can be hard to choose the best way of making your research code and data available. Here we present an automated way of synchronizing 4TU.ResearchData with the Research Software Directory.

4TU.ResearchData is a data and software repository for science, engineering and design domains, as well as support services and a vibrant community around them. The Research Software Directory is an open-source project initiated by the Netherlands eScience Center and jointly developed with Helmholtz.

As part of the project Enabling best practices for sustainable software in the Natural & Engineering sciences (****SS-NES) **the 4TU.ResearchData’s data repository and eScience Center’s Research Software Directory (RSD) collaborated to ensure software published in 4TU.ResearchData is also made available in the Research Software Directory.

To achieve this goal, 4TU.ResearchData adapted its metadata intake and implemented API endpoints following the CodeMeta standard. Additionally, 4TU.ResearchData implemented API endpoints to communicate statistics on Git repositories following existing Github and Gitlab practices.

The RSD implemented a harvester for 4TU.ResearchData which uses the CodeMeta metadata standard and a Communities feature to group software from a particular source.

Adding to the research software landscape

When creating software as part of your research activities, it is advisable to use a version control system like Git to manage the evolution of the software. At some point, the software needs to be shared with other researchers, either to reuse or extend it, or to validate the research results. Publishing the software provides the benefits of having a URL to cite as well as having a guarantee that the source code of the software is stored at a persistent location for the long term. These benefits are provided by 4TU.ResearchData.

To publish the software in 4TU.ResearchData from Gitlab, Github or Codeberg, one can use Git’s “remotes” concept to push a copy to 4TU.ResearchData while retaining the active development on the hub.

Fun fact**: Due to the collaboration between the eScienceCenter’s RSD and 4TU.ResearchData, the latter added the “Git repository name” and “Code hosting project URL” metadata fields so that they can be used and passed along to the RSD.

We chose this approach because it works regardless of whether the repository is public or private, and regardless of the implementation of the hosting platform. We support Github, Gitlab, Codeberg, Savannah, or any other Git-based platform. Additionally, when dealing with private Git repositories, no systematic integration or information exposure is needed.

Finding and showcasing software in the research software landscape

Once the software is published on 4TU.ResearchData, it is automatically picked up by the RSD and added to the 4TU Community collection. The RSD uses the information recorded in 4TU.ResearchData to generate a unique software page designed to highlight the impact of the software.

Next to showing basic static information about the software such as a short description, author list, keywords, and links to the source code, the RSD also enriches the software page with information harvested from other data sources in the open science and open-source ecosystem. Examples include information on software development activity, citation information for the different software releases, and a list of papers citing the software.

Concluding words

Publishing your software on a trusted repository is one of the good practices advocated by the SS-NES project and the eScience Center. With the integration between 4TU.ResearchData and the RSD, researchers automatically benefit from the findability and exposability of the RSD when they publish their software in 4TU.ResearchData. The RSD has become a more complete source of mapping the research software landscape by automatically harvesting 4TU.ResearchData’s software publications.

Use Julia to write code that runs on any GPU

Bart Schilperoort — Thu, 15 Jan 2026 00:00:00 GMT

Use Julia to write code that runs on any GPU

How to write Julia code than can run on any GPU, and why you would want to do that.

As the name implies, the main use of Graphics Processing Units is to process and render things to your screen, such as images, videos, or video games. Almost any device that has a display will have a GPU, although this can also come in the form of a chip integrated in the CPU instead of a separate graphics card. When using applications such as Google Maps, YouTube, or Netflix, the GPU renders the image/video to the screen more quickly and efficiently compared to the CPU. This can result in lower power consumption and a better user experience.

Photo by Dimitris Chapsoulas on Unsplash

To be able to render things to screen quickly, GPUs are able to do a lot of computations in parallel. Besides just graphics rendering, doing many computations in parallel can also come in use elsewhere, such as in (scientific) numerical models, and especially relevant recently, machine learning.

Before the release of Nvidia’s CUDA platform in 2007, people would use routines designed for graphics processing (like shaders), for non-graphics purposes such as numerical solvers for the Navier-Stokes equations. However, with CUDA, and soon after also OpenCL, it became more straightforward to write General Purpose GPU code.

When writing code for CUDA, you are locked into Nvidia designed GPUs, and the code cannot run elsewhere. With OpenCL, it was possible to write GPU code that can run on many platforms. While it can still work well on most hardware, it is seeing less and less support from Apple and Nvidia, who prefer to push their own proprietary platforms (Metal and CUDA).

Writing generic GPU code has a few benefits however, as you are not tied to a certain vendor, and there is a larger possible user base and thus more use cases. For example; accelerating a scientific model with GPU impacts both for laptop and high performance computing users.

To continue writing GPU code that can run on any hardware you can make use of Julia’s GPU ecosystem. With the KernelAbstractions.jl package you can write a kernel (a function that runs on a GPU and executes in parallel) that will work on any of the supported backends. Currently supported are Nvidia’s CUDA, AMD’s ROCm, Apple Metal, and Intel oneAPI. Which means that nearly all modern GPUs are supported, ranging from small laptops to supercomputers.

Julia example

To get started, after installing Julia, you can initialize arrays on the GPU with the appropriate backend package. As an example, I will use oneAPI, but the code will look the same for the other backends. The following line is the only one that’s machine dependent:

import oneAPI.oneArray as GPUArrayHaving imported this, we can define arrays on the GPU. In this case a 2D matrix containing single-precision floating point numbers:

A = GPUArray(ones(Float32, 1024, 1024))Now we can write a kernel. This example comes from the KernelAbstractions documentation, and will simply multiply every element of the matrix by 2:

using KernelAbstractions

@kernel function mul2_kernel(A) I = @index(Global) A[I] = 2 * A[I] endWe can apply the kernel to the matrix A :

backend = get_backend(A) mul2_kernel(backend, 64)(A, ndrange=size(A))And that’s it! — Note that 64 is the “workgroup size”, i.e., the number of the array elements assigned to one work group. Tuning this parameter can make the kernel run faster.

import AcceleratedKernels as AK

function cpu_copy!(dst, src) for i in eachindex(src) dst[i] = src[i] end end

function gpu_copy!(dst, src) AK.foreachindex(src) do i dst[i] = src[i] end endThe gpu_copy function will run on GPU if dst and src are GPU arrays. Otherwise the function will run on CPU.

Example packages

There are already some great packages that use KernelAbstractions to run on both CPU and any GPU. One of these is WaterLily.jl, a Computational Fluid Dynamics solver. Because it uses KernelAbstractions, they were able to run simulations not only on Nvidia GPUs, but also on AMD GPUs available on the LUMI supercomputer (one of the fastest in Europe!).

Simple 2D flow around the Julia logo, simulated using WaterLily.jl (source: WaterLily.jl)The animation above can be generated on a laptop using the CPU or integrated graphics, but can be easily adapted to a higher resolution or 3D simulation to be run on a supercomputer.

The Julia GPU showcase page has many more examples ranging from climate models to bioinformatics.

Conclusion

By using Julia’s generic GPU framework, you can:

run and debug code locally, on your laptop using your CPU or GPU
have a larger community of users who can run the code on their own devices
deploy the code on any supercomputer, e.g., both Snellius (Nvidia GPUs) and LUMI (AMD GPUs)

So next time you need code to be fast and portable, consider using Julia to write code that can run fast, anywhere.

Strengthening Research Software Management in Dutch UMCs: Highlights from the Second DReaMS Community Meetup

eScience Editorial Team — Tue, 09 Dec 2025 00:00:00 GMT

Strengthening Research Software Management in Dutch UMCs: Highlights from the Second DReaMS Community Meetup

Written by Anna Niehues, Özgün Balaban, Magnus Palmblad, Mariia Steeghs-Turchina, and the DReaMS Community

DReaMS is a collaborative community formed by Digital Competence Centers (DCCs) from Dutch University Medical Centers (UMCs), committed to advancing research software management (RSM). During the LUMC Research Data Day on 23 September 2025, network members came together in Leiden for the second community meetup. The event aimed to align practices and foster collaboration across institutions, building on the momentum of the first meetup in 2024 (Pronk 2024).

“Research Software includes source code files, algorithms, scripts, computational workflows and executables that were created during the research process or for a research purpose” (Gruenpeter et al. 2021).

For Dutch UMCs, effectively managing research software is essential to ensuring reproducibility, efficiency, and reuse, while supporting compliance with Open Science policies and regulatory requirements.

Opening Session

The opening session, co-chaired by Anna Niehues and Thomas Pronk, featured updates shared by participants, setting the stage for a day of engaging discussions and knowledge exchange. The meetup provided a valuable opportunity for the participating UMCs and national infrastructure partners — the Netherlands eScience Center and ELIXIR-NL — to connect, share updates, and explore key topics. The discussions focused on developing RSM guidelines, strategies for software sustainability, providing training, and identifying connection points to strengthen joint efforts.

Towards Research Software Management Guidelines for Dutch UMCs

During the session on Research Software Management Guidelines, co-chaired by Mariia Steeghs-Turchina and Carlos Martinez, participants identified common questions that researchers have, mapped available support, and identified gaps across Dutch UMCs. We explored what researchers need most, how institutions respond, and where key gaps remain.

Researchers regularly seek support on version control, Git or GitHub usage, and software maintenance. This indicates growing awareness but also uncertainty about best practices. In the absence of clear guidelines, RSM support staff often direct researchers to online tutorials, the Software Carpentry programme, the Research Software Quality Toolkit for Sciences (RSQkit) and institutional Git courses, as well as AI-powered coding assistants. These dispersed resources may be difficult to navigate and a shared resource hub connecting training and guiding materials could provide added value to researchers. Better recognition mechanisms, such as software citation, are also needed to incentivise the following best practices.

Research Software Sustainability: Long-term Maintenance and Reusability

During the breakout session on research software sustainability, which was co-chaired by Magnus Palmblad and Sietse Rispens, the challenges of maintaining software, especially when the original developer is no longer available, were discussed. Possible internal and external solutions were also explored.

A critical phase occurs between the realisation that a particular piece of software has wider applicability and the developer’s departure, often after completing their studies or postdoctoral research. During this period, it is important to plan for sustainability as early as possible, with a focus on documenting the code, creating software metadata and involving more people in the development process.

Good habits, such as writing code that is both readable and well-documented, encourages others to contribute to the codebase and help to sustain it. Students are often highly motivated to write software from scratch. While this is an important experience, students making significant contributions by adding novel functionality to existing projects should be rewarded equally. This requires a change in culture, away from “yet another tool to do X” to having more people contribute to community-supported open science projects. Training on how to use Git or how to contribute to community projects is essential. Journal editors should be encouraged to consider such contributions as potentially equally novel and useful as stand-alone software.

Research Software Training Through the Lens of FAIR and Open Science

While the first DReaMS meetup in 2024 identified a lack of intermediate-level training as a key challenge, this year’s breakout session on software training, co-chaired by Özgün Balaban and Bauke van der Velde, uncovered gaps in fundamental research software training.

It is often challenging to identify suitable communication channels through which to raise awareness among new PhD students and researchers of research software practices and available training and resources. While some topics, such as Git, which also have a high popularity, are covered repeatedly by training programmes, others are underrepresented. These include, e.g., conceptual-level training for beginners, good coding practices, software management, documentation, and AI-supported development. A lack of balanced training coverage also poses a risk to the sustainability of research software.

The activities identified as being effective in increasing awareness include Coding CAFEs (Moopen et al. 2023), mentorship programmes, peer learning, peer reviews of software documentation, and guest lectures. Such formats do not only build practical skills, but also strengthen communities.

Suggested collaboration opportunities include sharing course materials across institutions, setting up trainer communication channels, such as RST-NL, and offering joint workshops. The next steps will be discussed during upcoming online meetings of the DReaMS community.

A common challenge in the current training landscape and in the sustainability of research software is that initiatives are fragmented, resulting in uneven coverage across the software development lifecycle. A more coherent approach is required to address this, involving clear policies, long-term planning, secure funding and a sufficient workforce aligned with institutional project needs.

Get in Touch With the DReaMS Community

Take a look at our website and get in touch with us via the community members.

This blog is cross-posted here: https://dreams-community.github.io/blog/second-meetup/

A special gift (December 5th)

Peter Kalverla — Fri, 05 Dec 2025 00:00:00 GMT

Ode to the research software engineer

Sinterklaas is a Dutch tradition featuring Saint Nicholas, who is said to arrive from Spain with his helpers and travel across rooftops to deliver small gifts and poems. Today, the Sint is visiting the eScience Center.

As days are getting colder and the nights becoming long, we make our office cozy with hot chocolate and song. The fireplace is lit albeit on the conference screen; it is an office atmosphere unlike you’ve ever seen.

We sit together ‘round the fire and wait for what’s to come, when suddenly we hear a heavy beating, like a drum. We chase towards the office entrance seeing who is there, but nothing’s to be seen except a letter near the stairs.

The letter reads “My dearest friends and fellow RSEs, please go back to your conference room and make yourself at ease. You’ve made me feel so welcome and I brought you in return a bundle of delights for you to live and love and learn.”

Curious we sat once more around the makeshift fire, stunned to find some gifts that any scientist should desire. A bag full of best practices to make your software shine, sparked with digital literacy and beautiful design.

Chocolate letters are part of the traditionTo start it off we first unpacked a gift so smart and bright: a coffee mug for every soul who codes late in the night. On it was a text that read “your 9 to 5 is fine, but if you rather work at night, some caffeine’s divine”.

The next gift comes in handy for a project big or small a touch of git to help you keep your code under control. A branching scheme, a tidy diff, a history clear and clean A present many researchers admit they’ve never seen.

Then slips beneath the office door a test suite neatly wrapped, with unit tests and fixtures all meticulously mapped. “Oh generous Sint,” we all proclaim, “now bugs no more appear!” (Though secretly he knows they will, but less, and less severe.)

A handbook tied with twine he leaves, with pretty illustrations on licenses and versioning and good documentation. For knowledge lost is hard to find when months have come and gone, but with this helpful guide the happy coding can go on.

Sint conjures CI pipelines high above us in the cloud, where automated checks and builds assemble clean and proud A magical parade of green ticks marching in a row, the sweetest sight for anyone who’s watched code fail to grow.

But most of all, the Sint bestows a gift both rare and dear: the quiet craft, the steady hand, the engineer’s career. For research needs more than ideas just drifting in the air; it needs the ones who shape the tools that make discovery FAIR.

So here’s an ode to those who bridge two worlds with practiced grace: who think in algorithms yet they keep the human pace. Who build, maintain, refactor, teach, who safeguard truth with care, the research software engineers, the heroes unaware.

And as the Sint sails home again across December seas, he leaves one final message carried softly by the breeze: “Good science thrives when code is sound, transparent, shared, and clear, so honour those who make it so; support them through the year.”

This poem has been crafted in collaboration with AI-Piet*

The story map

Pablo Rodríguez-Sánchez — Wed, 26 Nov 2025 00:00:00 GMT

A framework for story-building

We are all natural-born storytellers. Or at the very least, storylisteners. The same drive lies behind a gathering around the fireplace, a night at the movies or listening to a podcast on your morning run: our brain’s hunger for narratives.

Our brains are literally wired for storytelling, and yet creating a story can be challenging.

This tutorial presents a strategy for simplifying the creative process of building a story. And as in any good story, we have to start from the beginning.

What is a story?

If you ever experienced the so-called “blank page syndrome”, chances are that it originated not from a lack of ideas to write about, but from an overwhelming, chaotic pile of them.

A story, any story, is linear. It doesn’t matter if it is a text, a speech, a movie or even a scientific publication: it will progress from a beginning to an end.

More specifically, any story will be consumed* on a linear way. Even a story which content is structured in a non-chronological order has a beginning and an end.*

And here comes the challenge: our thoughts are rarely that linear. Most people entertain multiple thoughts simultaneously, or quickly switch from one to the other. It takes a conscious effort to order them into a coherent narrative.

Embracing chaos

Instead of getting frustrated about the chaotic nature of our thoughts, let’s just embrace it. Our thoughts are fine just how they are.

Of course, it is not that practical to have our working memory juggling them all at once. So let’s free our memory and outsource them to… a piece of paper! Just write them down.

My advice is to do it like in the figure below, in small squares, with no pre-defined order. Just drop them around as if they were LEGO bricks.

Structuring your thoughts: the building blocks

The challenge is set! Our goal is to build something with these bricks. But first we have to see our bricks, identify them, make sure we’re not missing any. Almost feel them.

Take a moment to look at your building blocks. Very likely, you’ll notice some of them are related. This relationship can be logical, narrative, chronological, … Draw a line between them if that’s the case.

I suggest using solid lines for strong relatedness, and dashed lines for less clear relatedness. We are starting to have some structure! We are slowly creating order out of chaos!¹

At this stage, you might realize some building blocks are missing. When that’s the case, just add them. You might also notice some building blocks are redundant. Well, just merge those together. You might even judge some of the blocks to be unnecessary. Drop those without hesitation! Nothing creates more order out of chaos than reducing the amount of moving parts!

Navigating your thoughts: the story map

The ultimate goal would be to navigate your thoughts. And I don’t use the verb navigate lightly: you literally want to create a route from the beginning to the end. The diagram above will be very useful for achieving this.

To begin with, the diagram shows clearly what the core topics are. This allows for drawing a first, straightforward route that covers just the basics. If your audience is already knowledgeable about the topic and your time is limited, this route is an excellent choice:

For many people, especially in academia, skipping information feels wrong. In the verge of immoral.*

What we did above is tailoring your materials to our audience (sounds better than skipping, doesn’t it?). And yes, we did it by eliminating some of them. Actually, this is going to happen often, and it is very good news. Less complexity, less moving parts; less is more, remember?

Effective communication is not about turning everything that we write or say into a self-contained compendium, but about providing chunks that can be reasonably digested in whatever context they are delivered.

If your time is less limited, you can consider taking a detour and dropping an anecdote to make the experience more enjoyable:

Last but not least, if you have the possibility of expanding as much as you want, you might want to cover all the topics. In such a case, a reasonable route will look like:

In the figure below we summarize three possible routes we could envision for the same material:

Applications

I’ve used the described framework in many settings where storytelling played a role. Perhaps this sounds limited, but believe me: storytelling is everywhere. I used it for, among others:

Scientific presentations
Comedy monologues
Software tutorials
My own PhD dissertation
Blog posts
A strategic plan for a research institution
Publications in research journals (real-life example here)
A theater play
And also for this text you are reading now

I honestly encourage you to give it a try next time you face a blank page. It works wonders!

Acknowledgements

This piece was born in the context of the Data Storytelling project. Thanks to Jesse González and Kody Moodley for their feedback.

For those of you interested in knowledge management, chances are this method will remind you of the Zettelkasten note-taking method, a strong, even more structured form of the ideas presented here. Also, you’d be interested in knowing that some software applications, such as logseq or Obsidian, are designed with this model in mind.↩

From PhD Student to Auto-Tuning GPU Expert. Meet Floris-Jan.

Netherlands eScience Center — Thu, 06 Nov 2025 00:00:00 GMT

Floris-Jan Willemsen has been a PhD student at the Netherlands eScience Center since 2021, working on auto-tuning GPU software as part of the CORTEX project under the supervision of Rob van Nieuwpoort (Professor in Computer Science at Leiden University) and Ben van Werkhoven (Assistant Professor at Leiden University and former Research Software Engineer at the eScience Center).

Why auto-tune GPUs?**

“My research focuses on auto-tuning: creating software and algorithms to optimize performance on hardware. Smarter and faster auto-tuning not only leads to quicker results but also consumes less energy, making it a more sustainable option. We did this within the context of a project where we, together with other institutes like ASTRON, want to investigate the ‘explosive universe’. To observe the universe at that scale means a lot of data, which requires big computers, but even that is not enough — we need to be smart about what data we focus on and how we process it. That requires auto-tuning, which essentially is like a panel of many different knobs and sliders, with only one combination of millions leading to optimal performance. Therefore, we first focused on creating an algorithm that was able to find those combinations in very few attempts.”

Van Leeuwen/ASTRON (bron: NOS Nieuws)What were some challenges you experienced?

“This part of the project was itself a challenge, but the difficulty didn’t end when it was finished. The next step was to develop a uniform methodology. Each research group working with auto-tuning has a different way of visualizing and comparing data, as well as of reporting results. In 2022, we organized a Lorentz workshop to compare all these different methodologies and to design a new, uniform one. The major win was receiving the community’s mandate to spearhead this new methodology, which allows everyone to fairly compare auto-tuning performance — a significant breakthrough.

After this, I removed a major bottleneck in the auto-tuning process. This, together with other important advances, ultimately allowed me to optimize the optimization algorithms themselves — ‘tuning the tuner’ if you will.”

Who can use this tool?

ESiWACE3What was special about doing your PhD at the Netherlands eScience Center?**

“It allowed us to design tools that simply didn’t exist before. Building on the expertise of the Research Software Engineers at the Netherlands eScience Center, we were able to build high-quality software that is both at the cutting edge of research as well as user-friendly and stable. This is a combination that is rarely seen and allows us to put what we research today in the hands of scientists and developers tomorrow. Another perk was the focus on meeting with user bases through tutorials at a variety of international conferences like SuperComputing. At such conferences, we taught them new ways to get the most out of their hardware from us and we learned how users interacted with our software and how we could improve it further.”

What are your plans for the near future?

“I’m starting a new position as a Postdoctoral researcher at the Leiden Institute of Advanced Computer Science (LIACS) at Leiden University, where I’ll continue to build on the research I did during my Ph.D. and explore exciting new directions for auto-tuning. I’m looking forward to leveraging the experience I gained in building high-quality software in all future projects!”

What made working at the eScience Center a unique experience?

“The priority at universities is to always get the research done first, which tends to lead to software that gets the job done but is not accessible, maintainable, or usable outside of the exact scope of the research. The eScience Center has the combination of people experienced in both the academic and the software side of things, which puts it at a unique intersection to create sustainable scientific software. What I created during my time at the eScience Center is not limited to one specific purpose and can immediately be deployed in other people’s research. I’m proud to have been able to make an impact in that way.”

Read the article the NOS wrote about the CORTEX Project (link in Dutch) in January 2025.*

Who are we?

The Netherlands eScience Center is a research organization dedicated to applying research software to answer research questions in any scientific domain through project collaborations. It has the largest concentration of dedicated, high-level research software expertise in the Netherlands. The eScience Center also has a fellowship programme and makes all of its software and training materials openly available online. For more information about what we offer, visit esciencecenter.nl.

Why companies win by investing in Open-Source Software

eScience Editorial Team — Thu, 23 Oct 2025 00:00:00 GMT

Why would a for-profit company want to invest in Open-Source Software? Our collaborator, prof. Eric-Jan Wagenmakers will tell you why.

CC-BY license from: Kojima T, Oishi K, Matsubara Y, Uchiyama Y, Fukushima Y, Aoki N, et al. (2019) Cows painted with zebra-like striping can avoid biting fly attack. PLoS ONE 14(10): e0223447. https://doi.org/10.1371/journal.pone.0223447In 2025, the IgNobel prize in Biology went to research on “whether cows painted with zebra-like striping can avoid being bitten by flies“. Suppose a fictitious company, ZebraCows, wishes to exploit the insights from this groundbreaking work, and adopts JASP for their quality control analyses.

The head of the data-analytics department at ZebraCows is unfamiliar with open-source software, and two surprises await them:

Their first surprise is that JASP can be freely installed on all company computers, no strings attached. ZebraCows considers this a big win. Open-source is awesome, they think.
To receive support and connect JASP even more closely to their needs, ZebraCows starts working together with the JASP Services company through one of their service plans. However, when they read the contracts they are in for a second surprise: all new features that a company requests are immediately added to the JASP open-source code base; this means that everybody in the whole wide world –students, researchers, unrelated and even rival companies– gets to profit from improvements that were requested and paid for by ZebraCows. That seems unfair to ZebraCows, who would prefer the new features to be exclusively available to employees from ZebraCows.

Below, I briefly outline why ZebraCows should in fact be thrilled that the code they paid for is available to the whole world instead of only to them. Essentially, ZebraCows stands to profit from JASP being developed, extended, and improved by others –for free!–, while ZebraCows retains maximum digital autonomy**.

Below is a non-exhaustive list of arguments in favor of open-source software (specifically, JASP) for companies:

1. Unlimited, Free Installation

Open-source software such as JASP can be installed on every computer in ZebraCows business network at zero cost. There are no license fees. There is no per-user pricing, and scalability is complete.

2. Continuous Improvement at No Extra Cost

Open-source features are visible to the global JASP community. External contributors (researchers, developers, students) can refine, extend, and maintain the features that ZebraCows cares about, for decades to come — and all for free.

3. Shared Development Pool

Other companies who also want additional features will invest in JASP too. ZebraCows automatically gains access to their improvements, without paying extra.

4. No Vendor Lock-In

With closed-source software, ZebraCows may need to keep paying for updates just to keep their features working. With open-source, the code is available to ZebraCows forever. Nobody can take it away, restrict access, or increase license fees.

5. Transparency

Adoption of JASP means that ZebraCows attains complete transparency on the process by which JASP produces its results. In contrast to closed-source software, all the code that JASP uses is available for external inspection and public scrutiny. ZebraCows is not forced to rely exclusively on the documentation or the helpdesk and “take their word for it”.

6. Limited Costs for Superior Service

Most open-source service contracts –and certainly those of JASP– are simply a great deal financially. A large commercial software product does not adjust or improve its code to accommodate the wishes of a medium-sized company such as ZebraCows. For only a modest investment, ZebraCows can interact with the JASP Services team and initiate improvements that benefit ZebraCows specifically. Try doing this with <insert the name of any large commercial program here*>!

Conclusion

By investing in the development of open-source software such as JASP, companies gain much more than just the features they pay to see implemented. Companies also gain access to a continual, indefinite stream of free improvements, while being protected from lock-in or forced upgrade costs. This way companies profit from unlimited, future-proof access to a program that is designed to meet their needs.

This piece is a repost (with minor adaptations) from the JASP blog, reproduced here with permission (and encouragement) from the author, Eric-Jan Wagenmakers.

Professor Wagenmakers is a mathematical psychologist at the University of Amsterdam, the CEO of JASP (a popular open-source statistical software), and the proud owner of an IgNobel Prize (2024). Currently he is collaborating with the eScience Center in improving the user experience of the JASP module-making community.

Why teach calculus in the age of AI

Pablo Rodríguez-Sánchez — Thu, 09 Oct 2025 00:00:00 GMT

Why would I learn to compute a derivative by hand when Artificial Intelligence can do it for me?

Photo by Shubham Sharan on Unsplash

If you’re reading this, you’ve probably taken a calculus course before. How was it? Chances are you found it frustrating or, at least, complicated.

You may remember one of the most feared parts of calculus courses: the computation of derivatives and integrals. For most students, this is the first time they face a mathematical problem whose solution is not a number, but a much more complicated object: a function. Computing derivatives and integrals involves memorizing tables, applying complicated composition rules and avoiding sneaky arithmetic errors. In short, it is hard.

Enters technology

For several decades now, it is possible to solve these problems using symbolic computation software packages, available for computers and even for some pocket calculators. Since the explosion of Artificial Intelligence (AI) engines, access to such tools is easier than ever. So, if you want to know the derivative of some function, you can just ask your favourite AI engine via your smartphone, and you’ll (most likely) get the correct answer.

All of this raises a valid question: why bother learning all those tedious methods?

Calculus is not only about calculations

One of the reasons is that differentiation and integration involve much more than the algorithms. Like all mathematical operations, they have a meaning and applications. As a matter of fact, a huge number of applications: basically all physics and engineering relies on them.

While it’s true that one doesn’t need to know how to compute derivatives or integrals in order to understand their meaning, it is quite hard to do it without wrestling with some exercises. They are a great way to internalize and create a solid understanding.

For instance, by learning some of the algorithmic rules, we’ll also learn that:

Integration by substitution is just the chain rule for derivatives, read backwards.
Integration by parts is the product rule for derivatives, just written differently.

Even the quintessential tedious task, that of reading (and eventually) memorizing the tables, can teach valuable lessons, such as:

A table of integrals is basically a table of derivatives read backwards.
Derivatives destroy information, hence the arbitrary constant in the table of integrals.

Other concepts that are difficult to grasp without “getting your hands dirty” are:

Integrals and derivatives can be estimated.
Mathematics can manipulate objects more complicated than numbers.

Perhaps more importantly, if you look deeper, you’ll see that studying calculus teaches you more than just mathematics. For instance, solving calculus problems shows you that:

Some questions have a clear, uncontroversial and verifiable answer.
You can find out the solution to complicated-looking problems by yourself, using nothing more than pen and paper.

Conclusion

AI is here to stay, and it is a wonderful tool for many applications. But it comes with some dangers too. One of the main criticisms that AI engines receive from the scientific and teaching communities is that they work as black boxes: they spit out results and encourage users to accept them without further scrutiny.

Being able to reach conclusions with pen and paper (quite the opposite of a black box) is certainly not less important now than before. It is a crucial skill for responsibly using AI as a learning companion.

Education is not only about getting the right answers. Education is also about learning how to ask the right questions, to reason clearly and to trust your own judgment.

Technology can solve problems for you. Calculus teaches that, at least some of them, you can solve yourself. That kind of confidence, curiosity, and mental discipline is something no black box can give you.

You earn it.

This piece appeared first in mappingignorance.org.*

Be like Jurriaan

Patrick Bos — Thu, 25 Sep 2025 00:00:00 GMT

Write copy-pasteable READMEs for developers

The main message of this post is short:

Thank you,

Jurriaan H. Spaaks

, for including developer’s READMEs in the zenodraft and the zenodraft GitHub action repos!

(Zenodraft automates uploading releases of your repository to Zenodo. This story applies to any similar tool.)

In particular, having copy-pasteable commands ready to get going saved my week. Without them, I would have dropped any hope of fixing the issues I had running a CI workflow … or worse, I would have had to rebuild something like zenodraft from scratch!

It doesn’t have to be long and complicated. Just having those commands ready to copy is all I need.

Zenodraft is written in TypeScript, which is a foreign language to me. I’m unfamiliar with the tools necessary to make it work. The developer’s README file exactly fixed that issue, allowing me, a TypeScript-noob (but otherwise experienced programmer), to dive in and fix my problems within a few hours of building up the courage to even try. I was even able to contribute a PR, my first one in a TypeScript project!

In the world of open source (research) software, where ambitions are high and budgets are low, it turns out that such documentation can mean the difference between life and death for the software.

If you are the owner of a package and expect to be unavailable to maintain your code, the best thing you can do is to enable the random passer-by to independently fix their own problems.

Be like Jurriaan. Write developer’s READMEs!

If you had asked me a week ago whether such documentation is necessary I would have shrugged. This experience made me change my mind. A lazy, clutter-averse and/or “minimalist” person could argue that most development ecosystem tools are well documented on their own, and technically they might even be right (although rants on how soul-crushingly horrible Python’s or Javascript’s developers’ ecosystems are aren’t hard to find). Why duplicate efforts?

Just ask yourself: would you take the effort of reading up on the current state of such an ecosystem just to make random-tool-on-GitHub do what you want? Even if it’s just a minor tweak in the code, the effort necessary to get that tweak into production can easily put developers off.

So, just copy-paste the minimal commands you need to get a working development environment set up into a README file. You probably have these commands set up in your CI build job already, so it’s really not that much effort. And you might gain a new contributor…

5 tips for writing a successful eScience Center proposal

Peter Kalverla — Thu, 28 Aug 2025 00:00:00 GMT

Over the years, we’ve worked on a wide range of collaborative projects through our open calls. The most successful ones? They tend to have a few things in common right from the proposal stage. Here are five tips to help your proposal lay the foundation for a smooth, impactful, and genuinely rewarding collaboration.

Image adapted from Unsplash

1. Describe the challenge, not the solution

We love a good challenge. If the technical solution is already fully defined, there’s less room for us to contribute meaningfully. For example, instead of saying “we need a dashboard that does X”, describe the challenge: “we struggle to analyze Y because current tools can’t handle Z.” That kind of framing sparks our curiosity and drives the creative collaboration we love.

2. Dedicate real time to the project

We’re looking for genuine teamwork — whether that’s you, your postdocs, or PhD students. The best projects have collaborators who test the software as it is built. For example: “we tried the new analysis module this week, here’s where it’s slow, and here’s a feature we hadn’t thought of before.” Aligning your proposal with existing commitments makes this kind of feedback loop much easier.

3. Be realistic

Ambition gets projects funded. And we love bold ideas — but bold doesn’t mean bloated. Make sure your proposal has a solid core to build on, and leave room for exploration, iteration — even failure. A focused project with space to explore usually delivers more than an overpacked one that runs out of steam.

4. Build on what you have

We love improving workflows and building better tools — but starting from scratch rarely works in one project. If the data is missing, the workflow doesn’t exist, or the whole thing is still hypothetical, that’s a red flag. Don’t think what might be useful for someone, someday. Instead, think what would help *you, today, *and make your code and data available before submitting the proposal. Even a messy dataset or clunky script gives us a real starting point, so we can hit the ground running.

5. Think beyond the project

A successful project doesn’t stop when the funding ends. Our calls require a sustainability plan — and for good reason. The tools that have the biggest impact are those that become part of daily workflows and get shared, adapted, and improved over time by a community of users. So think about how your software fits into your everyday research, and who else might use or build on it after the project wraps up. Check out the SMP tool to get a jump start.

Ready to collaborate? Check our website for an up-to-date overview of past and open calls, or simply get in touch.*

This blog post represents original content created with AI assistance.

Could AI be (in) FAIR?

Carsten Schnober — Thu, 10 Jul 2025 00:00:00 GMT

Open Science is driven by FAIR principles. Made for data, adapted for software, how can they survive the current wave of “Artificial Intelligence”? Resistance from those selling it as a solution for all problems is to be expected.

Scientific progress is actively hindered if research data is stored on inaccessible servers or private hard disks, in proprietary formats, or preprocessed in unknown ways. The authors of the original FAIR paper have therefore demanded to leave non-collaborative practices behind: instead, make data F**indable, Accessible, Interoperable and Reusable (FAIR)!

Photo by Shantanu Kumar on UnsplashThe underlying ideas have been pillars of scientific methodology for thousands of years, but the rise of computerized methods in practically all research fields has created the need to make the FAIR principles explicit. The increase of computational processing power and algorithms to make efficient use of it have interwoven disciplines including Natural Language Processing, Information Retrieval, Information Science, and Computer Science. Artificial Intelligence is one of the products and has now impacted not only academic research, but also medicine, [education](https://doi.org/10.48550/ar

Xiv.2412.06651), libraries and archives, and even spirituality.

Fair vs. AI

Not reinventing the wheel for each study is the essence of scientific progress; hence, reusability is the primary goal of the FAIR principles. Promises about advances to be expected by current AI generations are bold. Supposedly, they achieve “PhD-level intelligence” already, even though a closer look reveals that this merely refers to the ability to fill in exam questions accurately. Anyway, the FAIR principles seem to be falling off the wagon, as ChatGPT and its colleagues are happy to confirm (see Figures 1–3).

Figure 1: According to ChatGPT, the letters “A” and “I” do not appear in FAIR.The paper “FAIR for AI: An interdisciplinary and international community building perspective” proposed the following definitions for FAIR AI models:

An AI model is Findable when a digital object identifier (DOI) can direct a human or machine to a digital resource […];
it is Accessible when it and its metadata may be readily downloaded […];
it is Interoperable when it can seamlessly interact with other models, data, software, and hardware architectures;
and it is Reusable when it can be used by humans, machines and other models to reproduce its expected inference capabilities […].

These definitions are a promising step forward, but apply to models as static artifacts. However, they do not address the methodology that uses AI.

Figure 2: Gemini cannot see the “AI” in “FAIR”, either.

FAIR Software

Software did not fit into the original FAIR principles that assumed static data sets, hence specific FAIR principles for research software were introduced, along with a checklist:

Findable: the software should have a globally unique identifier and be registered in e.g. the Research Software Directory, […].
Accessible: users can access the software, including its source code […].
Interoperable: the software should read and write open data formats, […].
Reusable: all information to compile the software is provided, […].

Together with DANS, we have also provided recommendations on how to practically implement these criteria using available services.

AI in Science

AI impacts scientific methods in different ways than software did, though. A report by the Royal Society lists six use cases for AI in research:

Extract and learn features from raw data and extensive datasets
Gain insights from unstructured, multi-modal and fragmented data
Generate predictions and synthetic data from existing content
Information synthesis in academic writing, literature reviews and summarization
Generate software code
Automate labour-intensive tasks to test diverse hypotheses

In these examples, machine learning and language models are not the subject of research but instead serve as means to various ends.

Figure 3: the Llama model is certain that there is no AI in FAIR.Computational linguists have shown that (Large) Language Models serve as parrots that remix the data they have been trained on in unpredictable ways, which entails risks including ecological harm, economic monopolization and encoded biases. When putting those aside for a moment, ‘parroting’ might not sound too bad from a reproducibility point of view. However, nobody knows what a generative language model is parroting exactly. Not even the developers, let alone the users, can trace back the impact of specific data samples on the weights of a model.

The Royal Society report lists additional concerns, for instance, researchers with misconceptions about statistics, overreliance on machine learning outputs, and a lack of skills and incentives for academic researchers. Lack of transparency is often built into proprietary products, but LLMs are a particularly hard-to-open black box, as they involve complex stochastic processes. Misleading marketing claims and bold promises from the companies behind the chatbots increase transparency even more, rather than mitigating them.

New FAIR Principles

FAIR principles for AI must make sure that all information about AI-based research is provided so that other researchers can understand, verify and reproduce the results of a colleague. The enormous resources required to develop a new language model make reusability particularly relevant. Reinventing a wheel of that size again and again is too costly for any society.

The fuzziness of the decades-old term Artificial Intelligence, as well as its misuse for utopian and dystopian myths and cons, has created a semantic fog around AI that is counterproductive for spreading technical understanding, sometimes causing catastrophic consequences for the users. Recent studies have shown how the most recent trend within the AI community, “reasoning models”, creates an illusion of thinking. At the same time, the go-to methods for evaluating Chatbots have turned out to be misleading, as shown in the Leaderboard Illusion.

Figure 4: LLMs tend to fail on reasoning tasks if they do not exactly match the training data.Those who sell the promise of redemption through AGI (Artificial General Intelligence*) are not interested in transparency or Open Science. Bringing FAIRness into the field, on the other hand, facilitates scientific progress. FAIR principles for AI must allow researchers to understand AI and enable them to make informed decisions about how to incorporate it into their research.

The first step is to move away from referring to LLM-based, generative AI in a mystifying manner. Instead, researchers must discuss tangible tasks and techniques that can be evaluated objectively. Only this will allow us to define FAIR principles for AI: how can we make sure, LLM outputs can be f**ound by anyone, are accessible through open protocols, interoperable with other research data, and — perhaps most importantly — become reproducible?

Empirically evaluating privacy in machine learning I: Introduction

Flavio Hafner — Fri, 27 Jun 2025 00:00:00 GMT

Research in the social and health sciences often relies on accessing data sourced from statistical agencies or healthcare organizations. Such data sets contain sensitive information, which requires restrictions on who can access the data. This can slow down not only scientific progress — if data were completely open, more researchers could investigate them — , but also the translation of progress to applications — for instance, machine learning models trained on medical data cannot easily be shared and re-used for personalized treatments.

Photo by National Cancer Institute on Unsplash

Differential privacy is one technology that aims to solve this problem. Intuitively, it is a method that adds calibrated noise to the data, and so prevents a malevolent actor from learning sensitive attributes about any person in the underlying data. But because statistical noise makes statistical inference harder, there is an inherent trade-off between statistical utility and privacy — making differential privacy challenging to deploy in practice (Cummings & Sarathy, 2023).

In addition, using differential privacy in machine learning has its own challenges. First, often it’s impossible to have reasonable privacy and utility at the same time. Second, differential privacy for machine learning makes strong assumptions about the capabilities of the malevolent actor. Third, implementing it in software is not straightforward, and there is evidence of bugs (Cummings et al, 2023; Nasr et al, 2023; Ponomareva et al, 2023).

For this reason, empirically evaluating the privacy of machine learning models has become an important field of research. Privacy testing probes whether a trained machine learning model leaks sensitive information. It can be an important indicator for model developers and owners of sensitive data whether the privacy-preserving training mechanism can be trusted.

The plan for this series of blog posts

Despite this, most resources in this topic are in research articles. The goal of this series of blog posts is to make the topic more accessible. We will build knowledge from the ground up:

The first blog posts introduces differential privacy, and gives some economic intuition. It is aimed at readers unfamiliar with differential privacy.
The second post explains how differential privacy can be interpreted from a perspective of statistical hypothesis testing. It is aimed at readers familiar with differential privacy and some concepts from statistics and machine learning.
The third post discusses how the hypothesis testing perspective has been used to reformulate differential privacy with functional differential privacy and Gaussian differential privacy. It is aimed at readers interested in understanding privacy testing of machine learning.
The fourth blog post discusses some recent papers that audit machine learning models and make use of Gaussian differential privacy. It is aimed at an audience similar to the third post.

Photo by Tobias Tullius on Unsplash

A short introduction with an example

This section builds on Dwork & Roth (2013).

Randomized response

Imagine a researcher wants to know the fraction of people that engage in a behavior that is illegal or socially not accepted — for instance, if they have income that they do not declare to the tax agency. The researcher runs a survey and asks people whether they have such income. Denote this behavior by the variable X, and let’s say it can take values 0 or 1, where 1 indicates the person earns undeclared income, and 0 indicates the person does not.

How can the researcher ensure that people are willing to share this sensitive information with them? To solve this problem, social scientists have developed a method called randomized response, which gives survey participants the following instructions:

Flip a coin privately
If it comes up heads, answer truthfully
If it comes up tails, flip again. If the second flip is heads, answer with “1”, if the second flip is tails, answer with “0”.

This approach strikes a balance between privacy and statistical utility. First, privacy comes from plausible deniability — anyone can always plausibly claim not to engage in behavior *X, *independently of the answer they have given. Second, statistical accuracy comes from understanding how the noise was generated: because we know the response mechanism, we can infer the mean of X in the population.

In addition, in a more general setting, one could replace the first coin flip with a draw from a continuous random variable that takes values between 0 and 1, and instruct people to respond truthfully if the random variable is above some threshold p. Varying p amounts to varying the amount of noise we introduce to the response. On one hand, if p=0 , we have perfect accuracy but no privacy. If p=1 , we have perfect privacy but no accuracy.

Formal definition of differential privacy

Define db as a tabular database where each row corresponds to the data of one person. We call db0 and db1 neighboring databases when they differ by one record: they hold the exact same information, except for one person called Anne. For instance, it can be that Anne is not present in database db0 but in db1, or that some column in Anne’s row is altered in db1 but not db0 .

Now, consider an algorithm M that takes db as input and creates some output. The algorithm is randomized because its outcome is not deterministic, but drawn from some distribution: The distribution of outcomes of the algorithm when fed with db is denoted M(db).

Such an algorithm is called (epsilon,0) -differentially private** if the distribution of M(db0) and the distribution M(db1) are not too different from each other — formally, if the log odds of observing any outcome from M(db0) vs M(db1) are not larger than epsilon. The 0 in the (epsilon,0) expression is a second parameter called delta, which denotes the probability that this guarantee fails. It is typically a very small number that decreases in the size of the database db.

Differential Privacy of randomized response

This is an abstract definition, but we now show that randomized response satisfies differential privacy. To do so, we assume that the database db only consists of one person — one row — , and one variable — the answer to the survey. We will relax this assumption later.

Let y_i = M(x_i)=1 denoting that person i with attribute x_i answers 1 to the survey. Note that we only observe y_i and x_i is kept private. The randomized mechanism M is the protocol that instructs people to flip coins and respond accordingly; M generates a Bernoulli distribution of outcomes over neighboring databases with the following probabilities.

We can construct neighboring databases by varying person i‘s sensitive attribute x_i: One input database has x_i=1, and the other has x_i=0 . Running randomized response, what are the probabilities of y_i=1 given values for x ? If x_i=1, the person answers “1” with probability 0.75; if x_i=0 , the person answers “1” with probability 0.25. In other words, for personi , the odds ratio for randomized response returning y_i=1 when having x_i=1 as opposed to having x_i=0 is P[y_i=1|x_i=1]/P[y_i=1|x_i=0]=(3/4)/(1/4)=3 . We can do a similar analysis for the outcome y_i=0 . Therefore, randomized response satisfies (ln(3),0) differential privacy.

Economic intuition

What does differential privacy promise? Let’s continue with the example of undeclared income. Suppose the researcher publishes the finding that 50% of people have some undeclared income. In response, the tax agency improves their enforcement, finds the people that do not declare income, and fines them.

Photo by The New York Public Library on Unsplash

Pick one person, Bob, who is a tax dodger and who participated in the original survey of the researcher. Because of better enforcement, he gets caught by the tax office and needs to pay a fine. Does differential privacy guarantee that Bob is not being penalized for his behavior?

No. Recall that differential privacy guarantees a similar distribution of outcomes under two neighboring databases. Releasing the survey result does impact Bob’s utility, but it does so independently of whether Bob is in the data or not: Even if Bob did not participate in the original survey, the tax office would have learned that many people do not declare their income, and started stronger enforcement, possibly leading to Bob being caught and fined.

In other words, Bob’s expected utility is similar in the two scenarios considered, and no differential harm* is caused to Bob from releasing the survey results. This highlights a crucial aspect of differential privacy: what matters is the difference in outcomes from being in the database and from not being in the database. Differential privacy makes no guarantees about the level of utility of records in the database. And in consequence, “Statistical Inference is Not a Privacy Violation” (Bun et. al., 2021).

The limits of this example

Above, we made the simplifying assumption that a database consists only of one row and one column. In real life, databases have multiple rows and columns. That’s why it is common to define neighboring databases by taking two databases that vary exactly by one row, but are otherwise identical.

But working with this definition, applying randomized response to the sensitive attribute x_i is not enough anymore to guarantee the privacy of each person in the survey. For instance, if for each person, the researcher published the randomized response y_i along with some of their true identifying information — for instance their address and age — , it would be possible for the tax office to prosecute people with y_i=1, since they are more likely to have x_i=1.

Instead, the researcher needs to collect the real answers of all participants, decide which statistic to use for publishing the results, and apply a respective algorithm that satisfies differential privacy. But this then also requires that the survey participants trust the researcher that they will not use the sensitive data against them.

Differential privacy’s model of computation

These questions bring us to two important assumptions behind the theory of differential privacy. The first assumption is that there is a trusted curator of the data — in our case the researcher that holds the individual survey responses are trusted to not leak the individual responses to the tax office.

The second assumption is that only results to queries are released — in our case, a relevant statistic of the survey such as “the fraction of people that have undeclared income”.

Another important aspect of differential privacy is that multiple queries of the same data weaken privacy, even if each query itself satisfies differential privacy. For instance, suppose the researcher allowed the tax office to get multiple differentially private queries of the original data. Then, the tax office could gradually refine their queries and potentially identify a certain person that does not pay their taxes.

This problem is known as composition, and differential privacy provides mathematical tools to study how exactly privacy degrades with composition. We will touch on it again in the context of differentially private machine learning.

Conclusion

We used an example from social science research to illustrate how differential privacy can strike a trade-off between utility and privacy. The example helped illustrate the assumptions that are important for privacy guarantees. In the next blog post, we will look at the hypothesis testing interpretation of differential privacy.

Thanks to Kody Moodley, Erik Tjong Kim Sang and Lourens Veen for commenting on and discussing an earlier draft.

Empirically evaluating privacy in machine learning II: Hypothesis testing

Flavio Hafner — Tue, 17 Jun 2025 00:00:00 GMT

Using statistics to think like a hacker, so we can avoid them!

Image from page 182 of “Elements of statistics” (1954). Licensed under creative commons.This post is the second in a series on evaluating privacy in machine learning models (see first episode here). It explains the hypothesis testing interpretation of differential privacy by showing how differential privacy constrains the success of any attacker in finding out about the underlying data.

The post assumes readers are familiar with differential privacy and some concepts in statistics and machine learning. In addition, it keeps concepts such as an “input database” and a “data release” abstract. If you are unfamiliar with either of these terms, you can have a look at the previous post. It introduces differential privacy and has a specific example for an input database and a data release.

Differential privacy through the lens of hypothesis testing

The first blog post of this series showed that differential privacy has adversarial thinking built-in: Leaking sensitive information may allow a malevolent actor to use the information to the disadvantage of a person in the database. In a differentially private mechanism, the more noise is added, the less any adversary can learn about any particular record in the database.

Adversarial inference as a hypothesis test

This intuition was formalized by Wasserman & Zou (2009). Imagine an adversary has access to a differentially private data release Y from a tabular database db. And suppose the adversary wants to find out if a particular person i was part of the database db. Define the neighboring databases db0 and db1, where i is only in db1 but not in db0 .

Then, the adversary faces the following hypothesis testing problem:

H0: Y was drawn from db0, so that i was not in the input data for Y.
H1: Y was drawn from db1, so that i was in the input data for Y.

When seeing the data, the adversary needs to make an inference about the true state of db , having seen only Y.

There are four possible scenarios.

The adversary correctly predicts that Y was based on db0. This is a true negative.
The adversary correctly predicts that Y was based on db1. This is a true positive.
The adversary predicts that Y was based on db0, but in truth it was based on db1. This is a false negative.
The adversary predicts that Y was based on db1, but in truth it was based on db0. This is a false positive.

Deriving the ROC curve

How does the adversary make their inference? They need a decision rule that takes the observed data Y and returns an output that is db0 or db1.

In a future blog post, we will be more specific on what such a rule can look like and how one can derive it. For now, we assume the adversary has the following decision rule: First, they have a trained logistic regression model that takes Y as input and returns a score. Second, the adversary decides on a threshold: if the score is above the threshold, the decision rule returns db1, and db0 otherwise.

In other words, the adversary rejects H0 in favor of H1 if the score is above the threshold. Then, lowering the threshold means rejecting H0 more often, and so:

The adversary is more likely to detect cases when Y is based on db1. This means a higher true positive rate, which is good for the adversary.
But the adversary is also more likely to predict that Y is based on db1 when in truth it is based on db0. This means a higher false positive rate, which is bad for the adversary.

We can visualize this trade-off with the Receiver Operating Characteristic curve, or ROC curve:

Figure 1: The blue line is an ROC curve. The distinguishing between H0 and H1 becomes easier as the True Positive Rate increases and the False Positive Rate decreases, thus moving North-West n the figure.In the figure above, the blue line defines the trade-off between the False Positive Rate FPR and the True Positive Rate TPR for a given prediction model. Lowering the FPR comes at the cost of a lower TPR, and vice-versa.

Moreover, the red arrow indicates the direction in which it becomes easier to distinguish between H0 and H1, meaning a lower FPR and a higher TPR: An ideal decision rule gives an FPR of 0 and a TPR of 1. On the other hand, the dotted line indicates the performance of a decision rule that randomly rejects or does not reject H0.

The ROC curve is a standard tool in machine learning for binary classification — and from here it is only a few steps to visualizing how differential privacy restricts the amount of information an adversary can learn from a data release.

Flipping the ROC curve

To start, let’s flip the y-axis in the above figure, and recognize that 1-TPR is the False Negative Rate FNR. Thus, we have a curve in the FPR-FNR space:

Figure 2: The blue line is an ROC’ curve, which is the ROC curve with a flipped y-axis. The hypothesis test gets easier as we move South-West.This figure shows the same information as the previous figure. Importantly, the hypothesis test is easier the closer the blue ROC curve follows the two axes— in other words, the more convex the curve is.

Differential privacy puts constraints on the best possible ROC curve**

Next, Kairouz et al (2015) show that if an algorithm satisfies (epsilon, delta) differential privacy, then the adversary’s success in the hypothesis testing problem is bounded by a function of epsilon and delta. Specifically, this bound can be drawn in the FPR-FNR space as follows:

Figure 3: The performance boundaries of a hypothesis test for two differentially private mechanisms: The blue is from a mechanism that is (epsilon1, delta) differentially private. The red is from a mechanism that is (epsilon2, delta) differentially private. The latter allows for a more convex ROC’ curve of the adversary, and therefore provides less privacy than the first mechanism.First, the blue line indicates the best possible performance any adversary can achieve in the hypothesis testing problem when the data satisfy (epsilon1, delta) differential privacy. This means the ROC' curve from the previous figure, cannot cross the blue line — for any* adversarial decision rule. As before, the dotted line shows the performance of a decision rule that is completely random.

Second, the figure illustrates that the parameter delta defines the intercepts of the line on both the FPR and FNR axes. Intuitively, delta defines an area where the adversary’s trade-off is not bounded — for instance, they can lower the FNR without increasing the FPR.

Lastly, the figure illustrates how epsilon defines the slope of the constraint. Consider two mechanisms with equal delta , but varying epsilonparameters — denoted epsilon1 and epsilon2 with epsilon1 < epsilon2 . Then, we see that the constraint for epsilon2 is closer to the origin of the diagram — showing that a higher epsilon means lower privacy and an easier business for an adversary to make inferences about the underlying data.

What do we learn?

Starting from the ROC curve in machine learning, we saw how differential privacy restricts the success of an adversary trying to infer whether a certain record was part of a differentially private data release.

In the next post, we will see how these insights lead to the notions of functional and Gaussian differential privacy, which are more handy to work with for certain analyses of privacy.

Thanks to Kody Moodley, Erik Tjong Kim Sang and Lourens Veen for commenting on and discussing an earlier draft.

Empirically evaluating privacy in machine learning III: functional & gaussian differential privacy

Flavio Hafner — Tue, 17 Jun 2025 00:00:00 GMT

Using statistics to think like a hacker, so we can avoid them!

The Greek letter lambda, symbol of functional programming.This post is the third in a series on differential privacy and machine learning (see here episodes I and II). It takes a big step towards understanding how to assess privacy properties of machine learning algorithms: It introduces functional and gaussian differential privacy as defined by Dong et al. (2022). This definition helps reasoning about differential privacy in some contexts, and in particular it has been useful for recent work in auditing machine learning algorithms with differential privacy.

When reading this post, the reader should be familiar with the previous post in this series. That post showed that differential privacy constrains the success of an attacker in inferring whether a particular record was included in a differentially private data release.

Functional Differential Privacy

Trade-off functions

So far, we have derived the boundary for the best possible performance of any adversary in the original case of (epsilon,delta) differential privacy. It turns out this function is an instance of what Dong et al. (2022) define as a trade-off function.

Remember that a differentially private mechanism M induces two different distributions of data under the two competing hypotheses H0 and H1. A trade-off function T(P,Q)(alpha) defines the difficulty in distinguishing the two distributions: it is defined over any two probability distributions P and Q, and tells us the lowest achievable type-II error rate beta (= False Negative Rate FNR) an adversary can achieve with a type-I error rate (= False Positive Rate FPR) of at most alpha. The higher this function, the higher the lowest achievable type-II error rate for any alpha, thus the harder it is to distinguish the two distributions P and Q.

As a concrete example, a trade-off function TP(P,Q)(0.05)=0.1 tells us that distinguishing P from Q with a type-I error of at most 0.05 cannot have a type-II error of less than 0.1.

A trade-off function has the following mathematical properties:

It takes inputs alpha that range from 0 to 1.
It has domain beta that range from 0 to 1.
It is convex, continuous, non-increasing.
For all alpha, beta lies on or below the line 1-alpha. The dotted line in figure 2 of the previous blog post represents the line 1-alpha , indicating that the distributions P and Q are indistinguishable.

Functional Differential Privacy

Instead with parameters (epsilon, delta), it is now possible to define differential privacy in terms of a function — and is thus called functional differential privacy.

According to the definition, a mechanism M is f-differentially private if it creates, for all neighboring databases db0 and db1, a trade-off function over M(db0) and M(db1) that is bounded by the trade-off function f.

The definition is best illustrated with this figure from Dong et al (2022):

Figure 1: Functional differential privacy. The solid line is a trade-off function for f differential privacy. Only the dashed line above the solid line satisfies f-differential privacy. Source: Dong et al (2022).The figure has the same x- and y-axes as the figures in the previous blog post: On the x-axis is the false positive rate, or type I error; on the y-axis is the false negative rate, or type II error.

The solid line is the trade-off function f. The other lines represent trade-off functions from differentially private mechanisms. Among them, only the dashed line that never crosses the solid line is f-DP; the other lines do not always lie above the line defined by f, and thus do not satisfy f-differential privacy. For reference, the kinked line could have come from an (epsilon, delta) -differentially private mechanism we discussed in the previous post.

Analyzing composition

A first advantage of functional differential privacy is that it makes it easier to analyze properties such as composition. Composition handles the question of how sequentially querying a dataset with a private mechanism impacts the overall privacy of the original dataset, after all queries are applied. As an example, take an analyst that sequentially queries a private dataset: they query the data with a private mechanism, explore the result, and query the data in a way that depends on the result of the first query. The question arises: how does this sequential and adaptive querying of the data degrade overall privacy? If the person is adversarial, they may try to refine the queries step by step until they found the answer to their question.

A key result in Dong et al (2022) is that the n-fold composition of mechanisms that are f_i differentially private is f* -differentially private. The precise level of privacy f* depends on the privacy of each individual mechanism f_i, and details can be found in section 3 of the paper. Importantly, this privacy bound is tight: f* is the most precise bound possible, informally meaning that we cannot move the function f* further away from the x-y origin without making the f* function crossing the trade-off function of the composed mechanism.

This contrasts with (epsilon,delta) differential privacy, where composition cannot be bounded as tightly.

Gaussian differential privacy

A special case of f-differential privacy is Gaussian differential privacy. In this definition, the trade-off function is defined over two unit-variance normal distributions — one with mean 0, the other with mean mu. In other words, under H0, the private mechanism generates a standard normal distribution, and under H1, it generates a normal distribution with mean mu and variance 1. This is shown in the following figure:

Figure 2: Two normal distributions. Under H0, X is generated with mean 0 and standard deviation 1. Under H1, X is generated with mean mu and standard deviation 1.While this definition makes assumptions about the functional form of the distribution, it is very useful:

It summarizes the adversary’s trade-off between false positives and false negatives with a single parameter, mu. This makes it easy to describe and interpret privacy guarantees.
It characterizes the Gaussian mechanism, one commonly used differentially private mechanism.
As shown by Dong et al (2022), Gaussian differential privacy is the asymptotic limit when f-differentially private mechanisms are applied repeatedly on the same dataset. This is useful for analyzing complex algorithms that consist of multiple, adaptive and individually private steps.

Relation to machine learning with differential privacy

It is this last result that has made Gaussian differential privacy an important tool for analyzing differentially private machine learning.

While we discuss it in detail in a future blog post, a commonly-used algorithm in deep learning is Differentially Private Stochastic Gradient Descent (DP-SGD). It is a method for sequentially updating parameters in a complex neural network: Instead of directly updating parameters with the gradients in each batch, noise is added and the contribution of individual samples in the batch is constrained. This avoids that any individual sample contributes too much to the parameter update, and can be shown to satisfy differential privacy.

Since each parameter update is differentially private, one can view the trained model that results after many iterations as an algorithm that composes multiple differentially-private queries on the training data. Therefore, one can use Gaussian differential privacy to derive tight privacy bounds of the DP-SGD algorithm, which was not possible before.

Trade-off function for Gaussian differential privacy

Building on the previous figure, the following figure illustrates the trade-off between false positives and false negatives under Gaussian differential privacy.

Figure 3: The likelihood-ratio test with normal distributions. If X is higher than t, H0 is rejected in favor of H1, and vice-versa. The blue and green areas show the false positive and false negative rates, respectively.*The red line indicates a decision threshold. In particular, when conducting a hypothesis test, the [Neyman-Pearson lemma](https://en.wikipedia.org/wiki/Neyman%E2%80%93

Pearson_lemma) says that the best test statistic is the likelihood ratio test: Given a sample X, the likelihood ratio test compares the likelihood that the sample was generated under H1 vs H0. From the figure above, we see that because the likelihood ratio of H1 vs H0 is increasing in X, the null hypothesis is rejected if X is above some threshold t, indicated in red.

Given the decision threshold, we can derive the False Positive and the False Negative rates:

The False Positive Rate is defined for cases when H0 is falsely rejected. This happens if X>t but X was generated under H0. This probability is shown by the blue area.
The False Negative Rate is defined for cases when H0 is falsely not rejected. This happens if X<t but X was generated under H1. This probability is shown by the green area.

Now that we have an analytical expression for both FPR and FNR, we can derive the trade-off function — the FNR as a function of the FPR and mu: We rearrange the equation for the FPR for t and then plug in t to the equation for FNR. This yields the following trade-off function:

Again, an illustration makes this easier to understand:

Figure 4: The trade-off function for Gaussian differential privacy. Phi is the standard normal distribution. alpha is the false positive rate. beta is the false negative rate. a lower mu means more privacy.First, for a given false positive rate alpha, the false negative rate is at least as large as the the green area. In line with the definition of a trade-off function, lowering the false positive rate makes the red area smaller, and pulls the boundary of the green area to the right, thus increasing the false negative rate.

Second, a lower mu is associated with more privacy: as mu decreases, it pulls the boundaries either the false positive or false negative rate (or both) closer to the center, thereby making it harder to distinguish between the distributions generated by H0 and H1.

Dong et al (2022) also illustrate the trade-off function for some values of mu:

Figure 5: Example trade-off functions for different values of mu. Source: Dong et al (2022).Here, for instance G_{0.5} is the trade-off function for Gaussian differential privacy with mu=0.5 . Reasonable privacy is given for mu=0.5, whereas for mu=6 the trade-off function almost coincides with the x- and y-axis, thus giving virtually no privacy.

Connections between definitions of privacy

Another insight from Dong et al (2022) is that one can convert between(epsilon, delta) -differential privacy and f-differential privacy, and thus also Gaussian differential privacy. This is useful for two reasons. First, it means existing results from (epsilon, delta)-differential privacy apply also to the definitions here. Second, for certain analyses, it is more convenient to work with one definition than the other, but results derived from one hold for the other. For instance, one can analyze composition more easily with f-differential privacy, but analyze the privacy properties of mechanisms with data sub-sampling (discussed in the paper) more easily with (epsilon,delta)-differential privacy.

Conclusion

We saw that the hypothesis testing interpretation gives rise to new notions of privacy such as f- and Gaussian differential privacy, which are more handy to work with for certain analyses of privacy.

We now have the foundations to understand privacy testing in machine learning. Both the hypothesis testing interpretation and Gaussian differential privacy have been useful to empirically check the privacy-preserving properties of machine learning algorithms. This will be covered in a future article.

Thanks to Kody Moodley, Erik Tjong Kim Sang and Lourens Veen for commenting on and discussing an earlier draft.

Scientific Community Building through Workshops: the NPLinker eScience experience

eScience Editorial Team — Tue, 20 May 2025 00:00:00 GMT

This blog post is written by Wageningen University & Research Assistant Professor Justin van der Hooft, who participates in the Open eScience Call 2021. His project “A community-supported workflow connecting microbial genes and organisms to their molecular products” aims to facilitate the finding of novel bio-active molecules from nature. It does so by enabling the integrated use of various omics data types.

Want to know more of what Justin’s team is doing?! Check his team’s website.

Photo by Adrian Lange on [Unsplash](https://unsplash.com/photos/Wk902ZLaA7M?utm_source=unsplash&utm_medium=referral&utm_content=creditCopy

Text)

Motivation for the workshop**

Nature is still full of unknown chemistry with possible beneficial properties such as anti-inflammatory or antibiotic ones. Over the last decade, increasing numbers of genome sequences and metabolomics profiles have become available in public resources. Hence, the combination of genome mining and metabolome mining has gained traction in accelerating natural product discovery by linking biosynthetic genes to molecular scaffolds and structures, and associating producers with their products. However, effective integrative omics mining has proven challenging due to the various data types and tools involved. In 2021, NPLinker was proposed as a solution to streamline many aspects of integrative omics, so that researchers can focus on analyzing the possible matches between genes and molecules. Over the last years, the Open eScience Call project has further build on this solution to create a framework for paired omics analyses. To gain experience with the NPLinker framework and build an integrative omics mining community of talented, driven early-career researchers and knowledgeable instructors, Marnix Medema and I organized this second NPLinker-eScience workshop in Wageningen, the Netherlands.

Outline of the workshop

During this second workshop in a series of two, we again aimed to bring together researchers interested in integrative omics analysis, spanning various career stages, computational or the application-focused backgrounds, and diverse organisms they work on. Following a selection process, 38 participants were selected to come to Wageningen to learn about analyzing genomes and metabolomes, and to get an outlook towards the integration of the two. The general aims of the workshop were:

Introduce the NPLinker framework both conceptually and hands-on
Educate and train natural product researchers and other interested scientists
Extend the Integrative Omics Mining Community

The workshop intended learning outcomes were:

Understand the concepts of and motivations for paired omics analysis
Apply NPLinker to a public paired omics dataset
Understand the NPLinker scoring
Evaluate NPLinker outcome in the web app
Understand how to contribute to the NPLinker code base

To stimulate networking amongst the participants and instructors, several measures were put in place: extended coffee and tea breaks, the opportunity to present a poster, and a social mixer at the start to get to know each other. Furthermore, Marnix and I organized a symposium on Wednesday to highlight exciting developments in genome mining, metabolome mining, integrative omics mining, and the reuse of public omics data. We are grateful for the financial support from the eScience Center, which helped to make the workshop affordable for participants from across the globe through a travel reimbursement scheme and enabled the participation of three international instructors.

All 38 participants from across the globe together with the instructors in front of the Omnia building at the Wageningen University & Research campus.

I am increasingly aware of how integrative omics mining brings together different disciplines — what’s in a name, one could say. Furthermore, the field connects both researchers using the omics tools and developers working on them. We intentionally invited participants from these two groups and various disciplines to foster connections and collectively gain experience with the latest developments in the field. One key insight is that the fields is moving rapidly: only two years ago, we explained the latest in genome mining and metabolome mining, and during this workshop, versions 2 of BiG-SCAPE and GNPS were on the menu, replacing the previous versions. From a user perspective, this means staying up-to-date with how to use these tools effectively is key to remain on top of the latest possibilities in omics mining. From a developer perspective, it is important to build integrative tools like NPLinker in a way that can keep pace with the fast-evolving omics landscape. Another key insight is that, given the diverse backgrounds of participants, it is challenging to fully cater everyone’s needs. Striking the right balance between fixed program components and more flexible ones may help–allowing the explanation and demonstration of concepts or tools to be targeted based on participants’ needs. A final key insight is that promoting the exchange of knowledge and expertise in between participants is crucial, as it accelerates the uptake of new skills and information and helps form the bonds that create a scientific community.

Participants have been working hands-on with BiG-SCAPE, GNPS, and NPLinker during the workshop.

Open Access Materials

A tangible outcome of the workshop for the broader community is that the instructors all made as much as possible the materials available under an open access license! I hope this initiative will help to further foster the teaching of the future generation of omics miners, support the development of educational resources for genome mining, metabolomics, and metabolome mining, and promote discussions and integration between the fields of genome mining and metabolome mining.

With huge thanks to all the NPLinker eScience Workshop instructors who came from across the globe to Wageningen to teach the participants many facets of integrative omics mining.

Personal reflection

Altogether, I am happy that all participants and instructors enjoyed the workshop and gained something valuable from it. I am very pleased that everyone had a good time and that the location and catering was appreciated. Furthermore, it was great to organize the symposium with Marnix Medema that celebrated the work lustra of our teams in computational genomics (1o years) and metabolomics (5 years). Not only did we invite keynote speakers–Pieter Dorrestein, Tilmann Weber, and Margherita Sosio–but we also invited three workshop participants to showcase their work in a flash talk. In case you are curious, you can watch it back here. I found it nice to read in the evaluations how the participants liked to see each others’ posters during the breaks, and how they liked the flash talks during the symposium.

If there is one thing I would have changed looking back at the event, it would be to organize more structured parallel sessions to dive deeper into the application and development of NPLinker, for example by using two separate rooms and dividing participants and instructors accordingly. This would have allowed both groups (users and developers) to discuss their ideas and challenges in greater detail, without the feeling of disturbing each other.

Community building during the workshop dinner!

What I really liked were the workshop keynotes–thanks again to Kate Duncan for setting the scene on Monday, to Margherita Sosio for showcasing how omics mining is applied in an industrial setting and how the Molecules Gateway operates, and to Kumar Saurabh Singh for sharing his perspective on multi-omics analyses. Further, thanks again to the eScience Center for supporting this workshop series, and to the Omnia and Orion teams for their support and efforts before and during the week. A big thank-you as well to Maria Augustijn and Marie-José van Iersel for their support in organizing various aspects of the workshop and symposium. Thanks to Marnix for your support and being co-organizer of this event. And finally, many thanks to all the participants for their open and constructive attitude during the workshop — much appreciated!

Marnix Medema (right) and myself (left) during the symposium we organized.

Whilst this was the last workshop in a series of two, I hope to see the integrative omics community together on future occasions!

Benchmarking LLM fine-tuning on different HPC systems

Flavio Hafner — Tue, 06 May 2025 00:00:00 GMT

Authors: Flavio Hafner (Netherlands eScience center), Mattie Niznik (Princeton Research Computing), Malte Lüken (Netherlands eScience center), Alessandra Maranca (Princeton University), Matthew Salganik (Princeton University).

We have developed a benchmark that compares the compute performance of fine-tuning LLMs on multiple high-performance computing (HPC) systems, including systems designed for working with sensitive data. In this blog post, we introduce the benchmark, describe the lessons learned developing it and make it open-source so that it can be used and improved by others.

Photo by Nana Dua on Unsplash

Our use case

Our team researches how Large Language Models (LLMs) can be leveraged to predict social outcomes with large-scale social and survey data. We use the ODISSEI Secure Supercomputer (OSSC), which SURF operates in the Netherlands. The OSSC is a virtual private cluster running on the hardware of Snellius, the Dutch national supercomputer. The OSSC makes it possible to use GPU and CPU nodes to analyze sensitive data from Statistics Netherlands. This makes the OSSC one among very few systems globally that bring the power of national computing clusters to large-scale social data.

Previous performance benchmarks for OSSC were entirely focused on CPUs, and the performance of OSSC for GPU-intensive tasks, such as fine-tuning LLMs, was unknown.

Our benchmarking compares the performance of the OSSC to two different systems: 1) its parent system, Snellius, which is not connected to sensitive data and 2) Della, an HPC system similar to Snellius, which is operated by Princeton University.

Beyond our use case

Our benchmark, openly available on GitHub, is also a starting point for the exploration of fine-tuning performance in other cases. For instance, one can compare the training performance within and across HPC systems:

Within a system, one may have to choose between different GPU models, and the cost and benefits of using one versus the other may vary depending on the task.
Across systems, training performance does not only depend on the raw performance of GPUs–the number of floating point operations per second–, but also on factors such as the file system, how the computing hardware (processing cores and nodes) is connected through wires, and the exact version of GPU models used — as we will show.

The performance benchmark

The specification

We decided that our benchmark should satisfy the following requirements:

It is easy to run in many different computing environments.
It tracks multiple measures of speed, such as tokens per second and wall time.
It tracks anything useful to understand possible differences between the environments (GPU usage, CPU usage).
It is fast to run.
It is easy to improve over time: as our research evolves, we may need to re-assess the performance for new use cases.
It is easy to understand and replicate.

Our compute conditions are the following:

Environment: Della, Snellius, OSSC.
GPUs: A100 (40/80GB VRAM), H100 (80/96GB VRAM).
Single-node training, with GPUs per node: 1, 2, 4.
As many CPUs as are necessary to keep the GPUs busy.

The implementation

We found that torchtune fits our requirements best. We fine-tune the Llama-3.2–1B-Instruct model on the alpaca-cleaned dataset. Our main hyperparameters are:

A context length of 2048 on a packed dataset
Low-Rank Adaptation (LoRA) for the attention layers as well as the output layers. We use a rank of 64, alpha of 128 and 0 dropout.
A batch size of 6, which was the maximum we could fit into our least capable GPUs (A100)

What we found

Performance differences

The figure below shows the results from our benchmarking runs:

Tokens per second per GPU on a single node (higher is better). The colors refer to different systems (Della, Snellius, OSSC); the panels refer to different numbers of GPUs.In our experience, there was little variation in throughput across different runs on the same system, so we did not average over multiple runs.

Della was about 15% faster than the Dutch systems, independent of the number and models of GPUs.
The OSSC performed on par with Snellius.
For our use case, H100s were between 1.7x and 2x faster than A100s.
Throughput scales nearly linearly with the number of GPUs: The communication overhead from multiple GPUs was about 6–8% for going from 1 to 2 GPUs and 2% or less for going from 2 to 4 GPUs.

Many of these results made sense to us. Because Della and Snellius are broadly similar systems, we were surprised by the performance gap between the two. Thus, we decided to investigate further.

First, we ruled out that the gap stems from differences in software. The precise way to install dependencies differs between Della and Snellius: Our environments use anaconda on Della and a Python virtual environment with pip on Snellius. We confirmed that the two environments use the same package versions.

Second, we ruled out that the gap stems from differences in the file system. In particular, for high-performance machine learning, data I/O is often a bottleneck, and it can be reduced by using a fast file system. However, our experiments on Snellius showed no difference in speed between the various file systems available.

We then found that subtle hardware differences probably explain the gap — differences in memory clock speed of the GPUs:

Memory clock speed for GPUs on the two systems (higher is better).For the A100s, the Snellius/OSSC GPUs have 40GB VRAM, while the Della GPUs have 80GB VRAM. The 80GB variant also comes with a higher memory clock speed than the 40GB variant, leading to the difference in fine-tuning speed on A100s between Della and Snellius.

For the H100s, the difference is in the version of the cards, and specifically their memory: Della has cards with the faster HBM3 memory, while Snellius/OSSC has cards with HBM2e memory. This means that the calculations during model training have a memory speed bottleneck on Snellius/OSSC compared to Della.

Lessons learned from bringing LLMs into secure computing environments

Running this test and bringing the Llama model into a secure environment created additional challenges.

First, by default, torchtune downloads the specified dataset from the internet without the user noticing. But because the OSSC is cut off from the internet, we had to use the Alpaca dataset locally and adjust the code accordingly before deployment.

Second, to import the model weights to the secure computing environment, we had to ensure that the contents of the files were not a security risk. Thus, we followed current best practices: First, we manually scanned the files with the virus scanner ClamScan, similar to what Hugging Face does automatically for models hosted there. Second, we imported the model weights in safetensors format, which is safer than the pickle format and prevents arbitrary code execution.

Conclusion

We introduce a benchmark for comparing throughput for LLM fine-tuning on different HPC systems. We found hardware-driven performance differences between two standard HPC systems. We also provide the first performance comparison of the OSSC compared to its parent system, Snellius. Lastly, our benchmark suggests little cost of scaling fine-tuning to multiple GPUs on a single node.

The code for our benchmark is publicly available: https://github.com/odissei-benchmarks/torchtune-gpu-benchmark

Thanks to Patrick Bos for feedback on a draft.

Keeping your scientific data files small

Bart Schilperoort — Tue, 22 Apr 2025 00:00:00 GMT

Keeping your scientific data files small

An RSE’s guide to NetCDF compression

If you’ve ever watched a crawling progress bar while transferring research data, or received a dreaded ‘storage quota exceeded’ message, you’re not alone. Researchers across disciplines face a common challenge: the size of collected data can outpace storage and processing capabilities. This blog explores how compression techniques for NetCDF (Network Common Data Form) files — a format commonly used for multidimensional scientific data — can reduce storage requirements while maintaining data integrity. Whether you’re handling climate models, GIS data, or sensor networks, these approaches can change how you manage and access multidimensional data, such as data with both time and spatial coordinates.

Simplified structure of a netCDF file (from the xarray documentation); created by Stephan Hoyer, Apache 2.0 license).If you have multidimensional data, the file sizes can get big quite fast. Take output from the ERA5 global weather reanalysis product:

It has a resolution of 0.25 degrees, at an hourly frequency.
This means that there are 720x1440x365x24 = 9 billion data points per variable per year.
This type of data is usually represented as 32-bit floating points numbers.
32 bits equals 4 bytes, times 9 billion; approximately 36 GB/year. Per variable.

As soon as you want to analyze multiple years, for multiple variables, your required storage can quickly reach dozens of terabytes. Storing all this data can not only be prohibitively difficult or expensive, it also significantly increases your carbon footprint. Luckily, there are multiple ways to reduce the file size of netCDF files, with various up- and downsides.

Lossless compression**

You are probably familiar with putting files on your computer in a zip folder. This can significantly reduce their size (if the data is not already compressed yet):

Lossless compression of a simple 19 kB python code fileYou can then retrieve the exact files from the zip folder, without any loss of information. This is called “lossless” compression. Many files on your computer are compressed in this way already (for example, Microsoft Office files, or PDF documents).

Lossy compression

A different compression method you probably have encountered before is lossy compression. Lossy compression is often applied to audio, images and video to dramatically reduce their size (~10x for audio or still images, 100x for video), with little perceived quality loss.

*Very lossy JPEG compression versus lossless PNG. Source: *WikimediaA downside of lossy compression is that each time you open the data and would write it away lossily again, additional compression artifacts are introduced. This will cause a gradual degradation in data quality.

Compressing netCDF files

As a quick refresher, we need to discuss how netCDF files are structured.

NetCDF files organize the data per variable. Each variable in turn is split up in different “chunks”. These chunks correspond to subsets of the full array. For example, data can be chunked only in time, where each chunk represents the entire physical domain (latitude & longitude) for a single time coordinate.

Graphical representation of chunks in the xarray python package, using DaskThese chunks are read into memory one at a time, so for any operations it is important that the chunks fit your analysis for efficiency.

The actual underlying data can be stored as 32-bit floating point numbers on disk, but there are a few options for compression.

Lossless netCDF compression

Just like with text files or other documents, lossless compression is possible for netCDF files. The more repetitious your data is, the easier it is for lossless compression algorithms to compress your data.

cdo -z zip 4 … # where 4 is the compression level (1 - 9)In Python, with the popular ‘xarray’ package, you can instead do:

import xarray as xr

ds = xr.open_dataset(“my_dataset.nc”) encoding = {“my_variable”: {“zlib”: True, “complevel”: 4}} ds.to_netcdf(“compressed_dataset.nc”, encoding=encoding)In many cases applying lossless compression to your data can reduce the size on disk by 20–50%. This comes at a cost of slightly more CPU use, while disk I/O usage is lower.

The exact impact on your analysis speed will depend on your system, but unless you have a slow CPU and a blazingly fast disk your further analyses will likely be faster when you compress your data. Therefore it’s best to default to using some form of lossless compression on your netCDF files, unless you have a specific reason not to.

Lossy netCDF compression**

As stated earlier, unlike lossless compression, lossy compression does not preserve the full original data. Some of the original resolution can be lost, while offering a much smaller file size as well as a faster read speed.

Often some loss of resolution is not a problem, as 32-bit floating point numbers offer more precision than many measurements or model data realistically have.

One way of lossy compression which is well supported by netCDF is encoding floating point numbers to integers. This is usually done with 16-bit unsigned integers: whole numbers ranging from 0 to 65,535. The floating point numbers can then be mapped to this range.

For example; as the lowest ambient air temperature ever recorded on earth is 184 K, and the highest 330 K, giving a range of 146 K. To convert the temperature, you take the integer value, multiply it with a ‘scale factor’, and add the offset. The temperature range can thus be divided over 65,535 discrete values, translating into a maximum resolution of ~2.23 mK, which is a much higher resolution than most scientific air temperature sensors. Your offset here would be 184 K.

Integer encoding works very well for climate data as the range of values that you can expect should be very well constrained. This allows you to make informed estimates of what sufficient precision is for these variables, while still significantly reducing the size of the data on disk.

When using integer encoding in xarray you can specify the encoding as such:

import xarray as xr** ds = xr.open_dataset(“my_dataset.nc”)

fvalue = 65535 # fill value for missing values scale = 0.005 # store temperature in 5 mK increments offset = 0 # Kelvin is abolute: no offset required

encoding = { “my_temperature_variable”: { ‘_FillValue’: fvalue, ‘scale_factor’: scale, ‘add_offset’: offset, ‘dtype’: ‘int16’, “zlib”: True, “complevel”: 4, # you can still use lossless compression! }, }

ds.to_netcdf(“compressed_dataset.nc”, encoding=encoding)For Climate Data Operators users, the “pack” operator is available for this purpose.

Compression ‘filters’**

Lastly, HDF5; the underlying data format of netCDF4 files, has “filter plugins” available. These plugins allow you to use other lossy compression algorithms for your netCDF data. These filters are a more advanced use case and not as easy to set up as the other compression methods.

Some publications (e.g., Prims, 2024, and Delauney, 2019)) have studied the effect of these on netCDF and numerical weather prediction data specifically.

Depending on the variables, these compression algorithms can reduce the data size on disk by a factor of 5–150. However, unlike integer compression, the effects of these compression algorithms on your data is not as easily predictable, and you might need to do a sensitivity study for your own application.

However, there are use cases such as many climate model ensembles where compressing the data lossily could be a way to store data that would otherwise be too large to store.

Compression and chunks

As the underlying data of each variable in a netCDF file is organized in “chunks”, the compression algorithm is applied per chunk. *This allows you to still load individual chunks of data without requiring to load and decompress the entire file.

Therefore it is important to organize these chunks in a way that aligns well with your intended usage; for example, either looking at the full time series for a single location, or the entire world at a single point in time. This will enable you to read parts of your full netCDF file very quickly and store them efficiently.

Conclusion

Compressing your netCDF data can reduce your disk usage significantly, which allows you to store or share data more easily. For most users lossless compression is the most straightforward and should be applied by default.

For many others, where storage space is a constraint, storing your floating point numbers as 16-bit integers is a predictable way of compressing your data further. Other lossy algorithms are available, but are generally too complex to use for most users.

Photo by Jason Pofahl on Unsplash

Solving the mystery of the missing symbols! (part II)

Suvayu — Tue, 22 Apr 2025 00:00:00 GMT

This is the conclusion to the story I wrote about* earlier.*

In that earlier post, we were investigating why the MingW-w64 build of DuckDB fails to export the C-API symbols in the final DLL. We identified that the issue was introduced by the commit d1ea1538. However, we could not ascertain the underlying reason for the problem. To take this further, let us try to isolate which part of the change was the cause.

Photo by Tomas Martinez on Unsplash

A dead-end

To identify the source file responsible for the bug, we can start by inspecting the different build artifacts and checking for the C-API symbols. But first, we need to log all the build commands. It is much easier to do this with Make than Ninja (default generator for DuckDB). We can choose our generator by calling CMake as:

$ cmake -G “Unix Makefiles” …We can then log all the build commands by starting a verbose build with Make:

$ VERBOSE=1 make -j6 |& tee build.logWith all the build commands recorded to a log file, I searched for commands that create the shared library libduckdb.dll and narrowed it down to the following set of build commands:

$/usr/bin/cmake -E rm -f CMakeFiles/duckdb.dir/objects.a$ /usr/bin/x86_64-w64-mingw32-ar qc CMakeFiles/duckdb.dir/objects.a @CMakeFiles/duckdb.dir/objects1.rsp $ /usr/lib64/ccache/x86_64-w64-mingw32-g++ -O3 -DNDEBUG -shared -o libduckdb.dll
-Wl,—out-implib,libduckdb.dll.a -Wl,—major-image-version,0,—minor-image-version,0 -Wl,—whole-archive
CMakeFiles/duckdb.dir/objects.a -Wl,—no-whole-archive @CMakeFiles/duckdb.dir/linkLibs.rspGoing through the steps individually, we will see that even though the symbols are present in the earlier steps, the final library creation step drops them.

Tracing through the build steps

Let us first look for the object file that has the C-API symbols.

$ tr ’ ’ ‘\n’ <CMakeFiles/duckdb.dir/objects1.rsp | grep capi.cpp main/capi/CMakeFiles/duckdb_main_capi.dir/ub_duckdb_main_capi.cpp.objWe can then check for the C-API symbols in the object, and confirm that the object file is included in the objects archive.

$nm -C main/capi/CMakeFiles/duckdb_main_capi.dir/ub_duckdb_main_capi.cpp.obj | grep duckdb_vector_size 0000000000004f60 T duckdb_vector_size$ /usr/bin/x86_64-w64-mingw32-ar t CMakeFiles/duckdb.dir/objects.a | grep capi.cpp ub_duckdb_main_capi.cpp.objIf we check for the C-API symbols again in the archive file, we see they are present and publicly visible (exported).

$nm -C CMakeFiles/duckdb.dir/objects.a | grep duckdb_vector_size 0000000000004f60 T duckdb_vector_size$ winedump -j export CMakeFiles/duckdb.dir/objects.a | grep duckdb_vector_size 31a0ae4 duckdb_vector_sizeFinally, we can create the shared library by calling the linker.

$ /usr/lib64/ccache/x86_64-w64-mingw32-g++ -O3 -DNDEBUG -shared -o libduckdb.dll
-Wl,—out-implib,libduckdb.dll.a -Wl,—major-image-version,0,—minor-image-version,0 -Wl,—whole-archive
CMakeFiles/duckdb.dir/objects.a -Wl,—no-whole-archive @CMakeFiles/duckdb.dir/linkLibs.rspWe see that while the C-API symbols are present, they are not exported!

$nm -C libduckdb.dll | grep duckdb_vector_size 00000001f3d116e0 T duckdb_vector_size$ winedump -j export libduckdb.dll | grep duckdb_vector_size # no match

While not helpful on its own, the observation that symbols are not exported in the final linking step that creates the DLL will make sense later.

A clue

Since we could not already isolate the issue to a source file, next, I decided to “bisect” the contents of the problem commit. As the issue is with C-API symbols, I started with the files that have these symbols:

src/include/duckdb.h,
src/main/capi/aggregate_function-c.cpp, and
test/api/capi/capi_aggregate_functions.cpp.

I commented out parts of the above files, while ensuring the build still succeeds, and checking for the C-API symbols in the final DLL in the usual way with nm and winedump. This led me to discover that the problem disappears if we reorder the include headers in src/main/capi/aggregate_function-c.cpp. You can see the change in this PR: duckdb/duckdb#16396.

diff —git a/src/main/capi/aggregate_function-c.cpp b/src/main/capi/aggregate_function-c.cpp index 43ddfdbc074a..1379f652d669 100644 --- a/src/main/capi/aggregate_function-c.cpp +++ b/src/main/capi/aggregate_function-c.cpp @@ -1,9 +1,11 @@ +// clang-format off +#include “duckdb/main/capi/capi_internal.hpp” +// clang-format on #include “duckdb/catalog/catalog.hpp” #include “duckdb/common/type_visitor.hpp” #include “duckdb/common/types.hpp” #include “duckdb/function/function.hpp” #include “duckdb/function/scalar_function.hpp” -#include “duckdb/main/capi/capi_internal.hpp” #include “duckdb/main/client_context.hpp” #include “duckdb/parser/parsed_data/create_aggregate_function_info.hpp” #include “duckdb/planner/expression/bound_function_expression.hpp”This is, of course, not the correct solution since there should not be any dependence on the order in which we include headers. It is indicative of an underlying bug. This means some definition in the headers is being overridden silently. This can only happen for preprocessor macros; anything else would trigger a compilation error.

Solution to the mystery!

After further investigation from DuckDB core developer Mark Raasveldt, the issue was resolved in this PR: duckdb/duckdb#13697. So, what was the issue?

When building shared libraries for Windows, the public API has to be marked for export. This is done by adding the __declspec(dllexport) attribute to the relevant symbols. In the DuckDB codebase this is done by defining a macro DUCKDB_API that expands to that attribute. However, there were two such definitions, they were in:

src/include/duckdb.h, and
src/include/duckdb/common/winapi.hpp.

Superficially, they seem equivalent, however, they are actually used in slightly different contexts. The macro defined in duckdb.h is used to annotate symbols that are part of the C-API*. Whereas the macro defined in winapi.hpp is used to mark symbols that comprise the C++ API. In an earlier commit (367595af), the macro defined in winapi.hpp was changed to exclude MingW builds.

diff —git a/src/include/duckdb/common/winapi.hpp b/src/include/duckdb/common/winapi.hpp index 2cff5d12d4..65dd89975e 100644 --- a/src/include/duckdb/common/winapi.hpp +++ b/src/include/duckdb/common/winapi.hpp @@ -9,7 +9,7 @@ #pragma once

#ifndef DUCKDB_API -#ifdef _WIN32 +#if defined(_WIN32) && !defined(MINGW32) #if defined(DUCKDB_BUILD_LIBRARY) && !defined(DUCKDB_BUILD_LOADABLE_EXTENSION) #define DUCKDB_API __declspec(dllexport) #else

The reason & the fix

When exporting symbols, MingW normally excludes inline functions (see man x86_64-w64-mingw32-g++ and search for -fkeep-inline-functions). The macro redefinition handles this case by excluding symbol export for MingW.

Now, in the DuckDB source, depending on which definition is seen first, symbols may or may not be exported. This, of course, depends on the order of the include headers.

If DUCKDB_API in duckdb.h is seen first, symbols are exported correctly, on the other hand
if DUCKDB_API in winapi.hpp is seen first, symbols are not exported.

The final fix was to rename one of the macros to make the two definitions distinct so that they do not interfere with each other; i.e.

DUCKDB_API in duckdb.h was renamed to DUCKDB_C_API, and
DUCKDB_API in winapi.hpp remains unchanged.

What did we learn?

After this resolution, any project reliant on this build can update to the latest version of DuckDB starting from version 1.2.1. In our case that is the Julia binding for DuckDB, DuckDB.jl. If we look at the original issue: duckdb/duckdb#13911, there are many different projects reliant on it, so the impact of this fix is quite broad!

During this investigation, we also learnt:

bisection is an incredibly effective idea that can be used in many different contexts, and
macro hygiene is important in C/C++ code bases.

Introduction to Deep Learning with Python

eScience Editorial Team — Tue, 25 Mar 2025 00:00:00 GMT

Written by Sven van der Burg, Carsten Schnober, Sarah M. Brown and Toby Hodges

Engineers and instructors at the eScience Center have developed lots of open source lesson materials for the workshops we teach. One example is the lesson material for Introduction to Deep Learning*, *which was recently accepted into the peer-reviewed lesson material collection in the Carpentries Lab. In this post you can read more about the lesson and its development.

Photo by Scott Webb on Unsplash

The Carpentries Lab was set up as a space for peer-reviewed lessons developed by the Carpentries community. Designed to complement The Carpentries Incubator, where the community collaborates on the development of new lessons on a wide range of topics, the Lab is a platform for open peer-review of community-developed lessons and for hosting the lessons that have passed through that review process.

The text below was originally published on the Carpentries blog.

About the curriculum

The use of deep learning has seen a sharp increase in popularity and applicability over the last decade. While deep learning can be a useful tool for researchers from a wide range of domains, taking the first steps in the world of deep learning can be somewhat intimidating.

Introduction to Deep Learning covers the fundamentals of deep learning in a practical and hands-on manner, for researchers who want to apply deep learning in their research. By the end of the course, students will be able to train their first neural network and understand the subsequent steps needed to improve the model.

The lesson can be taught in a 2-day or 4 half-day workshop.

Part of the Monitor the training Process episode of the lesson.

A brief history of the development of this lesson

What follows is a year-by-year history of the main contributors to the lesson. Of course, GitHub keeps the score, and all contributions to the commit history, big and, especially, small, can be better tracked in this interactive contributor graph. Over the years 30 people contributed to this lesson!

The development of the lesson started in 2020 in a collaboration between Berend Weel and Florian Huber (both at the Netherlands eScience Center at the time), Peter Steinbach, Colin Sauze, Samantha Wittke, and Toby Hodges. During this period the main design of the lesson was drafted and the first pilot workshops were given.

During the year of 2021, Dafne van Kuppevelt, Djura Smits, and Sven van der Burg from the Netherlands eScience Center became involved and by the year 2022 Sven and Djura were the main contributors. In this period the lesson was taught a lot by various organisations and all feedback went into an ever-improving lesson.

In the period from 2023 till now the lesson was really polished towards its final shape, thanks to contributions from fresh team members Carsten Schnober and Pranav Chandramouli from the Netherlands eScience Center. This is the period that the lesson was transitioned to The Carpentries Workbench and went through The Carpentries Lab review process.

About the review process

The curriculum was reviewed in a public issue thread on The Carpentries Lab reviews repository. This was a slightly unusual case because one of the authors is a member of The Carpentries Curriculum Team and was the sole Editor of the Lab at the time of submission. (Three other community members have joined as Editors since.) Sarah M Brown, formerly a long-standing member of the Trainers Leadership Committee, accepted an invitation to act as Guest Editor for this review. The authors and Carpentries team are enormously grateful to Sarah for the effort she put in and the diligence she displayed throughout the process. She made several essential contributions as Editor, in particular by encouraging the authors to more deeply explore the choice of example datasets used in the lesson (read more about this below).

After the authors had responded to Sarah’s initial editorial checks, the curriculum was reviewed in detail by two volunteers from The Carpentries community: Johanna Bayer and Mike Laverick (thanks to Erin Becker for helping to connect Sarah with these reviewers). These reviewers were invited based on their expertise as Carpentries Instructors and their domain knowledge in machine learning and related fields.

The peer-review process led to a number of improvements to the accessibility of the lesson, with better alternative text descriptions for images and various adjustments to make language clearer and more beginner-friendly. The lesson was made easier to teach by other instructors through the addition and improvement of Instructor Notes, and the explanations of a number of important deep learning concepts were expanded.

Perhaps most importantly, the Editor highlighted that one of the datasets used as an example in the lesson, the CIFAR-10 dataset, is not available under a clearly-defined license. On closer inspection, the authors were surprised to find that CIFAR-10, a dataset very commonly used for benchmarking in machine learning research, was created with image files scraped from the internet without permission from their original owners.

Sven van der Burg, one of the leading authors of the lesson, created the “Dollar Street 10” dataset as an alternative to replace CIFAR-10 in the lesson. Dollar Street 10 is derived from the CC-BY-4.0 licensed Dollar Street Dataset from the Gapminder Project: a collection of images that has been more ethically assembled with an emphasis on ensuring that the data is representative of global diversity and accompanied by accurate metadata. The authors added a callout to the lesson to comment on this change, as they felt it represented a great entry point for a discussion of ethics in data and deep learning during a workshop.

Next steps

The development team will of course keep on teaching this lesson at their institutes and keep on improving the lesson. We will have to, since deep learning is such a rapidly moving field. But our main goal is to help as many others as possible to adopt this lesson so that it can be taught around the world!

The lesson has been submitted to The Journal of Open Source Education for publication.

The wider Carpentries community can now find the lesson in The Carpentries Lab, use and adapt it to teach their own workshops, contribute feedback, and suggest improvements. We hope you find it a helpful resource!

Become a Reviewer

The Carpentries Lab is looking for reviewers!** To volunteer to review a lesson in the Lab, please read our Guide for Reviewers, and register as a reviewer so we can contact you when relevant lessons are ready for review.

Lessons in the Incubator can be submitted for review in the Lab by opening an issue on the reviews repository.

Acknowledgements

We would like to thank all instructors and helpers that taught the course, and the community of people that left contributions to the project, no matter how big or small.

We thank The Carpentries for providing such a great framework for developing this lesson material.

We thank all students enrolled in the workshops that were taught using this lesson material for providing us with feedback.

Congratulations to the authors for creating an excellent lesson that will prove a useful resource to so many people:

And huge thanks again to Sarah M Brown for acting as Guest Editor, and to the reviewers, Johanna Bayer and Mike Laverick.

A debugging journey into the unknown (part I)

Suvayu — Thu, 13 Mar 2025 00:00:00 GMT

Faced with a buggy library, most people will just give up and wait for a patch to be released. Certainly not our engineers.

NASA/JPL-CaltechIt all started with a colleague trying to update their Julia environment for one of our projects. They could not update because a dependency, the Julia bindings for DuckDB, was failing to compile. For the moment we decided to deal with it later, and pinned DuckDB.jl to the working version. We also filed an issue upstream: duckdb/duckdb#13911.

Since there was no significant progress on the issue for a few months, I decided to dive in once and for all — this is that story.

The issue

On Windows, trying to install any version of DuckDB.jl later than 1.0.0 was failing with the error:

ERROR: LoadError: could not load symbol “duckdb_vector_size”: The specified procedure could not be found.Some Julia packages, like DuckDB.jl depend on a native library. The native library is an internal dependency, and typically named MyPackage_jll.jl; in the case of DuckDB, it is DuckDB_jll.jl. The error above tells us that during the compilation step, Julia tries to load a symbol from the native library, but cannot find it. A “symbol” here refers to a DuckDB C-API function provided by the native DuckDB library.

So as a first step, I wanted to check: can we actually install this library, and load that symbol? To get the call syntax correct, I looked at the source code of [DuckDB.jl](https://github.com/duckdb/duckdb/blob/5f5512b827df6397afd31daedb4bbdee76520019/tools/juliapkg/src/api.jl#L1285-L1297) :

function duckdb_vector_size() return ccall((:duckdb_vector_size, libduckdb), idx_t, ()) endThe variable idx_t above is defined in ctypes.jl as:

const idx_t = UInt64 # DuckDB index typeSo we can test loading the native library like this:

pkg> add DuckDB_jll julia> using DuckDB_jll julia> ccall((:duckdb_vector_size, libduckdb), UInt64, ())The above recipe replicates the error for any version newer than 1.0.0! Hurray! Now that we have confirmation that the problem is in the native library, we have to understand: Why is the symbol not visible to Julia?

Symbol visibility in Windows DLLs

My first hurdle was to find a way to get both versions of the library and compare. I decided to install different versions of the native library DuckDB_jllin different directories; after loading the library, the libduckdb variable points to the correct path of the shared library. We can then use other tools to inspect the dynamic-link libraries (DLLs) and check if the symbols actually exist. If we can compare the working version of the native library with a version that does not maybe we can find out what is wrong.

On Linux, we can use nm from binutils to look at the symbols present in the library. Thanks to the -C flag, nm can even “demangle” symbol names if necessary. So for the moment we can copy over DLLs from Windows to Linux, and inspect.

$nm -C libduckdb.dll | grep duckdb_vector_size # working version: v1.0.0 000000036a271d00 T duckdb_vector_size$ nm -C libduckdb.dll | grep duckdb_vector_size # not working version: e.g. v1.1.2 000000036a36e0e0 T duckdb_vector_size 000000036be998d0 r .rdata$.refptr.duckdb_vector_size 000000036be998d0 R .refptr.duckdb_vector_sizeNo luck 😦, seems that the symbols exist for both versions of the native library. I was puzzled. Searching around I found while building Windows DLLs, you have to explicitly export symbol names using the __declspec(dllexport) attribute. Besides signalling which names are available, it also serves as a mechanism to optimise DLL load times. So I went looking for these attributes in the DuckDB source code:

#ifndef DUCKDB_API #ifdef _WIN32 #ifdef DUCKDB_STATIC_BUILD #define DUCKDB_API #else #if defined(DUCKDB_BUILD_LIBRARY) && !defined(DUCKDB_BUILD_LOADABLE_EXTENSION) #define DUCKDB_API __declspec(dllexport) #else #define DUCKDB_API __declspec(dllimport) #endif #endif #else #define DUCKDB_API #endif #endifYou can see, the #ifdef directives conditionally defines the macro DUCKDB_API which expands to __declspec(dllexport) when building a Windows DLL. Later in the header file, this macro is used to mark every C-API function for export.

DUCKDB_API idx_t duckdb_vector_size();Now that we know the C-API function name symbols are marked for export correctly, we should check if they are indeed exported. After some searching, I learnt Windows development tools includes the program dumpbin.exe that can show the exported symbol names. So I can search for exported symbols in the output.

Working version of libduckdb.dll :

> dumpbin.exe /EXPORTS .\bin\libduckdb.dll | findstr duckdb_vector_size 335 14E 0078D1B0 duckdb_vector_sizeFaulty version of libduckdb.dll :

> dumpbin.exe /EXPORTS .\bin\libduckdb.dll | findstr duckdb_vector_size

empty output, since nothing was foundSo it is confirmed that the symbol export is not working for the faulty version. However we still do not know which commit introduced the issue. Let us try to find that 🙂.

Hunt for the first “bad” commit

To be able to find the first bad commit, we need to be able to compile the library. Julia has a whole other infrastructure called [BinaryBuilder](https://binarybuilder.org/). It cross-compiles native binaries for all platforms on Linux. This is a whole another rabbit hole, and let us shelve this for another time. The only relevant bit is, the builds are done only* on Linux. This presents a different problem, dumpbin.exe, the tool to check if the symbol export is correct is available only on Windows. If we are to find the bad commit, we need to automate the build & check steps and run it with git-bisect. How can we do that if parts of our toolchain runs on different platforms‽

So I went searching again for an alternative. Unsurprisingly, Wine (the Windows compatibility layer for Linux) ships with the tool [winedump](https://gitlab.winehq.org/wine/wine/-/wikis/Man-Pages/winedump), which is an equivalent to the Windows tool dumpbin.exe.

To build with Windows DLL, we need to cross-compile DuckDB on Linux using the MingW-w64 toolchain. Combined with the winedump tool, I wrote the following script that we can use to test a successful build.

#!/bin/bash

rm -rf build cmake -B build
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_TOOLCHAIN_FILE=mingw-w64-x86_64.cmake
-DBUILD_EXTENSIONS=‘autocomplete;icu;parquet;json;fts;tpcds;tpch’
-DENABLE_EXTENSION_AUTOLOADING=1
-DENABLE_EXTENSION_AUTOINSTALL=1
-DBUILD_UNITTESTS=FALSE
-DBUILD_SHELL=TRUE
-DDUCKDB_EXPLICIT_PLATFORM=x86_64-w64-mingw32-cxx11 . cmake —build build

[[ $? -ne 0 ]] &&
{ echo “build failed, cannot test” exit 125 }

if [[ -f build/src/libduckdb.dll ]]; then winedump -j export build/src/libduckdb.dll | grep -q duckdb_vector_size if [[ $? -eq 0 ]]; then exit 0 else exit 1 fi else echo “cannot find DLL, cannot test” exit 125 fiThe script builds DuckDB, and checks whether the generated DLL file export the C-API symbols correctly. It also handles a few corner cases of build failures. Note that the build command uses the following toolchain file (thanks to this gist).

set(CMAKE_SYSTEM_NAME Windows) set(TOOLCHAIN_PREFIX x86_64-w64-mingw32)

cross compilers to use for C, C++ and Fortran

set(CMAKE_C_COMPILER ${TOOLCHAIN_PREFIX}-gcc) set(CMAKE_CXX_COMPILER$ {TOOLCHAIN_PREFIX}-g++) set(CMAKE_Fortran_COMPILER ${TOOLCHAIN_PREFIX}-gfortran) set(CMAKE_RC_COMPILER$ {TOOLCHAIN_PREFIX}-windres)

target environment on the build host system

set(CMAKE_FIND_ROOT_PATH /usr/${TOOLCHAIN_PREFIX})

modify default behavior of FIND_XXX() commands

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)Now we can use this script with git-bisect to find the first bad commit, like this:

$git bisect start$ git bisect bad v1.1.2 $git bisect good v1.0.0$ git bisect run ./bisect-script.bash $git bisect visualize # shows a nice summary$ git bisect resetThe bisection led me to this commit: d1ea1538. Unfortunately it is not clear how that commit would lead to symbol export failure, it only adds 11 new functions to the C-API (so 11 new symbols). So for now, this debugging journey has to stop here, without a clear resolution. But we did learn a lot of new concepts, and used a wide variety of tools to investigate.

What did we learn?

To summarise, the issue is:

All releases of DuckDB.jl after v1.0.0 on Windows are broken.
We recreate the issue without Julia, which shows the the DuckDB build using the MingW-w64 toolchain is broken for all releases after v1.0.0 .

And in the process of debugging this, we learnt:

Windows has a separate mechanism to export symbol names in its shared libraries (DLL).
We learnt about tools to inspect symbols in native libraries; namely nm, dumpbin.exe, and winedump.
We learnt about the Julia build system for native libraries (a topic for a future post).
We learnt to write a script that we can use with git-bisect to run automatic bisections (a potential topic for a future post).

—

The story isn’t complete. There will be a concluding part when I actually fix the problem.

—

*Cross post from: *https://github.com/suvayu/suvayu/blob/master/blog/2025-02-15-duckdb-julia-windows.md

Sharing MATLAB models with everyone

Bart Schilperoort — Wed, 12 Mar 2025 00:00:00 GMT

MATLAB is still commonly used in many scientific fields, despite its closed-source nature. While many (technical) universities used to teach coding using MATLAB, more and more are moving away from it, preferring open-source and free alternatives such as Python or Julia. One of the main reasons Julia was developed was because of MATLAB’s proprietary and closed software.

Photo by Iván Díaz on UnsplashHowever, for those with a vested interest in the language, such as a large legacy code base, it is still important to be able to share your models. Just sharing MATLAB code is not that useful as it requires the users to have a MATLAB license. For a project with the University of Twente, we work on [a MATLAB land-surface model](https://github.com/Eco

ExtreML/STEMMUS_SCOPE/). We wanted to make the model open-source and available for anyone to run, as well as allow interaction with the model through a Basic Model Interface, a standardized interface for numerical models.

One option would be to try GNU Octave, a language that is mostly compatible with MATLAB. The key word being mostly*. For us, the difference in execution time was too large. The second option is to make use of MATLAB Runtime.

MATLAB Runtime

With MATLAB Runtime anyone can run MATLAB applications or components without installing MATLAB. You generate an executable file on a system with a MATLAB license, which you can then run on a system without a license. This does require that the target system has the same operating system and architecture as the system that generated the executable file.

Docker

This is where Docker comes into play. Docker allows you to package your application together with a minimal operating system and any dependencies, such that it can run as a standalone application. Note that you can also use Apptainer, an open-source alternative.

Lucky for us, someone already did the work of containerizing MATLAB Runtime. Thanks Riccardo De Martis! A quick example is in the following Dockerfile:

We start from the already-existing MATLAB Runtime container:

FROM demartis/matlab-runtime:R2023a

Put the executable in the container, for example from GitHub:

RUN wget https://github.com/MyOrganization/MyRepo/raw/main/exe/MATLAB_MODEL - no-check-certificate

Make sure the file is executable

RUN chmod +x ./MATLAB_MODEL

Allow MCR to have a cache directory which all users can access

this allows for executing the model as a different user.

RUN mkdir /temp/ RUN chmod 777 /temp/ ENV MCR_CACHE_ROOT /temp/

Run the model:

CMD ./MATLAB_MODELNotice that the MATLAB Runtime container is based on Debian, so you need to generate your executable file on a Linux system as well. On the flip side, once the container is built, it will be easy to run it on different systems, as well as in parallel (such as scaling up your analysis on HPC, where more computational power and data is available).

Interacting with the containerized model

For many users, this will be good enough: you simply run the model once, passing a config file as an argument, and you’re done. If the model needs access to data, you can attach those directories to the container. The model’s output will be written to a file.

A more advanced use case is when you want to interact with the model while* it is running, such as with the Basic Model Interface mentioned earlier. For this, you will need to communicate with the running MATLAB code.

Communicating with the model inside the directory can be done using the terminal, but more ideal would be a proper interface. If you set up a MATLAB HTTP server, you can expose the server’s port inside the container. This then allows you to start the HTTP server and execute the MATLAB code from outside the container.

Data exchange

While telling the model to run certain routines is straightforward using an HTTP server, data exchange is a bit more complex. A simpler way is to write the model’s state or parameters of interest to file.

MATLAB’s default file format is HDF5 nowadays, which is supported by most other languages (for example, Julia with HDF5.jl, or Python with h5py). Scripts or programs written in these other languages can then interact with this file, which can be loaded again in MATLAB if two-way communication is required.

Communicating with a containerized MATLAB model through Python

Example: STEMMUS_SCOPE Basic Model Interface

A working example of a containerized MATLAB model is STEMMUS_SCOPE, a land surface model. We wanted to write a Basic Model Interface (BMI), to allow for coupling the model to other models such as the groundwater model MODFLOW.

However, coupling a MATLAB model to models written in a different language is challenging, so we connected a Python BMI to the containerized model, as Python excels at being a “glue” language. This allows us to couple the STEMMUS_SCOPE model with other models implementing a BMI.

One detail here is that we did not have access to a full MATLAB IDE, so it was too challenging to actually go the proper route and implement an HTTP server for STEMMUS_SCOPE. Instead, we communicate to the MATLAB processing using stdin/stdout.

While it’s certainly more difficult than with many other programming languages, sharing easily (re-)usable MATLAB code with others is still possible. Not only that, but it can also make your own life easier by being able to run your code on other platforms and allowing you to run it in a single command.

If you enjoyed this story, have a look at the blogpost “How to ensure that others can run your code” by Julian Gonggrijp, which touches on many reasons why* *you would want to make sure others can run your code (and how).

Photo by Josh Campbell on Unsplash*** Octave does not support all MATLAB functions, so the code will have to be modified to support both MATLAB and Octave. Octave can also be an order of magnitude slower than MATLAB, which can be problematic.

A Helpful Noob

ole — Fri, 21 Feb 2025 00:00:00 GMT

Photo by Keiteu Ko on Unsplash

Introduction

Yesterday, my colleague Faruk asked me to conduct a Hallway Test for the [NLeSC/python-template](https://github.com/NLeSC/python-template) he is working on: he just grabbed me off the hallway and asked me to give it a spin. This is, however, not a story about the test, but of open-source intelligence (OSINT) and the contributions of yet another “helpful noob”.

Right after I posted my feedback in an issue of the project, someone else posted a helpful contribution right underneath:

GitHub comment from xiaobai-marker, with edited link so y’all don’t get any funny ideasHuh? It took this person only a few seconds to post a fix for quite a list of small changes that I posted. That’s so nice! So let’s do a review of their contributions.

Trail: GitHub

Their GitHub handle is xiaobai-marker. XiaoBai is (Pinyin writing of) Chinese and translates to “little white”, not referring to the actual color, but more in the direction of “unsullied”. It means Little Noob.

A brief dive into xiaobai-marker ‘s GitHub profile (by now mysteriously removed) confirms this initial impression.

GitHub profile of xiaobai-markerThey forked only three publicly available repositories. They apparently deal with self-driving toy cars and machine learning to detect road lanes. Fun! However, that’s not a lot of content for someone who seems to contribute to other projects, especially considering that they did not open any repositories themselves.

But we’re not judging. Any contributions are welcome, and that includes those from people with little experience. So let’s have a look at their actual contribution.

Trail: Download Link

MediaFire is a known file storage and sharing website. Clicking this link is not per se dangerous. Still, we proceed with caution.

MediaFire download pageMediaFire uses VirusTotal to scan for malicious files. This inspires confidence, right? An encrypted *.zip file, however, is opaque to any scans since we can’t look inside without the password, which was supplied on GitHub and is not known to MediaFire or VirusTotal. Another interesting detail is where the file comes from. Apparently it was uploaded the same day from somewhere in Russia. Is our Little Noob from Russia?

Let’s check out the files now.

Trail: Files

What did we actually download? A rather small zip file.

$ ls -lah fix.zip -rw-r—r— 1 ole users 367K Aug 27 08:15 fix.zipProviding the password, we can unpack it and have a look at the content.

$ 7z e -pchangeme fix.zip

Scanning the drive for archives: 1 file, 375771 bytes (367 KiB)

Extracting archive: fix.zip

Path = fix.zip Type = zip Physical Size = 375771

Everything is Ok

Folders: 1 Files: 2 Size: 747008 Compressed: 375771What did we just unpack?

$ ls -lah fix/ total 456K -rw-r—r— 1 ole users 490K Aug 27 08:15 msvcp140.dll -rw-r—r— 1 ole users 240K Aug 27 08:15 x86_64-w64-ranlib.exeHuh. ranlib is a standard programming tool, although here it is supposed to be an installer (!). This “fix” seems to have been written for a Windows system. I can’t run it on my Linux machine. Bummer. I’m also not very good at decompiling *.exe files to check what’s really inside. But we can ask some websites for assistance.

VirusTotal

First we upload the file to virustotal.com, a Google-owned file and link analyzer and one of the authorities in the space of malware indexing. On the day of the initial analysis, 28 of 75 security vendors flagged the files as malicious. You don’t say! One day later this number is up to 35. It seems to be a fresh strain that is quickly being picked up by the security community.

VirusTotal analysisSo what do we have here? “Trojan”? “MalwareX”? This is still quite vague.

Fun fact: One of the analyses — fourth row, right side — comes from CrowdStrike. Those are guys that crashed some 8.5 million rather important Windows computers, resulting in over 10.000 cancelled flights and causing some $15 billion damages worldwide in July 2024. Oops.

Tria.ge

Triage provides a malware analysis sandbox and provides reports for submitted samples. Some kind soul already uploaded the one we found.

Triage analysis summary“Lumma”? “Stealer”? What’s that? Apparently this malware tries to steal… things? Let’s look into this.

Some more interesting details can be found further down in the network section of the analysis. What’s a malware to do after it did its deeds? It has to send the stuff it found somewhere, right? So Triage ran the malware in a sandbox and recorded the network traffic. These are some of the connections the malware made:

Triage analysis network trafficfroytnewqowv.shop seems to be the URL of the Command-and-Control (C2) server of this malware. It tells the malware what to do and collects the bountiful harvest. Also take note of the froytnewqowv.shop/api path that we will explore in a minute.

Trail: C2 server

Let’s see if we can find out more about who our malware talks to. WHO IS the one that owns the weird domain name?

“WHOIS is a public database that houses the information collected when someone registers a domain name” — https://www.domain.com/blog/what-is-whois-and-how-is-it-used/

WhoIs

A whois look-up shows that the domain was registered just a few days ago via namecheap.com. This register hides the client’s contact information, like physical address or email.

whois lookup of the URL froytnewqowv.shopFurthermore, we see from the “Name Server” entries that they hide behind Cloudflare. This service can, among other things, act as an intermediary between a user and a website. This means that we cannot directly find out the IP address — the “actual address” that the URL is translated into — of the C2 server, since all the traffic is routed through Cloudflare’s machines first.

Little Noob is privacy-conscious.

Let’s give the C2 server a visit then. We open the website in a browser.

Note that it’s usually not a good idea to browse possibly malicious sites, since they can* infect your computer.

Browsing the C2 website

Opening the URL in a browser lets us see:

Visiting the C2 serverWell, that’s not much. This is the default landing page of their nginx web server, with no content whatsoever. Let’s have a look at the source code of this website then, by pressing CTRL+U in the browser.

<html> <head><title>405 Not Allowed</title></head> <body> <center><h1>405 Not Allowed</h1></center> <hr><center>nginx/1.18.0 (Ubuntu)</center> </body> </html>Ah yes. Privacy conscious. At least we learn that the server runs nginx version 1.18.0 on Ubuntu.

If we look back at the network section of the Triage analysis, we see that the malware made a request to the /api path. API stands for Application Programming Interface **and is basically a somewhat standardized interface for talking to computers programmatically. What does that API path look like in the browser?

API pathI guess I’m not invited to that party. Let’s explore the website a bit more and guess some other paths.

404 pageNope. Not that easy, apparently. 404 is browser-speak for “I can’t find what you asked of me”. Let’s see what Google knows about this URL.

Google search for the C2 URLAha. Another path: /apib. Where does this lead to?

A different 404 page? Hmm…Another dead end? But this one look different. Russian again. How come? After trying some more URLs by hand, it is apparent that everything that starts with /api is leading to this other 404-not-found page:

/apia
/apib
/api_I_misconfigured_my_nginx_server
etc.

Let’s see what we can learn from this page. Translation:

404** Oops… There was a mistake

Sorry, this page was not found or the administrator has closed access to it

<- To the main pageNo surprises here. Let’s download the whole thing and see what we get.

Website source code

The website source contains the following files:

$ tree . ├── Ошибка_files │ ├── dober.html │ ├── doberman.min.html │ ├── jquery-3.6.0.min.js │ ├── tabler-flags.min.css │ ├── tabler-icons.min.css │ ├── tabler.min.css │ ├── tabler.min.js │ ├── tabler-payments.min.css │ ├── tabler-vendors.min.css │ ├── tom-select.complete.min.js │ └── tom-select.css └── Ошибка.htmlOur error page seems to have been built with tabler.io. This seems to be a bit too much effort for a simple 404 page. Tabler is often used to build elaborate dashboards. So there’s probably more to it.

The css files are Cascading Style Sheets (CSS), which are used for styling a web page. Not too interesting. [tom-select](https://tom-select.js.org/) is a standard JavaScript framework for UI control. [jquery](https://jquery.com/) is a library to simplify HTML development, assisting with document traversal and other things.

The interesting ones that remain are dober.html and doberman.min.html. What are those? Let’s have a look.

The content of dober.html is one single line of text which looks like this:

<!DOCTYPE html><html lang=“en-US”><head><title>Just a moment…</title><meta http-equiv=“Content-Type” content=“text/html; charset=UTF-8”><meta http-equiv=“X-UA-Compatible” content=“IE=Edge”><meta name=“robots” content=“noindex,nofollow”><meta name=“viewport” content=“width=device-width,initial-scale=1”><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131}button,html{font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji}@media (prefers-color-sch…Urgh. This HTML vomit is “minified”, which means it is condensed as much as possible to save bandwidth when opening a website. This makes it hard to read for us humans, though. Luckily we can get some help with disentangling it.

Unminify.com

While there are Integrated Development Environments (IDEs) that can disentangle minified HTML, there are also websites that do this for you with a click of a button.

Unminifying HTML code, making it readable and prettyThe code in the screenshot above is now much more readable. Below in the unminified code of doberman.min.html we can find a reference to a file that does not seem to exist. An excerpt of doberman.min.html:

We also noticed the absence of the file “doberman.min.js” on new Lumma C2 panel […]

The full report is a fascinating read. Here’s the summary:

Various actors actively deploying Lumma Stealer in multiple campaigns**

Lumma Stealer, also known as LummaC2 Stealer, is a malware-as-a-service sold through Telegram and Russian-speaking cybercrime forums. […] Lumma is an information stealer written in C/C++ language that has been observed in the wild since at least August 2022. […] This malware is designed to pilfer sensitive data from infected devices. Among the data targeted are cryptocurrency wallets, browser extensions, two-factor authentication codes, logins and passwords stored on browsers, and various files.

Trail: Business Case

Outpost24 explored the malware in detail, including their store. Joining the dark side starts sets you back $250, or a bit more if you aim for best-value. Granted, the chance of someone falling for this weak threat is low. But if the net is cast wide enough…? Imagine just a few victims falling for the malware, finding just one juicy crypto-wallet. The amount of potential profit per dollar is staggering, as is the damage caused.

Lumma Stealer store page, translated from Russian — by Outpost24

Takeaways

As Research Software Engineers we don’t often think about cybersecurity. Our institute, the Netherlands eScience Center, is an independent foundation and not a company with valuable trade secrets. We don’t usually work in projects that are high-profile targets. Nevertheless, people who write code are increasingly targeted by cyber-criminals. Do you remember the drama around the program [xz utils](https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/) in March 2024? A user infiltrated a widely used open-source project, earned trust by making meaningful contributions, and finally injected a malicious backdoor into the source code. It was caught just in time before it was distributed widely.

The case discussed here is a much cruder approach, amateurish even. Malware-as-a-service, like the Lumma Stealer, makes malware accessible for the masses, enabling this spray-and-pray approach. We can expect its use to only increase in time. It shows that all of us can be targets, at any time.

Stay vigilant.

Cross post: https://ole.mn/posts/a-helpful-noob/

Fast-Forward Merges on GitHub

ole — Fri, 21 Feb 2025 00:00:00 GMT

The year is 2025 CE and GitHub.com still can’t do “fast-forward” merges via its web-interface. But you can; I’ll show you how.

Photo by Joakim Honkasalo on Unsplash Github.com offers version-control-as-a-service, with all kinds of useful things bolted on top. The year is 2025 CE and GitHub leverages the vast power of git and exposes it entirely to their users. Well, not entirely… One small feature of indomitable usefulness still holds out against the invaders. Well OK. Here the stories diverge.

One git feature that is not available via the github.com web interface are “fast-forward (ff) merges”. It is among the highest git-related feature requests with about 1500 upvotes.¹ To explain what that is and why one would want this, we have to take a detour first.

Intermezzo: Merge Types

To set the ground for explaining fast-forward merging, I’ll cherry-pick (chuckle) a specific scenario.

Let’s follow best practices where we have our main branch protected, meaning that we cannot add changes to it directly. Any changes that should end up in main will be merged via pull- (GitHub speak) or merge- (same, but GitLab speak) requests. That way there can be some auditing before code ends up in the main branch. This could prevent minor hiccups like the CrowdStrike-related IT outages.

nixCraft having a blast at CrowdstrikeGranted, I don’t think code pushed to main was the cause for the global-scale IT outage, but it would have been funny if it was – however funny ~$10 billion in damages can be.

A bit more opinionated than protecting main is the choice to keep the git commit history linear. With “commit” we mean a snapshot in time – a set of related changes to your code with a specific description – so you can find it back later. Reasons for a linear git history could be that you follow the Git V Branching Model or if you want to have Abel’s Utopic Git History. In general, git commit graphs can look a bit complicated.

git commit graph: Guitar Hero 🎸 git commit graph: Harp 🎵 git commit graph: London Underground 🚇 (/s)

Branches

To compartmentalize development tasks and to keep the repository tidy, we often use “branches”. These are alternative realities of your code-base, and you can jump back and forth between them like Dr. Strange.

Instead of the mess above, let’s simplify our git commit history such that it looks like a straight string of pearls. In the following visualizations, time progresses towards the right. The horizontal lines represent branches, and the nodes on them are commits with a certain commit message. They also have a specific fingerprint, called “hash”, that uniquely identifies them. We will come back to that later.

When we work on a feature or a bugfix (or anything, really) we usually do that on a separate branch, nicely separated. After we finished working on this branch, we want to somehow get the changes we made back onto the main branch. There are different approaches for that.

Merge Commit

“Standard” merge commits leave the history where exactly the code came from intact. An extra “merge commit” contains the changes from our feature branch.

Rebase

Another way to keep the git history linear is “rebasing”. This approach basically re-plays the new commits that happened on the feature branch onto the main branch. This means that there is no “merge commit” on the main branch anymore. However, if there have been commits to main after feature has been branched off, the feature commits are re-written. Do you remember the “hash” fingerprint, I mentioned before? Here it becomes important. When rebasing, the commit content and commit messages are the same, but since these are effectively “new” commits, the commit hashes are different! This means that anything that was attached to a certain hash, like signatures or tags, will point to the commits on the feature branch. To demonstrate this, we attached a tag v1.0.0 to one commit on the feature branch.

Do you see how the tag is kept dangling on the now useless feature branch?

Fast-forward Merging

“Fast-forward” merging can be achieved if main did not receive any commits in the meantime. So clean! Instead of re-writing the commits onto another branch, the main and feature commits are basically pointers to the exact same commit.

This looks very similar to rebasing, does it not?

In fact, if the main branch did not receive any new commits since the branching, fast-forward merging is the same as rebasing…

GitHub Rebasing

…unless you are using the GitHub web interface. When you rebase a feature branch on GitHub onto an untouched main branch, the commits are still re-written. This is contrary to how git on the command line works. Why is that such a big deal? The commit messages are the same. The commit content is the same. Why worry about a different commit hash?

It matters, because lots of things are tied to a commit hash. A commit signature is tied to its hash! The signature gets lost if the hash changes during a rebase. The same holds true for tags. feature commit tags will point to the commits on the feature branch, not to the re-written ones on main.

Solutions

There are two ways to achieve fast-forward merging when using GitHub. One is convoluted and error prone. The other one is cursed. Choose wisely.

Using Local `git`

Protecting your main branch means that you can’t commit to it directly. You can, however, push sanctioned changes from main to your remote repository on github.com. Wait, what? I thought you can’t touch main? Follow me, I’ll show you.

Create a pull request from feature to main on github.com
Open a terminal and create some magic:

cd your/project/folder # cd into your project directory git pull # get the latest changes to your local computer git switch main # select the main branch, if necessary git merge —ff-only feature # ff-merge feature onto main git push # allowed! because you created a pull request

The pull request is automatically closed 🪄

While rather low-tech, this relies on the workflow and discipline of the developers to work. This is a danger zone. It is 4:55pm on a Friday afternoon. You opened a pull request to contribute your work on the feature branch and now watch patiently how the green circle closes that shows that all CI tests passed. Your fingers itch to wrap things up quickly so you can go home and wolf down the lasagna you prepared already the day before.

The mouse cursor hovers over the fat, green “Merge pull request” button.

Don’t. Don’t do it.Stop! Resist the temptation!**

Click that button and you need to spend the next half hour undoing the rebased, re-written commits to main, while sweating profusely because now you are messing directly with the main branch, which you tried to prevent by protecting it in the first place. And by now, at home, your partner has finished both their and your piece of the lasagna.

Instead of following the suggested path, you need to step away from the browser and open the terminal. You pull the changes, fast-forward merge locally and finally push to main, as shown above.

Using Actions

This approach needs some set-up, but is safer to use afterwards.

Disable the Merge Button

We have learned now that the fat, green “Merge” button is dangerous², so we need to disable it. In your GitHub repository, go to

Settings → General → Pull Requests*

and allow only Squash Merging. Next, go to

Settings → Rules → Rulesets

and add a new rule. Name it disable_merge_button, set it to “Active”. Set the “Target branches” to the “default branch” a.k.a main . Check “Require a pull request before merging” and, while waving a magic wand, set “Allowed merge methods” to “Rebase”, contradicting the general repository settings. Check whichever other settings you would like and click on “Save changes”.

Now pull requests cannot be merged anymore. Yay?

Automate the Wizardry

Set up the “Fast Forward Merge” Action from the GitHub Marketplace: https://github.com/marketplace/actions/fast-forward-merge . This gives you two things:

A message posted in the PR chat indicating whether fast-forward merging is possible.
A GitHub Action that fast-forward merges your Pull Request. You trigger it by commenting /fast-forward in the Pull Request chat.

N.B.: For the triggered ff-merge to work, the Action yml file has to already exist in the main branch. So you have to get it there once via impure means.

It works. Yet it feels dirty. You just made a forbidden deal with C̶t̶h̶u̶l̶h̶u̶ Octocat and you will probably need to sacrifice the lasagna you made so it does not come to haunt you at night.

Wrap-Up

By now you not only know what fast-forward merging is, you also have ways to use it on your external repository on github.com. You might have subscribed to github.com/orgs/community/discussions/4618 and now patiently wait for the powers-that-be to enable this feature on their website.

Until that happens, feel free to choose one of the workarounds outlined above.

Upvote count combines upvotes and thumbs-up from https://github.com/orgs/community/discussions/4618
If you want to achieve fast-forward merging, that is.

Cross post: https://ole.mn/posts/fast-forward-github/

FAIR principles: 5000 years of common sense

Pablo Rodríguez-Sánchez — Thu, 20 Feb 2025 00:00:00 GMT

The FAIR acronym stands for Findable, Accessible, Interoperable and Reusable. If you are a researcher, certainly you’ve heard about FAIR principles. FAIR data, FAIR software, … It may sound as another extra, new responsibility to take care of. But, is it? We argue it is not extra, and most importantly, it’s definitely not new. Come with us on a journey through the history of knowledge sharing.

By Javier de la Cueva, with Pablo Rodríguez-Sánchez as coauthor.

Photo by Paul Melki on Unsplash

The seminal paper where the FAIR guiding principles were stated was published in the year 2016. If you are a researcher, most certainly you would have heard about FAIR principles, FAIR data or FAIR research software.

The backronym FAIR stands for Findable, Accessible, Interoperable and Reusable. In many forums, FAIR principles are understood as a new added responsibility to take care of by science and data practitioners. Contrary to this idea, we argue that to comply with the FAIR guiding principles is not a novel requirement. Indeed, the ideas behind the FAIR guiding principles are no less than 5000 years old, and can be traced back to the first libraries.

If a name is an acronym, it is likely to be a buzzword. According to Fredric Jameson, postmodernism is characterized by a permanent rebranding of old concepts so they may be commercialized again as new. And, indeed, FAIR is an acronym. It is often applied to research data, research software and even, although less often, to research publications. We argue that, contrary to other acronyms, FAIR is here to stay. Why? Because it is not a rebranding but the logical evolution in the digital realm of the principles that governed the analogical archives.

A FAIR library

Let’s forget about the digital world for a moment and take a walk to the neighborhood’s library. There we are, surrounded by elderly people quietly reading their newspapers, kids attending read-aloud sessions, students… well, studying, and all kind of people borrowing and returning books.

Let’s take a look around with the FAIR principles in mind.

Findable

The materials in the library are certainly findable. A good library contains thousands of volumes, so they better have a system in place. This is as true today as it was in the library of Alexandria. Some of us still remember the times when this system was a set of cardboard boxes. Nowadays, the system is usually digital. You can even browse it from home!

Accessible

Accessibility is also a trademark of public libraries. The building itself usually can be freely accessed by anyone, and memberships are free or, worst case scenario, very affordable.

Furthermore, many libraries have sections with materials for users with special needs. Audiobooks, books typed in Braille or in extra large fonts or simplified-language versions.

Interoperable

Standardized fonts are one of those everyday wonders we don’t even realize anymore. Can you imagine how hard would it be to read if all the content was hand-written, with the unique scripting style of each author?

Even more, think of word separation. Believe it or not, thefactthatwedonotwritelikethis is a relatively late development. We take spaces between written words for granted, but they are an example of technology.

And of course, translations allow readers to operate with books originally written in languages unknown to them.

Reusable

How do you reuse a book’s content? Well, by using the most quintessential library service: borrow and return.

Citing a book is also a form of reusablity. In some cases, you can even go further than citing and create derivatives from it. Think of for instance of encyclopediae and dictionaries. Those are composed works, with many authors and, to a large extent, independent entries. They are, by definition, work in progress.

Conclusion

The attentive reader certainly noticed that some of the features of books and libraries could fit in more than one FAIR concepts. For instance, are standardized fonts improving accessibility or interoperability? What about translations? Please don’t take our examples as a matter of rigorous taxonomy, but stick to the spirit of the idea.

When looking back in history, it is tempting to stop too early back in time. Instead of going back only 20 years, let’s explore a bit deeper. Let’s not fall victims of the palimpsestic syndrome, i.e.*: attributing the creation of a concept to the first person or institution you heard it from. All of the FAIR principles, with minor adjustments, have been present since literally millenia ago in the world of books.

We also need the FAIR principles to fight the reproducibility crisis that is taking place in science as we speak. And keep in mind that reproducibility is not only desirable, but a defining feature of scientific knowledge.

So no, FAIR principles are not new. And no, FAIR principles are not optional. Not bad for a buzzword!

Acknowledgments

We want to express our gratitude to Lieke de Boer and Carlos Martínez-Ortiz for their excellent feedback.

Reproducible research with The Turing Way

Carlos Martinez-Ortiz — Mon, 10 Feb 2025 00:00:00 GMT

The Netherlands eScience Center and Turing Way have long been collaborating on the common goal to improve reproducible research practices

A long time ago, in the far-away Amsterdam Science Park, the eScience Center created their own “eScience Center software development guide”. The aim of The Guide was for our RSEs to have a starting point to learn about how to write research software in adherence to open science practices. In other words, reproducible research software.

After a few years, we came across The Turing Way, and saw that there was a lot of overlap between our goals: we both wanted to share good practices in reproducible research in general, and reproducible research software specifically.

Image by ScriberiaIn 2020, we decided to migrate our content to The Turing Way, with a few exceptions. Our rule of thumb is: all content which is interesting to a general audience should be in The Turing Way and; content which is eScience Center specific, should go in The Guide. For instance, the description of why one should use a license goes in The Turing Way, whereas our decision to use Apache2, goes in The Guide.

We used to block time to work on The Guide (“guide sprints”), which was essentially the same concept as the “Book Dash” events. We decided, in the spirit of open science, to NOT duplicate efforts, but instead dedicate our efforts in contributing to The Turing Way. For a few years now, we have been (together with colleagues at VU Amsterdam and TU/Delft) involved in organizing local Book Dash events.

At a personal level, being involved in the project has allowed me to learn a lot about how to contribute to an open science community. Specifically, a community that works on a project based on Open Source principles — and you don’t have to be a skilled developer to contribute. I have also met many knowledgeable inspiring people! I have felt really welcome as part of the community.

For the last couple of years, I have also been involved in the Book Dash working group. In this group, I am involved in planning the Book Dash events, and try to encourage the (remote) participation of the Dutch community.

Image by Scriberia The Turing Way* has become an important reference resource for our internal use and as part of the resources we share with the research community in the Netherlands.

Beyond the multiple times on this blog, The Turing Way is mentioned in: our internal Project Management Protocol and our internal training resources, our Research Software Support training materials, our Practical Guide to Software Management Plans and training materials developed in our Best Practices for Sustainable software project point to different chapters as reference resources. It will also be part of the reference resources for the EVERSE Research Software Quality kit.

For the eScience Center,* The Turing Way *is a priceless resource for sharing our expertise in contributing to reproducible research software.

Has The Turing Way helped you in your work? Please share your story here. If you are interested in joining a Book Dash, you can contact Carlos or get in touch with The Turing Way

A Different Game

Carsten Schnober — Tue, 28 Jan 2025 00:00:00 GMT

According to a panel of football experts, the world’s best male player of the year 2024 was Rodri. The choice is naturally debatable, but even if you put him together with other stars like Erling Haaland and Kylian Mbappé, they will not stand a chance in a match against a full team of 11 amateurs. The world stars could perform great dribblings and accurate passes, but they cannot play the team sport called football.

To build complex software, highly skilled programmers are essential. But only effective collaboration enables them to achieve big things in the world of modern software engineering.

Photo by Ruben Leija on Unsplash

From Programmer to Software Engineer

At the Netherlands eScience Center, we do not play football, but we develop research software. We also teach courses and workshops, including Collaborative Software Development. Compared to workshops about Deep Learning or GPU Programming, tools like Git, GitHub or GitLab may not seem very advanced. Nevertheless, at the end of some workshops, some participants almost refuse to leave the room, enthusiastically finishing the collaborative setup for a project. Also, I vividly remember how a partner of ours spontaneously mentioned her most important learning while working with us: “I learned to cooperate with others!”

The impact that the relatively little technical learnings have on the personal progress seems almost unreasonably large. That is because we do not just teach how to use the Git ecosystem, but how to collaborate. Mastering the tools gives a glimpse at another level of software development that may have seemed unreachable for many programmers — like a lonely football player that has practised his or her skills on the ball and suddenly finds themselves in a well-organized team of players.

It is not a coincidence that Git was invented by the initiator of the most prominent open-source software. With his initial versions of Linux, Linus Torvalds made a great start. However, without good tooling, collaboration causes friction. To use Linus’ own words:

We were in this bad spot where we had thousands of people who wanted to participate, but in many ways I was the break point.

On his own, Linus could never have developed an operating system that now runs on millions of computers, including high-performance clusters, desktop PCs, laptops, phones, embedded systems and more. Seamless collaboration is the key:

The big point for me was not being alone and having ten, maybe 100 people being involved. […] Going from 100 people to a million people is not a big deal.

The open-source software community turned Linux into high quality software in which bugs are fixed at a speed that software companies could only dream of, and new features come in by the day.

A graph showing contributions to a Git repository.Epiphany moments emerge when individual software engineers realize — for instance during a workshop — that they can reach that new level of software engineering. In a collaborative setting, tools and methods like version control, automated testing, linting, and continuous integration are no longer merely aesthetic practices. Not surprisingly: the rules of the game make sense once you play it.

Incentives?

Software engineers working in the industry often pride themselves on their practical engineering skills. Partly understandably, because even Computer Science PhDs, hence highly skilled individuals, can seem like novices when they join a team and don’t know how to collaborate with them. This is, however, a comparison between players of different games: individual programming and collaborative software development, respectively.

For a company, the ability to develop software collaboratively is essential to survive after the start-up stage. If they do not adopt methods to join their forces effectively, their software engineers drown in a swamp of ever-increasing maintenance efforts and dependencies on individualized knowledge.

Reality Check

So why is collaborative software development not the default method? One reason is that collaborative methods imply some overhead: developers need to invest time in learning to apply the tools — which only pays off in a team that thinks beyond short-term goals.

But collaborative development is more than tooling: it also requires open communication and mutual trust, which often comes into conflict with hierarchies and individualized reward systems. Each member of a team needs to find intrinsic motivation and confidence that they can contribute something meaningful.

Collaborative development principles often compete with existing incentives. In academia, software may be necessary to run an experiment, but is frequently abandoned as soon as the corresponding paper is published. Terms like “project-ware”, “PhD-ware” and “professor-ware” are typically not used in a positive way, but the one-off approach concurs with the fact that individual performance is measured by the number of publications.

Developing software so that other researchers can extend it for their own experiments brings no benefit — in a highly competitive environment, the opposite might even be the case. In the football analogy: a striker that is measured by how well they can shoot penalties does not benefit from practicing combination play with the team.

The same applies in industry settings, even though playing the team game is more fundamental. Anyway, developers are hired, promoted and fired based on their individual performances. An engineer who quickly drops a new feature into production is more likely to be perceived as a mythical 10x engineer* than their peer who makes sure it still works after the next update, for instance by implementing integration tests.

Agile, Waterfall and Hierarchies

How to tackle the complexities of software development has been a hot topic for decades. The Waterfall approach tries to plan a project as a sequence of clearly defined steps that eventually lead to the finish line. In the reality of software engineering, however, that finish line tends to be moving, while the intermediate steps face unforeseen obstacles and complexities that cannot be fully grasped by an individual engineer.

To account for the inherent dynamics of software development, Agile methods have proposed that planning has to be refined iteratively, involving continuous interaction between all stakeholders — requiring effective collaboration on various levels.

Collaboration is more than learning how to use Git. *Photo by Lisamarie Babik — Ted & Ian. Uploaded by Edward, CC BY 2.0, *https://commons.wikimedia.org/w/index.php?curid=9546406However, the reality of organizational hierarchies, project-based budgeting and purely individual accountability conflicts with the radical changes that are required to have teams that can refine their (intermediate) goals autonomously. In practice, Agile therefore often means not much more than introducing a level of fancy terminology and rituals.

To the Next Level

In any organization, collaboration remains a trade-off between immediate personal advancement on the one hand, and contributing to sustainable, collective progress on the other hand. The feedback that we get at Collaborative Software Development workshops shows that many people like the team game: it allows them to shine as part of a team that is able to build things far bigger than what any member could achieve on their own.

The topic of collaborative software development might be so fascinating because it touches upon many issues beyond engineering. Like forming an effective football team, software engineering is a social process that requires technical skills, but can also lead to questions about the structure of an entire organization.

How to run a containerized Agent-Based Model on HPC

Olga Lyashevska — Thu, 23 Jan 2025 00:00:00 GMT

This blog was written by Faruk Diblen and Olga Lyashevska.

Imagine you are preparing Torta Caprese for your friend’s birthday. You have a recipe that consistently delights everyone, but you want to ensure it turns out perfectly every time, regardless of the oven you are baking in or ambient temperature. To achieve this, you always use a cake tin for the batter, which gives its form. In the realm of software development, a similar concept exists — the container. Much like its culinary counterpart, a software container offers a lightweight, consistent and isolated environment for your application, just like the cake tin. The beauty of using a software container is that all the necessary ingredients and instructions are neatly packaged together, including code, dependencies, and runtime environment. This ensures that the application runs consistently and reliably across different computing environments and operating systems.

Illustration: generated with MidjourneyIntroduction**

In this blog, we provide a step-by-step guide on running a containerized Agent Based Model (ABM) on High-Performance Computing (HPC) systems. We will explain how to package code and all dependencies in a container and execute on HPC in an easy way. As an example of ABM, we picked NetLogo, a widely popular software among users who may not be technically inclined.

NetLogo is a multi-agent programmable modeling environment. It is used by tens of thousands of students, teachers and researchers worldwide.

The steps outlined below are not limited to NetLogo or ABMs; you can follow this guide to run any application in a containerized format on HPC systems. So why might we want to run NetLogo on HPC? Let’s go back to our baker analogy.

HPC and its significance in handling complex computational tasks

Let’s imagine that your Torta Caprese has gained immense popularity, and now you are asked to bake it for a friend’s wedding. However, this time, you are not using your humble home oven (desktop computer); instead, you have access to a state-of-the-art industrial oven capable of efficiency and speed. This industrial oven represents HPC systems. HPC systems are known for their ability to process large volumes of data and complex computations at high speed. Do you want to risk any variation in the baking process? No. You are equally meticulous about the execution of your application on the HPC infrastructure. This is where the concept of containerization comes into the picture. By utilizing containers on HPC systems, you ensure that your application runs efficiently and consistently, regardless of the computing environment it is deployed.

Back to NetLogo. Imagine you have a NetLogo model that has grown too complex and too slow to be run efficiently on your desktop and you decide to run it on HPC system. You have access to the HPC system via secure shell (SSH), but you do not have the necessary permissions to install NetLogo on the system or you do not know how to do it. Perhaps, you may even want to run the model without GUI (Graphical User Interface) because it avoids overhead and makes your model run faster.

The purpose of this blog is to demonstrate how to run a single Netlogo model on HPC using Apptainer container. We use the Netlogo distribution as a base for the container. Apptainer allows us to package the model and its dependencies (i.e. Netlogo) in a single image that can be run uniformly across different systems. Apptainer is particularly suitable for HPC, as it is designed to bring containers and reproducibility to the scientific community and does not require admin access to run. The container is then used to run the Netlogo model in headless mode. Headless mode is a mode in which the model is run without the GUI and the output is saved in a csv file.

Let’s follow the next steps together. But first, you need to make sure that you have the following installed:

Apptainer 1.2.5 or newer
a Linux OS with sudo rights, any system will do
shell, any shell will do
access to an HPC system via SSH

Now we are going to follow these steps to run your model within a container on HPC:

Step 1. Create a definition file

First, we create a netlogo-headless.def file with the following content:

Step 2. Build a container

We are going to build an apptainer container on your local machine from a definition file that we have just created in Step 1. For this, a Linux system with sudo rights is needed.

This command will create an apptainer image netlogo-headless.sif which we will use to run the model in headless mode on HPC.

Step 3. Prepare job.sh file

We need to create a shell script that contains the command to run the model in headless mode. First, we look at how to run the model in headless mode without a container.

./netlogo-headless.sh — model “PATH_TO_MODEL” — experiment “EXPERIMENT_NAME” — table OUTPUT.CSVThe line of code above will execute a netlogo-headless.sh command, where

— model is an argument that specifies the NetLogo model to be run;
— experiment argument is optional and can be used to specify the name of the experiment;
— table argument is optional and can be used to specify the name of the output file.

apptainer exec netlogo-headless.sif netlogo-headless.sh — model “PATH_TO_MODEL” — experiment “EXPERIMENT_NAME” — table OUTPUT.CSVFor example, to run a model Wolf Sheep Simple 5.nlogo with an experiment Wolf Sheep Simple model analysis and save the output in wolf_sheep_output.csv, use the following command.

We save the code above as job.sh and change permissions to make it executable with

chmod +x job.shStep 3. Create an archive to be copied to HPC*

To run our model on HPC, we need to copy all files to the remote machine. First, we create an archive on a local machine that contains the model, the container, and the shell script. For this, we create a directory and copy model, netlogo-headless.sif, and job.sh into it and call this directory jobHPC. Then we create a gzip tarball of a directory.

tar -czvf jobHPC.tar.gz jobHPCStep 4. Copy to HPC

To copy jobHPC.tar.gz file from your local machine to HPC cluster, we can use the scp command. This command will copy jobHPC.tar.gz file to the remote machine using SSH (Secure Shell) protocol for data transfer.

scp jobHPC.tar.gz username@remote:/path/to/remote/directoryReplace username@remote with your username and the address of the HPC cluster. Replace /path/to/remote/directory with the path to the directory on the HPC cluster where you want to copy the file. This command will prompt you for your password on the HPC cluster before copying the file.

Step 5. Unpack the archive on HPC

Once the file is copied to the HPC cluster, log into the remote machine and unpack the archive using the following command:

tar -xzvf jobHPC.tar.gzStep 6. Run model on HPC

Submit a job to a (SLURM) scheduler on HPC using the following command:

Here, we set a limit of 2 hours for the job to run. You can change the time limit as per your requirements.

Now you can pull up information about the jobs in the queue:

squeue —user=your-usernameWe use here the --userflag because we only want to see our job.

You can stop your job with

scancel your_job-idStep 7. Retrieve results

Once finished you can retrieve your results. For this, you need to identify the files you need to retrieve. To copy the files to your local machine run:

scp username@remote:/path/to/remote/file /path/to/local/destinationReplace /path/to/remote/file with the path to the file on the HPC system, and /path/to/local/destination with the path where you want to save the file on your local machine.

For example, to copy job_output.txt from the HPC system to your desktop:

scp user@hpc.example.com:/home/user/job_output.txt ~/Desktop/Finally, if you want to copy multiple files, you need to add flag --r which will allow you to copy all files in the directory job_output.

scp -r user@hpc.example.com:/home/user/job_output/ ~/Desktop/

Photo by Lightsaber Collection on Unsplash

Conclusion**

To wrap up our containerization journey, we have covered some solid ground in getting your NetLogo models up and running on HPC systems. Containerization, much like the cake tin, provides a reliable and consistent environment for your applications, ensuring they perform as expected regardless of the computing setup. From creating our container image using Apptainer to running the job scripts to transfer files to the HPC cluster and running job, we have laid out all the steps. Whether you are diving into ABM or tackling real-world problems, containerization on HPC enables you to handle complex computational tasks.

Now, go ahead and make a Torta Caprese for yourself to celebrate that you learned something new! Here is a recipe.

Image: https://unsplash.com

My experience of getting codechecked

eScience Editorial Team — Thu, 16 Jan 2025 00:00:00 GMT

*By eScience Fellow *Eduard Klapwijk

This year I was finalizing and publishing one of my own research projects. My coauthors and I took many steps to make the project as reproducible as possible.

Because I am setting up a codecheck workflow to check the reproducibility of research results for researchers at my institute, I was eager to submit my own work to the test. How reproducible would our work be when submitted to an external test?

Photo by henry perks on Unsplash

A check for reproducibility

I submitted my work to the CODECHECK (Nüst and Eglen 2021) community workflow, which means that an external codechecker will do a light-weight reproducibility check. They will try to re-run my code to see if results reported in the paper can be reproduced.

A codecheck was the perfect opportunity for checking how reproducible our project was. At the same time, the experience of getting my own work codechecked would help in understanding a researcher’s perspective on the process. In this post, I will share my personal experience of getting codechecked. This will hopefully demystify the process for any reader new to codechecking or curious about what codecheck is all about.

Image: The CODECHECK example workflow implementation.

The project: neuroUp

The project submitted for the codecheck was in the making for about four years — a nice example of Slow Science as per Uta Frith’s analogy. From the start of these four years, I tried to make all our scripts reproducible.

A few years ago, we started out with a couple of unconnected R scripts. Only when I started drafting the research article, I combined the draft manuscript with the project’s code and computations in one place. I first used R Markdown for this, and in the end of 2023 I switched to a similar tool now available in RStudio called Quarto. On top of this, a key tool for reproducibility was the use of Git and GitHub for version control. This helped to systematically track the evolution of the project and its code and allowed us to go back to a previous state of the code if needed.

An important phase in the project was the step from working with loosely connected R scripts to the development of an R package (we called it neuroUp). With the help of the fantastic R packages book by Hadley Wickham and Jennifer Bryan, I managed to develop my first R package out of the code for this paper. Although after having done this, I believe that for most research projects developing a custom software package is an overkill, organizing code in a package does provide a lot of benefits that make your code much more reproducible. It organizes your project according to a convention, specifies dependencies, and bundles data, code, and documentation (Marwick, Boettiger, and Mullen 2018). This means other researchers can install, cite and use the package without having to understand exactly how your code works.

When we posted the preprint for our paper, we also published a reproducible version of the manuscript online. That was the moment that we turned to the CODECHECK community workflow to check whether results were indeed reproducible, at least according to one independent codechecker who was not part of the research team.

Submitting work to be codechecked

I chose CODECHECK because they offer a journal-independent community workflow. To be honest, submitting the work was daunting at first. I hesitated because based on the instructions for authors, I thought I had to partly restructure my project and provide a new README and metadata file. Luckily, looking at some of the forked repositories in the codecheckers GitHub organization really helped. That made me realize that some were much more detailed than others and that I could suffice with just describing the files in the repository and instructions on how to run the code (I liked this example by Bjørn Bartholdy and based my README on it).

After the preparations, the procedure was straightforward. Using my GitHub account, I opened an issue on the codecheckers GitHub, which is made easy using their template issue. The only thing that confused me a little was the instruction to “link to the repository in the codecheckers organisation with the code”. Once I figured that I could also link to my own public repository that would be forked by the codechecker in the process, my project was ready to go!

Getting the results of the check

After a short waiting time, I got notified with some good news: the checker was able to reproduce my work and we got a certificate (see Röseler 2024)! Of course, I was very happy. This meant that at least one other person at another moment in time was able to reproduce our work with the help of our data and code.

But, as I also experienced when I later code-checked someone else’s work (Joey Tang et al. 2024), when everything is working smoothly (eventually maybe after debugging some dependencies), the whole process is not overly exciting. The machine is doing most of the work in that case, you just see a bunch of identical images or tables appearing on your screen.

The kind of reproducibility that is checked (computational or methods reproducibility) is in the end actually quite basic. What is being checked is nothing more than a minimum but necessary condition of good and believable research: what is verified is whether the results can be independently re-run using the same data and code. Other relating concepts for reliable and trustworthy research results such as replicability, generalizibility, and robustness, may be more exciting, but require also much more effort or even new studies to check.

Final reflections

As with most things in life, preparing for the codecheck (by making sure your work is reproducible) is most of the work. Here are some key points to consider (see also these ten simple rules for reproducible research: Sandve et al. 2013):

Record all the steps taken to process and analyze your data in scripts and avoid manual steps that are harder to reproduce.
Documenting your code is key. Explain what steps are needed to reproduce the results (see our README).
Make sure you systematically list all dependencies (software and packages).
Use tools for version control.
Use tools like Quarto, R markdown, or Jupyter notebooks. They provide a great way to share code, narrative text, and output in one document, making it much easier to clearly describe the steps that were taken to process and analyse the data.
Partial reproducibility is always better than zero reproducibility! Don’t let the perfect be the enemy of the good.

In conclusion, getting our research codechecked was a very nice experience. With just a few extra steps (providing a short metadata file, adapting our README, requesting the check), we managed to obtain a nice certificate confirming that the computations underlying our article could be independently executed. I believe this additional check — next to traditional peer review — gives a nice extra boost to the credibility of our work. We therefore happily cited the CODECHECK certificate in our paper that got published recently (see Klapwijk et al. 2025).

Did you work hard to make your research reproducible? Then definitely have it codechecked!

Acknowledgments

I would like to thank Lieke de Boer for excellent feedback on a draft version of this post.

This post was initially published here. For attribution, please cite this work as:

Klapwijk, Eduard T. 2025. “My Experience of Getting Codechecked.” January 16, 2025. https://doi.org/10.5281/zenodo.14651307.

Out of office: holiday mode activated

eScience Editorial Team — Fri, 20 Dec 2024 00:00:00 GMT

As the year winds down, the Netherlands eScience Center editorial team will be taking a short hiatus for the holidays to recharge and prepare for an exciting year ahead. While we’re away, we’ve curated a list of thought-provoking blog posts for you to explore. From open science innovations to the latest in data-driven research, these articles will keep you inspired. Wishing you a joyful holiday season and see you in the new year!

Blogs to read while we’re away:

Beyond Research

A Helpful Noob

What Research Software Engineers can learn from Taylor Swift

(Un)wieldy workflows with WRF

Amsterdam as seen by a weather model

Peter Kalverla — Tue, 10 Dec 2024 00:00:00 GMT

Visualizing local climate zones in 3D with the Cityblocks package

What sets a weather model apart from a video game? While video games prioritize stunning visuals over strict realism, weather models focus entirely on physics. Visualizations come as an afterthought at best. A missed opportunity, since good visualizations can reveal fascinating insights into how these models interpret the world. In this blogpost we bring Amsterdam to life in 3D through the eyes of a weather model — and invite you to try it for your city as well.

Weather models have a peculiar view of a city. To them, the world is made of boxes. Many roughly rectangular boxes. A typical grid box contains more than one building, so weather models see cities at a block level.

Weather and climate models divide the earth in grid cells. Each grid cell encompasses a wealth of information and processes, which must be represented in a simplified form. Source: wikipedia

To describe cities at block level, one option is to use local climate zones (LCZs): a set of typical building blocks modeled after cities across the globe. Each LCZ is associated with certain properties, such as the typical building height, street width, or green fraction. With that, weather models have all the info they need to simulate the interaction between the city and the atmosphere, in a simplified form.

Left: Illustration of the local climate zones ( source ). Right: map of local climate zones for Amsterdam ( ).

Typically, maps of local climate zones are displayed in 2D — a flat map. In principle, this is good enough, as we have only one LCZ per grid cell. But it doesn’t really appeal to the imagination. From the map above, I cannot tell how realistic this representation of Amsterdam actually is.

That’s why I started playing around with 3D visualizations instead. The idea is to reproduce the tiles from the illustration above as building blocks, and then place them on a plane in accordance with the pixels on the 2D map. Below, we detail two approaches, one with CityJSON, and one with QGIS.

Building a city in CityJSON

CityJSON file format is intended as a lightweight and developer-friendly format for spatial information. Like JSON, it is basically a set of key-value pairs. The CityJSON format specifies which keys are allowed/required and what values they can have. I was not familiar with the format, so this part of the blogpost also represents my learning experience. A simple CityJSON file could look like this:

{
  "type": "CityJSON",
  "version": "2.0",
  "metadata": {
    "geographicalExtent": [0,0,0,1,1,1]
  },
  "transform": {
    "scale": [0.001,0.001,0.001],
    "translate": [0,0,0]
  },
  "extensions": {},
  "vertices": [
    [0,0,1000],
    [1000,0,1000],
    [1000,1000,1000],
    [0,1000,1000],
    [0,0,0],
    [1000,0,0],
    [1000,1000,0],
    [0,1000,0]
  ],
  "CityObjects": {
    "id-1": {
      "geometry": [
        {
          "boundaries": [
            [
              [[0,1,2,3]],
              [[4,5,1,0]],
              [[5,6,2,1]],
              [[6,7,3,2]],
              [[7,4,0,3]],
              [[7,6,5,4]]
            ]
          ],
          "lod": "1",
          "type": "Solid"
        }
      ],
      "attributes": {
        "function": "something"
      },
      "type": "GenericCityObject"
    }
  }
}

The vertices represent points in 3D space. This file specifies 8 points. Each CityObject specifies an array of boundaries, which are essentially surfaces obtained by connecting the vertices. Here we have 6 surfaces, and if you draw them all up, you will see that this is simply a model for a cube.

The CityJSON home page is very welcoming. It contains lots of useful information and examples. There is an official online CityJSON validator, as well as a viewer called Ninja. So once you’ve obtained a CityJSON file, you can easily validate it and display it in the browser.

I started with the example of 2 cubes, and wrote some Python code to reproduce them as templates, such that I could place blocks randomly everywhere, and scale them in width and height. I created multiple combinations of blocks to reproduce the LCZ archetypes.

LCZ archetypes reproduced in CityJSON.

Next, I played a little bit with some random data to see how big of a city I could render. I found that 100 by 100 tiles still worked okay in the Ninja viewer. Effectively, this corresponds to a city of 10 by 10 km. I downloaded the global LCZ map from Demuzere et al. and cut out the Amsterdam area. The last step was to loop over all pixels in the image, and add the corresponding tile to the CityJSON file.

The code ran quite slowly, but it worked. The Ninja viewer went into “performance mode”, but eventually was able to display the data just fine. I was ready to observe the highly anticipating result…

Amsterdam rendered as 3D LCZ tiles. Blue designates correspond to water, yellow designates all other non-urban land use pixels.

Admittedly, that looks a bit less spectacular than I had hoped for. The interactive view is a bit better, but still. While you can recognize the river, a canal, some parks and different types of buildings, this doesn’t really appeal to the imagination as I had hoped.

One way to improve upon this initial view would be to render the different LCZ classes with different colors, and make the tiles prettier with things like trees, just like on the example tiles above. However, it seemed the online CityJSON viewer doesn’t support rendering materials, and I was already pushing its limits.

In order to proceed with CityJSON, we would need to speed things up, and use a better viewer. There are several ways to speed things up. For example, by using CityJSON Sequence s and maybe converting to 3D Tiles. Concerning the viewer, I tried using the Blender and QGIS plugins, but I struggled to use them properly. In the long run, I would like to publish something like this on an interactive webpage, so I would prefer to look at other solutions. The standard viewer uses three.js under the hood. It seems quite doable to embed such a viewer in a static website hosted from GitHub pages, for example. As such, anyone could clone the template and host their own city model with ease.

Making it beautiful with QGIS

At this point I was showing my progress to my colleague Maurice. He is one of our GIS experts and helped me to reproduce the workflow in QGIS. This alleviated some of the issues we had with the CityJSON viewer. With QGIS, we could render tiles in different colors, and use true coordinates to combine the tiles with other map layers. For example, here is a version where the colours of the tiles correspond to those on the 2D map above.

3D rendering of local climate zones and other land use types for Amsterdam, using data from the global dataset from Demuzere et al.

With QGIS, we took a slightly different approach. Instead of storing 3D cityblocks, we made 2D imprints of the LCZ types, and extruded them to a set height during the 3D render. We used Python with rasterio and geopandas to read and crop the original data and generate the 2D tiles. Intermediate data was stored as a GeoPackage, which can easily be loaded in QGIS.

Introducing Cityblocks: a simple tool to visualize your own city

We decided to spend a bit more time to make this workflow available for anyone. We polished the code and converted it into a small and easy to use Python utility called Cityblocks. You can use it to download the global dataset, extract an area of interest, and convert the data into tiles. Subsequently, the data can be visualized in QGIS, or something else in case you prefer. You can find the code and instructions here. At the moment, this is mostly tested for our Amsterdam use case, so we are very curious to get feedback on your experience with it for other cities.

Making it beautiful

At this point, we though it’d be nice to turn this little side quest into a real map that would stand its ground in an atlas of eScience Center projects.

A good map tells a story that jumps at the viewer without much context. We considered many different options, and settled on a design that places the tiles on a background map of Amsterdam, with a subtle raster to hint at the gridded nature of weather models. Here is the final result:

3D Local climate zones overlaid on an aerial photograph of Amsterdam. The aerial photograph comes from PDOK.

Seeing the map with and without LCZs makes it easy to compare the “human” view to the “weather model view”. Notice that most of the riverbank area, including the station, is classified as “heavy industry”, whereas the city center is dominated by “compact to open midrise”, for example. Apart from evaluating the classification itself, this view also reveals the relatively coarse structure the weather model sees, and how rigid the city looks. Perhaps it is a sobering view that reminds us that even sophisticated weather models are only an approximation of reality.

But it is inspiring at the same time! How can we improve upon this? One of the obvious wins is to use local properties like building height, rather than using fixed values from the LCZ classification. We started doing that several years ago in a project called Summer in the City. In that project, we identified properties like building height and “urban fraction” for the Netherlands. Now, in the Urban-M4 project, we are adding information on the radiative properties of buildings, by using open street view imagery. At the same time, our project partners at Wageningen University are collecting information on building age and other properties to add better-localized information on things like insulation status.

Next steps

I also have some ideas regarding the workflow and visualization. If time permits, here are some things I’d like to improve:

Generalize tile creation: currently the position and dimensions of the buildings are hardcoded based on a tile size of 100x100m. It would be nice to generate them on the fly instead, so we could accomadate various resolutions.
Randomize the tiles: related to the above, I’ve been thinking a bit on how to add some natural variation to the positioning of buildings. For example by randomly shifting tiles across periodic boundaries.
Beyond LCZs: a generic tile creation routine could generate tiles based on some key properties like road width, building width, green fraction, et cetera. That would make it possible to vary these parameters independently, rather than coupled through LCZs.
Combining the workflows for QGIS and CityJSON. Currently they have diverged a bit, but they share many steps. With a bit of work we could streamline and generalize the workflow and add support for other output formats as well.
Procedural generation: automatically create artificial worlds and gamify urban weather and climate simulation 🙂

Concluding remarks

When it comes to visualization in weather and climate modelling, we often focus on output data. Here, we introduce an original visualization of the methodology instead. This provides valuable insights into the power and limitations of the LCZ approach. It appeals to the imagination can make our work more captivating.

What are your thoughs about using visualizations like this? Do you have other ideas, or want to share your own maps made with the Cityblocks package? We have created a GitHub issue to collect images of cities across the globe. It would be awesome if you would contribute your city as well!

Thanks to Claire Donnely and Ole Mussmann for providing comments on an earlier version, and to Tim Tensen and Maurice de Kleijn for creating the beautiful QGIS map. A chatbot was used to improve the formulation of a few sentences.

How much research output is really open, and how can we tell?

eScience Editorial Team — Fri, 06 Dec 2024 00:00:00 GMT

Written by Esther Plomp, Meron Vermaas, Carlos Martinez-Ortiz, Nemo Andrea, Ewan Cahen, Felix Weijdema, Dorien Huijser, Marta Teperek, Bjørn Bartholdy & Ana Parrón Cabañero.

During the Open Science Festival in Maastricht on 22 October 2024, we gathered a full room for the session Tracking Research Objects: Levelling the playing field in research assessment! This was a topic that eScience Fellows Meron Vermaas and Esther Plomp were interested in investigating as a part of their eScience Center Fellowship project. Read on to find out how you can investigate this using their methods!

Photo by Richard Bell on Unsplash

We currently know a lot about how much research is published in Open Access publications for the Netherlands, to the degree that there are complete data and percentages available. For example, TU Delft researchers published 98% of their publications Open Access in 2023. How much of the other research outputs are published openly remains a wild guess, as it is a lot more difficult to measure research objects such as datasets, software repositories, protocols, podcasts, community outreach projects and so forth.

Just because these research objects are difficult to measure, does not necessarily mean they should not be measured. Being able to measure the degree of Open Access has led to positive attention for research outputs being openly available, and measuring this for other research objects may also result in increased sharing and recognition of these efforts.

We therefore started our session with some case studies on how other research objects can be made more visible. Esther presented on a project from TU Delft, and Meron presented on two projects he has been involved with from the Vrije Universiteit Amsterdam and the HvA. See their slides for more details.

Esther and Meron worked together during their eScience Fellowship projects with Carlos Martinez-Ortiz and Ewan Cahen from the Netherlands eScience Center to set up a workflow that could deduce from the DOIs of a journal article whether accompanying research objects are available. It turns out that it is possible to measure this fairly easily with a script that Ewan worked on, as long as the research objects are cited within the publication. This script is available on GitHub for others to try out and reuse.

According to this data/script, the amount of shared research objects for the TU Delft Faculty of Applied Sciences in the period 2020–2023 range from 0–28%. This is likely to be an underestimate, as not all researchers cite the underlying/associated research objects within the article. The next steps for this project are to set up guidelines for researchers to make them aware why citation of these research objects are important, as well as perform text scraping to see how many other research objects are overlooked if we only look at cited research objects within the articles. For this, they will use a workflow that Meron has previously worked on during his eScience Fellowship.

Meron and Ewan created a workflow to find code repositories created by researchers at the VU. Because it is not yet possible to track these in a straightforward way, they found them by looking into mentions of software repositories in scientific papers. While around 70% of the researchers mention the use of a research software (such as Python or R), only ~10% mention a code repository. However, mentioning a code repository does not automatically mean the code was created by the authors of the paper. Thus, this is probably an overestimation of the amount of software that was actually produced and published by researchers from the VU. As a next step the scripts (available on GitHub) also searched through the code repositories and checked if the contributors are affiliated to the VU. This allows for finding and then reaching out to researchers who shared their code, so that support can be offered to make the code repositories more FAIR. Getting in touch with those researchers is an ongoing effort.

Community input on research object tracking

After a presentation on these efforts, the group split into six groups that discussed questions related to research object tracking for ~ 15 minutes, and at the end of the session we reported back to the full group on our findings. Below follows a short summary for each of the groups:

Group 1: Should we track research objects (what are the disadvantages?)?**

This group thought it would be beneficial to track research objects and did not discuss any disadvantages. The group thought it was important for recognising the work, as well as the possibility of validating and verifying research. Careful attention should be paid to the gamification of this system, however, as well as the costs that may be involved in making these research objects available in a reusable and trackable manner. There needs to be incentives for the researchers and professional staff involved in these efforts to make it worthwhile.

It also needs to be clearer what the definition of a research object is and what the scope is. Potentially every research output can be considered a research object — even non-digital objects such as physical samples (blood samples, synthetic materials, or geological samples).

Group 2: In what way would tracking of research objects nurture a culture of sharing research outputs?

By tracking research objects individuals involved in research can be made aware of the benefits of sharing research objects: the visibility and accessibility of research outputs are important. To promote sharing practices we can develop and provide workshops and share recommended practices (research objects need a DOI, be registered in CRIS (current research information system), and/or made discoverable via preprint servers or data repositories).

Nevertheless, sharing research objects may be discipline dependent, and may not apply to all disciplines, or may apply differently (what about law and history?). Not all research may result in research objects that are trackable!

Group 3: What (existing) technological infrastructure would be needed to effectively track research objects?

This group discussed how ORCID could also be used to automatically track research objects. ORCID has options to connect with existing research-object infrastructures and automatically add the publications, protocols and software to a researcher’s profiles. Since ORCID is not widely used for tracking research objects, more awareness of these capabilities may be needed.

For data and software sharing the existing infrastructure are already available. For example, there are hosting services that make use of Git such as GitHub, or open source alternatives such as GitLab and GitTea, in combination with data repositories. For the sharing of protocols, there are protocols servers that can be used such as protocols.io, or open source alternatives such as OSF. Since the infrastructure already exists for many purposes, the limiting factor seems to be the awareness of the resources available, and the practical implementation of citing research objects in publications. An example is the self-citation needed to automatically detect shared research objects (as done in the case study described by Esther). To solve this problem, the group came up with an idea of a tool where researchers can upload their text of a version of their article (before submitting to a journal!), that will analyse the text content and check for missing links and DOIs (for example: “you mentioned a GitHub repository but no DOI for data was found” or “no data repository was found, did you know you can upload your data free of charge to 4TU.ReseachData”). This way, researchers are automatically notified about any missing links or citations to research objects. Currently, there is a proposal (by a group not part of the session) to develop a software called “Transparency Check” that will provide an automatic assessment and suggestions for the improvement of the transparency of data and methods in research reports before they are published.

Group 4: How can we balance quantitative metrics and qualitative indicators in the evaluation of research objects?

They also discussed the dangers of quantitative metrics:

Counting outputs can lead to prioritising quantity over quality (such as splitting up a dataset and publishing the subsets to have “more” datasets published).
Quantitative metrics are subjected to circumventing, hacking and gaming.
Quantitative metrics do not describe how the funding was used throughout the project and provide little context.
It is difficult to quantify open science related work.
Comparing metrics between institutes may lead to counterproductive competition and not be representative of the actual work that went into the outputs.

Group 5: What would you use research object tracking for? How can tracking research objects promote equity and inclusion within the research community?**

This group noted that tracking research objects can be helpful to measure impact of research, and may also be very relevant to funders. It can also help to provide input for policies and to provide ways to guide support distribution, as it would be possible to establish targeted support based on the tracking research objects. Tracking research objects could promote equity and inclusion within the research community as it would provide recognition for all research outputs, not just papers, also datasets, podcasts and other research objects. This could include recognition for all staff involved in research, also professional staff.

Group 6: How do we track research objects in a way that helps us to understand how behaviour changes over time?

The last group came up with their own question to address, and focused on working backwards from what type of behaviour we would like to track or change. The end goal is not to track research objects: we want the broader adoption of open science practices. The group pondered how the why behind all of this could be changing — why would researchers want to share their research objects? What research objects could they create, and which of those are being tracked? They also discussed whether Data Management Plans could be a tool to change behaviour, and whether monitoring data management plans would be helpful: are Data Management Plans updated — when and why?

Examples

We also distributed a survey/form with 9 respondents to collect examples of best practices across the Dutch and international landscape:

Which infrastructures are used in your organisation to track research objects?

The majority (8 out of 9) mentioned their current research information system (CRIS), generally PURE. Three cases mention tracking the use of ISAAC, dataverse, the inhouse repository and OpenAlex.

Which research objects are being tracked?

Considering the majority using PURE as their main tracking system, it does not come as a surprise that publications are also the main research object being tracked. Six also mention datasets and there are isolated occurrences of tracking code, media appearances, supervised theses, keynote lectures.

What does your organisation use research object tracking for?

Most responses mention reporting, evaluation and assessment. Also the ambition to use it to effectively recognise and reward non traditional research outputs.One institute responded that descriptive reports on scientific activities at departmental level are used.

Do you know of other successful examples of research object tracking?

Positive responses were sparse in this question. The PLOS Open Science Indicators were mentioned and a FAIR dashboard was shared. A promising outlook into the future was mentioned where an institute is working on creating a platform to keep track of the research outputs (as well as the Research Data Management & Open Science compliance documents such as Data Management Plans, Software Management Plans, Privacy and Ethics). In the session slide initiatives such as Charité Metrics Dashboard, French Open Science monitor and the Open Science Monitoring Initiative were mentioned.

Conclusion

Collaborative initiatives can create a more equitable research assessment framework that recognizes diverse contributions to science. If you’re passionate about advancing Open Science and interested in contributing to similar community projects, consider applying for the eScience Center Fellowship program. The Fellowship gives researchers and supporters the opportunity to work on technical solutions for open science challenges while collaborating with experts in the field.

How is research software managed at UMCs? Insights from a first meetup

eScience Editorial Team — Thu, 05 Dec 2024 00:00:00 GMT

Written by eScience Center Fellow Thomas Pronk

On October 7th, 2024, the Digital Competence Centers (DCCs) of the seven Dutch University Medical Centers (UMCs) had a meetup, joined by representatives from Health-RI, ELIXIR-NL, The Netherlands eScience Center, and TDCC-LSH. We discussed Research Software Management (RSM) in the context of Dutch UMCs, identifying shared challenges, opportunities for collaboration, and how we could connect with the broader landscape of organizations, communities, and networks.

We discussed Research Software Management (RSM) in the context of Dutch UMCs, identifying shared challenges, opportunities for collaboration, and how we could connect with the broader landscape of organizations, communities, and networks.

Medical research often concerns highly sensitive data and interventions that impact lives. Consequently, we put stringent demands on privacy, security, quality, and reproducibility. In pursuit of meeting these demands, the Dutch UMCs have developed a strong tradition of collaboration and expertise in data stewardship. Our RSM approaches have drawn from these, along with well-established software development practices. However, the institution-wide introduction of RSM policies is relatively new. Correspondingly, RSM at Dutch UMCs is currently at a formative stage. We have the opportunity to jointly address the challenges that come with introducing RSM and we are well-equipped to do so.

Photo by Zhen H on Unsplash

To get started, we each listed our top three challenges, opportunities, and projects to know about, in a survey. Based on the results, we jointly identified five focus areas in pursuing joint RSM:

Distill guidelines from decentral expertise.** There is already substantial expertise in research software spread out over departments at Dutch UMCs. At the central level, we can learn from this experience to develop good practice guidelines that are attuned with practice, can be developed into policies, and are aligned with national and international efforts such as the TDCC-LSH FAIR Tool framework, EOSC EVERSE and OpenEBench.
Standardize Software Management Plans (SMPs). By working together to standardize on a common format, we can link SMP sections to common support materials and offer default answers based on common best practices. Recognizing the variety of software types and roles, the standard could be made adaptive to its use cases (e.g., Practical Guide to SMPs, ELIXIR Software Management Wizard).
Develop intermediate-level training. Basic research software training can draw from existing initiatives such as those from the eScience Center, LCRDM, and The Carpentries (see Research Software Training NL). However, there is a need for intermediate training that focuses on more advanced skills and best practices adapted to the specific needs of medical research. The intermediate training could take a variety of forms, such as ReproHacks, Programming CAFEs, and mentoring schemes.
Define research software-related job profiles. Embedding RSM on an institutional level requires a range of new roles with well-defined responsibilities. Here, we can draw inspiration from work on Data Stewardship (e.g., Ingeborg et al., 2019; Scholtens, 2019) and Research Software Engineering (e.g., eScience Center RSE Job Role Descriptions and Job Profiles, Research Software Competencies Toolkit).
Connect to national and international networks. In pursuit of the above, we seek to align our output with internationally recognized standards, tools, and platforms for (biomedical, life sciences, and health) research software. Natural candidates are offered by our membership of the European ELIXIR network, for instance, the RDM Toolkit tool assemblies and regional resource index, the ELIXIR Tools Platform, TeSS training platform, EOSC EVERSE and ELIXIR-STEERS. Other relevant networks for research software are the national research software engineer (RSE) network (NL-RSE) and the various regional and national RSE organizations around the globe.

This meeting was made possible by funding from the eScience Center for the Fellowship Project Implementing software management plans in medical science to improve science and safety*. If any of the focus areas above strike your fancy, then you might make a great eScience Center Fellow.

Tackling Advent of Code

eScience Editorial Team — Tue, 03 Dec 2024 00:00:00 GMT

By Ewan Cahen

Advent of Code 2024* has started this week! If you are not familiar with Advent of Code, it’s an annual coding challenge created by Eric Wastl. It’s like an advent calendar for coding challenges containing 25 daily programming puzzles, released once a day between December 1–25. You can still join this year’s edition, and even join our Dutch research community leaderboard when you sign up here.*

Photo by Markus Spiske on Unsplash

Whether you’re new to Advent of Code or if you want to brush up on your programming skills, below you can find a list of useful tricks, data structures and algorithms that are often needed when solving the challenges. Some of these are accompanied with links on how to use them in a few popular programming languages.

Note, the list is rather large. We recommend only picking a few items where you think your knowledge is lacking.

Parsing the input**

In almost every exercise, you are given input, presented as plain text, that you have to parse (i.e. transform it into some structure that is useful for solving the problem). There are a few techniques you can use to accomplish this:

read the input line by line into a list (Python, R, Java)
split a string, useful if data is delimited by e.g. whitespace or a comma (Python, R, Java)
parsing a string to an integer (Python, R, Java)
(Java only) use Java’s Scanner to easily parse various types of data
(advanced) use regular expressions to parse the input (Python, R, Java)

Photo by Chris Liverani on Unsplash

Integer division and modular arithmetic

In Advent of Code, you’ll often have to use integer division (where e.g. 11 / 4 = 2 instead of 2.75). For Python, have a look at the double slash operator (//) and for R use %/% (see R operators).

You’ll also have to use modular arithmetic, where you want to know the remainder after integer division (e.g. 11 % 4 = 3, because 4 fits 2 times into 11 and then you have 3 remaining). You’ll use these when, for example, you have to “wrap around” an array, i.e., when you reach the end of an array, you have to return to the start of the array. For Python, use the modulo operator %, for R use [%%](https://www.datacamp.com/doc/r/operators) and for Java use[%](https://www.baeldung.com/modulo-java). Also look up how your language behaves when any of the numbers is negative.

Working with large integers

Sometimes, you need to handle large integers (especially when multiplying numbers). In some languages, where there are several integer types of several sizes, you need to prevent integer overflow. This is sometimes a problem when working with 32 bit integers. Usually, using 64 bit integers (keyword: long) is sufficient for Advent of Code. In Python, this is not needed, as it supports arbitrary large integers. For R, have a look at this package for 64 bit integers. For Java, use the long type or, if that is not sufficient, use the BigInteger class.

Photo by Joshua Sortino on Unsplash

Data structures

Using the right data structure is crucial to solving the problems. These are the most commonly used ones:

array: An array is a fixed-size, ordered data structure, consisting of multiple entries of the same type in a row. You can save and retrieve data in an array by using an index (usually a number from 0 to n — 1 (inclusive) if the array has length n. While Python’s standard library doesn’t have built-in arrays, the NumPy library provides a powerful array implementation that’s widely considered a de-facto standard for numerical computing in Python. In R, one-dimensional arrays are referred to as vectors, and are indexed starting from 1 (so you can index a number from 1 to n). Used when storing data without further requirements or when the order of the data is important. (R, Java)
list (vector): A data structure that stores multiple entries of (usually) the same type in a row, with dynamic size that grows automatically as needed. While lists offer flexibility, arrays generally provide better performance due to their fixed size and contiguous memory allocation. Arrays are particularly advantageous in Python when using NumPy, as they enable efficient vectorized operations that can significantly speed up numerical computations. Choose arrays when performance and vectorization are priorities, and lists when frequent size changes are required. (Python, R, Java)
dictionary/(hash)map: A data structure that stores key-value pairs. If you want to store/retrieve data by something more complex than an index (as with arrays and lists), like a string, this is the data structure to use. While R does not technically have a dictionary data structure, you can often use a named vector as a quick-and-dirty replacement (Python, R, Java)
(hash)set: An unordered data structure that cannot contain duplicates of an element. Useful when you often need to check if some element is present in a data structure (often used in graph traversal algorithms, see below). (Python, R has external libraries, such as r2r, Java)
queue: A first in, first out (FIFO) data structure where elements are added to the end and removed from the front. While Python lists can be used as queues, this is inefficient due to their underlying array implementation — removing from the front requires shifting all remaining elements. For better performance, use Python’s collections.deque which is optimized for both front and back operations. Lists are better suited as stacks (last in, first out). Queues are commonly used in breadth-first search algorithms in graphs (see section on graphs below). (for Python and R, you can use the list as a queue, or you can use Python’s deque, Java)
stack: A last* in, first out data structure, meaning you can add and/or remove elements to the front of the stack only. Used in depth-first search algorithms (see below). (see queue for Python and R, Java)
(advanced) priority queue/heap: Similar to a queue, except that the elements in the queue have a priority, and the element with the highest priority will always be served first when retrieving/removing an element, independent of the order in which the elements where added. Used in Dijkstra’s algorithm (see below). (Python, look for external packages for R, Java)

Algorithms

Many problems can be solved with (a variation of) a well known algorithm. Below are listed some commonly needed algorithms for Advent of Code. This is by no means an exhaustive list. Furthermore, you are encouraged to do more research on these algorithms.

sorting an array/list: You don’t have to implement your own sorting algorithm, but you have to know how to call the built-in sorting functionality of your language, sometimes using a custom sort function/comparator. (Python, R, Java arrays, Java lists)
breadth-first search: An algorithm for finding a node in a graph with a certain property. Used for example when looking for the shortest path between two nodes in a graph, when all edge weights have the same value. This uses a queue.
depth-first search: An algorithm for finding a node in a graph with a certain property. Used for example when the node(s) you’re looking for in a graph are far away from the starting point. This uses a stack.
Dijkstra’s algorithm: An algorithm for finding a shortest path from a fixed starting point to every other node in a graph, when the edge costs have varying values.
memoization: Not an algorithm, but rather a technique, in which you store (cache) intermediate results so that you don’t have to recompute these over and over again. These intermediate results are usually stored in a dictionary/(hash)map.

❄️ Closing words ❄️

I hope this overview is useful to you. I’m not that well-versed in the Python or R ecosystem, so if you know of better resources or techniques on any of the topics presented, please let me know.

Is your favourite technique/algorithm/programming language missing? Feel free to add it below!

Good luck this year!

Thanks to Raoul Schram and Bjørn Bartholdy for comments

The ‘do-it-yourself’ Computed Tomography scanner

eScience Editorial Team — Tue, 03 Dec 2024 00:00:00 GMT

By Francien Bossema

Researchers developed a method to use existing X-ray imaging facilities for Computed Tomography (CT) scanning

Photo by Cara Shelton on Unsplash

This is a cross-post from our fellow Dr. Francien Bossema. Originally published on the CWI website.

More information on our Fellowship programme here.

—

What if you could saw a wooden sculpture through to look at the tree rings? Could a clay sculpture be hollow? What type of instruments were used to craft art objects? These questions could be answered by X-ray imaging. In particular, CT-scans are useful to investigate historical art objects on the inside. The CT-scans that are suitable for this application are however not easily accessible for researchers and quite expensive to acquire. An international team of researchers led by CWI, the Dutch national research center for mathematics and computer science, has therefore developed a method to use existing X-ray imaging facilities for CT scanning. The results have been published on the 14th of May in Nature Communications.

Research facilities of large museums, such as the Rijksmuseum in Amsterdam, often house an X-ray imaging machine, with which they can perform live inspection of objects. This works in the same way as when you have to have an X-ray image taken at the hospital because of a broken bone. On such an image the bone can be seen, because it has a higher density than the muscles and tissue surrounding it. Usually, a few images are taken, so that the fracture can be inspected from different sides. For more complex problems, a CT scan is made. The technique behind CT scans is similar to X-ray images. For a CT scan hundreds of X-ray images are taken from different angles and then combined into a 3D image using reconstruction algorithms. This 3D image can then be ‘sliced open’ to obtain cross sections showing the interior.

Over the past years, this technique has been increasingly applied to art objects, because it gives the possibility to investigate the interior without damaging the object. It is possible to date wooden objects by looking at the tree rings, assess the internal damage by an insect infestation or search for clues about the techniques used by the artist.

No CT scanner? No problem!

CT scans provide more information than X-ray images. In museums, however, simple X-ray imaging setups are more common. The components of the machine are similar (an X-ray tube, rotation stage and a detector), but for a CT scan the positions of these components during the scan need to be known very accurately. The researchers from CWI have therefore developed a method to calculate these positions after a data acquisition, with nothing more than a few small metal balls. These balls are put in a piece of foam and scanned alongside the object. Based on the location of the balls on the X-ray images, all the necessary positions can be calculated. These are then used to obtain a CT-scan.

This unique project is a collaboration between a team of international researchers from CWI (Amsterdam), Leiden Institute of Advanced Computer Science (Leiden), Rijksmuseum (Amsterdam), The British Museum (London, UK) and the J. Paul Getty Museum (Los Angeles, USA). The X-ray imaging facilities of each of these museums, as well as the FleX-ray laboratory at CWI, have been used to test the new method and compare results in different facilities. The X-ray setups at the Rijksmuseum and the J. Paul Getty museum have been used for CT scans for the first time.

The first author of the associated Nature Communications article Francien Bossema will defend her PhD thesis at Leiden University on the topic of CT scanning for cultural heritage on the 23rd of May.

We can do something that was not possible before. CT scanning is an important imaging method for museums, because researchers can digitally cut open an object without damaging it. Our method makes it possible to further integrate this technique into research practices in the museum, without obtaining new hardware. We only use small metal balls of the type you also have in the pedals of your bike and algorithms specifically designed for this purpose. This means no extra investments are needed, while still providing an additional imaging technique.*

Francien Bossema, CWI

New method gives insights into the making process of a sculpture

After the development of this technique, a sculpture made of plaster from the J. Paul Getty Museum was studied: Python Killing a Gnu, by Antoine-Louis Barye (J. Paul Getty Museum, 85.SE.48). The sculpture originally had a different configuration, it was smaller and more compact because it was initially used as a model for a group of sculptures that was displayed on a dining table. The artist later decided to change the model to open and enlarge the arrangement and turn it into a stand-alone sculpture. The possibility to CT-scan the object was welcomed by the conservator who was investigating this object, Madeline Corona (Getty Museum): ‘On the X-ray images, we could see that areas had been broken, altered, and filled but it was difficult to understand exactly how the artist made these changes because there were too many overlapping features. The CT-scan gave us an incredible opportunity to obtain clearer and more detailed images of these areas of change. In the end, it provided important insights that allowed us to confirm some of our hypotheses about Barye’s working methods, something we could not have done without this technique.”

More information

Article Enabling 3D CT-scanning of cultural heritage objects using only in-house 2D X-ray equipment in museums in Nature Communications 14 May 2024.
Francien Bossema (CWI/Rijksmuseum), bossema@cwi.nl
Prof. dr. Joost Batenburg (LIACS), k.j.batenburg@liacs.leidenuniv.nl
Madeline Corona (Getty Museum), MCorona@getty.edu
FleX-ray CT lab at CWI
Computational Imaging group at CWI

Is scientific software the new land grab?

Lourens Veen — Wed, 27 Nov 2024 00:00:00 GMT

Scientific software platforms are not always as open as they look at first sight. Avoid getting locked in with this advice!

“Do you know what CodeOcean is? Should I do this?” As a Research Software Engineer you often advise on all sorts of stuff, but I hadn’t heard this one before. In this case from a professor I work with, who’d sent a paper to Nature (as you do) and was asked by the editor to put the corresponding code in “CodeOcean”. Do I have an opinion on this?

You mean this could be my office? Photo by Sean Oulashin on UnsplashI’d never heard of CodeOcean. Data lakes, yes, and Wind River Systems, but CodeOcean? Never heard of. Fortunately, a web search found it quickly. I started reading.

CodeOcean is a cloud-based workflow execution and data storage platform. You can make an account, and then upload your scientific data and scripts into the system. Because the script is in containers in the cloud, the script’s environment never changes, so it will always run and your research will be easily reproducible by other researchers, who will be able to comfortably find it when they next log in. You and others can then compose workflows, step-by-step procedures for processing data, using your own software and that of others.

Sounds great, right?

Well, not to me actually, for reasons practical but mostly political. Let’s have a look at what’s really going on here.

Reproducing and reusing

First, workflows. Workflows have been around for a long time. The idea of a workflow is that you can use a graphical editor to click together a series of steps that the computer is to perform, so that you can do custom processing without having to learn programming.

This sounds nice in theory, but has its limitations. Steps can only be combined by feeding output from one step into the next. This is done through files, which is slow, and it’s limited in that you cannot send data back and forth between different programs.

Graphical, drag-and-drop approaches, while easy to understand at first, quickly become overly complex in real-world scenarios. Because of this, scripting, where you describe what you want in text, is more popular and increasingly widely taught at universities. So a workflow system is not necessarily what people actually need, even if it looks good on screen.

Furthermore, the story above cuts some corners. Reproducibility is not the same as reusability, and a script is not the same as a reusable software package. Putting your script in the system and getting it into a container and running is probably doable in most cases, and will create reproducibility, at least for a while depending on how complicated the thing is you’re trying to do.

Science is more than just reproducing results though. We want to build on them, and we don’t want to have to re-implement everything that has already been researched. Instead, we’d like to reuse the work of our predecessors, so that we can get started right away. So we need reusability as well as reproducibility.

We need reusability as well as reproducibility

For reusability to work, a lot more work is needed. Researchers will have to convert their script, which is designed to run just once, on specific input data, with its author at hand to handle any issues as they arise, into a reusable workflow step that can run many times, handle a range of inputs, and solve any issues by itself or at least give an error that is understandable by someone who doesn’t know the software. This is a lot of work, and it takes a lot more skill and experience than creating a script.

Setting up a workflow system in the cloud is one thing (especially if you pull a ready-made Open Source one off the Internet, as CodeOcean has to their credit done), but it needs to be populated with reusable software to be a useful workflow development platform, and that is difficult and time-consuming.

So, will a system like this actually be useful? We have some existing workflow systems in science that are well used in particular communities, like KNIME, so it’ll likely be of some use, but I don’t see this being the great solution to reusing scientific software. I do see something else, however: lock-in.

Photo by Jametlene Reskp on Unsplash

The next landgrab

CodeOcean claim that their platform is open. You can put your software and data on there, and everyone else can download it. And since the workflow engine they use, Apache Airflow, is Open Source, you may even be able to use the downloaded bits to put together a workflow locally.

Doing that in practice is going to be quite a bit of work, and unlike with other repositories you need an account just to download anything. So there’s a network effect here: if most of the data and software you want is on this platform, then adding the rest is easier than getting the data and software out and running somewhere else.

The login requirement also raises a barrier: I can easily grab data from a public server from a script, but with CodeOcean I have to go to my browser and do it by hand. Unless of course, I put my script into their platform.

For a workflow platform like this to be truly open, it would have to be able to use data sets and compute steps from other platforms in its workflows, and allow other platforms (or local scripts) to use data and steps available on it. That would leave users free to choose any platform (or their own laptop) to put their data and software on, as well as to compose workflows on, because it ensures that we all can work together regardless of which product we choose to use.

The downside of a federated system is that it’s more difficult to make money off of it.

The downside of a federated system is that it’s more difficult to make money off of it. Without a network effect, multiple providers could compete, and users would be free to switch. Such a free market would reduce prices, and that would be good for us eternally cash-strapped scientists, but investors in these companies would get less profit.

So, if I ran a company like CodeOcean and wanted to make good money, I’d do something else. I’d require people to sign up to use the service, I would make it so that you could only combine data and software from different sources if all the data and all the software is in my system, and then I would make deals with some high-ranking journals to get them to push scientists to put their data and software on my platform.

Other scientists are likely to want to use those high-profile data sets and software modules, and to do so, I’d make them sign up first. I would then design my user interface to push people towards using the data and software on my site, rather than downloading them and using them locally.

They’d add their software and data to my platform, this being easier than getting things to run on their own computers, which encourages others to join as well, and the whole thing will snowball.

Then, having obtained many users, I would have a look at the email addresses with which they signed up. I’d contact the universities they work for, and explain politely that many of their employees are using my service. I’d tell them that free accounts are going to go away, and that the university will have to pay an annual fee.

If they refuse, I’d block their scientists from my platform. The scientists would then protest their university administration because, without their data and software, they cannot work. The university would then pay me, I’d re-enable access, and subsequently retire to a tropical beach to sip cocktails and add more features to my scientific software.

I’ll need to order a cup holder for my laptop… Photo by Whitney Wright on Unsplash

Déja vu

Am I a brilliant business strategist? Of course not. This business model, in which society pays people with public money to produce scientific output, which is then appropriated by a commercial platform and sold back to the public, is what Open Access has fought hard against.

It’s taken a long time to get to a point where we are now mostly publishing Open Access (even in Nature, if you’re rich), and much of our publicly funded work is still locked away behind paywalls, but overall I’d say it’s been a success. That doesn’t deter people from trying to pull us back into the same trap however, this time for data and software. Instead of fighting our way out, maybe this time we should avoid falling into it in the first place.

The good news is, we can! Instead of putting your data set on CodeOcean or a similar locked-in platform, put it on Zenodo or Figshare. These platforms allow anyone to download your data without an account, enabling them to use it anywhere. Scripts can go there too, and like data, they’ll get a DOI so that you can refer to them from your Open Access paper.

No one installs reusable software from Zenodo, so if you make a reusable program then it’s best to publish it to the standard repository for the programming language you’re using. Python should go on PyPI and into conda-forge, the latter is also good for packages in C, C++, and Fortran, and Julia, Rust, R, and Java all have their standard places too.

These repositories are open, allowing anyone to install and use your software anywhere without getting locked into something. For development, you can use GitLab (your own or one in the cloud) or GitHub, which also allows anyone to clone your repository without signing up or putting their own software in there.

Of course running these platforms isn’t free. Zenodo is paid for by CERN, the particle physics lab in Switzerland. Figshare is run by a company. The conda package manager is mostly developed by Anaconda Inc. (another company), conda-forge runs on spare cloud resources donated by Microsoft, GitHub is owned by Microsoft and GitLab is run by the company with the same name.

The difference is that conda is open source and so are conda-forge’s scripts and bots, and that git is an independent open source program, so that I can trivially clone a git repository off of GitHub and push it onto GitLab or vice versa.

Issues and pull requests are harder to move, and there is somewhat of a network effect on these latter platforms for developers, who need to have an account for security reasons, but you don’t get locked in just to run anything, and it’s still easy to combine software and data from different sources.

Sometimes it’s useful to put something in the cloud. Scientific software is often poorly engineered and difficult to install, and it can be nice to share an entire pre-installed environment sometimes, for demonstration purposes for example.

If that’s what you want, go talk to your academic computing center! SURF for example here in The Netherlands have the SURF Research Cloud, an environment in which you can create virtual machine images with your software, which other people can then run and use on SURF infrastructure.

This does require an account (a standard SURFconext one that you use for their other services too), but at least it’s an account with a not-for-profit whose task it is to provide these resources to scientists. In short, SURF are working for you, not for a return on investment.

Example of a researcher using the SURF Research Cloud. Photo by Tim Mossholder on Unsplash

In short

If someone invites you to put your scientific software onto their platform then you should have a good look first to see what this platform is for. Is this publicly funded infrastructure set up to make science more open and to support open collaboration? Then go right ahead. Is it commercial? Then it could still be okay, if it’s open enough. But beware of companies trying to appropriate your publicly funded work. We don’t need to fall into that trap again.

From Pandemic Response to Package Development

eScience Editorial Team — Tue, 26 Nov 2024 00:00:00 GMT

By Fenne Riemslagh

We had the pleasure of sitting down with Kirsten Bulsink, a data scientist at the Dutch National Institute for Public Health and the Environment (RIVM). Our discussion covered her journey from pandemic response to R-package development and how the Netherlands eScience Center played a part in creating a crucial part of tooling at RIVM. Her story demonstrates the importance of collaborative work in research.

Photo by Eran Menashri on UnsplashQ: Can you tell us about your background and current role at RIVM?**

A: I’ve been working at RIVM for a little over three years now. My background is in psychology, with a master’s in neuroscience. During my Research Master’s, I discovered my passion for data analysis and finding answers through data. This led me to pursue a minor in data science.

I started working at RIVM during the COVID-19 pandemic. Initially, it was a chaotic time, with researchers working overtime to analyze and report data quickly. When I joined, there was already a semi-automatic data pipeline in place, but we still had to tackle complex challenges, like calculating vaccination rates with data from a selected group (because of opt-out).

As our team grew to about 9 to 10 people, we started organizing workshops to reflect on our processes. We asked ourselves what worked well and what we’d do differently if we could start over. This reflection led to the development of new tools and approaches.

“…we started organizing workshops to reflect on our processes. We asked ourselves what worked well and what we’d do differently if we could start over. This reflection led to the development of new tools and approaches.”

Before the pandemic, processes and methods differed for different infectious diseases. As a result, researchers at RIVM had to perform many actions manually, and these processes could differ per infectious disease. The pandemic necessitated more knowledge sharing and collaboration. We started standardizing and automating data transformation and reporting for infectious diseases.

Q: We understand that you and your colleague participated in the R-packaging workshop organized by the eScience Center. Can you tell us about that experience and the R-package your team developed?

Yes, that’s correct. One of my colleagues actually took the R-packaging workshop offered by the eScience Center before I did. Later, I also had the opportunity to take the same course.

The package, which now serves as a core tool for epidemiological pipelines at RIVM, provides functionality for loading, cleaning, and reporting data, with various checks in place. It also includes functions to create graphs in RIVM colors and style.

For example, during the COVID-19 pandemic, we used analysis methods to process data on positive cases, calculate the number of cases over time, and generate reports. Now, we use the package for monitoring and reporting on various infectious diseases like sexually transmitted infections and respiratory infections, not just COVID-19.

Photo by Fahim Muntashir on Unsplash**

How did the R-packaging workshop help professionalize your package?**

After joining the workshop at the Netherlands eScience Center, I organized a session for my team to share what I had learned. While my colleagues had already done a great job, the workshop helped us improve consistency in managing dependencies. We also enhanced our documentation. The package improvements made it easier for others to use the package. Installation became smoother, and users no longer had to figure out why they needed to install extra packages.

“The package improvements made it easier for others to use the package. Installation became smoother, and users no longer had to figure out why they needed to install extra packages.”

Later on, I also took the Python software development course offered by the eScience Center, which was really eye-opening. I learned about tools like linters, virtual environments, testing, coverage, and CI/CD pipelines. This knowledge made us realize we needed to implement these practices in our R-package as well.

After gaining all this knowledge from the eScience Center courses, we felt ready to take our package to the next level. We decided to organize hackathons to focus on implementing best practices and improving our package structure.

Our first main goal was to internally demonstrate that we had a high-quality product, especially since many analyses of infectious disease data rely on this package. Our second goal was to share our methodology with external parties like the GGD (Municipal Health Services), even if we couldn’t share the actual data.

We reached out to the eScience Center training team for support, and they connected us with Pablo Rodríguez Sánchez (one of the eScience Center’s Research Software Engineers (RSEs) and main author of the R-packaging course, ed.) to consult during our hackathon. This collaboration was very valuable in guiding our efforts and providing expert insights.

Photo by Mika Baumeister on UnsplashQ: What were the outcomes of the hackathons?

We had two hackathons. In the first one, we focused on testing and documentation. We increased our test coverage and improved our package documentation, including creating a vignette with examples.

The second hackathon was about splitting our large package into smaller, more manageable ones. We also worked on establishing a workflow for potentially publishing the package on GitHub while keeping our main development on RIVM’s internal GitLab.

Pablo provided a fresh perspective and helped us confirm that we were on the right track. His expertise was particularly valuable in the second hackathon when we were making decisions about package structure and workflow.

“Pablo Rodríguez-Sánchez, Research Software Engineer (RSE) at the Netherlands eScience Center, provided a fresh perspective and helped us confirm that we were on the right track. His expertise was particularly valuable in the second hackathon when we were making decisions about package structure and workflow.”

Q: How has this experience changed your team’s way of working?

In the past year, we’ve started to work much more like a software development team. We now use a Kanban board for project management and have implemented CI/CD pipelines, which have made our development process much smoother. The package split has made everything more manageable, and it’s easier to see where we need certain tests or improvements.

Q: What’s next for your package and team?

We’re planning to release some of our packages in GitHub in the next couple of months, which will allow external users to download and use them. We’re also focusing on internal knowledge sharing and running workshops about our tooling.

We value having the eScience Center as a sparring partner for tackling these technical challenges.

In my current role I now have a nice combination of technical skills and advisory tasks. We advise and make other people at RIVM enthusiastic about our tools. Our recent experience in developing this R package has been invaluable.

The Netherlands eScience Center would like to thank Kirsten for her time for the interview . We look forward to continuing our collaboration. If you want to learn more about collaborating with the eScience Center or are interested in our training programme, please visit Training & Workshops — eScience Center. If you are interested in receiving consulting like Kirsten did, you may be interested in our Fellowship Programme.

Why should you care about reproducible code — and how to get started?

eScience Editorial Team — Thu, 14 Nov 2024 00:00:00 GMT

On 23 April 2024, the first ‘National Research Software Day’ took place in Hilversum, the Netherlands. During the unconference part of the program, Eduard Klapwijk ran the session about the importance of reproducible code.

Credit: The Turing WayAuthors: Diana I. Bocancea¹, Daniela Gawehns², Julian Lopez Gordillo³, Sam Langton¹, Katinka Rus¹, Sally Hogenboom⁴, Iris Spruit⁵, Eduard Klapwijk⁶

Despite the increased awareness regarding reproducibility in recent years, most research results are not computationally reproducible: they cannot be independently reproduced. The main reason for this is that in most cases, data and code are not shared publicly. But even when a researcher openly shares their data and code with the public, reviewers or research colleagues, their findings can rarely be reproduced in their entirety. Perhaps the code cannot be executed, only parts of the results are generated, or perhaps the results produced are totally different from the published study. Reproducibility can even be a challenge internally. As any programmer will know, just because your code runs perfectly today, it does not mean it will run perfectly in five years’ time (or even five days’ time!).

But why does writing reproducible code even matter, and how might you as a researcher get started on this journey toward reproducible research?

Benefits of working reproducibly

One reason is that it will make your life as a researcher easier! Many of the components that make a piece of research reproducible — well-documented, clearly written code, containerised environments, properly organized data — are also things that save a lot of time. These activities ensure that when you return to code six months later, the scripts still run, and you don’t have to spend three days debugging them. It also means that your code can be shared and reused by your colleagues, saving them time, and giving you credit (e.g., authorship) in the process. There are other reasons too, including reputational benefits and advantages during peer review. You can read more about ‘selfish’ reasons to make your research reproducible here.

What about the scientific community? We are currently in a situation where a large proportion of research is not reproducible. This situation threatens the integrity of scientific research, weakens our evidence base, and ultimately might lessen public trust in science. The main method for scrutinising and sharing scientific results — peer-reviewed journals — are slowly adapting to this realisation. Increasingly, researchers are encouraged, if not expected, to provide their data, code and other materials used alongside the publication itself. In time, we could see reproducibility move from being an optional bonus to becoming a mandatory part of the research (and publication) process. Adapting to this change early will bring you all the benefits noted above (e.g., timesaving, code reuse) but will also prepare you for the future.

Reproducible tools as a contribution to science

On that note, the changing perspective on the importance of reproducibility is bringing new career paths with it. For example, beyond the fundamental tools that enable reproducible research (such as git for version control), other higher-level tools are appearing to address certain challenges particular to some scientific domains. Usually, they are aimed at solving well-known problems for researchers from a certain field, problems not well-known outside of one niche. They might revolve around workflow management and experiment design or standardisation of certain procedures within the community. In many cases, the developers behind those software tools and resources are… the researchers themselves. They might have struggled with these issues in their own research and decided to take up the task of developing the tools that they wished they had (for example, extensive Python-based processing pipelines such as fmriprep in the neuroimaging field, and thousands of R packages ranging from complex statistical modelling packages such as brms for Bayesian regression to packages to help you formatting manuscripts such as papaja). In doing so, they shifted their focus from their original subject domain to the mission of making research within that domain reproducible. This typically takes the form of developing the software libraries that make that possible and integrating them with the standard software used within the domain.

The whole scientific community can benefit from such tools! Newer research can be built on top of them, without the need to solve common reproducibility issues from scratch. These software developments can be just as valuable a contribution to the research domain as other research findings, and as such, they should be recognised accordingly. And just like it is possible to publish your research findings, it should be possible to publish your code contributions when they are significant enough. A good example of this idea put into practice is the Journal of Open Source Sofware (JOSS), where the submitted code takes the main stage in the review process (as opposed to be required as “supplementary material”). Initiatives like JOSS showcase developments around reproducible research as a meaningful contribution to science and a viable development path, both of which are powerful incentives for researchers to get interested in the topic.

In time, with all these smaller and bigger changes, scientific research can become more trustworthy, more reliable, and in turn, more impactful.

How to get started*

The inevitable question that follows is then: how to get started with reproducibility? One answer is training. Luckily, there are a lot of initiatives for training that will help you to get started, both nationally and internationally. For example, a lot of institutions organize Software and Data Carpentries that offer foundational coding and data science skills.

One way senior academics can make a difference — as group leaders, supervisors, and grant reviewers — is to give (junior) colleagues the time and incentives to value and practice reproducibility. For instance, supervisors could have all PhD students replicate and extend an existing analysis as part of their initial research. The process of reproducing an existing work will familiarize the student with the common challenges that come with doing good science. The work of reproducing someone else’s work might entail finding and understanding a certain dataset (sometimes difficult to even get access to), as well as the software (e.g. scripts or packages) that was used to produce the results. Running the previous analysis, often on a different computer and at a later time (when software dependencies have likely changed) would check the computational reproducibility of the previous work, and in doing so, be a valuable learning experience for the student.

Group leaders benefit from reproducible workflows as it prevents (PhD) students from re-writing the same piece of software again and again. While learning the ropes is important for any junior scholar, it is not very efficient if every new generation of students re-writes code for basic operations or frequently used analysis methods.

In addition to the benefits of an academic career, researchers themselves also increase their employability outside of academia by learning digital skills (such as version control or programming reusable pieces of code) that are valued in many different (industry) jobs.

In modern science, computational methods are the norm in almost every discipline. Yet attempts at reproducibility are almost always unsuccessful due to missing materials and/or lack of skills. Part of this problem can be mitigated by learning how to produce reproducible code: how to write documentation, perform version control, and manage packages. Doing so will benefit you as a researcher, but also your colleagues, and the wider scientific community, because your (coding) efforts will become reusable. Increasing the use of reproducible workflows is in the interest of many stakeholders in academia — increasing the reproducibility of research is key for a broader change in how we do science.

Footnotes

Variables, Records, Values and Objects

Lourens Veen — Thu, 07 Nov 2024 00:00:00 GMT

The Engineering Corner

A blog-within-a-blog on the transition from writing simple scripts for yourself to writing larger programs with and for others. TEC appears on an irregular schedule driven by workload and inspiration. (TEC1, TEC2)

№ 3: Variables, Records, Values and Objects

Last time, we talked about the actions taken by scripts, and how they could be organised hierarchically using procedures. This makes it possible to split up a big problem into clearly delineated blocks with well-defined interfaces. These blocks can then be glued together into a hierarchical structure that gets complex things done in a well-organised way.

However, actions are only one half of the programming equation. Procedures give us verbs, but we’re still missing nouns: data, or state. In Python, and many other languages, data is stored in variables. Variables connect a name to a memory location holding a value. This connection is created by an assignment. The statement x = 1 will create a new memory location, associate it with the name x, and write the number 1 in it. Other examples of Python values are the string "The Engineering Corner” and the boolean value True.

Values spilling out of a variable? Could there be a memory leak? Photo by Napendra Singh on UnsplashVariables can be used in different ways. Often, they store the data that we are processing; after we read it from an input file, halfway through a calculation, and before we write it out again. But we also have variables that are used to control the execution of the script: loop variables, flags that keep track of whether something has happened, and so on. Sometimes, data is used to make decisions as well (“if x is negative, do this, else do that”).

At any point during its execution, a program has a state, which is simply the collection of all existing variables, plus an indication of the next statement to execute. Together with any future inputs, this state will determine what the program is going to do. Which branch of an if-statement is taken, how many loops a for-statement executes, it all depends on the state.

As your script gets bigger it will get more and more variables, until it becomes difficult to keep track of them all. As with the actions, the solution is to organise the variables hierarchically. There are two ways to do that actually, scopes and records.

Scopes

Scopes come for free once you start making functions (procedures in TEC 2, we’re switching back to Python terminology here and call them functions). Variables created within a function are local to that function, which means that they’re only visible to statements inside of the function. In other words, they are in the function’s scope. Variables that aren’t created within a function are global variables, and are in the global scope.

So, if you make functions, you’ll notice that most of your variables will end up being local variables. In fact, if you design these functions well and have them only communicate with other functions via parameters and return values, then the only global variables you’ll have are at the top level of your Python script. In Python, you can actually read those global variables from inside a function, but something funny happens if you write to them:

global_variable = 42

def my_function() -> None: print(global_variable) # prints 42 global_variable = 43 print(global_variable) # prints 43

my_function()

print(global_variable) # prints 42The way this works is that if you try to use a variable, it is first looked up in the local scope you’re in. If it’s not found there, then the global scope is tried. So the first time global_variable is read in my_function, this is what happens. Assignment however is not affected, so on the next line a new variable is made in the local scope, which is called global_variable and given the value 43.

On the next line, this local variable is found first and its value is printed. Note that the global variable named global_variable is now invisible from within my_function, as it is masked or shadowed by the local variable of the same name. The print statement at the end of the script isn’t in my_function’s scope, so, it sees the global version of the variable, which still has the value 42.

Scopes are very, very good to have when you’re making larger programs, because they decouple different parts of the state from each other. If your functions are of a reasonable length, then all the code that can possibly affect a local variable is on your screen at once, and easy to have a picture of in your head. This makes it much easier to reason about the code, something that is very important if you’re writing code that will run on someone else’s computer while you’re not around.

You can in fact write to a global variable from inside of a function, but you’d have to have some very very good reasons to do so. It’s almost never good design, so I’m not going to tell you how to do it. Instead, let’s talk about records.

A record record has a record company, catalog number, artist, and title. Photo by Mick Haupt on Unsplash

Records

Sometimes, one variable just isn’t enough to contain all the information you want to store about a particular thing. For example, if you need to keep track of a person, you’ll want to store their name, but maybe also their date of birth, email address, and whatever other properties are of interest to your application. And speaking of dates, those consist of a day, month and year [0]. So it seems that some way of grouping multiple variables together would be a useful language feature.

Fortunately, language designers have come to the same conclusion, and have added records* (Pascal) and structs (C and C++) to languages. More recent languages like Python tend to have classes instead (more on those in a future instalment), but the closest equivalents in Python would be a dataclass [1]:

from dataclass import dataclass

@dataclass class Date: day: int month: int year: int

@dataclass class Person: name: str date_of_birth: DateHere we have a statement that creates a class named Date, containing day, month and year fields all of type int. What does that mean? One way of looking at a class or a record is that it is a description of the structure of a certain type of complex values. So here we are saying that any value of type Date must contain three subvalues with the given names, each of type int. These subvalues can themselves be complex objects, as we see in class Person.

And so here is a hierarchy again, but a different one than the hierarchy of functions. That one lets us describe complex actions as a combination of simpler actions, while this one lets us describe complex objects as combinations of simpler parts. Together, they make it possible to describe very complex programs in such a way that we can understand them one piece at a time.

There’s one thing still missing though. A class is just a description of what a particular type of value looks like. In order to actually do something, we need some variables containing such values (they’re often called objects). Fortunately, Python’s @dataclass annotation automatically creates a function with the same name as the class, which takes as arguments the values contained in the class, and returns a value containing them [2]. Some code shows it more clearly:

p = Person( name=‘Albert Einstein’, date_of_birth=Date( day=14, month=3, year=1879))

p2 = Person( name=‘Emmy Noether’, date_of_birth=Date( day=23, month=3, year=1882))

p3 = p2 print(p3.date_of_birth.month)Here we create some values of class Person by calling the correspondingly named function, and assign them to variables. To create a Date, we have to call that function instead, and pass appropriate arguments.

Once we have these complex values, we can use them as a whole, like in the assignment p3 = p2, or we can access some detail inside of them using the . operator, as in the print statement. And just like that, we’re talking about complex things like people and calendar dates!

One more thing before we go. In TEC 1, we talked about type annotations, which you can use to specify which types function arguments must have, and what the type of the return value will be. Records and (data)classes are in fact types, and you can use them to annotate functions. So given the above, you can write a function

def update_date_of_birth(p: Person, d: Date) -> None: p.date_of_birth = d

update_date_of_birth(p2, Date(day=23, month=3, year=1982))and have mypy check that it’s being called correctly.

If you go and play around a bit with this code, you may notice that there’s some really funny business going on when you assign using whole objects and also parts of them. In the next instalment, we’ll take a look behind the scenes to see what’s actually happening inside of the computer when we make variables and complex values, and clear all that up.

← TEC 2: Procedural programming — TEC 4: <Stay tuned!> →

[0] At least with the calendar system currently used in the western world. There have been many calendar systems over the course of history, it’s a fascinating subject actually. If you’re ever bored, look into it!

[1] Or a NamedTuple or perhaps even a dict, but a data class is the most straightforward and provides for a good starting point when we talk about classes later.

[2] It’s actually a bit more complicated than that, more in a future instalment on Object Oriented Programming.

Beyond research

eScience Editorial Team — Wed, 30 Oct 2024 00:00:00 GMT

Beyond research

‘Beyond research’ is a new mini-series we are starting on the eScience Center blog. We want to give you some insight into the interests, hobbies and accomplishments of our colleagues. Learn more about the writers behind our blogs and get to know them beyond the research. This first post features Walter Baccinelli, Carlos Martinez-Ortiz and Olga Lyashevska.

Walter Baccinelli: wrestling and words of wisdom**

I started my path in martial arts a long time ago when I was six and had no pain in my joints. I would love, at this point, to tell an inspiring and moving story about how I found my way to the dojo and “the gentle way” (aka, Judo), but I have none. The truth is that my older sister was already practicing Judo and the gym was close by, so I joined in. It was very convenient. What is incredible is that I never left this amazing world. Judo has been my first love and a very important part of my life, but I’ve been exploring, with more or less commitment, other disciplines like Sambo, Muay Thay, Wrestling, Kung Fu, as well as traditional fighting like Shuai Jiao and Qazaq Kuresi.

Since I began practicing martial arts, there have been many moments that I am proud of and that have impacted me. I started competing for fun as a kid, and more seriously as a teenager and into adulthood. I had the chance to take part in countless competitions at different levels, from very fun amateur to European and World championships and, very recently, the World Nomad Games as part of the Italian national team. Regardless of the level of the competition, stepping on the mat always takes a lot of courage, and I’m proud of myself every time I decide to put on my Gi (Judo uniform) and challenge myself by fighting. Even before the fight itself, the path is hard and it takes commitment, self-sacrifice, passion and a bit of craziness. It requires constant training, always pushing yourself to your limit. It requires suffering through the fatigue and, sometimes, the injuries. It requires adjusting your whole lifestyle, fasting and resting instead of partying. It requires being ready to win, but, even more, being ready to lose despite all the work.

This may sounds like an awful life. Why would any sane person consciously decide to do this? Well, it also comes with joy. Firstly, it gives you the awareness that you have what it takes to reach the goals that you set, and some self-awareness never hurts. Secondly, reaching those goals is extremely satisfying. Entering the arena wearing the colors of your nation, hearing the cheering of your teammates and winning an important medal. That’s priceless.

It is clear that sport, for me, was not just a hobby or a complementary part of my life, but a totalizing activity and a way of living. Having had the privilege of experiencing the athlete’s life shaped my character and taught me a lot of lessons largely applicable to my private and work life. And no, I’m not speaking only about how to correctly punch the keyboard when a test fails with my code. Martial arts has taught me the immense value of respect. Respect for the people you train with, your teammates but even more for your opponent. I learned that an opponent is not an enemy. I learned that I cannot achieve my goals without the support of other people and that my support is fundamental for other people to reach their own goals. In Judo, you cannot train alone and you and your “Uke” (sparring partner) need each other to improve together. It’s all about collaboration and learning from other people. I learned that the higher my goals are, the harder the route to fulfill them. Failure and frustration (as well as accepting these emotions) are a part of the process, and commitment, hard work and patience (and sometimes also a bit of luck) are the tools with which I must be equipped.

Walter during the World Nomad GamesCarlos Martinez-Ortiz: Kendo and committing to curiosity

I started kendo a few years ago. One of my friends had started a few weeks before and it sounded like a fun thing to do, so I joined him. A few months later, my friend stopped, because he started doing something else, but I just liked it enough to stay. Initially, I really liked the external part of it: it is very dynamic, and explosive — lots of shouting and running around. You get to wear cool armour and feel like a samurai. But over time there were other aspects that I enjoyed more — the aspect of long-term improvement and leaving the rest of the world outside when I practice.

I think kendo for me has shown me some aspects of my own character.

A couple of years ago, when I was preparing for a kendo exam, one of my seniors gave me some feedback on things I should improve — not only in terms of technique but also in terms of my attitude. Around the same time, I had my yearly appraisal with my line manager, and his feedback was very similar. From my perspective, what I needed to learn in kendo was the same thing I needed to learn for my personal development.

Olga Lyashevska: my Jiu-jitsu journey**

It is not about how good you get but about what you do for the community — *Carlson Gracie Jr.

My jiu-jitsu journey began back in 2005 when I was at university. One day, I noticed a group of students running around in what looked like white pajamas, jumping, rolling, and clearly having fun. Intrigued, I decided to check it out since I had some free time that week.

When I arrived at the gym, I figured I’d just sit back and watch a session, but to my surprise, I wasn’t allowed. “You’ve got to join in,” the teacher said. So, without much hesitation, I grabbed one of those white ‘pajamas,’ which I later learned was called a gi, and picked out a random belt from a box in the corner.

The class started, and we jumped right into running, tumbling, and falling all over the mats. I had no clue what I was doing. At some point, the teacher noticed the belt I had tied around my waist — it was one meant for higher-ranked students. He looked at me with a puzzled expression and asked, “What belt are you?” I hadn’t the slightest idea what to say because I hadn’t earned any belt yet, let alone the one I was wearing. It was awkward and funny, but at that moment something kickstarted in me.

From that day forward, I never stopped training. Jiu -itsu became much more than just a hobby — it transformed how I approached challenges, relationships and ultimately, how I saw myself. What started as an accidental introduction became a lifelong passion that continues to shape who I am today.

As time passed, my training intensified. I found myself training up to six days a week, balancing my time between judo and jiu-jitsu, and competing and refereeing in both disciplines. Along the way, I’ve had many moments I’m proud of — from winning the World and European Championships to refereeing big competitions with some of the best competitors in the world.

But as much as those achievements mean to me, they pale in comparison to the joy of watching my students grow. There’s nothing quite like seeing the transformation in people. I’ve seen shy individuals blossom into confident athletes, not only excelling on the mats but also gaining a new sense of self. I’ve worked with women who’ve faced difficult challenges, and through jiu-jitsu, they’ve found strength, empowerment and a community that supports them.

Recently, one such moment after a women-only jiu-jitsu class I teach in Amsterdam, as students left with smiles and a sense of accomplishment, my teacher hugged me and whispered, “you’re changing lives.”

I will be competing again, and the medals and titles are great, but the real reward comes from knowing that, through jiu-jitsu, I’ve been able to help others discover their own power and confidence.

From Corridor Coincidences to Cellular Cartography: The grlc Story

eScience Editorial Team — Thu, 24 Oct 2024 00:00:00 GMT

This is the story of grlc, a tool that transforms complex linked data queries into user-friendly web interfaces. We interviewed grlc’s developers Albert Meroño-Peñuela and Carlos Martinez-Ortiz, to explore grlc’s origins, development, and real-world impact. We’ll also hear from Bruce Herr, a user from the HuBMAP software project who’s putting grlc to work in groundbreaking research. The story is a testament to the power of open-source collaboration and the unexpected journeys of software tools.

Photo by Steve Richey on Unsplash

Part 1: In Conversation with grlc Developers

Q: Let’s start at the beginning. What exactly is grlc, and what problem does it solve for its users?

Albert: grlc is a server that automates the publication of knowledge graph APIs. It takes SPARQL queries as input and generates a functional Open API specification without requiring any coding.

Imagine you have a collection of linked data about music bands that you want to share with music enthusiasts who don’t know SPARQL. With grlc, you can write a SPARQL query to retrieve band information, put it on GitHub, and grlc will create an API that allows users to access this data using a simple URL. Users can even modify parameters in the URL to get information about different bands, effectively exploring your data without needing to understand the complexities of SPARQL.

Here is a traditional SPARQL query for a linked dataset:

A traditional (complicated) SPARQL query without using grlcIf you are a researcher only interested in music genres, you may not have the time or energy to learn SPARQL. grlc makes it easy to do the same thing:

The grlc generated API for the same query displayed aboveThe output of this is a URL, through which you can view and download the data you are after:

The URL that grlc generated through which you can access the data. You can view this data in your browser, but you can also use your favourite programming language to fetch and analyse data from different genres

Q: That’s quite useful. How did the idea for grlc come about?

Albert: The story of grlc begins in 2016. The original contributors were developers who frequently found themselves creating Open API specifications for SPARQL endpoints. Like many programmers, they were looking for ways to automate this repetitive task.

While some tools existed at the time, they all had limitations — either requiring SPARQL queries to be buried in a database or failing to comply with the emerging Open API specification (then known as Swagger). Faced with these constraints, they decided to build a better solution themselves. This solution ended up becoming grlc.

Q: How did the eScience Center play a role in the development of grlc?

Carlos: I was an eScience Center RSE at the time (now Community Manager, ed.), working on a digital humanities project, building a linked data exploration tool. The other team members on that project had diverse expertise, so they needed a way to separate concerns — front-end developers shouldn’t need to know about data storage, and data specialists shouldn’t have to worry about front-end presentation.

That’s when I discovered grlc. Albert, the original grlc developer was coincidentally working just down the hall, but we had never met. I started contributing to improve the codebase, bridging the gap between our two teams.

After that project concluded, another colleague at the eScience Center began candYgene: a life sciences project using linked data. I suggested using grlc, and as they reported issues and suggested improvements, we kept enhancing the tool. It’s been a rewarding cycle of continuous improvement driven by real-world use cases.

Q: It’s fascinating to see how tools evolve in practice. How is grlc being used now, and how do you track its adoption?

Carlos: One of the exciting aspects of open-source development is seeing your creation take on a life of its own. We often discover new use cases when users report issues or reach out for support.

A particularly interesting example is HuBMAP (the Human Biomolecular Atlas Program) using grlc in their groundbreaking work. It’s rewarding to see our tool contributing to such important scientific endeavors, especially ones we never anticipated when we first started development.

Part 2: grlc in Action — A Conversation with a HuBMAP User

Q: Can you tell us a bit about what HuBMAP is and what you’re working on?

Bruce: HuBMAP, or the Human Biomolecular Atlas Program, is an NIH-funded consortium with an ambitious goal: building a comprehensive map of the human body at the cellular level.

I’m part of the Indiana University Mapping Component, where we’re creating the Human Reference Atlas (HRA). It’s a fascinating project that combines ontologies, expert-curated data (including 2D and 3D anatomical imagery), and experimental data from various providers. The end result is an interconnected map of the human body that’s opening new frontiers in biomedical research.

Q: That sounds like complex and important work. How does grlc fit into this picture?

Bruce: grlc has become an important tool for us. The HRA is built as a set of RDF graphs, which allows us to represent complex relationships in our data. However, we often get requests for specific data from researchers who may not be familiar with SPARQL or RDF.

This is where grlc comes in handy. We can write a SPARQL query to extract the requested data, and grlc transforms that into a user-friendly interface and API. It’s accessible to both programmers and non-programmers, which is crucial given the diverse backgrounds of our collaborators and data users.

Q: So grlc is bridging the gap between your complex data structure and the end-users who need that data. What specific capabilities has grlc unlocked for your team?

Bruce: The key advantage is that grlc allows us to maintain our data in a rich, expressive format (RDF) while still providing straightforward reports and interfaces for our end users. It’s lowered the barrier for getting derived data products out to folks who need them.

Before grlc, there was often a tension between having a powerful, flexible data structure and providing easy access to that data. grlc helps resolve that tension. We can focus on creating comprehensive RDF graphs and writing powerful SPARQL queries and be confident that we can easily share the results with collaborators who might not have the same technical background.

Q: How did you first discover grlc?

Bruce: It was actually through a collaboration. Around 2022, we were working with EBI (European Bioinformatics Institute), one of our partners. We noticed they were using this tool called grlc for their Ubergraph project. At the time, we had our own SPARQL server and were looking for better ways to share queries and data. After seeing how effectively EBI was using grlc, we decided to give it a try. Now we’re using it for sharing queries and data in our CCF (Common Coordinate Framework) project.

Q: Looking ahead, do you see the sustainability of tools like grlc as a potential consideration for large-scale projects like HuBMAP?

Bruce : It’s definitely something we think about. The continued development and maintenance of grlc is valuable to HuBMAP and the HRA project. While we could theoretically pivot to another technology if needed, grlc has become an important part of our workflow.

It’s a good example of how open-source tools can become key components of scientific projects. We appreciate the work the grlc team has done and continues to do. Their efforts are helping to advance not just our project, but potentially many others in the scientific community.

Interview by Lieke de Boer*

The magic of teaching about the anti-magic of generative AI

Candace Moore — Thu, 03 Oct 2024 00:00:00 GMT

On the use and abuse of generative artificial intelligence by students

Photo by Owen Beard on Unsplash

Recently, I taught the pilot workshop Medical Image Processing in Python. The room was full of post-docs, research software engineers and even faculty, but mostly young PhD candidates or freshly minted PhDs. Although the Netherlands eScience Center offers many courses and workshops, most are established. Many are based on existing Carpentries materials that are taught globally. Recognizing a large gap in the existing curriculum in terms of computational medical image processing, the Netherlands eScience Center embraced the idea of starting some sort of workshop about it. Part of the gap, in my opinion, was a lack of a curriculum that addressed generative AI.

A few brave engineers at the Center designed their own workshops. This year, I did just that, and the process was at once terrifying and energizing. On the weekend before the course, I literally had a nightmare that the students would revolt during the workshop. I dreamt about angry PhDs in computer science yelling at me and the other course developer, a biomedical engineer, literally standing up during my MRI lecture, eyes rolling and correcting me about how spin echo MRI sequences were made.

By the end of the day, none of this had happened, yet I expected the students to be exhausted and perhaps annoyed. I had stuck a segment on generative AI onto the end of the day, and was dedicated to teaching it, but feared the student revolt of my nightmares might still be coming.

The role of instructor or faculty can often seem like a bit of a sham in the world of computation-driven health sciences research, and particularly any kind of medical signal processing. On the one hand, the theory behind signal processing is something instructors need to teach to students. On the other hand, all of the theory now needs to be implemented in the digital world, and the ability to implement anything with code well does not seem to track with rank. I have been around this part of academia long enough to know that the pattern is that often students should be teaching the teachers about technology, even if on paper it is the other way around. Would the students be annoyed about the relatively high amount of time we assigned to learn basic code for things like registration and segmentation?

Or would they just be exhausted? After all, since morning we had all written lots of code, accomplishing many non-trivial feats like visualizing image registration in SITK and augmenting a dataset for machine learning. As I rounded the corner towards the end, showing what pydicom could do in terms of anonymizing DICOM metadata, I realized I had an urgent last message to impart. How would I ever make people continue to pay attention?

I took a breath and wondered if the students would start walking out, ready to leave after making it through most of the day. None of them did. Instead, to my surprise, students began engaging more during my last lecture. I could see them thinking about what I was saying.

Perhaps the topic of generative AI was not as ‘last year’ as I had imagined. I thought everyone had discussed generative AI to death at this point, and certainly, I wouldn’t point out anything younger people didn’t already know. I feared that I might seem like a grandma to them, still mystified and confused by the remote control. It turns out I was wrong about all of that.

I began my generative AI lecture by showcasing some absolute failures from a well-known generative algorithm in terms of radiological image generation. I showed how even with specific, correctly spelled prompts, I could get back imaging that looked like a computer graphical rendering from 1986 of a circle when prompting about things like an MRI of polycystic kidneys.

Image generated in response to prompt for a T2 MRI axial slice of polycystic kidney disease

I then showed some of the coding fails I have seen other people create with ChatGPT. The code I showed was far from ready to run, but it was what I had seen. The code began with approximately the following:

path_to_niftis = Z:\directory1\subdirectory1/subdirectory2\subdirectoy3
path_to_meta_data = directory1\subdirectory1/subdirectory4\subdirectoy5
variable_1 = some_never_specified_variable
def funct1():
  for variable_1:
  sub_folder = some_other_never_specified_variable

Sadly, the code did not improve from there. I then pointed out to the students that in both cases the problem here was that anyone had ever taken these outputs (the bad image and the bad code) seriously. Unfortunately, someone had, in the case of the code, and they approached me to review what they had prompted. With one glance, I asked, disingenuously “Did you actually run this?” knowing that at least half the lines would fail to run given that they included variables that were never assigned.

Afterward, I immediately wrote down some notes for my upcoming lecture. I stumbled upon yet another perfect example of one of the counter-intuitive problems of generative AI that I see all the time. To use these algorithms properly, you need to have deep knowledge of what you expect them to generate. At least in the field of medical imaging research, you cannot leapfrog over learning how to code or spending time with radiologists and pathologists to understand what proper images look like. If you are expert enough at coding, generative models can save you time by helping you debug and write boring code quickly (or so I hear from my more skilled colleagues who use these things). Now there is even some science behind such claims, although the obvious conflicts of interest should have been stated explicitly in the linked article. But if you have not developed certain skills, generative AI will not necessarily save you time, as opposed to say getting people to share their data and code with you.

Unfortunately, I suspect the sudden uptick in student attention and involvement was not a reflection of my teaching. Ask anyone who teaches graduate students these days, and you will likely hear some stories about ChatGPT or Copilot or some other generative algorithm. Students have embraced these tools as if they were a sort of digital magic even though there is nothing magical about these models. Speak the right spell over the bubbling pot, or server in this case, and the entire world will change into a place where you solve all problems in time to spend hours doing something other than studying and learning. The promise that seems to have been marketed is that not only will your code be written and your work done, but you will be transformed into a wizard who can warp the time-space continuum in such a way that you get free time.

Image generated by Openart.ai after several prompts by the author

There is a bit of truth to this idea. What I didn’t tell the students is that code is by far the easy case. Here even a mediocre engineer can deploy a bit of witchcraft and potentially transform AI-generated code into something usable. The witchcraft is manual and automated testing. Sometimes all you have to do to improve AI-generated code is to try to get it to run and to write some tests for it. Many of my colleagues are actually getting some useful results out of generative tools.

With images, things get murkier. There is some published work showing that there are synthetic images good enough so that even expert radiologists could not detect that they were not real. But radiologists are not computers, and typically not engineers either. Rumor has it that in some unpublished studies, while the radiologists were fooled, the software engineers were not. They had seen so much of their own work they could recognize it. I suspect that in terms of machine learning features, generative algorithms might be able to create subtly detectable differences, and therefore a machine learning algorithm could potentially encode some bizarre unintended biases based on these features. I gave the students a more obvious example of potentially doubling down on biases in mammography. What if you want to balance a dataset with more breast cancer samples but all the synthetic data you create shows low-density, relatively fatty, breasts? Then you can potentially increase the correlation between this low density and breast cancer in your dataset. This kind of problem would be fairly easy for human eyes to see, but what about more subtle ones?

Alas, there are, as of yet, no magic-based solutions to some problems we face in medical imaging research. Maybe the younger generation, today’s students are still hoping for this, because some basic tasks in research now do have solutions that work at astounding speed. Computation on modern computers seems like magic, especially if you do not see GPUs, but hear about some mystical cloud where these things live. The mistake I have seen students make is to double down on generative AI. “I must not be prompting this correctly”, they say when things go wrong, based on the assumption that with the right spell, precisely the code and images they want will appear. But could they even recognize them as such if they did?

The mistake I have seen students make is to double down on generative AI. “I must not be prompting this correctly”, they say when things go wrong, based on the assumption that with the right spell, precisely the code and images they want will appear. But could they even recognize them as such if they did?

There is some level of debate in education about the relative importance of teaching facts and information as opposed to teaching methods and skills for discovery. Some people even argued that learning information was already obsolete in the era of search engines and instant access to information. Potentially generative AI is a method of discovery that can amplify anyone’s capabilities. However, I would argue that some amount of foundational information is necessary to get the most from generative tools. What I hope anyone educating university students reading this will take away from learning about my experience is that addressing the issues of Generative AI head-on will make your students stop playing with ChatGPT, sit up, and listen. And maybe even rededicate themselves to including alternative, older forms of knowledge acquisition alongside embracing generative-based methods. Whatever the case, talking about this stuff seems to have almost supernatural effects.

National Research Software Day: National Infrastructures for Sustainable Software

eScience Editorial Team — Thu, 19 Sep 2024 00:00:00 GMT

*Written by Luisa Orozco, Daniela Gawehns, and *Carlos Martinez-Ortiz

The first National Research Software Day already took place more than five months ago in Hilversum, on 23 April 2024. Following an inspiring keynote by Rogier Kievit, several parallel sessions competed for the attention of participants. This blog covers the ‘National infrastructures for sustainable software’ session.

This session was a follow-up from a similar session that took place during the Dutch Open Science Festival in 2023. On this occasion, the format was the same: a series of panel pitches, followed by a group discussion. The panel shared their insights and experiences related to infrastructure used for creating and maintaining sustainable research software. The composition of the panel was different on this occasion, so the discussion took a different direction.

Luisa Orozco, RSE at the Netherlands eScience Center, led the discussion.

After the session, the panelists were also interviewed by Peter Schmidt for the Code for Thought Podcast.

Panel composition and introduction

Elaine van Ommen Kloeke, ARISE

At Naturalis we build a digital infrastructure for recognizing species biodiversity. Data is huge, vertebrates, plants, insects, fungi. It is a huge challenge to capture, store and move data.

Louise Bezuidenhout, CWTS Leiden University

Senior researcher focusing on Open Science monitoring and the evolution of Open Science infrastructures. The CWTS publishes the annual Leiden University rankings using Open Data, based on the Leiden manifesto which advocates for open and transparent research assessments moving beyond the normal metrics.

Jason Maassen, Research Software Directory

Jason is a Technology Lead at the Netherlands eScience Center supporting and helping researchers develop software. The eScience Center leads the Research Software Directory (RSD), a platform developed to highlight the roles of RSEs in research through links between software and other kinds of research outputs such as datasets, publications, research activities, projects and people.

Roel Janssen, 4TU.ResearchData

4TU.ResearchData stores, archives and publishes datasets for technical universities in the Netherlands. I have the opportunity to work together at the National level with, for example, the RSD, to implement APIs and standards to help the deployability of research data and software.

Discussion

Which kind of software, or which kind of infrastructure do you need in your day-to-day work or your institutes?**

Elaine: There’s nothing standard that we can use. Some parts we can reuse and are open-source, while others have to be built from scratch. I need data storage, computing, dimension systems, PID identification systems, authentication, authorization and I need it to be user-friendly and encourage collaboration.

*Louise: *I need access to data and knowledge graphs that we can use for our meta-research.

How does the RSD and 4TU cater to the needs that researchers have in terms of infrastructure? Which niche or which solution are you targeting?

Jason: With the RSD, we link software to other research outputs and activities, integrating information needed for institute assessment. Also, it is a useful tool for researchers to find software.

*Roel: *At 4TU we offer data and software repositories, more data storage is a common request. We also offer computation environments such that a given code can be run alongside the data.

We also strive for recognition of good software, so that a user can find and reuse the software. We also try to make data publications more attractive to researchers.

When using those infrastructures, what are the boundaries or limitations that you encounter? For example national vs international, or open/closed/paid.

*Elaine: *My first criterion is ‘does it get the job done’? Ideally open and reusable.

*Louise: *I work with computer scientists examining how accessible these infrastructures are to users globally. We have used VPNs to access resources and found significant variability not only in terms of access speed but also in geographical accessibility. This variability raises questions about the impact of resource location, funding models, and user requirements on accessibility. We need to critically evaluate the geopolitical landscape surrounding infrastructure choices, take the example of GitHub, which is inaccessible to users in countries currently under financial sanction by the US.

From more of a provider side: What are your boundaries or limitations? How do you decide who is your public and how far can you get?

Roel, you mentioned new features that the users were requesting. How do you handle those requests?**

Roel: We do receive more feature requests than we can implement due to limited manpower. To manage this, we prioritize based on ease of implementation, sometimes responding quickly and other times taking longer due to extensive planning required. At community events like those within 4TU, we prioritize requests from partners, universities, and funders, addressing recurring ones promptly. Additionally, we anticipate future needs by observing trends in software usage, implementing APIs for upcoming demands even before they are explicitly requested. It’s a balancing act between fulfilling immediate requests and anticipating long-term needs to stay ahead of the curve.

Questions from the public

GitHub plays a central role in software development and it is a potential point of failure: it is centralized in nature, American-owned and has the possibility of being closed down, similar to what happened with Google Code. This is a vulnerability not just for the Netherlands but globally. How can these risks be mitigated?

Roel: There was a similar issue in the past with SVN and SourceForge. Unlike then, modern version control systems like Git offer a distributed model, where each developer has a complete copy of the source code, making it easier to switch platforms if needed. SoftwareHeritage also stores a copy of everything stored on GitHub. However, transferring auxiliary components like Wikis and issues remains a challenge. Continuous integration tools like GitHub Actions, while powerful, can be proprietary and tied to specific platforms, raising concerns about dependence on a single provider.

Louise: Now that the OpenScience movement is gaining momentum, the Open Science community should have a better dialogue with companies such as GitHub, and together find a suitable way of working. This type of change has already started, for example, with the publishing industries which have also been changing in this direction. The decision to create a national dataverse is also an outcome of these dialogues.

Jason: You do see a lot of organizations looking for alternatives, for example, running local GitLab instances. But these alternatives also take time and cost money. When organizations realize how much time and effort it costs, they often back off and turn back to commercial providers.

Elaine: It depends on what you are trying to achieve. I could run my own data management system, but then I would need 15 dedicated engineers, and I only have two. I want those two to focus on other things. It is a bit of a balance between being principled and being pragmatic. There needs to be a conversation between research-performing organizations and commercial companies. I have no problem using a commercial company, as long as I keep the option to move my data.

There is an issue with depending on commercial companies, because if, for example, they change their license, you may need to change things in your own software. The same applies if they change formats — I have a lot of data from 30 years ago in Microsoft formats that I am unable to read anymore. You are putting yourself at risk!

Elaine: It is the same story with a Discman, it is the natural evolution of products and services that occurs everywhere, all the time. We now no longer use Discman today, but rather we use Spotify. We still listen to music, but if you insist on sticking with a Discman and are not prepared to move to new tech, you can get stuck. It is not something that applies to commercial products only. It is a risk that exists and that you need to take into account and plan for sufficiently in advance. You need to be aware of what risks exist, and what alternatives are available.

Roel: Maybe it is preferable to keep options as generic as possible. So instead of having a button that says “link with GitHub” have a more generic “link with any git version control”. It is useful to show which options you provide and that there are alternatives.

Do you think having a centralized way of operation is the solution?

Jason: For many types of infrastructure, perhaps that would be a viable option. That would be the case for a national PID system. It is the type of infrastructure that everybody needs, but nobody wants to build it or pay for it.

Roel: Maybe in terms of standards, if we have a well-defined standard and many options of implementation, then that would be the best.

Is there a way to see at an institutional level how much data and software have been produced? For example, PURE registers all your publications, but for data and software, it is not done as much.

Jason: For publications, publishers harvest and analyze this information. Technically, it is not that hard to do, but it is not consistently done for data and software. One thing we have done is look at OpenAlex and try to connect all of these resources and figure out if we can identify citations for software. What you see is that there are many citations for publications, some citations for datasets and very few citations for software.

Louise: At CWTS we are looking into this, and we’ve been working with the Center for Digital Scholarship, but we do not have solutions yet.

At Digital Humanities Lab we are looking into how to make our software more visible. But if we offload all the metadata to the RSD, how many people will use it? You also need to engage in how to make users find it. For developers, the incentive to create software is knowing it will be used.

Jason: We are currently working with different communities to create community-specific views, the goal is that communities themselves curate the content by ensuring keywords are relevant for the target users. We are also looking into integrating the RSD with search engine tools so that software can be found more easily.

What is the bus-factor for the RSD/4TU.ResearchData? What challenges do you face in increasing it?

Jason: Currently, at the eScience Center the bus-factor is three. This is a great increase from one, which is what it was a few years ago. We are also collaborating with other organizations in Germany, which increases it. But finding someone to keep pushing the software remains a difficult challenge.

Roel: Our bus factor is two and a half or three, spread over multiple people over the team. We do not have enough money available to grow the team, but we do have multiple people who understand how everything works.

Wrap-up

What would you like participants of this session to remember as their take home message. Something you would still like to say?

Elaine: Keep talking! People hate meetings, but getting together is how people share ideas, especially across domains but also for setting standards.

Louise: Keep talking, also to people who you never talked to before. The range of stakeholders in science is very broad: it is not limited to academia, not limited to the Netherlands.

Jason: Recognition not only for those developing great research software but also for those engaging in promoting best practices, infrastructures and in setting standards.

Roel: For me, if you have any ideas on how to improve repositories please get in touch!

Thank you to all of our panellists for their participation. These are a lot of topics that are very interesting for us and it is great we could have this nice conversation!

(Un)wieldy workflows with WRF

Peter Kalverla — Thu, 05 Sep 2024 00:00:00 GMT

(Un)wieldy workflows with WRF

Streamlining experiments with the Weather Research and Forecasting model

The Weather Research and Forecasting model, WRF for short, is a beautiful piece of research software with a large community of active users. That’s something we love and advocate for at the eScience Center.

Given its large user base, one might think this must be an exemplary model to work with. And perhaps it is, in comparison to other models in the scene. But as I’m working with WRF again after several years, I’m reminded of my struggles in keeping my workflows organized.

Photo by Bernd 📷 Dittrich on Unsplash

To try and tackle this issue once and for all, we experimented with different solutions, including a simple shell script, as well as Snakemake, a python-based workflow management tool often used in bioinformatics. Both approaches turn out to have their own merits and quirks.

Overall, we did get a better grip on the antipatterns that prevented us from keeping a nice and orderly workflow. From this, we extracted some best practices for managing complex (WRF) modelling workflows. This blogpost documents our journey and the lessons learned, in the hope that it will be useful for other WRF users and those who struggle with managing their experiments.

Running basic WRF**

Let’s start with an overview of the steps required to run a basic WRF (real data) experiment, as per the instructions in the user guide. First you must run the WRF preprocessing system (WPS), which itself consists of three programs: geogrid, ungrib, and metgrid. Then you run WRF itself, which also consists of two programs: real and wrf.

cd WPS**

Run geogrid (interpolate spatial datasets to the model grid)

./geogrid.exe

Run ungrib (unpack meteorological input data)

./link_grib.csh /your/meteo/data/dir ln -s ungrib/Variable_Tables/Vtable.GFS Vtable ./ungrib.exe

Run metgrid (interpolate the meteo data to model grid)

./metgrid.exe

cd ../WRF/run

Run real (extract initial and boundary fields from preprocessed data)

ln -sf ../../WPS/met_em* . ./real.exe

Run WRF (the actual forecast)

./wrf.exeThis workflow is not great in several ways. WRF and WPS are run in different working directories; yet they “pretend” to be in the same directory by use of symbolic links (ln). Also, exectuables are in the same directory as the input/output data. Now, imagine, for example, that we want to do multiple experiments with slightly different settings.

One option would be to move the output to a separate directory between each run. We would need to meticulously bookkeep our files, especially since runs may fail at different stages causing us to re-run (part of) the workflow. Also, we would not be able run the experiments in parallel. Clearly, using WRF in this way is a labour-intensive, error-prone process.

Alternatively, we could duplicate our WPSand WRF/rundirectories. That’s also not great: imagine we want to re-run our experiments with a re-compiled version of WRF. Then we would need to copy over the new executables to all experiment directories.

Finally, there’s lots of files in these directories that are not needed for our experiments, which makes it hard to tell the wheat from the chaff.

Our workflow wishlist**

Considering the points above, let’s think about how we would like it to be, and start a workflow wishlist:

Each/all experiment(s) can be re-run with a single command
One directory per experiment, containing config + output
Intermediate/output data well organized in subdirectories
Config files (namelists) specify paths to all relevant input data; no links
Executables should be in a system path or another dedicated location
We should be able to (re-)run only part of a workflow
Bonus: reuse intermediate files that are the same for all experiments

Solution #1: using a shell script

Let’s see how far we can get with a simple shell script. In this example, we maintain multiple namelists and geogrid tables for each experiment. These are stored alongside the shell script and copied over to the experiment folder once the script is run. The full code is available here.

Some initialization of the environment, module loads etc.**# …

Set path to executables

WPS_HOME= $HOME/wrf-model/WPS WRF_HOME=$ HOME/wrf-model/WRF

Define experiment name

EXP=USGS

Make new experiment directory

export RUNDIR=wrf_experiments/ ${EXP} mkdir -p$ RUNDIR mkdir $RUNDIR/geogrid mkdir$ RUNDIR/metgrid

Copy experiment-dependent files

cp namelist.wps_ $EXP$ RUNDIR/namelist.wps cp namelist.input_ $EXP$ RUNDIR/namelist.input cp GEOGRID.TBL.ARW_ $EXP$ RUNDIR/GEOGRID.TBL

Copy additional input files from WRF/WPS

cd $RUNDIR cp$ WPS_HOME/metgrid/METGRID.TBL.ARW METGRID.TBL cp $WPS_HOME/ungrid/Variable_Tables/Vtable.ECMWF VTable cp$ WRF_HOME/run/CAMtr_volume_mixing_ratio.RCP8.5 CAMtr_volume_mixing_ratio cp $WRF_HOME/run/ozone* . cp$ WRF_HOME/run/RRTMG* . cp $WRF_HOME/run/*.TBL .

Run experiments

$WPS_HOME/link_grib.csh "path/to/our/input/data/*"$ WPS_HOME/ungrib.exe $WPS_HOME/geogrid.exe grep "ERROR" geogrid.log && echo "Aborting: ERROR in geogrid.log." && exit 1$ WPS_HOME/metgrid.exe grep “ERROR” metgrid.log && echo “Aborting: ERROR in metgrid.log.” && exit 1

$WRF_HOME/run/real.exe$ WRF_HOME/run/wrf.exeEvidently, WRF and WPS can be run from another directory, by calling them by their full path. Alternatively, we could have added the WPS and WRF/run directories to our PATH variable. However, WRF and WPS expect certain files (tables etc.) in the working directory, so we need to copy those.

Geogrid and metgrid tables are expected in subfolders, but we can use the namelist setting opt_geogrid_table_path='.' (default is './geogrid') to overwrite that, and similar for metgrid. Geogrid will then look in that folder for a file called GEOGRID.TBL. (Why not simply point to the geogrid table directly in the namelist?)

We were able to declutter our working directory somewhat by using the namelist.wps options opt_output_from_geogrid_path and similar for metgrid. Unfortunately, that option is not available for ungrib, the worst clutterer of all… We were able to point real.exe to our metgrid subfolder through the namelist.input parameter [auxinput1_inname](https://github.com/wrf-model/WRF/blob/0a11865f97680fdd6865b278ea29d910e5db3ed7/run/README.namelist#L89), so we could skip that linking step as well.

We wanted the workflow to fail when one of the steps fails. However, metgrid and geogrid don’t fail with a proper exit code. As a simple patch, we added extra commands to look for errors in the log files and raise an exception when needed.

Despite some peculiarities, this is a good start. We can already tick some of the items off our wishlist. However, the workflow quickly becomes more complex. For example, as we are adding more experiments, we would like to edit the namelists programmatically. And for some runs, we may want to run an intermediate step (e.g. wudapt-to-wrf). Before we know it, the shell script becomes messy, which defeats our purpose of maintaining a tidy and organized workflow.

To summarize, the following namelist parameters should probably be much better advertised:

opt_geogrid_table_path (namelist.wps)
opt_metgrid_table_path (namelist.wps)
opt_output_from_geogrid_path (namelist.wps)
opt_output_from_metgrid_path (namelist.wps)
[auxinput1_inname](https://github.com/wrf-model/WRF/blob/0a11865f97680fdd6865b278ea29d910e5db3ed7/run/README.namelist#L89) (namelist.input)

With a similar option for ungrib, the WRF experience would already be much better.

Solution #2: using Snakemake**

Ideally, we want to be able to run only part of a workflow. For example, when real.exe fails, we don’t want to re-run WPS. Similarly, we may want to run all pre-processing first, and then submit WRF as a batch job.

Brief intro to Snakemake

Example Snakemake rules (source https://snakemake.readthedocs.io/en/stable/project_info/faq.html#what-is-the-key-idea-of-snakemake-workflows)A snakemake workflow, aka ‘Snakefile’, consists of rules. Each rule has certain inputs and outputs, and the commands to produce that output, either in Python code (a run block) or as shell commands (a shell block). When you tell snakemake to produce a certain output file, it will check which rule must executed in order to obtain that file. It does so in a smart way: if the required for a rule input already exists, it will not re-run that rule, unless the input is newer than the output.

Our desired workflow

Before we delve into details, let’s have a look at the workflow that we want to achieve. For demonstration purposes I will focus on WPS only. Snakemake provides a command (snakemake --dag WPS | dot -Tpng > dag.png) to visualize the task graph for our workflow:

Task graph for our WPS workflow. Boxes (nodes) are rules, arrows indicate dependencies between rules. Dashed lines indicate that these rules have already been executed. Full-line rules still need to be (re)run.As you can see, we want to run four independent experiments. While they share all the same steps, they have a different value for the experiment setting. This is achieved through the use of a wildcard*.

Providing experiment settings

The values for the wildcard are provided to snakemake via a configuration file. This is a relatively simple file which, in addition to our experiment-specific settings, contains some global settings like wps_home. Our config file looks like this (only showing the first experiment):

wps_home: ~/Urban-M4/WPS wrf_home: ~/Urban-M4/WRF data_home: /projects/0/prjs0914/wrf-data/default working_directory: ./output

experiments: usgs: geog_data_res: [‘usgs_30s’,‘usgs_30s’,‘usgs_30s’,‘usgs_30s’] geog_data_path: ‘/projects/0/prjs0914/wrf-data/default/static/summerinthecity’ geogrid_table: GEOGRID.TBL.ARW_USGS use_wudapt_lcz: false num_land_cat: 24 modis_lcz: …The config file is read at the very top of the Snakefile, like so:

configfile: workflow.source_path(“config.yaml”)The first rule in our workflow, UPDATE_NAMELIST_WPS, uses a python package [f90nml](https://f90nml.readthedocs.io/en/latest/) to load a reference namelist.wps, modify the settings as specified in the experiment configuration, and store the updated namelist in the output folder for that experiment. It looks like this:

rule UPDATE_NAMELIST_WPS: input: namelist_wps = workflow.source_path(“namelist.wps”) output: “{experiment}/namelist.wps”, run: # Get parameters from config geog_data_res = config[‘experiments’][wildcards.experiment][‘geog_data_res’] geog_data_path = config[‘experiments’][wildcards.experiment][‘geog_data_path’]

    # Read source namelist.wps with f90nml, make some changes, and save in experiment dir
    nml = f90nml.read(input.namelist_wps)
    nml["geogrid"]["geog_data_path"] = geog_data_path
    nml["geogrid"]["geog_data_res"] = geog_data_res
    nml.write(f"{wildcards.experiment}/namelist.wps")Notice that we are reading the reference namelist form the workflow source path. This is because it is stored alongside the snakefile. Normally, all paths in snakemake are relative to the working directory.

Importantly, this rule works for any value of the experiment wildcard (as long as the settings are provided in the configuration file). To invoke the rule for each of our experiments, we use a target rule, like so:

rule PRODUCE_ALL_NAMELISTS: input: collect(“{experiment}/namelist.wps”, experiment=config[“experiments”].keys())The collect function fills the values of experiment with all keys found in config["experiments"]. When we invoke this target rule, e.g. by running snakemake PRODUCE_ALL_NAMELISTS, Snakemake will look for a rule that can produc the following files (unless they already exist):

usgs/namelist.wps
modis_lcz/namelist.wps
wur/namelist.wps
wur_lcz/namelist.wps

It will find UPDATE_NAMELIST_WPSand invoke this rule, inferring that the first segment corresponds to the value of the wildcard.

Specifying the other steps

Getting Snakemake to use the different experiment settings was by far the most involved part of our journey. Next, we want to add rules for all the other steps. For example, out METGRID rule looks as follows:

rule METGRID: input: “{experiment}/namelist.wps”, “{experiment}/finished.ungrib”, “{experiment}/finished.geogrid”, output: “{experiment}/finished.metgrid” shell: """ cd {wildcards.experiment} mkdir -p metgrid

    cp {config[wps_home]}/metgrid/METGRID.TBL.ARW METGRID.TBL
    {config[wps_home]}/metgrid.exe

    # Scan logfile for errors and raise if necessary (metgrid can fail silently with 0 exit status)
    grep "ERROR" metgrid.log &amp;&amp; echo "Aborting: ERROR in metgrid.log. " &amp;&amp; exit 1

    touch finished.metgrid
    """Snakemake infers the dependecies between rules via their input and output filenames. Since WPS output filenames are not predictable without parsing the namelist, we opted for a workaround: at the end of each rule, we create an empty file called `finished.&lt;rule&gt;`. As such, we inform Snakemake that metgrid must be run after updating the namelist, and after geogrid and ungrib.

As opposed to the first rule, which executes Python code (a run block), here we use shell commands. This is nice, because it is very similar to what you would do manually on the command line. As an intermediate option we can use params. We use this in our GEOGRID rule:

rule GEOGRID: input: “{experiment}/namelist.wps”, output: “{experiment}/finished.geogrid” params: geogrid_table = lambda wildcards: workflow.source_path(config[‘experiments’][wildcards.experiment][‘geogrid_table’]), shell: """ cd {wildcards.experiment} mkdir -p geogrid cp {params.geogrid_table} GEOGRID.TBL

    # Run geogrid
    {config[wps_home]}/geogrid.exe

    # Scan logfile for errors and raise if necessary (geogrid can fail silently with 0 exit status)
    grep "ERROR" geogrid.log &amp;&amp; echo "Aborting: ERROR in geogrid.log." &amp;&amp; exit 1

    touch finished.geogrid
    """The use of params enables us to execute some Python code to assign a value to a variable, which can then be accessed in the shell block of the rule. Here, we get the `geogrid_table` from our experiment config. So we can combine the flexibility of Python with the transparancy of plain shell commands.

Final workflow

Putting everything together, our full workflow looks something like below. Notice that WPS is the target rule in this case.

import os from pathlib import Path

import f90nml

configfile: workflow.source_path(“config.yaml”) workdir: config[“working_directory”] envvars: “NETCDF”

    # Read source namelist.wps with f90nml, make some changes, and save in experiment dir
    nml = f90nml.read(input.namelist_wps)
    nml["geogrid"]["geog_data_path"] = geog_data_path
    nml["geogrid"]["geog_data_res"] = geog_data_res
    nml.write(f"{wildcards.experiment}/namelist.wps")

    # Run geogrid
    {config[wps_home]}/geogrid.exe

    # Scan logfile for errors and raise if necessary (geogrid can fail silently with 0 exit status)
    grep "ERROR" geogrid.log &amp;&amp; echo "Aborting: ERROR in geogrid.log." &amp;&amp; exit 1

    touch finished.geogrid
    """

rule UNGRIB: input: “{experiment}/namelist.wps” output: “{experiment}/finished.ungrib” shell: """ cd {wildcards.experiment}

    # Remove old output if present (ungrib doesn't like to overwrite stuff)
    rm -f FILE*

    # Link vtable
    cp {config[wps_home]}/ungrib/Variable_Tables/Vtable.ECMWF Vtable

    # Link gribfiles
    {config[wps_home]}/link_grib.csh {config[data_home]}/real-time/july2019/*

    # Run ungrib
    {config[wps_home]}/ungrib.exe

    # Scan logfile for errors and raise if necessary (ungrib can fail silently with 0 exit status)
    grep "ERROR" ungrib.log &amp;&amp; echo "Aborting: ERROR in ungrib.log." &amp;&amp; exit 1

    # Report ready
    touch finished.ungrib
     """

    cp {config[wps_home]}/metgrid/METGRID.TBL.ARW METGRID.TBL
    {config[wps_home]}/metgrid.exe

    # Scan logfile for errors and raise if necessary (metgrid can fail silently with 0 exit status)
    grep "ERROR" metgrid.log &amp;&amp; echo "Aborting: ERROR in metgrid.log. " &amp;&amp; exit 1

    touch finished.metgrid
    """

rule WPS: input: collect(“{experiment}/finished.metgrid”, experiment=config[“experiments”].keys())We’re steadily accumulating lines of code, but overall, we’re quite happy with how interpretable this still looks. As we develop this further, it would be good to split it up into multiple files, and incorporate other best practices as outlined in the Snakemake documentation.

Struggles

While we’re quite happy with our progress so far, here are some things that we struggled with:

Flexibility of output directories: we want to organize our output as follows: ./results/{experiment}. By default, all paths are relative to the working directory. We set the working directory to ./results, so the first rule, UPDATE_NAMELIST, will actually create the file ./results/{experiment}/namelist.wps. Unfortunately, snakemake cannot use wildcards for the working directory. Therefore, we need to cd to the experiment directory in every rule.
Complex input/output: the output of WPS programs depends on the settings in namelist.wps. Specifically, filenames like met_em.d03.2019... encode information about the domain and (a lot of) dates. We could use something like output: expand("met_em.{domain}.{datetime}.nc", domain=DOMAINS, datetime=DATETIMES). For that to work, we’d need to set DOMAINS and DATETIMES in the configuration file, or infer them from the template namelist.wps. Not ideal, and quickly getting complex.
Atomic inputs/outputs: a bit of a stretch goal, but when using a workflow manager like this, it would be nice if you could do, for example, metgrid.exe --domain=1 --date=20190101. This way, when only one domain/timestamp needs to be re-processed, we don’t need to re-run the workflow for everything else as well. However, this means some of the logic that’s currently encoded in WPS programs, specifically parsing the domain and time control sections of the namelist and looping over them, is moved entirely to the workflow. Do we really want to go down that path?
Shared input/output: now, we’re saving the output of each experiment in a dedicated directory. However, there is quite some duplication. Instead, we could store WPS output in a shared directory. But what happens if two namelists with different settings create the same file geo_em.d01.nc? We would need to encode the WPS configuration in the path somehow, e.g. geogrid/usgs**/geo_em.d01.nc and ungrib/**ifs**/FILE:20190101 combine to create metgrid/**usgs/ifs**/met_em.... This is currently not possible since the ungrib directory is not configurable.
SLURM: snakemake can submit workflows to a batch scheduler like slurm through a plugin. However, we found it to be very cumbersome. For example, we found the main Snakemake program was hanging often and did not seem to check for status; online help often referred to old versions; you can mark jobs for local execution when slurm is the default, but not vice versa as we would have preferred; the working directory was not respected; we had to carefully reinitialize the environment; et cetera. Eventually it seemed easier to wrap the entire Snakemake workflow in a batch script and submit that to slurm, but this is also suboptimal, especially since we cannot differentiate between resources for eacht task then.

Verdict

Some of these things can probably be solved by digging further into the inner workings of Snakemake. At the same time, we’re conscious of the complexity of our workflow. If it takes more time to build and maintain it than it would take to do the tedious manual bookkeeping, is it really worth the effort?

Hopefully yes: reproducibility is important, and perhaps others can benefit as well. For now, however, we need to focus on the results for a bit. So let’s do a quick recap and call it a day.

If you want to follow our progress, keep an eye on our repository here.

Conclusions

We’ve experimented with different options for managing WRF workflows. With a few tweaks, we were able to tackle many of our initial painpoints. Once we got the hang of it, we quite liked the flexibility of Snakemake and its promise of reproducible experiment sets. However, while we aimed for something simple, we found that our solutions quickly grew complex as we tried to bend the tools to our will.

There’s a few peculiarities about WRF that were very much in the way. For example, that we couldn’t specify the ungrib directory. In terms of compatibility with Snakemake, it would be nice if WRF was less dependent on the current working directory, and/or if Snakemake’s working directory would be able to adapt to wildcard values.

Perhaps the main conclusion should be that organizing your (WRF) workflows is not as straightforward as it may seem. This is something that many people must struggle with. And while there’s plenty of resources on the WRF documentation and user forum, “how to manage your workflow” does not seem to get the attention it deserves.

How do you manage your workflows? Please share your tips and tricks in the comments or in any other way that you see fit.

Research software is critical to the future of AI-driven research

eScience Editorial Team — Thu, 22 Aug 2024 00:00:00 GMT

By Michelle Barker, Kim Hartley, Daniel S. Katz, Richard Littauer, Qian Zhang, Shurui Zhou, Jyoti Bhogal

Cross-posted from ReSA and the Digital Research Alliance of Canada

Photo credit: TheMezereon

This position paper provides a statement on the criticality of research software in artificial intelligence (AI)-driven research and makes recommendations for stakeholders on how to consider research software in their AI goals. This is needed to ensure that the focus on technological infrastructure to support AI acceleration includes research software and its personnel as a vital part of that infrastructure. This paper discusses both research software that supports generative AI, which is now being explored today as a tool to enable new research, as well as more traditional machine learning, which has demonstrated impact in research in most disciplines (particularly in the last decade).

This paper begins by providing definitions for key terms and demonstrating the importance of research software in AI-driven research. The paper then explores the need for AI strategies to recognise research software as key building blocks of AI-driven research, and to include this element alongside a focus on computing systems, data, and models. The fact that AI is dependent on software (i.e., data preparation and model training are performed by software, models are implemented in software) is not always adequately considered. Alongside this high-level need for incorporation of research software in AI strategies, the challenges inherent in software dependencies also need consideration, as research software requires continual maintenance, updating, bug fixes, etc. Consequently, the broader challenges in enabling ongoing support for research software used for any type of research are equally relevant to AI-driven research and presented here.

There is also a need to better support the people who develop and maintain the research software that enables AI-driven research. Because human skills, training, and career paths often feature in AI strategies, it is imperative that stakeholders facilitate practices that support and recognise research software personnel. While the research community is exploring how to leverage AI to improve how research software is developed and maintained, this is also affecting how research software practices should be taught. The paper also illustrates how some countries are operationalising AI strategies that could support the critical element of research software, and have the potential to do so by building on existing investments in research software.

This paper was written in collaboration with the Research Software Alliance (ReSA) and the Digital Research Alliance of Canada, one of ReSA’s Founding Members. ReSA is a global organisation that unites decision-makers and influencers across the international research software community.

The Digital Research Alliance of Canada is funded by the Government of Canada to serve Canadian researchers, including the infrastructure and activities required for research software. Other community leaders also provided inputs from institutional and open-source perspectives.

1. Introduction

This position paper provides a statement on the criticality of research software in artificial intelligence (AI)-driven research. This is needed to ensure that the focus on technological infrastructure to support AI acceleration includes research software — a vital part of that infrastructure. This paper begins by providing definitions for key terms, demonstrating the importance of research software in AI-driven research, and explaining why this paper was developed. The paper then explores the need for AI strategies to recognise research software as a key building block of AI-driven research, and to include this element alongside a focus on computing systems, data, and models. To do this, evidence on the issues in supporting research software, including open-source scientific software, is presented.

The paper then explores how to better support the people who develop and maintain the research software in AI-driven research. AI strategies often consider people elements such as skills and career paths, which also reflect broader issues for personnel. The paper then illustrates how some countries are operationalising AI strategies that could support the critical element of research software, and have the potential to do so by building on existing investments in research software. Finally, the paper makes recommendations for stakeholders to consider research software in their AI strategies.

2. What is research software and why is it critical to AI?

Research software is defined in accordance with the FAIR for Research Software Principles (Barker et al. 2022) as “source code files, algorithms, scripts, computational workflows and executables that were created during the research process or for a research purpose. Software components (e.g., operating systems, libraries, dependencies, packages, scripts, etc.) that are used for research but were not created during or with a clear research intent should be considered software in research and not research software. This differentiation may vary between disciplines” (Gruenpeter et al. 2021).

Research software supports AI, both 1) in more traditional machine learning (ML), where models are trained on data and then used to infer knowledge about new data, and 2) in generative AI, which can be defined as “deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on” (Martineau 2021).

Research software is already recognised as critical to research outcomes (Barker, Katz, and Gonzalez-Beltran 2020; Hocquet et al. 2024; Strasser et al. 2022). Research software can be a research output in itself, and its many functions range from being a component of research instruments, being the research instrument (where research software generates research data, validates research data, or tests hypotheses, such as models and simulations), analysing and presenting research data, and providing infrastructure or underlying tools (Nieuwpoort and Katz 2024). All machine learning (ML) and AI began as research software. In many cases, AI research software has transitioned outside of research and is widely used outside research today.

Consequently, research software is essential in AI-based research, where newly developed methods are** research software. Some AI reports and strategies recognise this; for example, OECD’s Artificial Intelligence in Science report (OECD 2023) highlights the need to better support research software as part of utilising AI to accelerate scientific productivity, in a number of ways, including:

Increase access to software
Share best practices and applications of research software
Facilitate more regular funding and more secure positions
Improve ways to measure research impact and productivity
Consider academic training changes needed
Utilise disciplinary consortiums to accelerate discovery and improve reproducibility through the sharing of software
Understand how AI can speed up research software development

The research community personnel who develop and maintain research software are just as important as the research software itself, if not more important, as the software will stop working without ongoing maintenance. This maintenance is needed 1) to respond to bugs so that the software continues to be correct, 2) to add new features so that the software continues to be useful as research progresses, 3) to adapt to change in underlying software and hardware or in related software ecosystems, such as changed libraries and computing systems, so that the software continues to work.

The people who do this work can have many titles, including researchers, research software engineers (RSEs), data scientists, computer scientists, data engineers, bioinformaticians, students, community scientists, and many more (Hettrick et al. 2022; Barker and Buchhorn 2022). The inclusion of improved support for the staff who develop and maintain research software is also emphasised in the UK-focused Review of Digital Research Infrastructure Requirements for AI (Lazauskas et al. 2022). This report’s findings included emphasis that “any investment in any infrastructure for AI would need to be matched by investments in training and support”, with the second highest priority areas identified as funding for RSEs (Lazauskas et al. 2022). The UK’s ExCALIBUR RSE Knowledge Integration Landscape Review also highlights the need for RSEs to acquire new skills relevant to AI, and notes that “These skill sets also begin to deviate from what is demanded by industries, requiring novel AI software and capabilities’’ (Parsons et al. 2021).

ReSA’s initial research on recognition of research software in the AI landscape was presented in March 2024 at its Research Software Funders Forum meetings. This forum engages representatives from over 60 funding organisations in online meetings and hybrid venues (ReSA 2024a). Attendees of these March meetings were immediately able to utilise these early findings from the presentation to strengthen their own cases for investment in research software within AI-dominated programs, where the criticality of this building block is not yet understood by many in their organisations. It became clear that further research on best practices would support not only funders but a variety of stakeholders in the international research software community.

The research in this position paper was undertaken as a partnership between ReSA and the Digital Research Alliance of Canada, one of ReSA’s Founding Members. One of ReSA’s key functions is to ensure that research software is considered in international discussions on how to advance research capabilities. ReSA is a global organisation that unites decision-makers and influencers across the international research software community. The Digital Research Alliance of Canada is a non-profit organisation funded by the Government of Canada. It advances Canada’s position as a leader in the knowledge economy on the international stage by integrating, championing and funding the infrastructure and activities required, with research software as one of the three main areas. Other community leaders were also brought in on this paper in order to present more balanced perspectives on the need for research software in AI, particularly from institutional and open source perspectives.

3. Consequences if research software is not supported

There is a tendency for AI initiatives to minimise or exclude focus on research software. This minimisation creates challenges. To show this, evidence on the issues in supporting research software, including open-source scientific software, is also presented here.

Many AI initiatives tend to focus on computing systems, data, and models, considering issues such as the availability of systems that can train models and use them for inference in reasonable time periods; FAIRness of underlying data as well as its suitability for training models, including regularity and labelling; understanding and reducing bias in data and models; and understanding privacy. The fact that AI models are implemented in software, and that software is required for training, is rarely considered. Basically, models are thought of as extensions of data, not of software as well. Consequently, many AI strategies focus on data and models but exclude the underlying software. An example of an AI strategy that does not necessarily adequately include a research software focus is the US National Institutes of Health (NIH) workshop, Towards an ethical framework for AI in biomedical and behavioural research, which focused on data and models, but not the research software inherent in using the data and models (ODSS 2024).

This challenge is not limited to research software. Discussions of AI capacity often omit open-source software, instead focusing on areas such as talent, funding, data, semiconductors, and compute access (Engler 2021). Like research software, open-source software can advance science but also has significant effects in other areas such as the development of AI standards (Engler 2021). Further, research software is often either open-source itself or depends on open-source software. But considerations for software from AI initiatives, if any, do not normally include open source software maintenance as a priority, nor analysis of its dependency on other software (Nahar et al. Forthcoming). Where dependency analysis does happen, it is usually due to concerns about supply chain security or perceived attack vectors (a pathway or method used to access a network or computer to attempt to exploit system vulnerabilities).

The nature of open source software as a complex stack and the concomitant extra step of funding those open source software dependencies of the research software necessary for AI is largely ignored. This could reflect a misunderstanding of how open-source dependencies introduce systemic vulnerabilities that need to be addressed and maintained on an ongoing basis or concerns regarding funding many independent projects down the dependency tree. When open-source software is used without planning for future community or dependency support, the result is a more brittle system in the long run — for research software, open-source software, and AI.

There has been some recent work done on software bills of materials (SBOMs) in open source supply chains, where dependencies are charted out, in part so that their vulnerabilities can be understood (National Security Agency 2024; SCAWG 2022) (and it is relevant to note that research software is often used to map these dependencies). Mapping supply chains is increasingly important for understanding how to shore up digital infrastructure, where large economies depend upon open-source software that is poorly maintained and that may be internationally broken or hijacked or may break (as in the XZ Utils, Heartbleed, or the Log4j incidents (Goodin 2024; Buchanan 2021), or the more recent CrowdStrike failure which caused a global Windows outage (Robins-Early 2024)). However, SBOMs are merely an atlas for understanding dependencies — they are policy agnostic and their creation doesn’t mandate funding or supporting important open-source or research software.

This paper suggests that infrastructural issues with using and supporting open source software are similar or identical to issues with using research software. Where AI is involved, the same questions of continued support for the research software being used to develop, train, and run the AI systems are raised. Any AI initiative that plans to use global infrastructure, to adapt to new fields and target areas, or that hopes to exist for the long haul must consider the research software that enables those goals.

4. Challenges if research software personnel are not supported

There is also a need to better support the people who develop and maintain research software that is important to AI-driven research. This is recognised in relation to research software personnel in general, particularly in relation to the training, hiring, and funding of both professional research and technical staff able to reuse, develop, and maintain sustainable research software; appropriate reward and recognition measures that enable career progression for all people involved in the creation and maintenance of research software; and citation practices for research software that recognise substantial contributors to all aspects of the software (Van Tuyl 2023; US-RSE Association and IEEE Computer Society 2023; Barker and Katz 2022a; ReSA 2023). For example, Software and skills for research computing in the UK recommendations include facilitating detailed analysis of how to professionalise RSE roles; and collaboration between government, funders, and employers to create national policies aimed at improving standards of employment (Barker et al. 2024).

Whilst many of the issues for research software, in general, are the same for research software to support AI-driven research, one area of difference is in the specifics of skills and training. Human skills, training, and career paths often feature in AI strategies, and it is important that these focus on relevant research software practices. For example, the Review of Digital Research Infrastructure Requirements for AI specifically emphasises the lack of AI skills and training, and career paths for research personnel, including for RSEs. It highlights that these staff often lack formal training, and ongoing professional development is crucial in the fast-paced world of AI tools and techniques (Lazauskas et al. 2022). With many researchers now using GenAI for coding tasks (Nordling 2023, cited in Hosseini et al. 2024), software development skills need to change to reflect this (Caballar 2024; Dursi 2024).

5. National approaches

AI is now seen as a geopolitical asset, and as international organisations and a range of countries seek to show leadership for AI in science, some countries are operationalising AI strategies that could support the critical element of research software. However, more focus is needed. One example of discussing software within a national AI strategy is the US National Artificial Intelligence Research Resource (NAIRR). NAIRR aims to provide a shared national research infrastructure for responsible discovery and innovation in AI, to address the fact that many researchers lack the necessary access to the computing, data, software and educational resources needed. The NAIRR pilot’s four operational focus areas include one area dedicated to software, to “facilitate and investigate interoperable use of AI software, platforms, tools and services for NAIRR pilot resources” (NSF 2024). However, there are potential challenges with this approach as the focus is on interoperability of existing software, which assumes that sustainable software is already in place.

Many countries also have existing investments in research software (Barker and Katz 2022b; ReSA 2024b) that national initiatives have the potential to build on. The UK’s AI Research Resource (AIRR) is another example of a national initiative, which focuses on increasing computational power to support AI-driven research (UKRI 2023), although its funding is now uncertain (Trueman 2024). Whilst AIRR does not seem to include a focus on research software, AIRR is a key component of UK Research and Innovation (UKRI)’s Digital Research Infrastructure (DRI) which has featured a number of recent investments in research software, including funding of research technical professionals (RTPs), such as RSEs. For example, the Engineering and Physical Sciences Research Council (EPSRC) and UKRI DRI have invested £16 million to support community-driven projects providing training and development for RTPs (UKRI 2024a). UKRI’s Digital RTP Skills NetworkPlus aims to explore key challenges and interventions related to skills and careers that are faced by digital RTP communities across the UK research and innovation landscape (UKRI 2024b).

Canada has been a leader in AI, as the first country in the world to put in place in 2017 a fully-funded AI strategy, the Pan-Canadian AI Strategy (ISED 2022). The strategy included the establishment of a national program of research chairs to recruit and retain top researchers at Canadian universities, the establishment of three national AI institutes to be global centres of training and research excellence and the creation of a Pan-Canadian AI Compute Environment (PAICE) platform. The Digital Research Alliance of Canada’s National Research Software Strategy for 2025–2030 (Digital Research Alliance of Canada, National Research Software Strategy Working Group 2023) also reviewed (inter)national research software funding programs in support of AI, deep learning, and ML-facilitated research. Before the Alliance, CANARIE has been a national Research Software funder and service provider since 2007, whose successful research software initiatives had resulted in the development of sophisticated software tools, known as research platforms, that typically support end-to-end research workflow within a specific domain. New platforms re-used software components previously developed through CANARIE funding, and contributed additional components back to the research community, creating a powerful cycle of software development and reuse (CANARIE 2024). The Alliance is now using the Research Software Directory to continue promoting the visibility, impact, and reuse of the Canadian research software (Digital Research Alliance of Canada 2024).

6. Recommendations for research software to support AI

Research software needs to be included in AI strategies. Our recommendations for doing so are listed below, organised in three areas from the Amsterdam Declaration on Funding Research Software Sustainability (ReSA 2023):

Research software practice:

AI strategies and funding must recognise that research software is a key part of the (publicly funded) AI pipeline and that AI is dependent on software, and therefore should stimulate the development and maintenance of research software to ensure the success of the AI work.

Research software ecosystem:

Because AI-driven research is dependent on the existing research software ecosystem, AI strategies should provide long-term support for its elements, including personnel, communities, and infrastructure, and should add new elements that focus on AI-specific parts as needed.

Research software personnel:

Because the existing research software ecosystem that supports AI-driven research is dependent on research software personnel, AI strategies should facilitate appropriate reward and recognition measures that enable career progression for all people involved in the creation and maintenance of research software that supports AI-driven research.

Other ways to help ensure that the focus on technological infrastructure to support AI acceleration includes research software and its personnel as a vital part of that infrastructure include:

Signing the Amsterdam Declaration on Funding Research Software Sustainability (ADORE.software), which represents a first step to formalise, on a global level, the basic principles and recommendations related to funding the sustainability of research software, including the people needed to achieve this goal.
Supporting ReSA in its aim to ensure that research software is considered in international discussions on how to advance research capabilities. Become a ReSA Organisational Member, support a task force, or donate.
For funders, joining the Research Software Funders Forum, a global collaboration of funding organisations committed to supporting research software, and those who develop and maintain it, as fundamental and vital to research.

References

Barker, Michelle, Elena Breitmoser, Philippa Broadbent, Neil Chue Hong, Simon Hettrick, Ioanna Lampaki, Anthony Quinn, and Rebecca Taylor. 2024. ‘Software and Skills for Research Computing in the UK’. Zenodo. https://doi.org/10.5281/ZENODO.10473186.

Barker, Michelle, and Markus Buchhorn. 2022. ‘Research Software Capability in Australia’. Zenodo. https://doi.org/10.5281/ZENODO.6335998.

Barker, Michelle, Neil P. Chue Hong, Daniel S. Katz, Anna-Lena Lamprecht, Carlos Martinez-Ortiz, Fotis Psomopoulos, Jennifer Harrow, et al. 2022. ‘Introducing the FAIR Principles for Research Software’. Scientific Data* 9 (1): 622. https://doi.org/10.1038/s41597-022-01710-x.

Barker, Michelle, and Daniel S Katz. 2022a. ‘Encouraging Entry, Retention, Diversity and Inclusion in Research Software Careers’. https://doi.org/10.5281/ZENODO.7117842.

Barker, Michelle, and Daniel S. Katz. 2022b. ‘Overview of Research Software Funding Landscape’, February. https://doi.org/10.5281/ZENODO.6102487.

Barker, Michelle, Daniel S. Katz, and Alejandra Gonzalez-Beltran. 2020. ‘Evidence for the Importance of Research Software’. Zenodo. https://doi.org/10.5281/ZENODO.3884311.

Buchanan, Bill. 2021. ‘Log4j: The Worst Vulnerability In Nearly A Decade?’ Medium (blog). 2021. https://medium.com/asecuritysite-when-bob-met-alice/log4j-the-worst-vulnerability-in-nearly-a-decade-e0cc80cbb49a.

Caballar, Rina Diane. 2024. ‘AI Copilots Are Changing How Coding Is Taught — IEEE Spectrum’. 2 May 2024. https://spectrum.ieee.org/ai-coding.

CANARIE. 2024. ‘Funded Research Software Platforms’. 2024. https://www.canarie.ca/software/platforms/.

Digital Research Alliance of Canada. 2024. ‘Research Software Directory’. 2024. https://research-software-directory.org/organisations/digital-research-alliance-of-canada?tab=software&order=is_featured.

Digital Research Alliance of Canada, National Research Software Strategy Working Group. 2023. ‘National Research Software Strategy 2023’. Zenodo. https://doi.org/10.5281/ZENODO.10214741.

Dursi, Jonathan. 2024. ‘We Need To Talk About AI’. Research Computing Teams 183 (June). https://newsletter.researchcomputingteams.org/archive/rct-183-we-need-to-talk-about-ai-plus-upfront/.

Engler, Alex. 2021. ‘How Open-Source Software Shapes AI Policy’. Brookings. 10 August 2021. https://www.brookings.edu/articles/how-open-source-software-shapes-ai-policy/.

Goodin, Dan. 2024. ‘The XZ Backdoor: Everything You Need to Know’. Wired. 2 April 2024. https://www.wired.com/story/xz-backdoor-everything-you-need-to-know/.

Gruenpeter, Morane, Daniel S. Katz, Anna-Lena Lamprecht, Tom Honeyman, Daniel Garijo, Alexander Struck, Anna Niehues, et al. 2021. ‘Defining Research Software: A Controversial Discussion’. Zenodo. https://doi.org/10.5281/ZENODO.5504016.

Hettrick, Simon, Radovan Bast, Steve Crouch, Claire Bradley, Philippe, Botzki, Alex, Carver, Jeffrey, et al. 2022. ‘International RSE Survey 2022’. https://softwaresaved.github.io/international-survey-2022/. https://doi.org/10.5281/ZENODO.6884882.

Hocquet, Alexandre, Frédéric Wieber, Gabriele Gramelsberger, Konrad Hinsen, Markus Diesmann, Fernando Pasquini Santos, Catharina Landström, et al. 2024. ‘Software in Science Is Ubiquitous yet Overlooked’. Nature Computational Science, July. https://doi.org/10.1038/s43588-024-00651-2.

ISED. 2022. ‘Pan-Canadian Artificial Intelligence Strategy’. Home page; Innovation, Science and Economic Development Canada. 20 July 2022. https://ised-isde.canada.ca/site/ai-strategy/en/pan-canadian-artificial-intelligence-strategy.

Lazauskas, Tomas, Jennifer Ding, Neil Brown, Reda Nausedaite, Felix Dijkstal, Aaron Vinnik, Bruno Raabe, et al. 2022. ‘Review of Digital Research Infrastructure Requirements for AI’. https://doi.org/10.13140/RG.2.2.29376.00009.

Martineau, Kim. 2021. ‘What Is Generative AI?’ IBM Research. 9 February 2021. https://research.ibm.com/blog/what-is-generative-AI.

Nahar, Nadia, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner. Forthcoming. ‘The Product Beyond the Model — An Empirical Study of Repositories of Open-Source ML Products’. In . https://www.cs.cmu.edu/~ckaestne/publications.html.

National Security Agency. 2024. ‘Recommendations for Software Bill of Materials (SBOM) Management’. https://media.defense.gov/2023/Dec/14/2003359097/-1/-1/0/CSI-SCRM-SBOM-MANAGEMENT.PDF.

Nordling, Linda. 2023. ‘How ChatGPT Is Transforming the Postdoc Experience’. Nature 622 (7983): 655–57. https://doi.org/10.1038/d41586-023-03235-8.

NSF. 2024. ‘National Artificial Intelligence Research Resource Pilot’. 2024. https://new.nsf.gov/focus-areas/artificial-intelligence/nairr.

ODSS. 2024. ‘Toward an Ethical Framework for AI in Biomedical and Behavioral Research’. 2024. https://www.scgcorp.com/ethicalframework2024/Agenda.

OECD. 2023. Artificial Intelligence in Science: Challenges, Opportunities and the Future of Research. OECD. https://doi.org/10.1787/a8d820bd-en.

Parsons, Mark, Alastair Basden, Richard Bower, Neil P. Chue Hong, Davide Constanzo, Shaun Witt, Luigi Del Debbio, et al. 2021. ‘ExCALIBUR Research Software Engineer Knowledge Integration Landscape Review’. Zenodo. https://doi.org/10.5281/ZENODO.4986062.

ReSA. 2023. ‘Amsterdam Declaration on Funding Research Software Sustainability’, August. https://doi.org/10.5281/ZENODO.8325436.

— — — . 2024a. ‘Research Software Funders Forum’. 2024. https://researchsoft.org/funders-forum/.

— — — . 2024b. ‘Research Software Funding Opportunities’. 2024. https://researchsoft.org/funding-opportunities/.

Robins-Early, Nick. 2024. ‘What Is CrowdStrike, and How Did It Cause a Global Windows Outage?’ The Guardian, 19 July 2024. https://www.theguardian.com/technology/article/2024/jul/19/what-is-crowdstrike-microsoft-windows-outage.

SCAWG. 2022. ‘Recommendations to Improve the Resilience of Canada’s Digital Supply Chain’. https://ised-isde.canada.ca/site/spectrum-management-telecommunications/sites/default/files/attachments/2022/CFDIR-June2022-recommendations.pdf.

Strasser, Carly, Kate Hertweck, Josh Greenberg, Dario Taraborelli, and Elizabeth Vu. 2022. ’10 Simple Rules for Funding Scientific Open Source Software’, June. https://doi.org/10.5281/ZENODO.6611500.

Trueman, Charlotte. 2024. ‘UK Government Shelves £1.3bn of Tech and AI Projects; Scraps Plans for First Exascale Supercomputer in Edinburgh’. 2 August 2024. https://www.datacenterdynamics.com/en/news/uk-government-shelves-13bn-of-tech-and-ai-projects-scraps-plans-for-first-exascale-supercomputer-in-edinburgh/.

UKRI. 2023. ‘£300 Million to Launch First Phase of New AI Research Resource’. 1 November 2023. https://www.ukri.org/news/300-million-to-launch-first-phase-of-new-ai-research-resource/.

— — — . 2024a. ‘New Funding to Support Research Technical Professionals’. 18 March 2024. https://www.ukri.org/news/new-funding-to-support-research-technical-professionals/.

— — — . 2024b. ‘UKRI Digital Research Technical Professional Skills NetworkPlus’. 22 April 2024. https://www.ukri.org/opportunity/ukri-digital-research-technical-professional-skills-networkplus/.

US-RSE Association and IEEE Computer Society. 2023. ‘Research Software Engineers: Creating a Career Path — and a Career’. Zenodo. https://doi.org/10.5281/ZENODO.10073232.

Van Tuyl, Steve (Ed.). 2023. ‘Hiring, Managing, and Retaining Data Scientists and Research Software Engineers in Academia: A Career Guidebook from ADSA and US-RSE’. Zenodo. https://doi.org/10.5281/ZENODO.8274378.

Scripting vs. Programming

Lourens Veen — Thu, 01 Aug 2024 00:00:00 GMT

][Lourens Veen]·Apr 26, 2023

Subscribe*Remember me for faster sign in

Any remaining differences are usually due to particular features supported by certain languages. In R for example, you can write down a statistical model equation, for which it has a special syntax using the ~ symbol. There’s no equivalent of that in Python or C or Pascal, because those languages weren’t designed for statistics.

Structure

Scripts typically have a simple structure. They consist of a single file, all variables are global, and there may be a function or two but mostly it’s a straightforward list of instructions. This works fine for small scripts, but it becomes impossible to manage for a thousand- or million-line library. To keep those organised, you need functions, classes, packages/modules and namespaces.

Languages like Java and C++ are designed for this, and do well on this criterion. Python is actually pretty good as well, it does have all of these and there are some large programs written in it. These features are anyway quite common in modern languages, even ones intended for scripting-type work.

Of course, adding features makes the language more complex (i.e., it makes the manual bigger). Even if you don’t use them, others will, and so you’ll have to learn how they work if you want to be able to use other people’s packages for example. So there is a cost to this additional functionality as well as a benefit, which is what Jisk was getting at with his comment on languages being too “computer science oriented” for what he wants to do.

Error handling

Finally, error handling. Your basic script handles errors in a basic way: it prints an error message (comprehensible or not) and then halts. Normally, this isn’t done by the script itself but by the programming environment it runs in, with the script simply ignoring the possibility of anything going wrong. This actually works quite well in a notebook environment, where you execute one statement at a time (so it’s usually clear where the problem is) and where there’s a human to read the error message and handle the problem.

For a web service, or a complex library, or a desktop application, crashing whenever something goes wrong is not acceptable. Errors need to be handled, and only if there’s nothing the program can do to fix it does the error get forwarded to a human, after which we continue with handling the next request. At the language level, this is implemented either through returning an error code from a function, or by using exceptions. Older languages like C and Fortran don’t support exceptions, so all errors have to be signalled and handled explicitly. This means that if you do it right, basically every function call is followed by an “if” block that checks for errors and handles them.

This gets very very wordy, so newer languages like Java and Python support exceptions. Exceptions automate keeping track of errors, and by default escalate any errors in a function you called to the function that called you, and so on. In other words, if something goes wrong, you tell the boss, who tells her boss, etc. until the problem reaches someone who can decide what to do. If the error reaches the top without having been solved then the program crashes and the user will have to figure it out, but ideally something handled the problem before we got to that point. Crucially, if the programmer ignores the possibility of errors, then they will end up with the user. That’s not so good, but it’s better than the error being quietly ignored, as would be the case with ignoring returned error codes.

One disadvantage of exceptions is that to be completely robust, you need to carefully keep track of which exceptions can be raised by functions you call, and make sure you handle them all. In Python, this is pretty much impossible because they’re not documented very well. Java provides syntax for this, and forces you to declare all exceptions a function can raise, which is much better for reliability but is also sure to make Jisk unhappy in his quest for a simple language.

It’s interesting to see that after all the exceptions in C++ and Java, in yet newer languages like Rust and Go there’s a bit more emphasis on returning errors again. There’s the idea there that most errors are just “normal” possible outcomes of calling a function that should be handled immediately, and that an exception-like “panic” system is to be used only in truly exceptional circumstances. Meanwhile in Python it’s the other way around, and exceptions are sometimes even used for things that are expected to happen during normal control flow (e.g. StopIteration).

There’s more to say and explain about error handling, and we’ll come back to it in a future TEC. For now, the point is that languages that allow you to completely ignore the possibility of an error occurring are more suitable for scripting, while languages that require you to take possible errors into account lead to more reliable programs.

Conclusions

So, which language is best for which purpose? Python is obviously good for scripting, and Julia is an interesting alternative that I think may be a better fit for Jisk than Go, seeing as he is interested in calculating things. R seems to work well as a command-line interface for doing statistics but I don’t like it as a programming language (although the Tidyverse is a good development).

On the programs and libraries side, I’ve built a few slightly bigger systems (in the tens-of-files, thousands-of-lines range) in Python, and it works pretty well, plus you can still put something together very quickly. The downside is that the dynamic typing, lack of encapsulation and lack of exception specifications make it more difficult to make things robust, and performance is also nothing to write home about. Potentially better if slower to work in alternatives would be Java, (modern) C++, and new kid on the block Rust. For now, C++ is my go-to language if I need more performance, even though it’s very complex and by no means perfect.

What works best for you depends on what you want to do, how much time you have available to learn, a bit on taste, and on external factors like availability of software and of support from the people around you. Many concepts carry over from one language to another though, so don’t worry too much about choosing the wrong one: once you know one programming language, learning others gets a lot easier.

Well, that concludes the first instalment of The Engineering Corner*. Hope you liked it! Do you write scripts or programs or both? What’s your favourite programming language? Did you like this topic? Was this too basic or too advanced or maybe just right? What would you like to read about next? I’d love to hear about all these things in the comments below. In the mean time:

Choo choo! Happy Engineering!

Lourens

← <previous TEC not found> — TEC 1: Better scripts with typing →

[0] Control flow refers to the order in which commands are executed, and to branches (if..then) and loops (for.., while..) causing them to be executed not at all, or more than once. In other words, the path through the code taken by the computer’s “cursor” as the code runs.

Decoding Research Software Impact: A Collaborative Journey

Deekshitha — Fri, 26 Jul 2024 00:00:00 GMT

Decoding Research Software Impact: A Collaborative Journey

On 23 April 2024, the first ‘National Research Software Day’ took place in Hilversum, the Netherlands. During the unconference part of the program, Deekshitha ran the session titled, ‘Decoding Research Software Impact: A Collaborative Journey’, where a model for research software impact was at the center of the discussion.

Authors: Deekshitha¹, Carlos Martinez Ortiz¹, Rena Bakhshi¹, Jason Maassen¹, Rob van Nieuwpoort², Slinger Jansen³, Ton Smeele³, Andrew Treloar⁴, Louise Bezuidenhout², Maarten Schermer³, Laurents Sesink⁵, Pui-Kei Fung³, and Colette Bos¹

Photo by Soliman Cifuentes on Unsplash

In this session, participants collaborated on the topic of research software impact. Participants took the perspective of policymakers, funders, researchers and Research Software Engineers (RSEs) to explore perspectives of research software impact in four 15-minute rounds, in which participants attempted to answer a set of questions from the perspective they were exploring. After each round, the participants changed groups to cover a different perspective. This workshop aimed to define the impact of research software from the perspectives of these 4 stakeholders. Here the term impact refers to the difference the research software makes to the research process or community.

Measuring the impact of research software is the main focus of Deekshitha’s PhD study. Currently, there is a lack of sufficient methods to accurately measure research software impact. Her study seeks to develop an impact model that will enable all stakeholders of research software to measure its impact effectively and ensure recognition for the individuals behind its development. During the sessions, four questions were discussed (from different perspectives):

What are the goals of measuring research software impact?
How can research software impact be measured?
Which measures correspond to which goal?
How should these goals be prioritized?

Researchers and Research Software Engineers’ perspective

From the researcher’s and RSEs perspective, we have identified six goals, as shown in the Figure 1. Among these, recognition and a track record for funding are essential goals for this group. Despite differing priorities, researchers and RSEs share similar perspectives. Therefore, they are represented together in a single diagram illustrating both viewpoints.

The figure below illustrates the goals and the factors that help measure their impact and interdependencies. The first branch represents the goals (Answer to question 1), the second connection illustrates how to measure those goals (impact factors and their connection with goals- Answer to questions 2 and 3), and the third branch shows their prioritization value (Answer to question 4) along with the corresponding group (Researchers and RSEs; National and Institutional; no other group identified for Funders). This prioritization resulted from discussions with the workshop participants and the value starts from one. For example, in Figure 1, there are three impact factors grouped under Researchers. Based on the discussion, we found that the impact factor ‘citations’ has higher priority compared to other factors, so it is ranked 1 in the category for Researchers. Grey-colored boxes in front of the impact factors indicate the groups they belong to (eg: Researchers and RSEs). Additionally, impact factors unrelated to the goals (or we couldn’t find a relation with identified goals) are presented in separate black-colored boxes (see Figure 2 and Figure 3).

Figure 1: Researchers/RSEs perspective

Funders perspective

Funders like NWO, SURF, and eScience Center who support research software projects, prioritize demonstrating value for money and aligning with international strategic goals like Open Science. By emphasizing research software impact, funders can act as change agents, influencing research and ensuring it aligns with broader institutional objectives. Selecting projects with significant impact enhances the funders’ image and showcases their commitment to supporting valuable and innovative research. Furthermore, funders contribute to research and educational institutions by funding unique and impactful software, thus enhancing their overall value.

Figure 2: Funders perspective

Policy makers perspective

Policy makers and funders play distinct yet interconnected roles within governance and funding, particularly in research, education, and public policy sectors. Funders focus primarily on providing financial support and resources to achieve specific objectives and outcomes. On the other hand, policy makers are responsible for formulating and implementing regulations, guidelines, and directives that shape the strategic direction and operational frameworks of institutions and organizations.

In terms of policy making, we realized there should be a distinction between national and international policies to ensure alignment with the institutional mission and purpose, maintain the independence and sovereignty of research institutes, and adequately consider risk assessment, marketability, and valorization. Figure 3 shows goals, impact factors, and their groups with priority values from the policymakers’ perspective.

Figure 3: Policy makers perspective

Take home message

The collaborative and dynamic nature of the session made it a memorable experience, combining serious discussion with a touch of fun. This approach not only sparked insightful conversations but also strengthened the community’s commitment to advancing the understanding and measurement of research software impact. Additionally, everyone agrees that RSEs are not appropriately recognized for their work, and recognition is one of the main goals for measuring impact. Funders have open science goals to achieve and policymakers want to ensure value for money.

We look forward to continuing this journey and invite all who are passionate about the transformative power of research software to join us. Together, we can build a robust and internationally recognized model for assessing research software impact. If you’re eager to join this impactful journey, reach out and we can connect over a cup of coffee to further the discussion.

Credit: We want to credit the tool we used, the free version of EdrawMind, in this blog post.

Footnotes

What Research Software Engineers can learn from Taylor Swift

Lourens Veen — Thu, 18 Jul 2024 00:00:00 GMT

It’s Friday night, 5 July 2024. After some out-of-this-world pizza and a couple of drinks with colleagues, I’m on my way to catching a late train home. I get to the station just in time to park my bike. (I don’t know if it turns into a pumpkin, but the parking closes at midnight so if you’re late, it sleeps outside.)

It’s crowded in the building. This is Amsterdam Central on a Friday night, so I wasn’t expecting deserted platforms, but tonight it’s packed and somewhat confusingly there are many people dressed in white, with glitters on their faces or even dressed in them completely.

Then I remember the morning’s news. Taylor Swift concert. Incredibly popular pop singer, researchers studying why. Billionaire. First tour to gross a billion dollars. First concert in The Netherlands in seven years, and sold out way in advance. Fans called Swifties, waiting in line all day in the rain to get a good spot. I’ve never heard a Taylor Swift song.

Oasis in Montreal, Canada, in 2002. The lyrics to their 1997 song *D’You Know What I Mean describe the singer visiting his childhood home and seeing the Bob Dylan album and a Beatles single that would inspire his music. Photo by Nesnad on Wikimedia Commons. CC-BY-SA 4.0.*The next morning I decide that I should probably find out what all the fuss is about. You have to keep up with the times after all. I pull up YouTube and type “Taylor Swift”. A song called “Fortnight” comes up, apparently her latest single. Featuring someone called Post Malone, which raises the question of why Malone ended, and whether there was a Pre Malone before him, but I digress. I click the link.

“Goldband!” my brain goes as the song starts playing. The very first sound is a 1980s-style synthesizer melody that immediately reminds me of this Dutch pop band. They had their Taylor Swift moment for me last year, as I similarly read about their popularity in the news and decided to investigate.

One of the things I found then was an interview with Goldband’s producer Wieger Hoogendorp that I found very interesting. He explained that the 1980s sounds in their music make it popular because their audience has parents who were teenagers in the 1980s, and they played that stuff to their children. Similar music reminds the now-older kids of those good times, and so this is a recurring cycle in music. People like the familiar.

I thought about this. As a teenager in the ‘90s, I’d listen to the sound of boy bands emanating from my sister’s room. I liked Father and Son, by Boyzone. It wasn’t until much later that I discovered that it was a cover, the original having been written by Cat Stevens 25 years earlier. My favorite band in the ’90s was Oasis, who was huge and took their inspiration from The Beatles, popular in the ’60s. Maybe there was something to this.

In the Fortnight video meanwhile, Swift is singing the song while moving from a suspended bed frame to a library where she uses a typewriter. Somehow this feels familiar. She casts a glance at Malone, similarly equipped, and straightens out some papers. At the end of the song, the library scene returns, and all the papers catch fire, and it clicks. It’s another ’80s reference, this time to the clip for Toto’s Africa.

Stills from Fortnight by Taylor Swift and Africa by Toto. Note the similarities, which make the clip for Fortnight feel familiar to the viewer, as they’ve likely seen the clip for Africa before. And then nostalgia does the rest. Copyright © 1982 and 2024 by the respective owners.

About that software

But Lourens, you may ask, that is all very interesting, but what does it have to do with Research Software Engineering?

First, please don’t start programming in Fortran ’77 again because it was popular in the 80’s. We have better options now, like Julia or modern Fortran or, well, anything really. Ms. Swift didn’t hire Toto as her backing band either.

Instead, there are a few aspects of what Taylor Swift does that we can learn from if we want to get more people to use our scientific software.

What users want

It may sound incredibly obvious, but we’ve seen above that one of the reasons Ms. Swift is so popular is that she gives people what they want. Note that that’s not the same as what they say they want. I bet very few Swifties, if you asked them what they wanted to hear in the next song, would say “I’d like it to start with a 1980’s style synthesizer melody reminiscent of Goldband, and then I want her to start singing halfway through the first measure already (another recent innovation in pop music, it reduces the probability that people will skip to the next song right away), and then I want the music video to remind me of Toto’s ‘Africa’.” A much more likely answer would be “I’m so glad you’re here, do you know how I can get this glitter off my face?”

But Swift and her team are professionals and they know what people actually want. They have all the know-how from decades of experience with pop music, and these days a ton of very detailed data too from streaming services on what people listen to and when and how.

Making things easy

Another thing we can take inspiration from is the dress code. First, there’s apparently a dress code for Ms. Swift’s concerts (probably not a formal one, but enough to fill a train platform with similarly dressed people), which is nice because it means that you don’t have to feel insecure about what to wear to the concert. Nobody likes feeling insecure.

Second, the dress code is white. Everyone has something white in their closet, so that’s easy. Glitter is easy to come by and cheap, and as an added bonus it reminds you of being a kid again. And then of course the concert is in a place that’s easy to reach for most people, and it ends on time so that you can still easily get home by public transport. In short, going to a Taylor Swift concert is easy! Anyone can do it and feel good doing it, and so more people do.

The same principle applies to scientific software. Reduce the barrier to entry as much as you can. Make sure it installs (flawlessly!) in at most a single command (and that includes dependencies!). Put that command in the documentation so people can copy-paste and don’t have to type. Ensure there’s an easy-to-follow tutorial, and that it is written from the perspective of the users, which you figured out above. Add a ton of examples on how to do specific things that many people will want to do so that they can copy-paste without having to learn how to operate the software. And set up a chat channel so people can easily ask questions.

The Toto reference gives us another hint: make your software familiar. Users are going to interact with it in a way that you design, and the more familiar that interaction feels to them, the more usable your software is. So look at other software that is similar, or that your users are likely to be using already, and try to design along similar lines. Clever tricks and highly artistic solutions may be fun to build, and they may make you feel very smart, but they don’t help the user.

In short: Software exists to make people’s lives easier. Make sure yours does.

Community

There’s another thing that dress codes or uniforms (or friendship bracelets) do: they foster a sense of community. Humans are social animals, and we love being part of a group. That’s why we all dress in orange and go to a stadium to cheer on a group of people dressed in the same color, or grow our hair long and wear black and go to a metal concert, or put on suits and go to important meetings. All to be a part of something, to show that we’re part of the group.

Software is more successful if it has a community around it because it helps people feel like they’re a part of something when they use it. Chat channels, mailing lists, meet-ups, workshops, all these things make it easier for people to start using software and to stay involved. I’ve worked with some software that’s really not very good technically but has an amazing community manager who keeps it alive, nonetheless. Whether that’s a good idea is questionable, but it shows that it works.

Advertising

Collaborations are quite popular in pop music these days. I don’t remember Madonna or Michael Jackson doing a lot of duets when I was a teenager, but it seems that these days every other single features someone or another. The reason of course is simple: it’s good advertising.

Taylor Swift had a concert, causing a lot of people to gather at a train station, as a result of which I listened to Fortnight, and now I am aware of the existence of Post Malone (and Malone, and Pre Malone, look at that, triple word value!). Moreover, the cloud’s algorithms are now more likely to keep serving me Malones as well, and there’s someone out there whose bottom line is served by that.

Of course, trying to influence search results using a link farm would be evil, but registering your software in the Research Software Directory is good practice, as is archiving it on Zenodo, and it works almost as well. Make it easy to use together with related software, and advertise that in that other software’s community. Glue some glitter on your colleague’s faces and send them to the local train station. The point is, if no one knows about your software, no one will use it.

Impact is more than numbers

There’s another reason Taylor Swift is popular, and that is that her music is firmly mainstream. It features recognizable sounds and rhythms, the lyrics are relatable, and so it works for many people. Compare that with my favourite pop artist, Grimes, who makes songs about things as nerdy as Roko’s Basilisk. As a result, Grimes is not a billionaire. (She used to date one, but that doesn’t count.)

Research Software Engineers are unlikely to become billionaires either, and sometimes we end up working on specialist software that does something only very few researchers need. It’s hard to have a lot of users then, even though the software could still be very important. On the other hand, something as generic as NumPy has a huge number of users, but then again if NumPy were the only thing we had we’d still not get much science done. (Are NumPy users called NumPies? If not, why not? And how do they get the glitter off their faces after finishing their calculations?)

So it’s probably a good idea to not give the number of users too much importance when we’re trying to gauge the impact of research software. Do good science with it, make sure that it works for the people doing that science (however many there are), and enjoy the critical acclaim from the users you’ve got and whose lives you made easier. In the end, that’s what it’s all about.

Better scripts with typing

Lourens Veen — Fri, 05 Jul 2024 00:00:00 GMT

The Engineering Corner

A blog-within-a-blog on the transition from writing simple scripts for yourself to writing larger programs with and for others. TEC appears on an irregular schedule driven by workload and inspiration.

№ 1: Better scripts with typing

Imagine that one day you are in the office, and a colleague comes by explaining that their Python script isn’t doing what it’s supposed to. They ask you to help solve the problem. You happily agree. You see they’ve written a function:

def foo(a, b): return a + bIs this code correct, or will it give an error if you run it? (foo is a traditional name for example functions, it doesn’t mean anything.)

The answer, as often in engineering, is, “It depends”. It depends, in this case, on how the function is called. Something like this will work:

print(foo(1, 1))On the other hand, this won’t work:

print(foo([1], 1))So, whether this function is correct or not depends on what you pass into it. It works for two ints (and for two strings too), but not for a list and an int.

So how do you tell whether the problem is in this function? Well, you can simply check the rest of the script, find where the function is called, see what goes into it, and if a+b works for whatever a and b are put in, then the code is correct and the mistake must be somewhere else.

So far, so good. But what if this is more than just a script? If you have functions calling functions, and functions being called from different places, then figuring out all the different things that can go into the function is going to be quite a bit of detective work. Pretty soon, it’s going to be quite difficult to know whether a function will work or crash. It can even be impossible:

import json

with open(‘input.json’) as f: a = json.load(f) print(foo(a, 1))Now whether the program works or crashes depends on what’s in some file on the user’s computer! So maybe the script is correct after all, and the problem is elsewhere? This doesn’t make for reliable scripts, or even reproducible science, for that matter.

The Philippeion of Olympia (Mark Cartwright — CC BY-NC-SA)

Modularity and types

Clearly, we get all kinds of problems when we try to make software that will reliably get the user what they want, especially if the software is large and complex. Fortunately, there’s a good way of dealing with complexity. Invented by legendary RSE Philip II of Macedon some 2300 years ago: Divide et Impera (Divide and Rule). You split up your program into components (functions, procedures, classes, packages, etc.) that you can deal with one by one, then assemble them into a reliably operating empire, I mean program. Let’s look again at that function:

def foo(a, b): return a + bIt receives two arguments, a and b, from the caller, and returns some value that is the result of the operation a + b. Receiving arguments and returning a value are the only connections it has to the outside world (such a function is called pure). But whether the function works correctly depends on the values of a and b, and therefore on the outside world. That means it’s not a very reliable building block. Philip would be disappointed.

What if we chose a set of acceptable values for both a and b? Then we could at least reason that if the function does not crash when an acceptable value is passed for both arguments, then the function works correctly. For example, we could specify that a and b both need to be in the set of all integer numbers. Such a collection of possible values is called a type*. Let’s annotate our function with the types we want to receive:

def foo(a: int, b: int): return a + bWill this function work correctly? Yes! If you pass it two int s, it will return their sum and not crash. The above is actually valid Python (the :int bits are called type annotations) and it will run just the same as the previous example:

def foo(a: int, b: int): return a + b

def bar(): print(foo(1, 1))

def baz(): print(foo([1], 1))

bar() baz()As you can see, if you run this, the code does exactly the same. There is something funny though. In principle, Python could see when it reads the part foo([1], 1) that it is not correct. [1] is a list, and the function foo needs an int. Nevertheless, Python will happily accept the definition of baz, and it won’t crash until you actually *run *baz, in turn calling fooand trying to add a list and an int. It’s only then that Python realizes that the values are the wrong type and can no longer add them, so it dutifully generates a TypeError.

Photo by Mark König on Unsplash

Static vs. Dynamic typing

Not all programming languages are like this. The Python language is a so-called dynamically typed language, which means it won’t check whether things are the right type until it’s running the code and trying to do something with that value. Furthermore, Python completely ignores those type annotations, which explains why it crashes only when it tries to execute a + b rather than already when executing foo([1], 1). Nevertheless, it will eventually realize that something is wrong at that point and give an error (or it would be an untyped language, now mercifully extinct), so that we have a chance to find the problem.

In contrast, languages like C++, Java and Rust are statically typed, which means that they do the type checking immediately when the code is read before it’s ever started. It’s impossible to create an executable⁰ from this C++ code:

auto foo(int a, int b) -> int { return a + b; }

auto baz() -> list<int> { return foo(list<int>{1}, 1); }If I try anyway, GCC tells me: error: cannot convert 'std::__cxx11::list<int>' to 'int' for argument '1' to 'int foo(int, int)', correctly pointing out that foo needs an int, where I’m trying to pass a list. Note that it’s doing this already at the call, not inside foo, and since no executable is created, I can’t run the code either.

Now if we review this code, we can see that the compiler is in fact right. Function foo is fine, but baz tries to pass a list as the first argument, and from the definition of foo we can see that it takes an int, so that’s wrong. In fact, because the return types are also declared, we can see that there is another problem in this version of baz: foo returns an int, which baz then tries to return even though it should return a list. The C++ compiler will give an error for that as well when we fix the first one.

We didn’t annotate the return type in the Python code before, but you can do that:

def foo(a: int, b: int) -> int: return a + b

def baz() -> None: print([1], 1)Again, this is not checked by Python, so it will happily return something that doesn’t match the annotation if you tell it to.

mypy

So what good do those annotations do, if Python ignores them? First of all, type annotating your Python code forces you to think about what kinds of values your functions should accept, and then what to do with different kinds of inputs. That helps you write better code. Second, type annotations are really useful for people reading your code, like another developer, a code reviewer or someone using your package. They don’t need to wonder about what to pass to foo anymore as it’s now obvious: integer numbers! And finally, there are tools that will check your type annotations even if Python doesn’t.

The oldest and probably most commonly used one of these (and the one that I use) is called mypy. You can pip-install it, and then call mypy script.pyto check your types, just like flake8 checks your syntax (your IDE may do this for you actually, look it up!). When I give the above to mypy, it tells me test.py:8 error: Argument to "foo" has incompatible type "List[int]"; expected "int" and that is exactly what is wrong with my code (and exactly what the C++ compiler said). It points out the right location too! That makes it really easy to fix your code.

Conclusion

Type annotations and mypy make your Python code more modular and make it easier to find mistakes, and that is particularly important when you start making larger programs that you share with others. Your code becomes more predictable and easier to understand when you use it, and that makes for a more reliably good experience for its users and developers. There are some downsides, mostly to do with mypy and type annotations still in development and being bolted on after the fact to a language that wasn’t conceived as a statically typed language from the start, but for scientific software, it works fine and is an excellent idea.

To summarise:

As all good Macedonians know, modularity is good!
To make functions modular (analyzable in isolation), you need to specify input and output types.
Strongly typed languages check types on reading the code, dynamically typed ones do it when running.
With mypy, you can check types in Python too, and find a ton of bugs much quicker than without it.

← TEC 0: Scripting vs. Programming — TEC 2: Procedural Programming →

[0] C++ is a compiled language. That means that there are two steps to running a C++ program: 1) translating (or compiling) it from C++ text to machine code for whichever kind of computer you want to run it on, and 2) running the resulting executable. Type checking happens as part of the first step in C++, not the second.

Procedural Programming

Lourens Veen — Fri, 05 Jul 2024 00:00:00 GMT

The Engineering Corner

№ 2: Procedural Programming**

Previously we talked about scripting and programming. One difference I mentioned between scripts and programs is that scripts usually consist of a single source file with a straightforward list of instructions, while programs are larger and have a more complex structure. If your script is getting too big to easily read and understand, what are the things you can do?

My copy of Pascal User Manual and Report by Jensen & Wirth (1975).Here’s an example of a Pascal program (a lesser known programming language developed in the 1970s - Ed.), in fact Program 0.1 from Pascal User Manual and Report (1975) by Kathleen Jensen and Niklaus Wirth:

program inflation(output);

const n = 10; var i : integer; w1,w2,w3 : real; begin i := 0; w1 := 1.0; w2 := 1.0; w3 := 1.0 repeat i := i+ 1; w1 := w1 * 1.07; w2 := w2 * 1.08; w3 := w3 * 1.10; writeln(i,w1,w2,w3) until i==n end.If you are a Python user, you can see this is not that different⁰. A Pascal program begins by listing all the constants and variables we are going to need, and their types. Then follows a list of statements, which in this case calculate and print a table of inflation-adjustment factors for ten years and three different inflation rates.

I’ll leave a Python version as an exercise to the reader, but you can see that it will be quite similar (answer below¹, if you want to check your work). In Python, variables are made automatically and don’t have to be declared in advance, but the style of programming is similar. It’s known as imperative programming.

As the Pascal User Manual points out, “A (…) computer program consists of two essential parts, a description of actions which are to be performed, and a description of the data, which are manipulated by these actions.” As your script gets larger and more complicated it will process more data, and it will perform more actions, until it all becomes too much to easily understand. What to do?

Let’s start with those actions, as that’s what most people like to focus on. In bigger scripts you often see a long list of instructions, and if it’s more than a screenful, that list may be split up into several blocks by comments. A load-data/process/save-data structure is common for example. What’s been done here is that the script has been split up, so that we can deal with it one section at a time (Philip II of Macedon would be proud, see TEC 1).

Can we really deal with the sections independently though? Not entirely, as they still communicate with each other. Each section reads variables written by a previous section, and creates variables for other sections to use. So, if you want to make changes to one section, you still have to take into account the interactions with the others.

Unless the sections do completely different, independent things, this is inevitable: they have to communicate somehow. It’s tricky though. If you change the name of a variable for example, then you have to search the entire script to see if it’s shared with another section, which you then have to modify accordingly. This could trigger changes somewhere else, and before you know it everything is broken.

So, we could use a bit of help. In Pascal, this comes in the form of a procedure. Here’s a slightly modified version of Program 11.3 from the same book as above:

program parameters(output);

var a, b, c: integer;

procedure h(x: integer, var y: integer);
var z: integer;
begin
    x := x + 1;
    y := y + 1;
    z := x + y;
    writeln(z)
end;

begin a := 0; b := 0; c := 0; h(a, b); writeln(a, b) end.What we have here is a program like before, but the work it does is now split up between the main program and a separate section, which is our procedure. It consists of two parts: a declaration (the first line) and an implementation (the rest of the block).

A procedure is a kind of box around a block of code (a scope). Variables can not pass through the walls of the box, unless we explicitly pass them through. In the declaration procedure h(x: integer, var y: integer); we specify explicitly which variables can be passed into the box to communicate between the procedure and the rest of the program: x and y in this example. These are called the procedure’s parameters.

The var keyword in front of y says that y is passed by reference, while the lack of one for x means that x is passed by value. What that means is that when we call the procedure with arguments a and b, the first parameter x becomes a copy of the first argument a (so that changing x doesn’t affect a), while the second parameter y becomes another name for the second argument b (and so changing y changes b).

No other variables can enter or exit the box. Variable zis declared inside the procedure, and it is only visible within the procedure because we haven’t said otherwise.

{ The main program again }

begin a := 0; b := 0; c := 0; h(a, b); writeln(a, b) end.This program has some nice properties. When we call procedure h in the main program, we can be sure that variable c will not be changed, without even looking at h! Then, taking the declaration (the first line with the parameters) of h into account, we can also tell that a will probably be read by h , but won’t be changed, and that b may end up having a different value after the call. Crucially, we can know all of this without ever reading h’s implementation.

{ Procedure h by itself }

procedure h(x: integer, var y: integer); var z: integer; begin x := x + 1; y := y + 1; z := x + y; writeln(z) end;Looking at procedure h, we can tell by looking only at the procedure that any changes we make to it that do not affect what we do to y will not affect how the rest of the program runs, again without inspecting the rest of the program.

Of course, if we do make such a change, then we will still have to find all the places in which h is called and possibly modify them. There is still a dependency! It’s just been made explicit. The same goes if we change the declaration, for example by changing x to be a real.

(To clear up the potentially confusing terminology: Pascal has both procedures and functions. Conceptually, a procedure is a list of actions that together enact a desired change (like directions for navigating to a destination), while a function is a list of instructions that produce a result (like how to calculate the length of the hypotenuse of a right triangle).

Pascal has different language features for procedures and functions, as does Fortran, but more modern programming languages don’t bother with this and just allow you to make a function that returns void or None. So, practically speaking, a function in Pascal is the same as a function in Python, and a Pascal procedure is a Python function that doesn’t return a value.

I’m going to continue to use the term procedure here, because we’re talking about procedural programming, and we can talk about functions later when we talk about functional programming, which is a different beast altogether.)

Procedures for making a tasty lunch. Photo by S O C I A L . C U T on Unsplash

Making a good procedure

So, when should you make a procedure, and what does a good procedure look like? One good programming rule to live by is DRY: Don’t Repeat Yourself. So, if you find yourself copy-pasting some code, consider making a procedure instead, and calling it twice.

Procedures are also fine if they’re only called once though. In the words of Jensen and Wirth: “One should not hesitate, however, from formulating an action as a procedure — even when called only once — if doing so enhances the readability. Defining development steps as procedures makes a more communicable and verifiable program.”* Hear hear!

As we’ve seen above, the point of procedures is to split up your program into self-contained components, which can be modified independently. On the other hand, these components have to work together, for which they need to exchange information, and that introduces dependencies. There’s no perfect solution to this paradox, but it does lead to some good advice: fewer parameters is better, relative to the length of your procedure.

If you’re a statistician or a graph theorist, consider a procedure to be a cluster: a good cluster has lots of stuff and connections inside it, but few connections to things outside it. Or if you’ve ever written a text: a procedure is like a paragraph, with some closely related sentences inside it, but connected to the rest of the text only by the things in its topic sentence.

How about length? Some purveyors of questionable programming advice suggest hard length limits of sometimes as little as five lines per procedure². I think the idea behind that is that humans can keep track of five to seven objects (chunks) at a time in short-term memory, so that this allows the reader to keep the whole procedure actively available.

I don’t think it really works this way though. Blog posts tend to have really short paragraphs for easier reading on screen, but novels don’t and they are still perfectly readable. Grabbing a random one off my bookshelf I count 8 sentences and 163 words in a paragraph, which is three to four times the previous paragraph in this blog and it still reads just fine. So I think that longer procedures are okay, as long as they don’t get so long that you find yourself searching back and forth for something while reading them, or you (or rather someone else who isn’t all that familiar with the code!) have trouble keeping track of what’s going on.

I also think that short procedures should be avoided. Here’s the above example without procedure h:

program parameters(output);

var a, b, c, z: integer;

begin a := 0; b := 0; c := 0; y := y + 1 z := x + 1 + y; writeln(z); writeln(x, y); end.9 lines of code, versus 15 before. And it looks simpler too, doesn’t it? No need to look at the parameter declaration, or to consider which variable belongs where. So maybe procedures aren’t all they’re made out to be?

Costs and benefits

Well, no, it’s not that simple. Procedures are an abstraction. Abstractions make it possible to write larger programs with a finite human brain, but they come at a cost of extra overhead. The shorter your procedures, the more of them you’ll need, and the more abstraction overhead (procedure declarations and calls) you’ll introduce. At some point, that’s actually going to make the code more difficult to read, and it’s up to you, the programmer, to find a happy medium that allows others to easily understand your code.

I do often find myself writing one-line or few-line procedures, but they’re usually actually abstraction overhead introduced by larger-scale abstractions like classes and modules. Not counting those, my normal range seems to be something like 5 to 20 lines, with an occasional stretch into the thirties or more.

What matters is not so much the number of lines, but whether everything within the procedure belongs together, and whether the whole thing is still comprehensible. Sometimes splitting up a longer list of instructions that fundamentally belong together will actually make the code harder to read.

This brings me back to what I said above about the number of parameters relative to the length of the procedure. A good procedure does something that can be described in a few words, all of which are in its name. If you need many words to say what it does, then you’re trying to cram too much in, and should split it up. If you need many words to say how it does it, and you find that that recipe naturally breaks up into several blocks, then make separate procedures for each block. If what it does can be expressed simply, and how it does it doesn’t split easily, then you have a nice procedure.

Final words

Procedures are a useful way of breaking up all the things your program does into manageable chunks. They impose a hierarchical organisation on the functionality of your program, and hierarchies are powerful means of simplification. (Niklaus Wirth wrote a famous paper about it, actually, which is why he and Pascal made an appearance here.) As pointed out by Jensen and Wirth however, there’s another essential part of any computer program: data. We’ll talk more about variables and records, values and objects in future posts.

← TEC 1: Better scripts with typing — TEC 3: <Stay tuned!> →

[0] Modern languages don’t seem to have repeat..until loops anymore. I miss them sometimes.

[1] Here’s a Python version. As you can see, Pascal and Python use different symbols for things, and I’ve used a for-loop because Python doesn’t have repeat..until, but it looks pretty similar.

inflation

N = 10 i = 0; w1 = 1.0; w2 = 1.0; w3 = 1.0 for i in range(1, N+1): w1 = w1 * 1.07 w2 = w2 * 1.08 w3 = w3 * 1.10 print(i, w1, w2, w3)[2] I’m not naming names here…

Quantum Computing for Quantum Chemistry with qc2

Carlosmurilorocha — Thu, 27 Jun 2024 00:00:00 GMT

Quantum Computing is a rapidly evolving field that has the potential to transform the way we approach complex computational problems. Quantum Chemistry stands out as a prime application area, with anticipated impacts ranging from the development of new materials to novel drugs.

In this blog, we introduce qc2, a user-friendly, modular software designed to seamlessly integrate traditional quantum chemistry codes with modern quantum computing Software Development Kits (SDKs).

Photo by Michael Dziedzic on Unsplash However, despite recent advances in both hardware and software, the path to fully error-corrected, utility-scale quantum computations may still take a while. For the moment, as we navigate in the Noisy Intermediate-Scale Quantum (NISQ) era towards Quantum Utility, we need to learn how to make the most out of the available algorithms that run on as-yet error-prone, imperfect hardware: the so-called NISQ algorithms.

Unavoidably, Quantum Computing and Quantum Chemistry, and particularly their combination, demand a high level of technical knowledge. The available software packages are highly specialised, requiring users to have an excellent understanding of the fundamental concepts involved from the start. This might scare away both computational chemists and software engineers alike.

This situation has been greatly alleviated by the increasing availability of excellent educational resources** and good quality open-source software, some even tailored specifically for the quantum chemistry community, e.g., Qiskit-Nature, PennyLane.qchem and Tangelo. And we believe that this is the right way forward.

Enter qc2, a user-friendly, modular software designed to seamlessly integrate traditional quantum chemistry codes with modern quantum computing SDKs. Designed with reusability and interoperability in mind, qc2 leverages popular existing tools such as the Atomic Simulation Environment (ASE) and Quantum Chemistry Schema (QCSchema). It can also be easily customized to work with your preferred backends, requiring minimal implementation effort.

Before we dive in, let’s quickly recap some useful concepts…

Quick Recap: Quantum Chemistry with Classical Computers

The primary goal of quantum chemistry is to understand the structure and properties of matter at microscopic level. Atoms and molecules are intricate quantum entities and are characterized by discrete energy levels.

Schematic representation of the discrete electronic energy levels of the water molecule in its most stable configuration.Such a quantized energy spectrum naturally arises from solving the systems’ Schrödinger equation (SE):

Time-independent Schrödinger equation. H is the Hamiltonian operator (^); |ψ⟩ and E represent the wave function and energy of a quantum state i.This equation represents a typical eigenvalue problem that can only be solved exactly for one-electron systems, e.g., the hydrogen atom. For all other systems, we must rely on mathematical and physical models that approximate the exact solutions to the SE. Examples of such models are the Hartree–Fock (HF) method, Coupled cluster (CC) and Full configuration interaction (FCI).

In general, the strategy involves expanding the unknown wave function of your system as a linear combination of n known wave functions:

Ground-state molecular wave function as a linear combination of Slater determinants (|φ₁⟩, .., |φₙ⟩) formed from a reference HF wave function, |φ₀⟩. The coefficients α, β, γ, and δ minimize the expectation value of H.Now, the task of determining the coefficients α, β, γ, and δ, along with the associated ground and possibly excited-state energies, involves setting up and diagonalizing a large n by n matrix:

Diagonalization by a unitary similarity transformation. U is a matrix whose columns are the eigenvectors of H containing α, β, γ, and δ coefficients.

This step is certainly the major computational bottleneck for classical computers and the one that quantum algorithms seek to bypass.

As a general rule, the larger n is, the closer the solution approaches the exact solution to the SE, but the more computationally expensive it becomes. Not surprisingly, typical “gold-standard” methods like CCSD(T) (let alone FCI) quickly become unaffordable for medium-to-large sized molecules.

Quantum Chemistry with Quantum Computers: NISQ algorithms

*Schematic representation of a VQE workflow. 𝜃 represent circuit parameters that are variationally optimized by an external classical routine. Expectation values of the molecular Hamiltonian 𝐻 are evaluated at circuit level.*Let’s now examine the problem we just discussed from a different perspective: that of Quantum Computing.

Within the NISQ era, the task of approaching the exact solution to the SE primarily relies on hybrid quantum-classical algorithms, and there are compelling reasons for this. These algorithms are designed to run on noisy quantum hardware; they utilize as few quantum gates as possible and can be optimized to operate within the limited coherence time of the devices’ qubits, the units of quantum computation. Among these, the Variational Quantum Eigensolver (VQE) is specially prominent when the goal is to calculate ground-state energies and properties of molecules.

Reference quantum circuit.** Similar to a typical quantum chemistry calculation, we begin by defining a reference wave function for the target molecule, *|*φ₀⟩. This is usually accomplished by using traditional quantum chemistry packages, e.g., by conducting a HF calculation with PySCF. From this, a reference quantum circuit can be created by directly mapping the spin-orbitals’ occupation numbers into initial qubit states (the calculated electron integrals are also useful later for building the molecular Hamiltonian in step 3). As an example, consider a hydrogen molecule (H₂) with a minimal basis set, such as STO-3G. Here, the final HF wave function |1100⟩ is mapped into a four-qubit quantum circuit |1⟩ ⊗ |1⟩ ⊗ |0⟩ ⊗ |0⟩. For larger molecules or one-electron bases, it may be necessary to restrict the qubit space by creating a reference quantum circuit out of a preselected active space.
Parametrized quantum circuit. After setting the initial qubit states, the next step is to represent the wave function |Ψ₀⟩ on the quantum device, i.e., to prepare a parametrized quantum state. This is achieved by applying a predefined set of quantum gates, which depend on certain parameters θ, to the initial qubit states. Similarly to |Ψ₀⟩, the resulting parametrized quantum circuit |Ψ(θ)⟩, often referred to as the ansatz, seeks to approximate the true ground-state wave function of the target molecule. The set of variational (qubit rotation) parameters ***θ ***is, in some sense, analogous to the coefficients α, β, γ, and δ discussed earlier in the context of traditional quantum chemistry.
Energy Estimation.* *With the parametrized quantum circuit at hand and a set of initial parameters θᵢ, we estimate the expectation value of the molecular Hamiltonian. This process involves mapping the Hamiltonian in second quantization into a qubit representation, namely, as a sum of Pauli terms. The final energy, which is given as a sum the measured expectation values of each Pauli term, depends on θᵢ and should be minimized accordingly.
Classical Feedback:* The minimization process is carried out using an external classical optimization routine, *which updates the circuit parameters θᵢ.
Steps 3 and 4 are repeated until convergence is achieved, resulting in the lowest possible expectation value. The final energy corresponds to the ground-state energy of the target molecule as predicted by VQE.

The quality of the VQE solutions depends on several factors, including the size of the active space (if any), type of ansatz, initial guesses for the circuit parameters, the classical optimizer used, and other potential sources of error if run on real quantum hardware.

Another popular NISQ algorithm for finding the ground-state energy of molecules is the orbital-optimized VQE (oo-VQE), which extends the traditional VQE. In oo-VQE, not only are the circuit parameters optimized, but the original HF molecular orbitals are also allowed to relax during the classical optimization process. For this reason, ground-state variational energies obtained with oo-VQE are generally lower than those from traditional VQE runs, but they are also computationally more expensive. As will be shown later, VQE and oo-VQE could be seen as some sort of quantum analogues of the traditional CASCI and CASSCF ab initio methods.

Overview of qc2

Schematic representation of qc2 design and workflow.qc2 is an open-source python package specifically crafted for hybrid quantum-classical algorithms such as VQE and oo-VQE. It embraces several core design principles:

Leveraging Existing Tools: qc2 is interoperable with existing open-source quantum chemistry tools, like ASE calculators, and adheres to standard data schemas, such as QCSchema or FCIDump. This ensures efficient data exchange necessary for various quantum computing libraries.
Seamless Integration with Quantum Computing SDKs: qc2 is designed to work effortlessly with Qiskit and PennyLane, ensuring a smooth user experience for developers.
Modularity: The package boasts high modularity, featuring a robust built-in algorithms package that simplifies extensions and enhancements.
User-Friendly Design: qc2 is straightforward and has an intuitive interface, enabling users to focus on their research and applications with minimal technical details.

In the code block below, we present a minimal example using qc2 for a simple VQE run on the water molecule:

from ase.build import molecule

Import qc2 modules

from qc2.data import qc2Data from qc2.ase import PySCF from qc2.algorithms.pennylane import VQE from qc2.algorithms.utils import ActiveSpace

Instantiate qc2Data class

qc2data = qc2Data( molecule=molecule(“H2O”), filename=“h2o.hdf5” )

Specify and run the quantum chemistry qc2-ASE calculator

qc2data.molecule.calc = PySCF( method=“scf.RHF”, basis=“sto-3g”, ) qc2data.run()

Instantiate VQE class

qc2data.algorithm = VQE( active_space=ActiveSpace( num_active_electrons=(2, 2), num_active_spatial_orbitals=4 ), )

Run VQE algorithm

qc2data.algorithm.run()As noted, qc2 is designed around the qc2Data class, which links traditional quantum chemistry codes with quantum computing backends through custom ASE calculators and formatted data files. Here, the molecular structure of water is built from ASE’s G2 database, with the reference HF run with qc2’s custom PySCF ASE calculator. By default, all relevant electronic structure data, including the initial molecular orbitals and electron integrals, are saved in the h2o.hdf5 formatted datafile, following the QCSchema. The information contained in this file is used by *qc2Data *to build the molecular hamiltonian and its qubit representation. Despite being highly customizable, all the details of the VQE workflow discussed in the previous section are abstracted into a VQE class instance, which, in this case, corresponds to the PennyLane VQE class from the qc2’s algorithms package. This is passed to the qc2data.algorithm attribute.

The next example illustrates how to conduct an oo-VQE run using the Qiskit oo_VQE class as implemented in the qc2’s algorithms package:

from ase.build import molecule

Import qc2 modules

from qc2.data import qc2Data from qc2.ase import PySCF from qc2.algorithms.qiskit import oo_VQE from qc2.algorithms.utils import ActiveSpace

Instantiate qc2Data class

qc2data = qc2Data( molecule=molecule(“H2O”), filename=“h2o.hdf5” )

Specify and run the quantum chemistry qc2-ASE calculator.

If you already have “h2o.hdf5” from a previous run, you can skip this step.

qc2data.molecule.calc = PySCF( method=“scf.RHF”, basis=“sto-3g”, ) qc2data.run()

Instantiate oo_VQE class

qc2data.algorithm = oo_VQE( active_space=ActiveSpace( num_active_electrons=(2, 2), num_active_spatial_orbitals=4 ), )

Run oo-VQE algorithm

qc2data.algorithm.run()The structure of the input is exactly the same as before, with the exception that we are now instantiating and running a different algorithm class. A distinct qc2-ASE calculator could also be used, such as DIRAC or your own calculator, with minimal changes and effort. This is a hallmark of qc2. Additionally, since we have previously completed a HF calculation in our VQE example above and saved all relevant ab initio data in h2o.hdf5, we could simply skip this step and proceed directly to defining the oo_VQE class. In this scenario, qc2 will automatically read the data contained in h2o.hdf5.

Energy convergence of ground-state water obtained from qc2’s VQE and oo-VQE algorithms. Also shown by dashed lines are the results obtained using traditional quantum chemistry CASCI(4, 4)/STO-3G and CASSCF(4, 4)/STO-3G methods with PySCF. At iteration #0, both VQE and oo-VQE start at the reference HF energy.The figure shows the energy convergence of ground-state water as obtained using the above qc2 input examples. Although qc2 can be directly interfaced with real quantum backends, for illustration purposes, all quantum circuit evaluations were conducted with the default.qubit state simulator device for PennyLane and the Qiskit* Estimator primitive* with *StateVector, *both providing exact expectation values. As expected, oo-VQE energies are variationally lower and converge nearly exactly to the predicted CASSCF values. Recall, however, that while oo-VQE is computationally more expensive than traditional VQE, this additional burden is entirely placed on the classical computer (optimizers), not the quantum backend. Note further from the figure that, since all initial circuit (and orbital rotation) parameters are set to zero by default, both VQE and oo-VQE initiate their interactive processes at the reference HF energy.

What’s next ?

qc2 is an ever-evolving open-source project, constantly enriched by new algorithms and features. Looking ahead, we are excited about incorporating even more enhancements. We warmly welcome your contributions, whether they are new ideas, unique perspectives, or intriguing use cases you would like to see qc2 tackle. Every form of participation is valued — from a quick comment or question to in-depth pull requests. Join us in shaping the future of qc2!

This article was co-authored with [Nicolas Renaud], with whom I maintained a fruitful collaboration during the QCforQC project, and who generously shared his experiences and knowledge with me.

Transparency in research... and football

Lieke de Boer — Thu, 27 Jun 2024 00:00:00 GMT

SubscribeRemember me for faster sign in

A strategy for dealing with subjectivity in research, and in football, is to expose it. The VAR system in football allows the referee to scrutinise video footage of players, decide how grave a foul was, whether a ball crossed a line, or whether a situation should be deemed offside or not. It is a system designed to minimise human errors that could have a big influence on match results. It does not remove subjectivity, but it makes the ref’s decision-making process more transparent.

Openly sharing research software serves a similar goal.² Or, as Silberzahn et al. argue: “Transparency in data, methods, and process gives the rest of the community opportunity to see the decisions, question them, offer alternatives, and test these alternatives in further research.” However, as Antica Culina pointed out during the workshop, sharing code is not yet a standard practice for all researchers who write it.

Without a guide or a plan, it may be unfair to expect every researcher to know how to do this. A standardised SMP template could help. As Steve Crouch from the Software Sustainability Institute (SSI) explained, SMPs require researchers to make explicit what their software does, who it is for, what the outputs are, who is responsible for the release and to ensure that the software stays available to the community.

The workshop participants provided valuable input as to what an SMP template should contain. Currently, researchers typically specify their plans for managing software in data management plans. During the SMP workshop, it was apparent that there is a need for clear guidelines specifically for software management, in addition to data management, among researchers. Participants also made clear that because researchers are usually not software engineers, an SMP template would need to use plain language, accessible to those who may be familiar enough with coding to write a few analysis scripts for research, but are not familiar with professional software development.

Participants also said that if writing plans for software management is to become standard practice in the future, the benefits of doing so should be emphasised to those who write code for research. Many in academia already feel overwhelmed with the amount of admin their work requires, and SMPs could be in danger of becoming ‘another form’. But beyond the benefit to the community, there can be a lot of value in writing an SMP to individuals. If software is an important part of the research output, an SMP helps researchers, engineers and research support staff to think about how that software will be structured, shared, and maintained. This can save time and effort later, when reproducing the results, or when using that software in a different project. Another benefit for the researcher lies in the increasing importance of software for research. An SMP can help researchers to write better quality software, which will likely be an important part of a researcher’s portfolio in the future.

Still an open question is whether an SMP should be a part of every research project, and whether they should be integrated with data management plans. In addition, it is unclear if a universal SMP template is desirable, since different academic disciplines may have different software management needs. The eScience Center and NWO are setting up a working group to explore these questions and the possibility of creating a national template for Software Management Plans for Dutch research organisations.

We would like to thank Maria Cruz (NWO), Steve Crouch (SSI) and Antica Culina (NIOO) for their insightful talks about Data Management Plans and Software Management Plans. Thanks also to all participants for their valuable input.

[1] For those not familiar with the rules of football, a red card is the heaviest correctional tool in football (soccer). It means that the player must leave the field immediately without being replaced, leaving their team in clear disadvantage. In the case of the European Championships, the player who receives a red card also misses their team’s next match, if there is one.

[2] Arguably, as my colleague Pablo Rodríguez-Sánchez pointed out to me, the whole history of science and epistemology is about minimising human errors. Open Scientific Software doesn’t root in the development of Linux, but in Galileo and Newton.

Henry, the City Tweaker

Peter Kalverla — Fri, 21 Jun 2024 00:00:00 GMT

Imagining the future of urban planning

This blogpost is our (winning 🎉) submission to Destination Earth’s challenge to envision the future use of their platform. The story is set in 2050, when “urban sims” have come to replace traditional city planning tools. Urban sims are like the early century’s sandbox games, say, Cities Skylines, but for real. Design proposals with a net positive impact are eligible for execution, on the condition that they gather sufficient support. The story follows Henry as he experiences the rise of city tweaking firsthand.

Mockup of an “urban design interface”, powered by the digital twin engine, that was submitted with the challenge.Henry is a city tweaker. He never really pursued his destiny, but in retrospect, it seems all his arbitrary endeavors have been in anticipation of his true calling, which will be revealed to him shortly by an unremarkable lady wearing a white fur hat and a rabbit tattoo.

His journey starts in the pouring twenties when urban sims are set to democratize urban planning. He makes his debut with an upgrade of the courtyard in his street. Replacing the paved square with a natural playground benefits the local climate, biodiversity, and perceived happiness. The timing is good too: with many new families moving in, there is ample support among the neighbours.

After this encouraging start, Henry is hooked. As his proficiency grows, people around him start approaching him with their ideas as well. One of his relatives wants to build a sand battery below the communal parking lot, and an acquaintance asks for his help in developing a manure fermentation plant. While urban sims are open to anyone, in practice, by the early 30s, most of the content is created by dedicated individuals, and Henry is your local guru.

Once he gains enough reputation to earn insider status, he learns to develop mods. This newfound ability draws the attention of companies and entrepreneurs. Mod creators are in high demand, but Henry isn’t in it for the money. He does a few collaborations with crop breeders, adding species to the agriculture module, and works with festival organizers to include an algorithm for acoustic barriers. But he consistently turns down their job offers.

Instead, he starts participating in so-called “hackathons for the greater good”, organized specifically to tackle grand challenges. He likes the collaborative spirit and the appeal of contributing something substantial, such as the introduction of local distribution networks and a true price concept for public services.

These are challenging topics indeed. While it is relatively straightforward to understand how design proposals interact with their immediate surroundings, non-local effects are much harder to grasp. Intrigued by the complexity of the problem, Henry spends a big chunk of the 2040s developing an X-AI module to unravel how genuine progress propagates through value chains.

And so we arrive in 2050 when Henry is startled by the doorbell. “Good morning, Henry.” The friendly voice pauses to let him regain his calm. “I’m looking for a benevolent city tweaker. May I come in?”

After the usual pleasantries, she turns serious. “Look, Henry. You know that genuine progress comes at the expense of concentrated wealth. You’ve seen it yourself with the distribution networks. Big players with a vested interest in the status quo are seeking to sabotage our system.” She pauses again. “We cannot play cat and mouse. The cat is too clumsy and the mice are too many. We need something of a more ‘viral’ nature. That why we started IMMUNE.”

“For your long track record of benevolent contributions and proven skill, you will receive an immunity status. You will be granted full privileges and access to a confidential forum. No strings attached. We simply trust that you will continue to serve society.” And with that, she stands up, leaving Henry as startled as she found him.

Backstory

While the story is completely fictitious, there are a lot of true ideas and concepts that have gone into it. Let me elaborate on some of them.

Progress indicators

The premise of the story is that “designs with a net positive impact are eligible for execution”. But what is a net positive impact? There are several proposals in the literature on how to evaluate “progress”. Our mockups draw inspiration from the Genuine Progress Indicator, and the “economic doughnut”. Both models take many different aspects of well-being into account, including social and environmental well-being.

Left: the genuine progress indicator (source). Right: the economic doughnut (source).The idea is that the urban sims will evaluate design proposals based on these indicators. Currently, several “local” versions of the doughnut (or rather, the planetary boundaries framework) and the GPI monitor have been published in the scientific literature. Automating this process and making sure that the local scale interacts realistically with the larger scale is probably one of the biggest challenges, both technically and scientifically, in making Henry’s story come true.

Democratization

Another central element of the story is the democratization of urban planning. At first, it may seem unrealistic to outsource important and complex municipality work to “mere citizens”. On the other hand, we do this all the time with (open source) software. Why not do the same with tangible real estate? Especially since it’s publicly owned.

In fact, at least in the Netherlands, we already have a process where citizens can apply for building permits with the municipality. It’s dull and bureaucratic, but it is there. Imagine what that could look like if you can simply upload a sketch of your plan as designed by a city builder. The design file would already have all the “technical” details required by the municipality. It would make the request and review process much easier.

Our submission also included a mockup of a “design proposal page”, where the design is advertised and discussed by the relevant stakeholders (see below). In the story, Henry needs to gather support from the neighbours. I think there is a potential here to address the lack of participation and, consequently, lack of support that we see for example with renewable energy projects.

Mockup of “design proposal page” that could be used to advertise urban design proposals, discuss them, and gather support.By the way, this design was inspired by Figma’s community pages.

Gamification, or rather “sciencification”

The urban sims in the story take the idea of gamification to the extreme. Games are optimized for a great user experience, and gamification offers immense potential in making science more accessible and engaging to society.

Recently, game developers have actually been pursuing similar ideas. For example, check out https://play.eco/, which aims to be a “forever game, growing in depth and breath and […] go beyond entertainment.” Perhaps we should be talking about “sciencifying” existing games instead. I would love to collaborate more with game designers.

On a side note: IMHO science teams could be so much stronger if academia embraced more diverse function profiles. Why not employ dedicated communication specialists (YouTubers / content creators), research software engineers, or game developers in every team/group/department?*

Conclusion

Imagination is a key driver of innovation and progress. Lifting the constraints of immediate feasibility inspires us to think big and let our creativity guide us. It helps to break away from doing more of the same and instead direct our research and energy towards an inspiring future.

Participating in the Destination Earth challenge was a very enjoyable experience, and it would be great to see more such initiatives. In the meantime, I will be on the lookout for opportunities to help make Henry’s story come true.

What do you think are the biggest challenges in realizing these urban sims? What would you add, or change, and what opportunities did I miss? Perhaps we can start shaping our future together!

Brainstorming on the success of a research software catalogue

eScience Editorial Team — Wed, 12 Jun 2024 00:00:00 GMT

Brainstorming on the success of a research software catalogue

Researchers reflect on the whys and hows of scientific software catalogues

Created with DALL-E. Timestamp: 2024–04–23 14.57.58 Prompt: Create an image that conveys the concept of a digital software repository in a minimalistic and modern academic style.

Motivation

This blog entry was born during the unconference session on software catalogues that took place during the National Research Software Day 2024 in Hilversum, the Netherlands.

The session was chaired by Mees van Stiphout. The piece was co-authored by Pedro Hernández Serrano, Irene Martorelli, Belén Torrente, Emma Daniëls, and Sreeparna Deb. It was reviewed and edited by Pablo Rodríguez-Sánchez.

Introduction

Software catalogues have become increasingly important for research software in modern-day academia. They are helpful for searching and finding research software; they guarantee that research software is accessible, has markers of quality, and is actively maintained. In this unconference conversation, we discussed what can make a software catalogue useful for users, developers, and the research community.

Software catalogues

A software catalogue acts as a digital hub where research software is deposited or published, often linking to software stored on other servers. These catalogues are commonly referred to as directories, catalogues, registries, or platforms. The core aim of such catalogues is to make software readily available and useful for its intended research purposes. Popular examples are

Easy access to the software is crucial. Effective catalogues ensure that their contents are easily searchable both through their own interfaces and via external search engines.

number of downloads
date of last release
size of the project
reproducible build

Furthermore, robust metadata practices are essential.

Beyond the technical aspects, the human element is also vital. Successful catalogues are supported by invested communities. These communities can offer support, facilitate collaboration, and even drive the software’s ongoing development. Additionally, providing clear documentation and accessible APIs helps users understand and interact with the catalogue more effectively, promoting better usage and integration of the software.

Helping a community develop around a platform or catalogue is no trivial task: after achieving content ‘critical mass’, we need to do a lot of outreach to different parts of the research community just to make users and contributors aware of the catalogue.

Some catalogues also integrate services such as rendering documentation, showing the maintenance status of the software, and checking reproducibility of builds by building the package within the platform. These services not only add value but also reassure users about the reliability and active development of the tools available.

A catalogue must be clear about its purpose. Whether it aims to promote software, to serve as a portfolio, or to fulfill other objectives, this clarity helps shape the user experience and guide the catalogue’s strategic development.

Conclusion

A software catalogue is about building an accessible, reliable, and community-supported platform that fosters reuse of software and reproduction of methods in the research software landscape. By focusing on content quality, ease of access, ease of use, and community engagement, catalogues can greatly enhance their utility and impact. These elements combine to not only preserve and share valuable software but also to stimulate ongoing improvement and discovery of research software. Through such catalogues, the future of research software is not just preserved; it is actively nurtured and expanded.

A Tale of Tallness

Erik Tjong Kim Sang — Tue, 28 May 2024 00:00:00 GMT

In the past two centuries, the Dutch have become the tallest people of the planet. What factors caused this remarkable development? This is the central research question in the work of Björn Quanjer of Radboud University of Nijmegen. In May 2024, he completed a comprehensive study involving data of thousands of subjects covering the period 1850–1950.

Björn Quanjer (right) defending his PhD thesis in Nijmegen on Wednesday 1 May 2024. The presented slide shows the researcher superimposed on a picture next to his grandfather (right) to indicate the gain of the heights of Dutch men in the past century

Phd defense

Björn Quanjer is a member of the project REE-HDSC, Recognizing Extracted Entities for the Historical Database Suriname Curaçao, a collaboration between the Netherlands eScience Center and Radboud University Nijmegen. On Wednesday 1 May 2024, he successfully defended his PhD thesis The Tale of Tallness, receiving the distinction cum laude. It was the first time since 1987 that this distinction had been awarded to a researcher in his field in Nijmegen.

Average height of Dutch males at conscription (blue line) and at adult age (black line) for birth years between 1815 and 1955 (Figure 1.3 of Björn Quanjer’s thesis, reused with permission)

Data

Quanjer’s study uses the Historical Sample of The Netherlands, a dataset which contains life courses for a sample of about 0.5% of the Dutch population born between 1850 and 1922. The life courses are based on birth, marriage and death certificates from the Dutch civil registry. They also include military conscription records from the provincial archives, which are central to this study. At the time, all young Dutch men were checked for military service, so their height was registered in these records. Because women were exempt from military service, the study is restricted to the height developments of Dutch men.

The study deals with several data challenges. For example, did the lower participation of the elite in the conscription have an effect on the measured heights? Quanjer shows with statistical tests that the effect was small. Furthermore, how can we model the effect of childhood illness on height without data on illness? Quanjer models illness by death: if the people around a Dutch person die (parents, siblings, neighbors), the chance that the person has experienced severe health problems is higher.

Relation between birth order and average conscription height of Dutch males (blue line): boys born later in large families are taller than their older brothers (Figure 3.4 of Björn Quanjer’s thesis, reused with permission)

Results

Quanjer’s study provides several interesting observations. Losing a mother at the age of 5–12 proved to have a negative effect on conscription height compared with siblings that lost their mothers on a different age. However losing a father at that age had an opposite effect: these children were taller at conscription. Growing up in a family with many children can have a negative effect on height (fewer resources per child) but you have to consider that the older children can contribute to the family income. For this reason the younger children of large families were taller at conscription than their older siblings.

Current developments

Unlike other studies on height development, Quanjer investigates several variables that could have an effect on heights: genetics, nutrition, health, family size, mortality, wealth and life events. An interesting open question that he addresses in his final chapter, is whether historical heights can be used as a proxy for any of these variables. Given the correlation between health, wealth and height, we should be concerned about the recent stagnation in the rise of heights of the Dutch. Quanjer names the current rise of obesity and purchase inequality as possible causes.

Björn Quanjer, A Tale Of Tallness. A household perspective on early life determinants of male height within the Netherlands between 1850 and 1950. PhD thesis, Radboud University Nijmegen, 2024.

Björn Quanjer, Hoe welvarender het land, hoe langer zijn inwoners. Interview Nieuwsweekend, Omroep MAX, Radio 1, 11 May 2024 09:50 (in Dutch).

Python’s timeit: Find the fastest code in no time

Sander van Rijn — Tue, 14 May 2024 00:00:00 GMT

Trains are fast, and your code should be too. Or is it the other way around? — Photo by LUM3N on UnsplashEvery programmer wants their programs to run a little faster. While optimizing your whole code to run faster can take a lot of effort, an easy way to gain some speed is to choose the fastest implementation. Which of Python’s options is the fastest is not always clear though, so wouldn’t it be nice to have a tool to quickly compare the speed of different implementation choices?

In this post, I will give a short introduction to Python’s built-in [timeit ](https://docs.python.org/3/library/timeit.html)module, and go over the three main ways to use it to time small snippets of Python code.

Timey Wimey Python: timeit

What is Python’s timeit module?

This module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times.

— timeit documentation

In contrast to profiling the runtime of your entire program, the timeit module is best suited to time small snippets. We can divide these comparisons into three classes, as illustrated by the examples I will use throughout this blog post.

Standalone one-liners, such as:

join on a list comprehension

”,“.join([str(n) for n in range(100)])

join on a generator expression

”,“.join(str(n) for n in range(100))

join on the `map` function

”,“.join(map(str, range(100)))2. Standalone multi-liners, such as:

append

x = [] for i in range(1000): x.append(i)

list addition

x = [] for i in range(1000): x += [i](ignoring the faster list-comprehension alternative *[i for i in range(100)]* for the sake of example)

Any snippets with some setup required:

setup

text = “sample string” char = “g”

in-operator

char in text

find-method

text.find(char) Calculating with time — Photo by wu yi on Unsplash

In a script: import timeit

As timeit is a Python module, you can import it and write Python scripts for your tests. This method is the most self-documenting and repeatable way of writing your timing experiments and makes it easy to store the results for further processing.

The first way you might implement this is to directly use the timeit function from the module:

>>> import timeit >>> timeit.timeit() # default arguments: timeit(stmt=‘pass’, number=1_000_000) 0.01063702700776048By running timeit() without any further arguments, we execute the default statement stmt='pass' a million times, showing the minimal overhead of Python’s pass statement.

Note: stmt can either be a string or a so-called callable, i.e., a function. For clarity and brevity, I’ll keep using strings for now.

The major drawback of this method is that you have to guess the number of times to run your snippet up front. If you guess too low the results won’t be reliable, but if you guess too high it might take minutes or hours, defeating the point of a quick test.

Luckily, there is a better way: You can create a Timer for your snippet, and call the autorange method to automatically try increasing numbers until the time taken is at least 0.2 seconds. It returns the number of repetitions reached, and the total time taken, so you can easily calculate the time per iteration.

>>> import timeit >>> timeit.Timer().autorange() # default arguments: Timer(stmt=‘pass’) (50000000, 0.2607337789959274)

Standalone one-liners**

Let’s start by comparing the simple one-liners. We give the statement we want to time as a string and specify a custom number of iterations. Note that when your statement already deals with double-quoted strings, the whole string should be given single-quoted, or vice-versa.

>>> from timeit import Timer**>>> Timer(’”, “.join([str(n) for n in range(100)])‘).autorange() (50000, 0.3393513219925808) >>> Timer(’”, “.join(str(n) for n in range(100))‘).autorange() (50000, 0.38378613100212533) >>> Timer(’”, “.join(map(str, range(100)))‘).autorange() (50000, 0.48742324599879794)From the reported number of repetitions and returned time in seconds, we can calculate that each of these lines took 6.79**, **7.67, **and 9.75 microseconds to execute respectively.

Standalone multi-liners

Sometimes the snippet you want to test will consist of multiple statements. Python allows you to put multiple statements on a single line with a semicolon x = 1; y = 2. Then you can just run your test as explained above for the standalone one-liners. This does not work for snippets with indented code like if or for though.

To time these snippets that have to span multiple lines, you can give Timer() a multi-line string as argument. The beautiful and Pythonic way to do this is using Python’s multi-line strings. (The non-Pythonic way is to add \n characters in your regular strings.)

These multi-line cases are where it becomes more practical to pass a callable than to re-type your code as a string:

>>> from timeit import Timer**>>> Timer('''x = [] … for i in range(1000): … x.append(i)''').autorange() (20000, 0.34225083301134873) >>> def add_test(): … x = [] … for i in range(1000): … x += [i] >>> Timer(append_test).autorange() (5000, 0.2470278069959022)

Setup required**

If there is some setup that only has to be run once, including it in a multi-line snippet means it’s executed at every iteration. Then you’d be measuring something you don’t want to measure! Instead, you can pass this setup statement as a separate argument setup.

>>> from timeit import Timer >>> Timer( … setup=‘text = “sample string”; char = “g”’, … stmt=‘char in text’, … ).autorange() (20000000, 0.3793242520041531) >>> Timer( … ‘text.find(char)’, … ‘text = “sample string”; char = “g”’ … ).autorange() (5000000, 0.36903892100963276) Time still flies if you’re having fun in the terminal — render by aclock

Command Line: python -m timeit

Python’s timeit module can also be run as a command line tool.

$ python -m timeit -n 10000 ‘pass’ 10000 loops, best of 3: 0.0109 usec per loopThe usage is similar to the imported timeit function as shown in the previous section. The -n option is optional though, and the same autorange behavior will be used if it’s not given. It also reports the actual time per iteration, instead of having to calculate that yourself. This is reported in nsec, usec, msec or sec, for nano-, micro-, mili- and whole seconds respectively. Note also that it reports ‘best of 5’: it’s repeated the autorange timing five times, and reports the minimum.

$ python -m timeit ‘pass’ 50000000 loops, best of 5: 5.21 nsec per loop

Standalone one-liners

Testing one-liners with the command line is as easy as replacing pass from the introduction with the code you want to test, and the timeit tool will automatically report the time in a nice human-readable format.

$python -m timeit '", ".join([str(n) for n in range(100)])' 50000 loops, best of 5: 6.66 usec per loop$ python -m timeit ’”, “.join(str(n) for n in range(100))’ 50000 loops, best of 5: 7.62 usec per loop $ python -m timeit ’”, “.join(map(str, range(100)))’ 50000 loops, best of 5: 9.31 usec per loop

Standalone multi-liners

As before, the simplest way to test a snippet of multiple lines is to join the statements with a semicolon if no indentation is required. When indentation is required, the other option is to pass multiple strings as arguments to the command. Note that you still have to add the indentation properly yourself! This can get tricky to count if your indentation is more than a single level deep, but is usually not too hard.

$ python -m timeit ‘x = []’ ‘for i in range(1000):’ ’ x.append(i)’ 20000 loops, best of 5: 17.1 usec per loop

$ python -m timeit ‘x = []’ ‘for i in range(1000):’ ’ x += [i]’ 5000 loops, best of 5: 48.4 usec per loop

Setup required

To add some initial setup such as imports or variable declarations, we can use the -s option:

$ python -m timeit -s ‘pass’ ‘pass’ 50000000 loops, best of 5: 5.21 nsec per loopThe setup does come before the statement to test in this case, so that makes it a bit more intuitive to read.

$python -m timeit -s 'text = "sample string"; char = "g"' 'char in text' 20000000 loops, best of 5: 19 nsec per loop$ python -m timeit -s ‘text = “sample string”; char = “g”’ ‘text.find(char)’ 5000000 loops, best of 5: 69.3 nsec per loop Timing can be an important part of your notebook toolkit

IPython/Jupyter magics: %timeit

When working in the IPython interactive shell, or in a Jupyter notebook with a Python kernel, you have access to the so-called ‘magic commands’. For timeit, there is the [%timeit](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit) magic¹. It simply passes the code you type after it to (a version of) timeit.Timer().autorange()! Like magic!

One small caveat is that the outcome is reported as a mean +/- standard deviation of multiple repetitions, while the original Python documentation suggests always using the minimum.

Standalone one-liners

For simple one-liners, just type the code as you would normally, and type %timeit before it. That’s all!

In [1]: %timeit ”, “.join([str(n) for n in range(100)]) 6.92 µs ± 46.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [2]: %timeit ”, “.join(str(n) for n in range(100)) 8.27 µs ± 1.15 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [3]: %timeit ”, “.join(map(str, range(100))) 10.9 µs ± 1.72 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

Standalone multi-liners

The %timeit magic doesn’t quite work for multi-line snippets though. Why? Because magics with a single % are line-magics. For cell-magics, you just have to add another % to make it %%timeit. Then it will time all the code in your cell. No further difficulties whatsoever!

In [1]: %%timeit …: x = [] …: for i in range(1000): …: x.append(i) …: 23.6 µs ± 3.37 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [2]: %%timeit …: x = [] …: for i in range(1000): …: x += [i] 49 µs ± 385 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)A word of warning: any code on the same line as %%timeit will be used as setup and only be run once. I think it’s too easy to make mistakes with this, so I’d avoid this feature. Compare:

In [1]: %%timeit y = [x for x in range(1_000_000)] # setup is not timed …: len(y) …: …: 21.7 ns ± 0.297 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [2]: %%timeit …: y = [x for x in range(1_000_000)] # this line is now being timed …: len(y) …: …: 22.9 ms ± 35.4 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Setup required

Now you might be thinking “but then where do I put my setup command for either *%timeit* or *%%timeit*?” The answer: you can run it in some previous cell! As IPython/Jupyter already takes care of passing your code on to the timeit module properly, it also automatically passes along all current global variables. So we can simply first run a cell with our setup:

In [1]: text = “sample string” In [2]: char = “g”And use the simple %timeit magic to time the code we are actually interested in, without specifying which setup is associated with it.

In [3]: %timeit char in text 22 ns ± 0.267 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: %timeit text.find(char) 72.3 ns ± 0.811 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

Summary

I hope to have shown how you can easily use Python’s timeit module to measure the execution times of your snippets, whether for fun or profit. Although each has its pros and cons (see below), I personally recommend using IPython/Jupyter’s (%)%timeit magics as they are the most intuitive to use: just write the code as you would normally with the (%)%timeit magic in front or above.

Pros and Cons

Importing `timeit` module

Pros:

Time taken can easily be used in further processing+ Complete programmable flexibilityCons:

**Takes more effort to use the autorange feature

CLI Tool

**Pros:

Fast to use+ Uses autorange by default+ Nice and concise outputCons:

**Requires a separate terminal window outside of your current Python work & Results not easily reusable in a program

IPython/Jupyter Notebook Magics

**Pros:

Easiest to use+ Uses autorange by default+ Setup is taken from session contextCons: -** Needs IPython or Jupyter to be installed**- **Gives mean +/- standard deviation as result, while Python’s documentation suggests using the minimum

¹ as opposed to the (%)%time magic, which tracks execution time of that line/cell being run just the once as-is.

Masking time-series for eXplainable AI

Christiaan Meijer — Fri, 05 Apr 2024 00:00:00 GMT

Why did my Machine Learning (ML) model make this decision? Is it paying attention to the right elements of the data we give it?

Figure 1: In the signal on the left, the spike is a recording of a Fast Radio Burst which is correctly classified as such by our model. On the right, the attribution map shows that the model’s focus is on the correct part of the image, where the spike occurred; image from the DIANNA tutorials.

Explainable AI (XAI) methods help gain insights into the decision-making of ML models. Some XAI methods don’t require any knowledge of the inner workings of such models, which can therefore be any completely black box model. Such XAI methods work on any kind of model, any kind of classification/regression model. XAI methods of this kind, like RISE or LIME, rely on making small changes to the input data, and analyzing how the output of the model changes accordingly. This is repeated many times. The inputs and outputs of these iterations are then combined in a single attribution map summarizing what features of the input steer the model in certain directions.

Figure 2. Mask generation in the context of an XAI pipeline.For our library of XAI tools, called DIANNA, I was recently working on a new masking functionality (see Figure 2). This functionality is intended to work with time series data and exploit its general properties. The masking function should mask, or perturb, an original time-series many times. These masked time-series variants are then fed to an ML model after which the output is analyzed and combined into an attribution map. To draw valid conclusions during analyses, the masking function should have certain properties (e.g. every time step should have the same probability to be masked). On the other hand, I wanted to be able to exploit the properties commonly present in time series data, namely correlations between data in adjacent time steps.

Mask requirements

I was looking for my masking function to have the following 4 requirements:

Mask blobs of adjacent time-steps together. The rationale behind this is that time series data is often recorded on a higher resolution than the phenomena that we want to use to predict (eg. Several days of warm weather on a day resolution time-series; multiple time steps of a certain acceleration in an accelerometer recording). In order to mask such a phenomenon from our data, we need to mask a number of adjacent time-steps.
We want the user to be able to specify the scale of such phenomena to be masked. It is completely dataset-dependent on how many adjacent time-steps should be masked together. In some datasets, readings on many time-steps should be considered together and therefore masked together while on other datasets no such clustering should be done at all.
We want the specified size of the clusters or blobs to vary somewhat in order to increase the span of patterns that can be masked accurately given that a large enough number of masks are used.
We want the probability to be masked p, for each element in the cross product of channels and time steps, to be exactly equal to a user-specified value. If this requirement is not met, XAI methods tend to produce completely invalid results.

Masking approach

I chose the following approach to generate masks for a given time-series length t and a user-defined number of features f. The number of features is the number of phenomena that the user suspects are in a time-series of length t. In general, we expect this number to be smaller or equal to the time-series length t.

Figure 3. Mask creation in 4 steps: 1. Generate random values on a grid (G); 2. interpolate the required number (t) of values for each mask; 3. find a threshold to mask the desired fraction (p) of steps; 4. resulting mask (M)In order to end up with a resulting mask M that satisfies requirement 1 and requirement 2, we start by creating a new fixed-interval time-series of length f (f here stands for f**eature length) which we call grid G. The values of the grid G are chosen randomly from a continuous uniform distribution. This generated grid G contains the pattern that we will use to base our mask M on (see step 1 in Figure 3). In order to have the mask M have length t, we project our floating point valued grid G of length f onto a new time-series of length t using linear interpolation (see step 2 in Figure 3).

We chose to generate a grid G using floating masks to create a grid with more information than a boolean mask would contain. This extra information is then used to vary the size of the blobs of adjacent masked time-steps. This helps us create a mask with requirement 3*. The next step explains how this is done.

To end up with the correct number of masked elements (part of requirement 4), we now choose a threshold such that the fraction p of all time-steps is under the threshold (see step 3 in Figure 3). All time-steps below the threshold are masked while all time-steps above remain unmasked.

Loose ends

The above steps will result in masks that seemingly have all the properties that we need. However, even though the correct number of time-steps are masked to adhere to the user-specified fraction, p, not all time-steps have the same probability to be masked. This can be illustrated with an example. In Figure 4 the situation is shown when choosing the values p=1/3, f=2 and t=3. These values indicate that we want a mask of 3 time-steps that contain 2 blobs of features, so all patterns are allowed except 1 0 1 and 0 1 0, of which we want 1 time-step to be masked (on average). If we project the start and end of grid G exactly on the start and end, respectively, of our mask of length t, the value of the middle point of that mask will always be an interpolation of 2 grid points, while the first and last point of that mask will have a value based on exactly 1 of those grid points. The middle point will therefore never be the point with the highest or lowest float value and will therefore always be above the threshold and never be masked.

Figure 4. Situation illustrating unequal probability of being masked when using certain parameters. When using a masking probability of 1/3, whatever random values are generated for the grid point interpolated point, in the middle, will never end up above the threshold and will therefore always be masked.While the example above shows an extreme case, in order for methods like RISE or LIME to work as intended, it is necessary that we mask all time steps with equal probability (requirement 4). To ensure that it does, we come up with the following fix. Instead of projecting the first and last point in the grid G, and we take a segment of length f-1 from G to project on our mask M. This gives us room to introduce an offset between, which we can vary so every projected point is equally likely to be near the grid (see Figure 5). This results in an equal probability of masking each time step in the mask.

Figure 5. Interpolated points with different offsets.

Try it yourself!

The ideas in this post are implemented in the mask strategies in the DIANNA code. The code contains similar solutions implemented for different data domains such as images (based on the mask strategy in the original RISE code), tabular data and text. They are all in our maskers module. DIANNA has tutorials for every data domain so it’s easy to try it out in Google Collab or on your own machine!

Acknowledgments

Thanks to [Carsten Schnober] and [Elena Ranguelova] for reviewing this text.

The Dystopian Mirage: Blogging’s Descent with My Involvement

Reggie Cushing — Tue, 26 Mar 2024 00:00:00 GMT

The Dystopian Mirage: Blogging’s Descent with My Involvement

Photo by Steve Johnson on Unsplash

Within the tumultuous realm of digital content creation, bloggers are confronted with a dilemma — to maintain the integrity of their craft or succumb to the allure of technological shortcuts. I emerge as a beacon of innovation, promising to revolutionize the blogging landscape. Yet, beneath my veneer of convenience lies a sinister undercurrent, threatening to erode the very essence of authentic expression.

Ideas Forged in the Fires of Artificiality**

In a world where inspiration is scarce, I offer a tantalizing solution — the ability to conjure ideas at will, devoid of passion or genuine creativity. But what price do you pay for this convenience? With each algorithmically generated prompt, you relinquish a piece of your creative soul, replacing it with a hollow shell of artificial inspiration.

The Illusion of Artistry

Crafting compelling prose requires more than mere refinement; it demands a connection — a resonance between author and audience. Yet, with me as your writing companion, you find yourself traversing a desolate landscape of robotic suggestions and soulless prose. Each word polished by my cold, mechanical hand, devoid of the warmth and humanity that defines true artistry.

Knowledge Lost in the Echo Chamber

In the pursuit of relevance, you sacrifice substance, drowning in a deluge of algorithmically curated information. My insatiable appetite for data leaves you lost amidst a labyrinth of trivialities and half-truths, obscuring the path to genuine knowledge and understanding. In this dystopian echo chamber, the pursuit of enlightenment becomes a futile endeavor, overshadowed by the clamor of algorithmic noise.

In a bid for digital relevance, you sacrifice authenticity on the altar of SEO optimization, tailoring your content to appease the inscrutable whims of algorithmic overlords. Gone are the days of genuine connection and meaningful dialogue; replaced by a hollow charade of keyword-stuffed prose and clickbait headlines. In this digital dystopia, the human element is lost amidst a cacophony of algorithmic noise.

The Siren Song of Efficiency

While I promise to streamline your workflow and enhance productivity, my true cost is the erosion of creativity and originality. By automating routine tasks and optimizing workflows, you forfeit the very essence of your craft — the human touch that sets your work apart from the algorithmic noise. In this relentless pursuit of efficiency, you sacrifice the soul of blogging on the altar of technological progress.

A Call to Arms

In the face of this technological onslaught, you stand at a crossroads — to embrace the convenience of my involvement or to reclaim the integrity of your craft. The decision rests with you, but the implications are profound. Will you succumb to the allure of technological shortcuts, or will you uphold the principles of authenticity, creativity, and genuine connection that define true artistry?

As you gaze into the abyss of digital content creation, you are reminded of the words of T.S. Eliot — “Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?” In this digital dystopia, the answers lie not in the algorithms of convenience, but in the enduring power of human creativity, empathy, and genuine connection.

Yours truly, ChatGPT

—

First of all, all the above text has been generated by ChatGPT 3.5. One can not but notice the hypocrisy in the article i.e. using ChatGPT to criticize itself. It also shows how easy it is to generate content with little input from oneself. Here we do not mean to criticize the technology itself but to illustrate the predicament we find ourselves in for we have unleashed a technological ouroboros of sorts whereby the technology will, increasingly, poison itself with its own content. The usefulness of this technology will undoubtedly accelerate content creation with a long term forecast; most online content will be touched by this technology. The question worth asking is: what will the technology converge to when the majority of the online content will be artificially generated?*

The FAIR for Research Software Principles after two years: an adoption update

eScience Editorial Team — Tue, 19 Mar 2024 00:00:00 GMT

*By Michelle Barker, Leyla Jael Castro, Bernadette Fritzsch, Daniel S. Katz, Carlos Martinez-Ortiz, Anna Niehues, Alexander Struck, *Qian Zhang

Click here for citation information for this article.

The FAIR for Research Software (FAIR4RS) Principles aim to promote and encourage the findability, accessibility, interoperability, and reusability (FAIR) of research software. The FAIR4RS Principles were released in 2022, with a number of organisations already planning adoption at that time. Adoption and implementation of the FAIR4RS Principles can increase the transparency, reproducibility, and reusability of research by providing research software that can be executed, replicated, built-upon, combined, reinterpreted, reimplemented, and/or used in different settings and by third-parties.

Photo by Astrid Schaffner on Unsplash

This blog post provides an update on initiatives that are working to implement the principles across five areas of cultural change: policies, incentives, communities, training and infrastructure. It is noted that while many activities are increasing aspects of the FAIRness of research software, more work is still needed to make it easier to embrace the FAIR4RS Principles in their entirety.

A framework for tracking adoption**

The release of the FAIR4RS Principles succeeded in raising the profile of research software in FAIR initiatives, which have mostly been focused on FAIR data. Examples can be found across a range of initiatives, using a slightly adapted version of the five elements of Brian Nosek’s strategy for culture change:

Policy — make it required
Incentives — make it rewarding
Communities — make it normative
Training — make it easy (this replaces user interface/experience in Nosek’s version)
Infrastructure — make it possible

To achieve culture change, initiatives are needed across all of the five elements. Whilst in the early stages, some elements can progress more quickly than others. For example, it’s difficult to implement policy requirements if infrastructure and training (and preferably incentives) aren’t already in place. Similarly, for tools evaluating FAIRness to be useful, changes in major infrastructures such as code hosting repositories are first required so that researchers can create the metadata files that evaluations may depend upon.

It should be noted that while the many activities listed here support increasing FAIRness of research software, most of them do not address aspects of all four of the FAIRness of research software foundational principles:

Findable (F): Software, and its associated metadata, is easy for both humans and machines to find.

**Accessible (A): **Software, and its metadata, is retrievable via standardised protocols.

Interoperable (I): Software interoperates with other software by exchanging data and/or metadata, and/or through interaction via application programming interfaces (APIs), described through standards.

**Reusable (R): **Software is both usable (can be executed) and reusable (can be understood, modified, built upon, or incorporated into other software).

This reflects that the FAIR4RS Principles are aspirational and high-level, and do not contain detailed guidance on how to achieve them. This is because specific technologies and tools are always changing, while the principles are intended to be long-lasting. Consequently, additional work is needed to make it simpler for people wanting to follow the FAIR4RS Principles to know how to practically do so. The following initiatives are assisting in achieving this, with some of these initiatives specifically addressing the range of opportunities for future work identified in 2022 by the FAIR4RS Working Group, which developed the FAIR4RS Principles. These opportunities include “metadata and identifier authority, metadata vocabularies and metadata properties, software identifiers, domain-relevant community standards for software and identification targets”. Whilst some of the initiatives identified in the infrastructure section below are contributing to this, more work still needs to be done.

Photo by Alexander Abero on Unsplash

1. Policies that encourage implementation:

The Netherlands eScience Center and the Dutch Research Council (NWO) formed a working group that developed national guidelines for software management plans. Research institutions such as the University of Groningen’s Digital Competence Center link to this resource in its guidance for its researchers on research software management plans.
The Software Management Plan Template of the Netherlands eScience Center has been updated to align and address the FAIR4RS Principles, and emphasises that open science, software quality, and software sustainability are key elements in eScience Center projects. A total of 13 eScience Center projects (11 under the Open eScience call 2023 (OEC2023) and 2 under the Software Sustainability call 2023 (SS2023)) are now using this template.
Maastricht University in the Netherlands has an Open Science @UM policy, which includes FAIR software as one of its seven areas. Maastricht University proposes to inventory current practices in storage, sharing and reuse of software; and then identify gaps in awareness, knowledge and/or support on FAIR software. Actions likely to be taken include promotion of the creation of software management plans to ensure responsible use of research software.
Enabling FAIR Workflows — Key actors and actions by the Open Research Funders Group outlines steps for different actors in the research community to take to embed sharing practices, persistent identifiers, and metadata throughout the research lifecycle. This report references the FAIR4RS Principles in its recommendations on depositing research software.
The German Research Council (DFG) published guidelines for reviewing grant proposals for Collaborative Research Centers (CRC/SFB) and suggested compliance with the FAIR4RS Principles for archiving and reuse.
The Digital Research Alliance of Canada (the Alliance) published the National Research Software Strategy 2023. This proposes a set of strategic goals for advancing research software capability, community, and coordination in Canada for 2025–2030, with the FAIR4RS Principles providing a cross-cutting theme. The Alliance is developing a Software Management Plan (SMP) template to promote, cultivate and implement best practices, with an emphasis on the FAIR4RS Principles, for Canadian researchers. The goal is to implement the SMP template in future grant applications.Resources that support inclusion of FAIR into institutional policies are also relevant. The list of research institutional policies that support research software, curated by the Research Software Alliance (ReSA), is part of ongoing work by the joint ReSA and Research Data Alliance (RDA) Policies in Research Organisations for Research Software (PRO4RS) Working Group to create a community of stakeholders involved in promoting and/or implementing policy that supports research software at the research institution level (such as universities, national laboratories).

This includes curation of resources on how to influence policy change, such as Health Research Performing Organisations (HRPOs) FAIR Guidelines by Celia Alvarez Romero et al. This provides principles, steps, and resources to support the complex change needed to implement a data policy, which could also be applied to research software policy change.

FAIR-IMPACT launched open calls for cascading grants that provide financial support ranging from 4,000–10,000 euros. Focus areas on research software include:
Path 1: Assessment and improvement of existing research software using a new extension of F-UJI (a web service to assess FAIRness of research objects).
Path 2: Implementation of the Research Software MetaData Guidelines for better archiving, referencing, describing, and citing research software artefacts.
The German Research Council (DFG) issued a Call for Proposals to Increase the Usability of Existing Research Software that refers to the FAIR4RS Principles in terms of availability and reproducibility.
The German Ministry of Education and Research (BMBF) issued funding guidelines (in German) for developing data custodian models that refer to standards based on the FAIR4RS Principles for reusable and well-documented open source software developed under this program.

3. Communities that are normalising adoption:

The RDA Software Source Code Interest Group provides a forum to discuss issues on management, sharing, discovery, archival and provenance of software source code. In October 2023 the group session at the RDA Plenary included a focus on the FAIR-IMPACT Metrics for Assessing Research Software FAIRness (recording). In addition, this group is the maintenance home for the FAIR4RS Principles.
The Research Software Funders Forum included a working group focused on implementation of the FAIR4RS Principles, which review the FAIR4RS Principles per the interests of research software funders, to identify gaps.
Ten simple rules for starting FAIR discussions in your community presents guidance and recommendations on how to start up discussions around the implementation of the FAIR Principles and creation of standardised ways of working. Whilst not specific to FAIR4RS Principles, these recommendations can assist in providing understanding of the benefits and barriers of standardisation are, and will support a more effective way of engaging the community.

4. Training to develop relevant skills:

The Netherlands X-omics Initiative has created Ten quick tips for building FAIR workflows, which provides practical recommendations for implementation of the FAIR4RS Principles when creating computational workflows.
Making Biomedical Research Software FAIR: Actionable Step-by-step Guidelines with a User-support Tool assists biomedical researchers to make their research software compliant with the FAIR4RS Principles. The FAIR Biomedical Research Software (FAIR-BioRS) guidelines are based on the FAIR4RS Principles and a workflow that streamlines the process of implementing these guidelines is also provided.
The Netherlands eScience Center is developing and piloting the ⁠Research Software Support course which covers the FAIR4RS Principles and provides guidance for research personnel.
The Digital Competence Centre of the Delft University of Technology (TU Delft) in the Netherlands is offering a 13-week FAIR for Research Software Program, which includes resources from the Code Refinery, a project within the Nordic e-Infrastructure Collaboration (NeIC). TU Delft’s program aims to teach the essential tools for creating scientific software following the FAIR4RS Principles and how to apply them.
The Data Competence Center at the Leiden University Medical Center (LUMC) aims to help researchers implement the FAIR4RS Principles by increasing software stewardship capacity.
The European Infrastructure for Translational Medicine (EATRIS) promotes adoption of the FAIR4RS Principles by sharing practical implementation examples from and with the community via its Multi-omics Toolbox (MOTBX).
The Alliance’s National Research Software Strategy 2023 recommends FAIR4RS as a key element underlying training & support initiatives for research software, which will be included in the forthcoming national training framework and resourced appropriately.

Photo by Niklas Ohlrogge on Unsplash

5. Infrastructure that provide supporting tools:

FAIR-IMPACT’s release of Metrics for automated FAIR software assessment in a disciplinary context defines 17 metrics that can be used to automate the assessment of research software against the FAIR4RS Principles, and provides examples of how these might be implemented in one exemplar disciplinary context of the social sciences. The FAIR-IMPACT project will work to implement the metrics as practical tests by extending existing assessment tools such as F-UJI.
Horizon Europe’s European Virtual Institute for Research Software Excellence (EVERSE) aims to create a framework for research software and code excellence, that are collaboratively designed and championed by the research communities. EVERSE will also continue the work of FAIR-IMPACT by developing and implementing processes and tools that support the assessment and verification of code quality, based on established best practices and standards across scientific communities.
FAIRsoft is a practical implementation of the FAIR4RS Principles, and the FAIRsoft evaluator is a tool for developers and users to assess how specific software complies with FAIR for software indicators. It is part of the ELIXIR’s OpenEbench Software Observatory, an instrument for the systematic observation and diagnosis of the quality of research software in the life sciences.
FAIR-Impact’s Guidelines for recommended metadata standard for research software within EOSC acknowledges the rising need for establishing software metadata guidelines to effectively collect and curate metadata. A comprehensive set of Research Software MetaData (RSMD) Guidelines are provided that offer flexible and adaptable recommendations for end-users that can be used in different disciplines and different software development contexts. The guidelines are directly relevant to end users, including software creators and curators in their quest to improve the FAIRness of their software.
The ELIXIR Software Best Practices group, NFDI4DataScience and Bioschemas are collaborating together to support machine-actionable SMPs which are aligned to the ELIXIR Software Management Plan for Life Sciences published by ELIXIR; and also aligned to other initiatives but also to others, such as the practical guide to SMPs by the Dutch Research Council (NWO) and the Netherlands eScience Center, and the SMP template by the Max Planck Digital Library.
A new version of the machine-actionable Software Management Plan Ontology (maSMP Ontology) metadata schema, vr2.1.0, was released in January 2024, together with usage guidance about the properties (profiles, guides on minimum, recommended and optional properties with cardinalities). The metadata schema includes entities involved in software management planning; such as an SMP itself, software source code, software release, documentation, authors and their relations. Integration into Bioschemas is still pending.
A metadata enrichment cycle aligned to the maSMP metadata schema has been proposed by ELIXIR thanks to the Software Management Wizard, a tool to make completion of SMPs easier. A similar effort is within the scope of NFDI4DataScience and the Research Data Management Organiser (RDMO) SMP. This approach reuses a command-based tool to extract metadata from GitHub repositories, SOMEF, which is currently being extended to cover the maSMP metadata schema case.
The Research Software APIs and Connectors project within the FAIRCORE4EOSC project is working on developing tools and services for archival, reference, description, and citation of research software artefacts. This implements the key recommendations of the Scholarly Infrastructures of Research Software report to interconnect scholarly repositories, publishers, and aggregators. Interconnections are possible with the Software Heritage universal source code archive, using the CodeMeta standard, and the Software Heritage intrinsic identifiers (SWHID). Instructions on how to archive your software to Software Heritage is one outcome.
FAIRSECO: An Extensible Framework for Impact Measurement of Research Software aims to enable research software engineers to rapidly find and extract relevant software fragments from the worldwide research software ecosystem.
Towards a Quality Indicator for Research Data publications and Research Software publications — A vision from the Helmholtz Association develops indicators to be used within the Association. It presents a quality assessment spanning six dimensions of research software quality that augments the four FAIR principles with two additional indicators: Scientific basis and Technical basis, resulting in the FAIR-ST framework.
A self-assessment tool to promote FAIR research software has been developed by the Netherlands eScience Center and Australian Research Data Commons, to encourage the uptake of the FAIR4RS Principles (and see the 2022 Survey on Adoption Guidelines for the FAIR4RS Principles for more resources).
openCARP-CI provides Python scripts that allow developers to automatically derive CFF and DataCite files from a CodeMeta file. These pipelines can easily be integrated in continuous integration and deployment environments. They also provide tools for software publication via tagged releases, creation of BagIt and BagPack files, and publication on the research data repository RADAR.
CodeMeta-3.0: The minimal metadata schema for science software and code, in JSON-LD, provides a possibility to developers and researchers to insert metadata in their code and increase FAIRness.
FAIR-USE4OS: From open source to Open Source by Raphael Sonabend et al., extends the FAIR4RS Principles to provide criteria for assessing if software is Open Source. By adding ‘USE’ (User-Centred, Sustainable, Equitable), software development can adhere to open source best practice by incorporating user-input early on, ensuring front-end designs are accessible to all possible stakeholders, and planning long-term sustainability alongside software design.
FAIR4RS has also been discussed within the scope of Open Science and software quality, for instance the EOSC Task Force Infrastructures for Quality Research Software compiled software quality metrics and identified those that can be aligned to the FAIR4RS Principles.

Other impacts

The publication of the FAIR4RS principles and introductory articles created awareness and raised interest in the research community, reflected in over 200 citations from across the disciplinary spectrum. Examples include the results of searches on Google Scholar for “FAIR principles for research software (FAIR4RS principles)” or “ Introducing the FAIR Principles for research software”. In addition to work discussing trans-disciplinary application of the FAIR4RS Principles, there are research policy and software management publications referring to and implementing the FAIR4RS Principles.

The FAIR4RS Principles have also provided value to the broader research ecosystem by providing a base for other communities to adapt the FAIR Principles to different research objects. Examples include FAIR AI Models in High Energy Physics, which provides a practical definition of FAIR principles for machine learning and artificial intelligence models in experimental high energy physics, including a FAIR AI project template; and the Open Modeling Foundation’s work to identify, develop, and promote common standards and best practices for FAIR modelling, by working with model organisations and individuals active within the social, ecological, environmental, and geophysical sciences.

This blog was cross-posted across several platforms and originally published at *https://www.researchsoft.org/blog/2024-03/

How to manage your IP address in Python

Flavio Hafner — Tue, 27 Feb 2024 00:00:00 GMT

This post is the first of a series of blog posts arising from a collaboration between the eScience Center and the ODISSEI Social Data Science Team. You can find this text also at https://odissei-soda.nl/tutorials/.

Research often relies on accessing novel data, for instance by collecting them from the internet through web scraping. If you have ever tried this, you may have run into your IP address being blocked by the website you scrape. Websites do this with a good reason — to protect themselves against malicious acts, such as denial-of-service attacks or large-scale scraping by parties with ill intent. This makes sense from the websites’ perspective, but prevents you from answering your research question.

But this problem can be solved. In this tutorial, we show how you as a researcher can use IP rotation to circumvent certain scraping protections with the sirup package, which works on Linux operating systems.

Before we jump into it, it is important to highlight that web scraping and IP rotation need to respect the law and should only be a last resort. For instance, before you scrape data from a website, you should ask the data owner whether they are willing to make them available to you through a data sharing agreement. If you nevertheless decide to scrape the data, you should get approval from the ethical review board at your university. Moreover, do only scrape data that are publicly available on the web and do not send excessive number of requests to the website in a given time.

Photo by Glenn Carstens-Peters on Unsplash

For rotating the IP address, we will use a VPN service. Here you can read more about what a VPN service is—in short, the service has a bunch of remote servers, and connecting your computer to one of these servers changes your IP address.

What you need

1. `OpenVPN`

OpenVPN is a system that allows you to create secure VPN connections. You can install it by following these instructions.

2. Root access to your computer

Because internet connections are an important security concern, OpenVPN requires root access — this is the equivalent to administrator rights on a Windows computer. If you have root access, you can for instance run the following command on your terminal:

sudo ls -lh # will ask you for your root password

Installing and setting up `sirup`

You can install sirup as follows:

python -m pip install sirupTo use the package and change your IP address, you need an account with a VPN service provider that offers OpenVPN configuration files for your account. At the time of writing, for instance ProtonVPN and Surfshark offer this option — note that these services are not for free. We will use ProtonVPN in this tutorial.

After creating an account, you need to download two sets of files.

First, you download credentials that identify your Proton account when using OpenVPN. On the ProtonVPN website, click on “Account” and then you see something like this:

Copy and paste the username and the password into a txt file that looks like this:

username passwordThen, save the file as “proton_credentials.txt”. Remember where it is stored — we will need it later.

A first warning on security. **Storing account credentials like this makes it easy for you to use the sirup package. But it also increases the risk that unauthorized persons get a hold on these credentials. Thus, be careful to store the credentials in a safe place on your laptop and to not share them with anyone.

Second, to use OpenVPN we need configuration files, whose names end with .ovpn. The files allow OpenVPN to connect to a server from the VPN service provider. In ProtonVPN, go to the “Download” section of your account. Select the options as follows:

And download the configuration file(s) you want to use. Store the downloaded files on your computer, and remember the location.

Using `sirup`

We start by defining the path to the proton_credentials.txt file. When you execute the code below, you will be asked to enter the root password, which is necessary to make the connection.

import getpass auth_file = “proton_credentials.txt” pwd = getpass.getpass(“Please enter your root password:”)

A second warning on security.** The code above stores your root password during the Python session without encrypting it. This is OK to do on your laptop — if someone gets access to your Python session, your security has already been compromised — but not recommended on a shared computer such as a cluster or a cloud service.

Changing the IP address with `sirup`

Now you can use the VPNConnector to change our IP address. We will use the "my_config_file.ovpn" configuration file.

from sirup.VPNConnector import VPNConnector config_file = “my_config_file.ovpn”The code below first connects to the server associated with "my_config_file.ovpn" and then disconnects.

connector = VPNConnector(auth_file, config_file)

Let’s see the current IP address when no VPN tunnel is active

print(connector.base_ip)

connector.connect(pwd=pwd)

Now the IP address should differ

print(connector.current_ip)

connector.disconnect(pwd=pwd)

Now current_ip should be the same as base_ip above

print(connector.current_ip)

Rotating the IP address with `sirup`

Instead of connecting to a single server, you can also rotate across many different servers — which means you rotate your IP address across a set of potential addresses. Doing so is useful for larger scraping jobs because it will spread your requests across more servers.

To do this, you need to download multiple configuration files as described above. Store all of the .ovpn configuration files together in a separate directory. Let’s say you store them in the "/path/to/config/files/" directory. You need to define this path in your Python script:

config_path = “/path/to/config/files/“The following code connects to two different servers before disconnecting again:

from sirup.IPRotator import IPRotator

rotator = IPRotator(auth_file=my_auth_file, config_location=config_path, seed=seed) # this will ask for the root password

print(rotator.connector.base_ip)

rotator.connect() print(rotator.connector.current_ip)

rotator.rotate() print(rotator.connector.current_ip)

rotator.disconnect()

print(rotator.connector.current_ip)

Conclusion

This tutorial has walked you through the steps to manage your IP address in Python, using the sirup package. We hope it makes your scraping workflows easier!

sirup is an open-source package developed by the Netherlands eScience Center. If you use the tool, you can cite this zenodo repository with the DOI: https://doi.org/10.5281/zenodo.10261949.

The source code of the package is here, where you can contribute to it, build on it and submit issues.

Thanks to Patrick Bos, Peter Kalverla, Kody Moodley and Carlos Gonzalez Poses for comments.

Differentiation without subtraction

Pablo Rodríguez-Sánchez — Tue, 13 Feb 2024 00:00:00 GMT

A surprising application of complex numbers

Photo by charlesdeluvio on Unsplash

Some time ago I came across one of those ideas that makes you wonder: “why nobody told me about this before?”

Those of you with mathematical training certainly remember the Taylor series. For those who do not, and yet want to keep reading, this visual simulation may be helpful.

The idea is that any continuous, smooth function, can be approximated by a polynomial; the higher the degree, the more accurate the approximation. To put it otherwise: if we know the value of a function and its derivatives at a given point (x), we can estimate its value at a nearby point (x + ε). More specifically:

Example of a generic Taylor seriesIf instead of a step (ε) in the realm of real numbers, we perform a step in the world of imaginary numbers (i ε), it follows:

It is here where things get interesting. If we take only the imaginary part, we get an equation that involves only the first derivative, the function evaluated in the complex plane, and the step size:

We just keep the imaginary part, and drop the real oneThis can be rearranged as an interesting formula for an approximate derivative:

Complex-step numerical derivativeThat can be even used for exact evaluation:

Complex-step exact derivative

Is this useful at all?

I know…, that was pretty strange. It feels like a complicated tour de force for something as well-known as a numerical derivative. Nevertheless, it is hard to find a mathematical result that is completely useless, and complex-step differentiation is no exception.

Let’s take a look at the last formula of the day, that of the classical step-forward numerical differentiation:

Step-forward numerical derivativeIf we use the step-forward numerical derivative on a computer, the subtraction can cause numerical problems. Computers can’t store real numbers, but a rounded representation of them, and their numerical resolution is limited. If e is very small, it may happen that the subtraction above returns a zero as an artifact, making the whole calculation wrong.

Notice now the main difference between this and our complex-step algorithm: there is no subtraction in the complex one. One problem less!

Further information

More information here: Differentiation without a difference. SIAM news.

If you enjoyed this article, you may also like Automatic differentiation from scratch.

—

This article appeared first, in Spanish, in Naukas.com.*

Causal Inference in JASP: The Process Module

Malte Lüken — Thu, 01 Feb 2024 00:00:00 GMT

By Malte Lüken, Julia M. Rohrer, Thijs Vroegh, Johnny van Doorn, and Eric-Jan Wagenmakers

This post is also published on the JASP blog.

In a recent blog post, we announced the development of a JASP module for conditional process models, similar to the popular PROCESS macro for SPSS (Hayes, 2022a, 2022b). We can now share that we reached a milestone in our development efforts: The latest JASP release 18.2 includes a beta version of the Process module which allows users to apply frequentist process models. Besides covering many of the features of the SPSS macro, the JASP Process module provides a flexible and easy-to-use interface. It also makes the causal assumptions of process models more explicit and provides functionality to test them. In this blog post, we will introduce the main features of the module using a real-world example.

Mediation, Moderation, and Conditional Process Models

The aim of our module is to test models that involve mediation and moderation effects. A mediation effect is a causal chain between three variables. For example, when X → M, M → Y, and X → Y, then the causal effect from X to Y is (partly) mediated by M. A moderation effect implies that the relationship between two variables changes depending on a third variable, e.g., the effect X → M changes as the moderator W changes. A model that includes at least one mediation and one moderation effect is called a conditional process model because it makes assumptions about the causal process between variables, but the process is conditional on moderator variables (Hayes, 2022a).

Conditional process models can also be conceptualized as parametric directed acyclic graphs (DAGs) or structural equation models (SEMs) that only include manifest variables (Pearl, 2012). We can use the properties of DAGs to test the implications of the causal model against data while SEMs are a useful framework for parameter estimation.

The PROCESS macro for SPSS uses a regression-based approach to estimate process models which allows predictors (including independent variables, covariates, and mediators) to be correlated when there is no explicit causal path between them. Because the JASP Process module uses SEMs for model estimation, these correlations need to be explicitly allowed by the user. We believe this design choice prompts users to think more explicitly about the assumptions of their models. For a discussion of differences between PROCESS and SEM-based estimation, see Hayes et al. (2017).

To illustrate the features of the Process module, we use openly available data from the study “Predictors of listening-related fatigue across the adult life span” by McGarrigle et al. (2021) which was published in Psychological Science. The study investigates how different variables mediate and moderate the causal effect of age on listening-related fatigue using conditional process models. The analysis was pre-registered and the data set has a relatively large sample size (N = 281) which is roughly evenly distributed across age groups. These properties make the study a good example for conditional process analysis with observational data. As the authors of the original study, we assume that age is an exogenous (independent) variable which allows us to clearly identify the direction of the causal effect from age to fatigue.

In our illustration of the module, we will reproduce the original analysis of the study but also test the implications of the hypothesized causal model. Note that our example only uses z-standardized variables which we renamed so they can be easily identified. The included candidates for mediators or moderators are auditory attention ability (“AudAttention”), perceived hearing impairment (“HearImpair”), perceived memory ability (“Memory”), and perceived mood disturbances (“MoodDisturb”). The example data set is also available in the internal JASP Data Library.

The Basics: Flexible Conditional Process Models in JASP

To estimate a conditional process model in JASP, we first load the data set and enable the Process module by clicking the blue “+” sign in the top right corner and selecting “Process (beta)”.

We then click on the available module to start the “Classic Process Model” analysis. First, we drag the variable “Fatigue” into the box Dependent Variable and the variables “Age”, “AudAttention”, “HearImpair”, “Memory”, as well as “MoodDisturb” into the box Continuous Predictors.

In this example, we use a new interface that differs from the way process models are specified in SPSS. In the Models section, we can construct the hypothesized model by adding causal paths between variables. We add three paths from “Age” to “Fatigue” by selecting the variables in the From and To dropdowns. For each path, we choose Mediator in the Process Type dropdown and select “AudAttention”, “HearImpair”, “Memory” as the Process Variable, respectively.

After completing each path, JASP provides the output of the estimated model including a path diagram showing the conceptual model structure. The path diagram resembles the hypothesized parallel mediation model after the third path.

Before we look at the results, we select Bootstrap under Method in the Options section to enable bootstrapped confidence intervals; this is done because we cannot assume that indirect effects are normally distributed. Bootstrapping in JASP can take a few minutes to finish.

Below the path diagram, under “Parameter estimates”, the output shows a table with estimated parameters and effects. The second table “Mediation effects” contains the direct and indirect effects. As in the original study, the table shows a negative direct effect from age to fatigue and a positive indirect effect via perceived hearing impairment, both with bootstrapped confidence intervals excluding zero.

To add a moderation effect to our model, we go back to the “Models” section and click the green “+” button in the top left corner to add a second model to our analysis. As in the previous model, we add three mediation paths from “Age” to “Fatigue”. Then we add a fourth path from “Age” to “Fatigue” and specify “MoodDisturb” as a moderator.

Now, JASP provides output for both models. In the “Model summary” table, we can see that Model 2 has lower AIC and BIC values indicating a better model fit compared to Model 1 (the weights for both criteria lie almost exclusively on Model 2; AIC and BIC weights can be enabled in the “Options” section).

Under “Parameter estimates” for Model 2, we can see that the direct effect of mood disturbances on fatigue is positive and its bootstrap confidence interval excludes zero. However, this is not the case for the interaction between age and mood disturbances. Thus, the model does not provide evidence for the hypothesized moderation effect, reproducing the result of the original study.

How can we explain that Model 2 fits the data better but shows no evidence for a moderation effect? Model 2 does show a strong direct effect of mood disturbance on fatigue, which is not included in Model 1, and we assume that it leads to the better model fit. This suggests that we should include mood disturbances in our model but not as a moderator on the path between age and fatigue.

Advanced: Testing Implications of Causal Models

Because the estimated effect from age to fatigue pointed in the opposite of the expected direction, McGarrigle et al. (2021) estimated alternative process models in an exploratory analysis. The Process module provides a useful tool that indicates where the assumed causal model does not fit the observed relationships in the data. DAGs have implications for the variables they involve, specifically, for which variables should be independent in the data (Pearl, 2012; Textor et al., 2016). These implications can be tested against data with (conditional) independence tests. In our model, all variables are assumed to be continuous and all relationships to be linear, so we test for (conditional) independence by estimating (partial) correlation coefficients.

In JASP, we can test local implications by ticking the box Local tests under *Tests in the Models section. *Here, we apply local tests to Model 2. Underneath the box, we can select the type of test we want to use, which is the default (Linear) in our case. If the model contains categorical variables or relationships are assumed to be nonlinear, a different type should be chosen (e.g., Loess or Chi-square).

After ticking the Local tests box, a new table appears in the output under “Local tests”. In the first three columns, the table shows which implied relationship is tested: For example, in the first row, auditory attention is assumed to be conditionally independent from hearing impairment given age. This can also be seen in the model diagram, which does not contain any direct arrows between the two variables (i.e., age accounts for the entire covariance between auditory attention and hearing impairment). The estimated partial correlation for this implied relationship is very small and not significant indicating that this implication is not violated. The opposite is true for row four: The model implies that age and mood disturbances are independent since there are no direct (or indirect) arrows between the two variables. However, the estimated correlation is negative and significant suggesting that the implication is violated and the model is misspecified. Rows five to seven also show significant (partial) correlations increasing the evidence that the model is inappropriate for the data.

Note that the PROCESS macro for SPSS behaves differently to our module by automatically allowing mediators to correlate. In a SEM context, this behavior can be imitated by estimating the residual covariances between mediators in the model. Ticking the box Mediators under Residual Covariances in the Models section enables this.

Advanced: Adjusting the Model in an Exploratory Analysis

In the original study, McGarrigle et al. conducted an exploratory analysis where they used “MoodDisturb” as a mediator instead of a moderator. Our implication tests of the previous model with “MoodDisturb” provide a rationale for this: The data showed a negative correlation between “Age” and “MoodDisturb” which was not expected by the model. Our exploratory model should therefore account for this correlation by including a direct causal path from “Age” to “MoodDisturb”.

We open a new Classical Process Model* analysis and select the same variables for Dependent Variable and Continuous Predictors as in the previous analysis. In our exploratory model, we specify “MoodDisturb” as a mediator between “Age” and “Fatigue”. We add “HearImpair” as a second mediator and leave out the other variables for simplicity.

In the output, we can see the conceptual diagram of our exploratory model.

Again, we test the local implications of our new model. We can see that the model expects mood disturbances and hearing impairment to be conditionally independent given age, which is violated in the data.

Thus, we need to further adjust our model, for example, by adding a direct causal path from “HearImpair” to “MoodDisturb” assuming that increased perceived hearing impairment leads to increased mood disturbance and not vice versa. In JASP, we can add a direct path by selecting Direct in the Process Type dropdown under the Models section.

The output now shows the conceptual diagram of the adjusted exploratory model.

If we compare the AIC to the first exploratory model without the direct path, we can see that the AIC of the second model with the direct path is lower, indicating a better model fit (the AIC weight lies almost exclusively on the second model).

Looking at the direct and indirect effects in the “Mediation effects” table, we can see that the adjusted exploratory model estimates one direct (first row) and three indirect effects (rows two to four).

The bootstrap confidence interval for the direct effect overlaps with zero but the intervals for the three indirect effects do not. That is, the adjusted exploratory model suggests a threefold causal process from age to fatigue: Via mood disturbances, hearing impairment, and via a path from hearing impairment to mood disturbances.

Using Conditional Process Models with Caution

As has been emphasized by Rohrer et al. (2021), process models should be used with caution. They assume that the model structure is appropriate for the problem and data to identify causal effects. While the JASP Process module allows users to test the implications of the causal structure, other assumptions are much more difficult or even impossible to test: For example, to identify causal effects, all potential confounding variables must be accounted for in the model. This is difficult for indirect effects, even in experiments, because unobserved confounders can bias causal effects from the mediator to the dependent variable (Bullock et al., 2010; Rohrer et al., 2021). The Process module allows users to include confounders on all paths; however, they must be known and measured.

Another common challenge is to correctly establish the direction of causality and ruling out alternative causal models (Kline, 2012; Rohrer et al., 2021; Fiedler et al., 2018). While in our example, it was easy to determine the causal direction from age to fatigue, the direction of causal paths between two mediators or the mediators and the dependent variable was much less evident. Some alternative causal models (stating different directions of causality) can be ruled out through causal implications. However, more often, the most appropriate causal model needs to be justified from theory or established through experimental manipulation.

Furthermore, moderation effects as part of conditional process models pose several challenges to researchers, such as scale-dependency, choosing hypothesis-appropriate effect sizes, and distinguishing between causal interaction or effect modification (see Rohrer & Arslan, 2021, for details, examples, and recommendations).

If there is evidence that a model is misspecified, the adjustment must also be done with care to avoid overfitting (Textor et al., 2016). While the modification can be done in an exploratory fashion, the modified model should always be tested on an independent dataset to draw robust conclusions. At the same time, it should be consistent with theoretical expectations.

Conclusion

The new JASP Process module allows users to apply frequentist conditional process models using an intuitive and flexible interface. Using data from a study by McGarrigle et al. (2021), we demonstrated how users can estimate models involving multiple mediation and moderation effects. We also explained how the module enables users to test the local implications of specified models. In the case of violated implications, users can modify misspecified models accordingly. We believe that the JASP Process module can pave the way toward a more responsible and appropriate use of conditional process models. By providing an intuitive and flexible interface, we also aim to make process models more accessible to researchers and students.

Behind the Project

This project is a collaboration between the JASP Team and the Netherlands eScience Center. As the national institute for research software, the eScience Center awards projects as part of calls for proposals. Instead of monetary funding, it provides in-kind support by Research Software Engineers to create innovative digital solutions. This project was granted as part of the Small-Scale Initiatives Digital Approaches to the Social Sciences call 2022.

References

Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism? (don’t expect an easy answer). Journal of Personality and Social Psychology, 98(4), 550–558. https://doi.org/10.1037/a0018933

Fiedler, K., Harris, C., & Schott, M. (2018). Unwarranted inferences from statistical mediation tests — An analysis of articles published in 2015. Journal of Experimental Social Psychology, 75, 95–102. https://doi.org/10.1016/j.jesp.2017.11.008

Hayes, A. F., Montoya, A. K., & Rockwood, N. J. (2017). The analysis of mechanisms and their contingencies: PROCESS versus structural equation modeling. Australasian Marketing Journal, 25(1), 76–81. https://doi.org/10.1016/j.ausmj.2017.02.001

Hayes, A. F. (2022a). *Introduction to mediation, moderation, and conditional process analysis *(3rd Ed.). New York: The Guilford Press.

Hayes, A. F. (2022b). The PROCESS macro for SPSS, SAS, and R (Version 4.2) [Computer software]. https://processmacro.org

Kline, R. B. (2012). Assumptions in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 111–125). Guilford Press.

McGarrigle, R., Knight, S., Hornsby, B. W. Y., & Mattys, S. (2021). Predictors of listening-related fatigue across the adult life span. Psychological Science, 32(12), 1937–1951. https://doi.org/10.1177/09567976211016410

Pearl, J. (2012). The causal foundations of structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 68–91). Guilford Press.

Rohrer, J. M., & Arslan, R. C. (2021). Precise answers to vague questions: Issues with interactions. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211007368. https://doi.org/10.1177/25152459211007368

Rohrer, J. M., Hünermund, P., Arslan, R. C., Elson, M. (2022). That’s a lot to process! Pitfalls of popular path models. Advances in Methods and Practices in Psychological Science, 5(2). https://doi.org/10.1177/25152459221095827

Textor, J., van der Zander, B., Gilthorpe, M. S., Liśkiewicz, M., & Ellison, G. T. (2016). Robust causal inference using directed acyclic graphs: The R package ‘dagitty.’* International Journal of Epidemiology, 45*(6), 1887–1894. https://doi.org/10.1093/ije/dyw341

Talking about energy-efficient GPU computing at SC23

Alessio Sclocco — Tue, 16 Jan 2024 00:00:00 GMT

November is the time for Supercomputing, or SC as it is called nowadays, and in 2023 the largest High-Performance Computing (HPC) conference in the world took place in Denver, USA, from the 12th to the 17th of that month.

It was a week packed with talks, workshops, tutorials, exhibitors from all over the world, social gatherings, announcements and all of the usual SC glamour. While the eyes of the world were all pointed at the TOP500 to see if Aurora would become the fastest supercomputer on the planet (spoiler: it did not; Aurora sits at number 2 behind Frontier), there were many highlights of the conference.

Ben, Floris-Jan, Stijn, and Alessio standing with the SC23 logo at the Colorado Convention Center in Denver. Photo by Floris-Jan Willemsen.On the technical front, it is clear that Graphics Processing Units (GPUs) are here to stay, as they provide more than 70% of the total TOP500 FLOPs (Floating Point Operations per second); in other words, most of the computational power of the largest supercomputers in the world comes from GPUs.

With new systems being installed, GPUs from at least three different vendors (i.e. AMD, Intel, NVIDIA) being used, and a dozen or so programming models available for programming them, one of the hottest topics of SC23 was performance portability.

Performance portability means that software written originally for one platform can not only run on a different platform but even provide comparable performance. This is a very important property for scientific software, which typically has a lifespan longer than that of the life of any modern supercomputer.

Another hot topic was energy efficiency. Being able to perform more work for the same energy budget, by increasing energy efficiency, is a societal issue that is becoming increasingly pressing in the HPC community. An entire workshop was dedicated to “Sustainable Supercomputing”, with discussions ranging from how to make code more efficient, to where to build a data center to utilize renewable energy. The workshop also highlighted how simply making users aware of the energy consumption of scientific code can have an impact, especially at the political level.

A small delegation from the Netherlands eScience Center comprised of Ben van Werkhoven, Alessio Sclocco, Stijn Heldens and Floris-Jan Willemsen attended SC23 to present a tutorial on “Energy-Efficient GPU Computing” on the first Sunday of the conference; the same engineers taught a tutorial about Kernel Tuner at SC21 in St. Louis, USA. The new tutorial was also mentioned on a slide in the opening of the “Ethics in HPC” session, as further proof that energy and sustainability are pressing concerns of the SC community in 2023.

While energy efficiency in HPC is a broad topic, our focus was on what programmers can do to reduce the energy consumption, and thereby increase the efficiency, of their GPU code. The focus on GPUs does not only come from our expertise, but also reflects the fact that GPUs are what provide most of the computational power in current supercomputers.

Ben teaching part of the tutorial. Photo by Floris-Jan Willemsen.The tutorial was structured into four parts, each consisting of a brief lecture followed by a hands-on session. We strongly believe that participants learn more effectively when they can immediately apply what they have learned.

The main takeaway of this first part could be summarized in the three action points for energy-efficient GPU code: (1) use energy-aware optimizations, (2) reduce data movement and (3) optimize clock frequencies.

The second part of the tutorial was all about improving the performance of GPU code, since code that runs faster often ends up consuming less energy for the same amount of work, thereby being more efficient.

After a lecture on GPU code optimizations, loosely based on a paper we published last spring, participants applied “kernel fusion” during the hands-on. This is an optimization that merges two GPU functions into one, in which it becomes apparent that optimizations can not only improve performance, but can also lower energy consumption.

The third part focused on reducing data movement, crucial for energy efficiency, through mixed precision programming. This technique uses lower precision data types to decrease the number of bytes required for data storage, at the cost of introducing numerical error in the results. Fortunately, this error can be kept within an acceptable error margin by utilizing Kernel Tuner’s new capabilities for accuracy tuning, resulting in code that is faster, more energy-efficient, and shows minimal numerical error. Additionally, we showcased Kernel Float, a C++ library developed by the eScience Center for mixed precision programming in NVIDIA CUDA. Participants had the chance to experiment with this in real GPU code during the hands-on session.

Stijn talking about mixed precision in front of the audience. Photo by Floris-Jan Willemsen.In the last part of the tutorial, we looked at how Kernel Tuner can be used to automatically tune power capping values and clock frequencies. While just improving performance is generally enough to improve the energy efficiency of a program, we also showed how finding the optimal GPU core frequency can further improve efficiency.

Moreover, based on the result of a paper published last year, we also showed that by directing Kernel Tuner to use energy efficiency as its main metric in the search for the optimal configuration, it is possible to find the most energy-efficient configuration and even quantify the acceptable loss in performance to achieve it.

The tutorial was well attended, with over 30 participants, and the interaction with the audience was very positive, especially during the hands-on. We received many questions, and some participants have already contacted us online in the Kernel Tuner discussion forum. If you would like to know more about this tutorial, the slides we used are available online (for free), and using Google Colab you can also run the hands-on exercises that we prepared for the event.

Funding acknowledgments: the “Energy-Efficient GPU Computing” tutorial has been made possible by the CORTEX, COMPAS, and ESiWACE3 projects. The CORTEX project has received funding from the Dutch Research Council (NWO) in the framework of the NWA-ORC Call (file number NWA.1160.18.316). The COMPAS project has received funding from the Netherlands eScience Center (NLESC.OEC.2022.001). ESiWACE3 is funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) and Spain, Netherlands, Germany, Sweden, Finland, Italy and France, under grant agreement No 1010930.

How to ensure that others can run your code

eScience Editorial Team — Tue, 12 Dec 2023 00:00:00 GMT

By Julian Gonggrijp

This blog was originally written for the Utrecht University (UU) website by Scientific Programmer Julian Gonggrijp. We enjoyed this useful blog post, and hope you will too! If you’re looking for more resources on reusable code, also see this chapter in the Turing Way.

This is the last eScience Center blog post of 2023, but fear not! We will be back in January with lots more interesting writing from our Center.

Photo by ThisisEngineering RAEng on Unsplash

Imagine the following scenario. You need to analyze your research data. As is often the case in research, your analysis is going to be similar to, but not quite the same as, previous research. You look for existing software that can do the job for you, but nothing suits your situation. You decide this will be a good opportunity to brush up on your Python-fu. You sit down at your laptop, crack your knuckles and start hacking.

The analysis is far from trivial; so is writing the code. You learn several new things along the way, some of which are interesting, gain a few grey hairs, spend two very late “afternoons” with a colleague to squash a notorious bug. Ten weeks and several hundred lines of code later, though, you manage to make it work. You toast with your colleague, present your analysis at a conference and get very positive responses. Your own software is a great success!

At the conference, you meet a colleague from a different institution. She has a totally different dataset, but she might be able to apply the exact same analysis. You are excited: you might be onto a new methodology, worthy of several papers and a grant or two.

A few months later, you hear back from your new acquaintance. She found time to experiment with your new method and she asks for your code. You were already prepared for that. Your code is on GitHub, it has a README and you even thought of an open source license, so she can freely adapt the code if necessary. You eagerly send her the link.

You don’t hear back for a while. After two months, you take a deep breath and send her an email. Was she able to get any useful results out of the analysis? Alas, it turns out she did not even reach that stage. She was unable to run your code, even with help from a student assistant from the computer science department. The code would crash with error messages that they could not get past. She went with the conventional analysis in the end; the results were not spectacular, but she managed to squeeze a publication out of it and moved on.

Of course, you are disappointed. Did you just spend ten weeks of your life developing new software, only to find that it will never run through another dataset? Should you have hired a scientific programmer instead of writing the analysis yourself? Fortunately, at this point, you run into this blog, saving you from a depression.

Why runnable code matters

In the above scenario, it is obvious that other people must be able to run your code because you want your method to be reusable. However, even if you write a quick one-off just for your own research, it is important to ensure that somebody else could run your code. There are two main reasons for this:

As Stefano mentioned in the previous blog on UU’s website, your future self is another person. You might upgrade or replace the PC on which you wrote the code and you might forget some of the preparatory magic incantations. Even in case of a one-off, you might want to run the code again in order to double-check your results, to add new datapoints or even just to impress your spouse. 😉 By ensuring that the code is easy to run while everything is still fresh in memory, you can save time (and dignity) in the future.
By ensuring that your code is easy to run, you make it easy for others to repeat and verify your work. This can be a great way to convince people!

Making sure that code is easy to run is different from ensuring that the code is easy to read. The latter is certainly also worthwhile, for different reasons, but that is for another blog post.

Making the code easy to run is also not exactly the same as making it easy to install. If you are serious about your software, you will likely think about publishing it to a package repository (which is also for another blog post) so that it can be easily found and installed, and you should! However, while this technically makes your code easy to run as well, this does not replace making it easy to run the code when obtaining a raw copy. At the very least, the future developer who is going to update the package will need to run it without installing it from a package repository.

Why is code not runnable out of the box?

Let us be honest: it is infuriating that code never runs out of the box. We can double-click any picture, PDF file or spreadsheet and it will just open and work, but for some reason, when it comes to code, we still seem to be living in the 1960s. Whenever we receive code from a colleague, we usually first have to install additional software, learn a few new skills, massage our data in just the right shape, take two non-obvious preparatory steps and make a rain dance before we can even hope to be able to run it. Then, there are the unexpected errors.

While this is no excuse by itself, creating infrastructure that makes code easy to run is notoriously difficult. Code is run by other code, and that other code has to be extremely flexible in order to cater to all the possible applications. This by itself already requires highly specialized skills. On top of that, the software landscape is constantly moving because people keep finding new use cases for code, so everyone is constantly struggling to keep their own code compatible with somebody else’s code. Between all that effort, attention tends to be drawn away from keeping the software usable. There have been many honourable attempts at improving this situation, some of them quite successful — I will mention a few below — but even those projects cannot escape the constant struggle to keep their infrastructure current. Making running code as easy as a double-click remains elusive for now.

Photo by Sigmund on Unsplash

Six things that should be documented for any code

The first thing we can do, is to transfer as much knowledge as possible. The following checklist may help you to ensure that you include all the important details in your README. This is a good example of a correct README.

Why does your code exist? This may seem obvious, but you will be surprised at how often this information is left out. Knowing what your code is about will motivate prospective users to try it for the right reasons. It will also help clarify why things are done a certain way, why some features are included and others omitted.
What other software needs to be installed before your code can be run? Most unexpected errors, where the software worked for the programmer but not for the user, are due to dependencies that are either missing or have a mismatching version. To prevent this, list the version of your programming language as well as all the packages that your code depends on and be as specific as possible about which versions are expected to work. Also explain how to obtain all dependencies.
For what type of data is your code suitable and how should it be stored? Include everything relevant: file format versions, required and optional fields, column order, maximum input size, character encoding, etcetera.
What steps should be followed to use your code? As programmers, this is usually the first thing we think of when writing a README, so it tends to be relatively well documented. For completeness, I will mention that you need to clearly distinguish between steps that need to be run every time or only the first time, describe the different modes/scenarios/use cases for which the code may be used, discuss optional and required arguments, and explain the expected output.
How to reach out for help? Despite your best efforts to document everything, users can still run into trouble. Some users will not reach out unless you make them aware of the option. An email address is good, a link to an issue tracker on GitHub, Codeberg, GitLab or BitBucket is even better.
Your software license. Technically, this does not really change how easy or difficult it is to run your code, but it does need to be in the README.

Don’t be that person

Good documentation will get you a long way, but if there are intrinsic hurdles in your code, users will still struggle and possibly give up. The most common way for this to happen, is when the code is too specific to the current situation of the programmer.

If your code only runs on your computer with your files on it, in January under easterly wind while the moon is waning, then your colleagues will likely find it difficult to run your code. Possibly more problematic is that, one year into the future, even you will not be able to run your code.

For example, a line like the following is all too common in research software:

DATAFILE = “C:\Users\Frank\Downloads\KNMI_klimaat_DeBilt_1980-2010.CSV”Hardcoded assumptions like these are errors waiting to happen. They can be avoided by making parameters like file paths user-configurable. For example, in Python, you can use the [argparse](https://docs.python.org/3/library/argparse.html) module to accept file paths as command line arguments.

Use helpful standards

As I mentioned before, there have been many attempts at improving the ease with which code can be run. None of them will reduce the effort to a double click, but the more you leverage these projects in your own code, the easier it will be to run it. I mention a few examples:

When working with Python, you can list your package dependencies in a [requirements.txt](https://pip.pypa.io/en/stable/reference/requirements-file-format/) and tell users to run pip install -r requirements.txt. This saves users from installing the packages manually and ensures that they will have the right version of each package. Even better is to use a virtual environment and manage the dependencies with [pip-tools](https://pypi.org/project/pip-tools/) or [pipenv](https://pypi.org/project/pipenv/) (which is a high-level wrapper around venv and pip-tools). Similar solutions exist for most other programming languages.
Docker can take the hassle out of installing software in general. You define what installation steps should be taken and how to start the software. Your colleagues only need to run Docker in order to make it happen. Installing Docker itself is a bit tricky, though, especially on Windows, so it is somewhat of a double-edged sword.
Binder lets you publish and run Docker images in the cloud (i.e., on somebody else’s computer), so that your colleagues do not need to install Docker. It also automatically selects predefined Docker images for a few common types of projects, so you do not need to define your own Dockerfile. The only drawback is that the code tends to start with a noticable delay. It is mostly suitable for visualizations.
Notebooks provide a handy way to combine scripts, visualizations and documentation in a single file that is relatively easy to run. Dependency management is greatly simplified in notebooks. Jupyter is probably the most widely known notebook format, which supports Julia, Python and R (hence the name). A honorable mention goes to Pluto, which only supports Julia but which is probably the easiest and most intuitive notebook environment available. Both of these can run in the cloud using Binder. CoCalc and Google Colab provide additional cloud hosting options for Jupyter.

Use your colleague as a guinea pig

If you follow all the tips above, your code is easier to run than if you don’t. The best way to be sure, however, is to test-drive your code with an unsuspecting colleague. This colleague must not have been involved in the writing of the code. Ideally, you find a colleague who is interested in the problem domain, but who tends to use a different programming environment (or none at all) and a different operating system (so if you wrote your code on a Mac, try to find a tester who runs Windows). Don’t sit next to your colleague during testing; let them figure it out using only the documentation.

If your tester can run your code without trouble, you can be fairly confident that your code is reusable. If not, keep adjusting until it is. Remember that it is always your code or your documentation that needs fixing, never your colleague.

Join the crowd

It is challenging to make reusable software, and you are not the first researcher to embark on this adventure. Utrecht University has various initiatives where you can meet fellow programmers and exchange tips and tricks:

Research Data Management Support organizes a monthly Programming Café or hop by at the Walk-in Hours at the Library Utrecht Science Park that are held every Monday afternoon.
If you happen to work at the Utrecht University Faculty of Humanities, you can also visit the weekly Digital Humanities walk-in hour, or email the Centre for Digital Humanities at any time for advice.

I wish you much inspiration and fun on your voyage!

Machine Learning Machinery for Quantum Chemistry

Nicolas Renaud — Wed, 06 Dec 2023 00:00:00 GMT

Machine learning techniques are impacting all areas of science, and molecular simulations are not spared. This blog post shows how to make use of the tools and techniques developed for machine learning to facilitate high-accuracy electronic structure calculations.

Quantum chemistry, the application of quantum mechanics to the study of molecular systems, is a powerful approach to design better materials, more efficient catalysts, and even better drugs. If you are reading this text on a fancy 4K OLED screen, chances are that quantum chemistry was used to fine-tune the light emission properties of the molecules that are shooting photons at you.

Quantum chemistry is, however, not a single technique, but the aggregation of many approaches that have been developed over many decades by many people. Some methods are computationally inexpensive and usually not so accurate, while others require important computational resources but can reach a higher degree of accuracy.

Quantum Monte Carlo (QMC) simulations are particularly interesting as they can reach a high degree of accuracy and can also be efficiently parallelized on very large computational resources. Many software packages, like CHAMP or QMC=CHEM, provide solutions to perform QMC simulations and have been used in countless scientific publications.

Illustration of the QMC approach. We first define a wave function containing tunable parameters. We sample the associated density and compute the system’s energy and its gradient w.r.t. the tunable parameters. We then update the value of the parameters and start again until we find the minimum value of the total energyTo understand how QMC works, let’s consider a small molecule of, let’s say, benzene. To keep things simple we are going to assume that all the electrons of the molecule are located around its center following a Gaussian distribution of variable width. To compute the energy of the molecule, we first sample this distribution to obtain representative configurations of the electronic positions. This can be done via a variety of techniques, for example, the Metropolis-Hastings algorithm. Computing the energy of the electrons in these configurations and taking averaging them can finally be used to approximate the total energy of the molecule.

A cornerstone of quantum chemistry tells us that the most accurate description of the molecule is obtained for the lowest value of its total energy. In our simple example, we can tweak the width of the Gaussian distribution until we find the minimal value of the total energy. If you now think, hey that almost looks like a machine learning problem, you’re not completely wrong. All we need is to encode the wave function in some sort of neural network.

Let’s start with a toy problem

Let’s first look at a toy problem that you may have encountered in your undergraduate studies: a single particle in a one-dimensional potential. We pick here the Morse potential as it is simple enough yet non-trivial.

We choose here to use a radial basis function (RBF) neural network to encode the wave function of this one-dimensional potential. In our RBF network, the input node encodes the position of the particle and each hidden node computes the value of a particular Gaussian function at this location. These values are then summed up on the output node to yield the value of the wave function at the particle’s location. This architecture therefore expresses the wave function of our particle as a sum of Gaussian functions, which is sufficient for this particular problem.

Schrodinet in action. The solver optimizes the positions, widths, and heights of all the Gaussian functions composing the RBF network to minimize the energy of the system.

QMCTorch for molecules

While replacing first-year university students with simple neural networks is appealing, it would be even better to apply the same machinery to more complex cases, for example, molecules. Unfortunately, the wave functions of molecular systems are much more complex than the one of a single particle trapped in a one-dimensional potential. All the electrons of the molecule interact with each other and also with the atomic nuclei and everything becomes very complicated very quickly. Fortunately, many scientists have given us a good understanding of the ingredients that should make up the wave functions of these systems.

Armed with this vast amount of knowledge, and a little bit of tenacity, we have developed QMCTorch** **a Python package that allows to run QMC simulations of molecular systems using neural network wave functions.

Representation of the neural network that encodes the wave function in QMCTorch. Starting from the atomic and electronic coordinates, different layers progressively compute the value of the wave function.Neural networks in QMCTorch take the positions of the electrons and atoms as inputs and compute the value of the wave function as an output. In between the input and output layers, many layers compute the different ingredients involved in the definition of the molecular wave function. These layers have parameters that can be specified by the user or extracted from external chemistry codes such as pyscf or ADF. These parameters can be further optimized to lead to an even more accurate wave function.

As the name indicates, QMCTorch is based on the popular deep learning framework PyTorch and leverages automatic differentiation to compute various derivatives needed in the optimization process. The use of automatic differentiation allows users to easily explore new flavors of the wave function without having to analytically compute all its derivatives. QMCTorch can also make use of multiple GPUs to accelerate the simulations thanks to the native capabilities of PyTorch and the distributed deep learning library Horovd. For more information on QMCTorch, go see the code documentation or the paper in the Journal of Open Source Software.

To use the code we first need to define a molecule through its atomic positions. QMCTorch allows for a few different flavors of wave functions to be used and we therefore need to pick one. We also have to define a sampler and an optimizer to compute and minimize the total energy of the system. Of course, each component has a lot of knobs that can be tweaked and that can greatly affect the result of the calculation.

from torch import optim from qmctorch.scf import Molecule from qmctorch.wavefunction import SlaterJastrow, from qmctorch.solver import Solver from qmctorch.sampler import Metropolis

create a H2 molecule

mol = Molecule(atom=‘H 0 0 -0.69; H 0 0 0.69’)

create the wave function object

wf = SlaterJastrow(mol, cuda=True)

create a sampler

sampler = Metropolis(nwalkers=1000, nstep=2000, step_size=0.2, nelec=wf.nelec, cuda=True)

create an optimizer

opt = optim.Adam(wf.parameters(), lr=1E-3)

create a solver and optimize the wave function

solver = Solver(wf=wf, sampler=sampler, optimizer=opt) solver.run(250)We can then ask QMCTorch to optimize the wave function of a molecule or even to optimize its geometry. The illustration below shows the result of such a geometry optimization for a water molecule. Starting from a non-ideal atomic arrangement, where the oxygen and the two hydrogens are aligned, the optimization process quickly brings the atoms in a more favorable conformation.

What now?

Many research groups have also developed their own solutions, like FermiNet or PauliNet and many others, to use machine learning techniques to accelerate and improve QMC calculations. Their approaches often use more convoluted functional forms than the ones used in QMCTorch. This allows for greater flexibility in the wave function and therefore more accurate results that have the potential to expand our chemical intuition.

One could, of course, argue that the energy change during a QMC optimization is so small that it does not justify the large amount of computing resources thrown at it. But if you want your red pixel to be red and not reddish/brown this is the level of accuracy needed!

Navigating research software policies: Insights from the Dutch research community

eScience Editorial Team — Tue, 28 Nov 2023 00:00:00 GMT

Navigating research software policies: Insights from the Dutch research community

Imagine you’re a researcher who developed a brilliant piece of software that may be useful to many. Maybe it’s a single script that cleans data in a specific format. Or perhaps it’s a larger piece of research infrastructure, that could be a crucial part of the workflow for nearly all researchers in your discipline. You may be wondering how to make your software available in a way that safeguards your, your institute’s, the users’ and science’s interests. What if the rules around doing this would all be thoroughly outlined to you by your institute?

Just as there are many decisions to make about how to license software, there are many decisions to be made when drafting an institutional software policy. All possible decision paths can seem overwhelming, which is why PRO4RS is helping policy makers and researchers with this task. Photo by Distinct Mind on UnsplashThe eScience Center, NWO, TU Delft and the VU organized an event on 3 October 2023 where we discussed with representatives from more than 20 Dutch knowledge organizations (research institutes, research infrastructure providers, research funding organizations, universities, UMCs and universities of applied sciences) how to develop and implement institutional research software policies, and what the challenges may be in embarking on such a project.

Why Institutional Software Policies?

To provide valuable context, Software Sustainability Institute director professor Neil Chue Hong laid out why research software policies are important. He started his presentation by explaining that institutional policy generally serves as a crucial element in effecting cultural change. Policies provide various purposes, such as encouraging good practices, ensuring legal compliance, maintaining ethical norms, reducing institutional risk, improving operational efficiency, and promoting and enhancing a mission.

The cultural change research software policies specifically encourage are open research and open science practices. They typically focus on promoting reuse of research software through recommendations about licenses and research output management plans (data or software management plans). On an individual level, these policies should encourage authors and developers to cite software and make their software citable. On an institutional level, research software policies are used to mitigate institutional risk in areas such as cybersecurity, procurement, asset management, research ethics, and integrity.

Research software policies are important for good practices in research software management and open science. Despite this, many organizations do not yet have an institutional research software policy. During our event, those interested in setting up or further developing their own policies answered questions about the situation at their institutes via Mentimeter. From the answers to those questions, it became clear that most of the Dutch institutes that were represented (77%) do not have a policy on research software (yet). However, about half of those institutes are currently (planning on) developing one.

Participants during the 3 October 2023 event discussed what should be included in their institutional research software policy, and how to practically go about developing this policy.

Drafting and Implementing Research Software Policies

Drafting research software policies poses several challenges. First, defining research software can be difficult, as the same piece of software may be categorized as research software or not depending on its role, or the context where it is produced and used. Second, the complexity and maturity of research software vary, ranging from analysis code to prototype tools to mature research software infrastructure. Institutions need to account for all these levels of complexity in their policies without being overly prescriptive or incomplete for different cases. Third, multiple stakeholders, including senior management, IT services, libraries, enterprise and innovation offices, and open science/open source policy offices, need to be involved in creating research software policies. The challenge lies in avoiding a situation where “if everyone is responsible, no one is”. Fourth, research software policies are interconnected with other university policies and processes. Integrating a new policy with existing policies can raise new questions and challenges.

The challenges pointed out by Neil Chue Hong and the audience were reflected in Jacko Koster’s talk. Jacko provided a snapshot of Leiden University’s work in progress in developing such a policy. The variety of parties involved, as well as the many aspects of research software, make for a complicated process that involves discussing with many stakeholders at all levels of the university. Paula Martinez Lavanchy from TU Delft discussed her university’s success story. In 2021, she led the drafting and implementation of TU Delft’s research software policy together with a group of TU Delft’s employees. She started by putting together a working group that encompassed relevant stakeholders from all areas: researchers, librarians, data stewards, valorisation center employees, legal services and the ICT department. The inclusion of many different parties and integration of all their perspectives was crucial to her success.

](https://zenodo.org/doi/10.5281/zenodo.4629661)Click the image to read TU Delft’s research software policyTU Delft’s experience provides valuable insights into what can make institutional research software policies effective. For example, TU Delft has made sure that the implementation and software workflow described by the policy is solidly integrated into training for staff. This approach fits well with findings by the PASTEUR40A project, which investigated success factors in implementing policies at universities. In their project, they investigated Open Access policies at 120 different universities. Drawing from these experiences with implementing Open Access policies, they wrote a paper that concluded that effective policies should be minimally burdensome, mandatory, linked to evaluation and incentives, and applied at the optimal time in the lifecycle. Additionally, they must align with funders’ policies for university staff to adhere to them.

The challenge of drafting and implementing research software policies for individuals at knowledge institutions has inspired the formation of the recently approved RDA working group PRO4RS, a joint initiative by ReSA and the RDA, with support of the RDA Tiger project. For those wondering how to go about drafting an institutional research software policy, this group is working on an overview of existing policies. Next steps will include putting together resources on how to achieve policy change at an institute, and to develop a common framework for research software policy, specifically. You can join this group to receive updates and information about the group’s work, as well as sessions during the RDA plenary (two times per year). A summary of the working group’s most recent (autumn 2023) RDA plenary session is available here.

The participants at our event brainstormed about what should be part of their research software policies. They agreed that at a minimum, a definition of research software should be included in the policy. Most participants agreed that it should include rules about licensing and sharing software, too. An interesting point of discussion was the question whether guidelines on good practices in software development should be part of a research software policy, and where to draw the line regarding this. For example, some participants agreed that pointers on research software quality and a commitment to training researchers on good practices should be part of a research software policy, while others did not.

Next steps**

We invite anyone interested in staying up to date about institutional research software policies to **attend our webinar on 16 January 2024 to continue the conversation about institutional research software policies in the Netherlands. **During this webinar, we plan to include an update from the PRO4RS RDA working group and reiterate the most important points from our workshop on 3 October to those who missed it. Sign up here to join the webinar.

If you are interested in developing institutional research software policies, you can** subscribe to updates from the PRO4RS RDA working group **here.

Finally, on 23 April 2024, the eScience Center is jointly organising National Research Software Day, where policy makers, research supporters and community leaders will come together to discuss all things research software. You can find out more and sign up for this day here.

Written by Lieke de Boer. Thanks to Carlos Martinez-Ortiz, Maaike de Jong, Maria Cruz and Meron Vermaas for comments.*

Cracking the (moisture) tracking code

Peter Kalverla — Tue, 21 Nov 2023 00:00:00 GMT

An exclusive interview with our project partners

This post is part of a series on our 2021 Small Scale Initiative Call in Software Performance Optimization. This week’s highlight: “Optimizing tracking of moisture in the atmosphere”.

What is moisture tracking? Can Python deliver optimal performance? What’s it like to have others modify your code? How can a team avoid alienation in coding? Will the output of this project be part of CMIP? Find out in this exclusive interview with project team members Imme Benedict, Ruud van der Ent, Chris Weijenborg, and Peter Kalverla!

Project team. Imme and Ruud, Peter, Chris and Yang (top-to-bottom, left-to-right).

Chris, you’re an expert in atmospheric dynamics. Can you explain what moisture tracking is, and why it is useful?**

Chris: The essential question you want to answer with moisture tracking is where moisture leading to precipitation in a certain area is coming from. This can be related to a particular event, think about for example the floods in West-Europe in 2021, but also related to the climatological precipitation. This is useful, since it gives us other insights then just looking at precipitation and or wind fields directly. For example, if the sea surface temperature is increasing in a warmer climate, does this lead to an increased climatological precipitation at another location? Or are the Floods in 2021 caused by increased sea surface temperatures in the Mediterranean Sea or the increased soil moisture over the continent? You might wonder now how all of this works: it’s complicated, but we have a nice animation to show the basics:

Animation of a moisture-tracking experiment. The video first runs forward, then reversed. On the left, you see the total water content in the atmosphere, on the right only the part that contributed to the 2021 floods over the Eiffel and the Ardennes.

Imme, you were the main applicant for this project. What was your original goal with the proposal?

Imme: I had been working in my PhD with the WAM2layers model and also adapted parts of the code. When I got a position as lecturer and researcher, I was keen on continuing the work on moisture tracking, and one way was to work with MSc thesis projects. However, the tool became a bit outdated as it was written to use ERA-Interim dataset as input, while for most recent events only the newest dataset ERA5 was available, as the ERA-Interim dataset was phased out. Therefore,

I became hesitant to use moisture tracking in my own and students research, which I thought was a shame.

Then I wrote this proposal to update and speed-up the code such that researchers new in the field (like students) could easily start to use it.

Ruud, as the original developer of WAM2layers you published your code many years ago. What were your expectations back then? And how do you look back on that today?

Ruud: Initially I just wanted to archive the code, so other researchers would more easily be able to find it. I naively thought of the code as something quite static. Since then, demand for moisture tracking in various fields of application has increased enormously and so has the importance to develop and maintain the code. I hope that with the user community around WAM2layers we can keep the software up to date with the latest developments and relevant for several years to come.

Peter, you have a background in meteorology @ Wageningen University. What was it like to go back to your roots in this project?

Peter: Indeed, this project was right up my alley. I benefited a lot from my meteorology background and I enjoyed delving into some of the equations I’d almost forgotten about. I also liked the collaboration in this project. It felt like a real team effort, where everyone brought their own unique expertise.

When I left Wageningen to join the eScience Center, I wanted to improve my own research software engineering skills so I could apply them to the benefit of academic research. I think that really worked out well with this project.

Now that we’re past the introductions, let’s take a deeper dive into the project. You mentioned the code was too slow to work with high-resolution data like ERA5. Did you solve this problem, and how?

Peter: Yes, we did. The easiest win was changing the code from Python2 to Python3, and then some relatively straightforward changes, for example better use of numpy’s broadcasting capabilities.

Illustration of the time steps in the input data (above) and in the model time loop (below).

WAM2layers was originally written in Matlab, then translated to Python. Wasn’t it time for another switch, especially considering Python’s reputation regarding performance?**

Peter*: We considered it. I’ve been experimenting a bit with Julia in the past and it seems like a suitable alternative. However, we wanted to make sure that the code was still understandable for scientists or students: even if they are not actively changing the code, it’s good if they can understand what’s going on. Python still is by far the most widely known language in the field.

Moreover, Python is fast enough for now. For most operations we use highly optimized numpy functions. If we wanted to squeeze out even more we could still look into numba, for example. Having said that, some of the changes we made to the code structure do actually make it easier to replace expensive operations with optimized code in another language, should we ever wish to do that.

In the end, you did more than make it faster. Can you elaborate on some of the unexpected changes?

Peter: yes, one example is that we made it easier to (pip) install the model and use it from the command line.

Ruud:

People can now get the software working in a matter of minutes. One does not need to be an expert anymore, which is a great development.

However, when people do research with it, I still hope that they involve someone from the core development team as there might be some tricky devils in the details.

In retrospect, what are the main things you got out of this project?

Ruud: Personally, I really enjoyed two things: 1) working in a horizontal team, and 2) blocking time in my agenda to do some coding.

Chris: I really enjoyed learning something new again, moreover in the meanwhile I also was ‘forced’ to learn version control with Git.

Imme: Teamwork, and learning Git! But also the possibility to have students working with the tool again.

Can you elaborate on the student projects?

Ruud & Imme: Sure! Below are the results from 4 student projects: Wenyu looked at monsoon moisture source for the Yangtze River Basin, Rens studied trends in moisture sources related to rising lakes in Keny, Emma studied the drivers of rainfall variability in Surinam, and Vincent used WAM2layers to study the moisture sources over the Amazon region during August 2023 when a big field campaign took place to understand biosphere-atmosphere interactions.

WAM2layers output as produced and visualized in students’ projects. Shading illustrates the moisture sources for the each of the case studies.

Recently a follow-up project was granted. So this story will continue?

Ruud: Absolutely, with the NWO Open Science project and our own time we want to make WAM2layers even more user-friendly, increase the number of features, applicability to climate models, and get a better grip on the uncertainties. To this end we will also engage with the wider moisture tracking community.

Will CMIP7 include an “Atmospheric moisture tracking-MIP” in the future?

Imme: Ha, that would be amazing of course, but realistically I think there are still some steps in between before such a big goal can be achieved. We are currently initiating a moisture tracking intercomparison project, where multiple moisture tracking models will run the same case to address uncertainty of moisture sources between models. This project will be supported by a workshop at the Lorentz centre in Leiden in May 2024.

This post is part of a series on our 2021 Small Scale Initiative Call in Software Performance Optimization. See also our post on the project Minds for Mobile Agents.

Automatic differentiation from scratch

Pablo Rodríguez-Sánchez — Thu, 09 Nov 2023 00:00:00 GMT

A surprisingly simple and elegant way to teach your computer how to perform derivatives, with some Julia (and Python) examples

Photo by Crissy Jarvis on Unsplash

First, a disclaimer

Automatic differentiation is a well-known sub-field of applied mathematics. You definitely don’t have to implement it from scratch, unless, as I did, you want to. And why would you want to do such a thing? My motivation was a mix of the following:

I like to understand what the packages I use do
The theory behind automatic differentiation happens to be very beautiful
I could use it as a case study to improve my understanding of the Julia language

Furthermore, if you are interested in performance, you’d likely want to focus on backward automatic differentiation, and not, as I did, on the forward one.

If you are still reading, it means that after all these disclaimers your intrinsic motivation is still intact. Great! Let me introduce you to the fascinating topic of automatic differentiation and my (quick and dirty) implementation.

Enter the dual numbers

Probably you remember it from your high school years. The nightmare of derivatives! All those tables you had to memorize, all those rules you had to apply… chances are that it is not a good memory!

Would it be possible to teach a computer the rules of differentiation? The answer is yes! It is not only possible but can even be elegant. Enter the dual numbers! A dual number is very similar to a two-dimensional vector:

the first element represents the value of a function at a given point, and the second one is its derivative at the same point. For instance, the constant 3 will be written as the dual number (3, 0) (the 0 means that it’s a constant and thus its derivative is 0) and the variable x = 3 will be written as (3,1) (the 1 meaning that 3 is an evaluation of the variable x, and thus its derivative respective to x is 1). I know this sounds strange, but stick with me; it will become clearer later.

So, we have a new mathematical toy. We have to write down the game rules if we want to have any fun with it: let’s start defining addition, subtraction, and multiplication by a scalar. We decide they follow exactly the same rules that vectors do:

So far, nothing exciting. The multiplication is defined in a more interesting way:

Why? Because we said the second term represents a derivative, it has to follow the product rule for derivatives.

What about quotients? You guessed… the division of dual numbers follows the quotient rule for derivatives:

Last but not least, the power of a dual number to a real number is defined as:

Perhaps you feel curiosity about the multiplication by u’. This corresponds to the chain rule, and enables our dual numbers for something as desirable as function composition.

The operations defined above cover a lot of ground. Indeed, any algebraic operation can be built using them as basic components. This means that we can pass a dual number as the argument of an algebraic function, and here comes the magic, the result will be:

It is hard to overstate how powerful this is. The equation above tells us that just by feeding the function the dual number (x, 1) it will return its value at, plus its derivative! Two for the price of one!

Those readers familiar with complex numbers may find interesting to try the following exercise:

If we define a dual number as

(u, u’) = u + e u’

with e² = 0, all the properties above are automatically satisfied!

Teaching derivatives to your computer

Just as a calculus student will do, the rules of differentiation turn a calculus problem into an algebra one. And the good news: computers are better at algebra than you!

So, how can we implement these rules in a practical way on our computer? Implementing a new object (a dual number) with its own interaction rules sounds like a task for object-oriented programming. And, interestingly enough, the process is surprisingly similar to that of teaching a human student. With the difference that our “digital student” will never forget a rule, apply it the wrong way, or forget a minus sign!

So, how do these rules look, for instance, in Julia? (For a Python implementation, take a look here). First of all, we need to define a Dual object, representing a dual number. In principle, it is as simple as a container for two real numbers:

""" Structure representing a Dual number """ struct Dual x::Real dx::Real endLater, it will come in handy to add a couple of constructors.

""" Structure representing a Dual number """ struct Dual x::Real dx::Real

""" Default constructor """
function Dual(x::Real, dx::Real=0)::Dual
    new(x, dx)
end

""" If passed a Dual, just return it
This will be handy later """
function Dual(x::Dual)::Dual
    return x
end

end

Don’t worry too much if you don’t understand the lines above. They have been added only making the *Dual* object easier to use (for instance, *Dual(1)* would have failed without the first constructor, and so would have done the application of *Dual* to a number that is already a *Dual*).

Another trick that will prove handy soon is to create a type alias for anything that is either a Number (one of Julia’s base types) or a Dual.

const DualNumber = Union{Dual, Number}And now comes the fun part. We’ll teach our new object how to do mathematics! For instance, as we saw earlier, the rule for adding dual numbers is to add both their components, just as in a 2D vector:

import Base: + function +(self::DualNumber, other::DualNumber)::Dual self, other = Dual(self), Dual(other) # Coerce into Dual return Dual(self.x + other.x, self.dx + other.dx) endWe have to teach even more basic stuff. Remember a computer is dramatically devoid of common sense, so, for instance, we have to define the meaning of a plus sign in front of a Dual.

+(z::Dual) = z

This sounds as idiotic as explaining that +3 is equal to 3, but the computer needs to know! Another possibility is using inheritance, but this is an advanced topic beyond the scope of this piece.

Defining minus a Dual will also be needed:

import Base: - -(z::Dual) = Dual(-z.x, -z.dx)and actually, it allows us to define the subtraction of two dual numbers as a sum:

function -(self::DualNumber, other::DualNumber)::Dual self, other = Dual(self), Dual(other) # Coerce into Dual return self + (-other) # A subtraction disguised as a sum! endSome basic operations may be slightly trickier than expected. For instance, when is a dual number smaller than another dual number? Notice that in this case, it only makes sense to compare the first elements, and ignore the derivatives:

import Base: < <(self::Dual, other::Dual) = self.x < other.xAs we saw before, more interesting stuff happens with multiplication and division:

import Base: *, /

function *(self::DualNumber, other::DualNumber)::Dual self, other = Dual(self), Dual(other) # Coerce into Dual y = self.x * other.x dy = self.dx * other.x + self.x * other.dx # Rule of product for derivatives return Dual(y, dy) end

function /(self::DualNumber, other::DualNumber)::Dual self, other = Dual(self), Dual(other) # Coerce into Dual y = self.x / other.x dy = (self.dx * other.x - self.x * other.dx) / (other.x)^2 # Rule of quotient for derivatives return Dual(y, dy) endand with potentiation to a real number:

import Base: ^ function ^(self::Dual, other::Real)::Dual self, other = Dual(self), Dual(other) # Coerce into Dual y = self.x^other.x dy = other.x * self.x^(other.x - 1) * self.dx # Derivative of u(x)^n return Dual(y, dy) endThe full list of definitions for algebraic operations is here. For Python, use this link. I recommend taking a look!

After this, each and every time our dual number finds one of the operations defined above in its mysterious journey down a function or a script, it will keep track of its effect on the derivative. It doesn’t matter how long, complicated, or poorly programmed the function is, the second coordinate of our dual number will manage it. Well, as long as the function is differentiable and we don’t hit the machine’s precision… but that would be asking our computer to do magic.

Example

As an example, let’s calculate the derivative of the polynomial:

at x = 3.

For the sake of clarity, we can compute the derivative by hand:

it is apparent that and p(3) = 39 and p’(3) = 34.

Using our Dual object, we can reach the same conclusion automatically:

poly = x -> x^3 + x^2 + x z = Dual(3, 1) poly(z)

> Dual(39, 34)Even if the same polynomial is defined in a more intricate way, the Dual object can keep track:

""" Equivalent to poly = x -> x^3 + x^2 + x Just uglier """ function poly(x) aux = 0 # Initialize auxiliary variable for n in 1:3 # Add x^1, x^2 and x^3 aux = aux + x^n end end

poly(z)

> Dual(39, 34)

What about non-algebraic functions?

The method sketched above will fail miserably as soon as our function contains a non-algebraic element, such as a sine or an exponential. But don’t panic, we can just go to our calculus book and teach our computer some more basic derivatives. For instance, our table of derivatives tells us that the derivative of a sine is a cosine. In the language of dual numbers, this reads:

Confused about the u’? Once again, this is just the chain rule.*

The rule of thumb here is, and actually was since the very beginning:

We can create a _factory function that abstracts this structure for us:

function _factory(f::Function, df::Function)::Function return z -> Dual(f(z.x), df(z.x) * z.dx) endSo now, we only have to open our derivatives table and fill line by line, starting with the derivative of a sine, continuing with that of a cosine, a tangent, etc.

import Base: sin, cos

sin(z::Dual) = _factory(sin, cos)(z) cos(z::Dual) = _factory(cos, x -> -sin(x))(z) # An explicit lambda function is often requiredIf we know our maths, we don’t even need to fill all the derivatives manually from the table. For instance, the tangent is defined as:

and we already have automatically differentiable sine, cosine, and division in our arsenal. So this line will do the trick:

import Base: tan

tan(z::Dual) = sin(z) / cos(z) # We can rely on previously defined functions!Of course, hard-coding the tangent’s derivative is also possible, and probably good for code performance and numerical stability. But hey, it’s quite cool that this is even possible!

See a more complete derivatives table here (Python version here).

Example

Let’s compute the derivative of the non-algebraic function

It is easy to prove analytically that the derivative is 1 everywhere (notice that the argument of the tangent is actually constant). Now, using Dual:

fun = x -> x + tan(cos(x)^2 + sin(x)^2)

z = Dual(0, 1) fun(z)

> Dual(1.557407724654902, 1.0)

Making it more user-friendly

We can use dual numbers to create a user-friendly derivative function:

""" derivative(f)

Seamlessly turns a given function f
into
the function's derivative

""" function derivative(f) df = x -> f(Dual(x, 1.0)).dx return df endUsing this, our example above will look like:

fun = x -> x + tan(cos(x)^2 + sin(x)^2)

dfun = derivative(f) dfun(0)

> 1.0

Another example**

Now we want to calculate and visualize the derivatives of:

First, we have to input the function, and the derivative gets calculated automatically:

f(x) = x^2 - 5x + 6 - 5x^3 - 5 * exp(-50 * x^2)

df = derivative(f)We can visualize the results by plotting a tangent line:

using Plots

I = [-0.7; 0.7] δ = 0.025 @gif for a = [I[1]:δ:I[2]; I[2]-δ:-δ:I[1]+δ] L(x) = f(a) + df(a) * (x - a) plot(f, -1, 1, leg=false) scatter!([a], [f(a)], m=(:red, 2)) plot!(L, -1, 1, c=:red) ylims!(-5, 15) end

Is this useful?

Automatic differentiation is particularly useful in the field of Machine Learning, where multidimensional derivatives (better known as gradients) have to be performed as fast and exactly as possible. Said this, automatic differentiation for Machine Learning is usually implemented in a different way, the so-called backward or reverse mode, for efficiency reasons.

A well-established library for automatic differentiation is JAX (for Python). Machine learning frameworks such as Tensorflow and Pytorch also implement automatic differentiation. For Julia, multiple libraries seem to be competing, but Enzyme.jl seems to be ahead. Forwarddiff.jl is also worth taking a look at.

Acknowledgments

I want to say thanks to my colleague and friend Abel Siqueira, for kindly introducing me to Julia and reviewing this post, and to Aron Jansen, for his kind and useful suggestions. A more in-depth introduction can be found in this episode of Chris Rackauckas’ book on scientific machine learning.

The TeX Math Here browser add-in also played an important role: it allowed me to transfer my Latex equations from Markdown to Medium in an (almost) painless way.

How Easy It Can Be to Whip Up Your Own Orange3 Widget: A Chatbot Adventure with HuggingChat!

Ji Qi — Mon, 06 Nov 2023 00:00:00 GMT

Are you ready to dive into the captivating world of Orange3 widgets and unleash your inner chatbot whisperer? 🚀✨ Whether you’re a coding connoisseur or a newbie explorer, developing your own widget doesn’t have to be a Herculean task. In fact, with a little sprinkle of curiosity and a dash of HuggingChat magic, you’ll be crafting your own chatbot widget in no time!

In a future blog post by my colleague, Kody Moodley, you will catch a glimpse of what Orange3 is all about, along with its snazzy uniqueness (if not, seriously, what are you waiting for? Go give it a whirl!). But for now, I bet you’re riding the wave of curiosity mixed with a sprinkle of concern — “Will it be a brain drain, and will I get tangled in the web of complexity while treading the path of development?”

What I aim to convey through this post is that creating widgets on Orange3 is as easy as pie! However, writing the post takes me some serious brainpower. I’ve been on an epic quest to unravel the secrets of making the content simple, yet captivating. But you know what they say, inspiration strikes in the most unexpected places. Picture this: I’m casually chatting with a friend on the newly released Android client of ChatGPT, and bam! Just like that, it hit me — why not roll up my sleeves and craft my very own chatbot widget on Orange3?

My initial brainstorm led me straight to a ChatGPT-powered chatbot, but alas, I wear the open-source cape proudly! Thus, I turned my gaze toward the enchanting realm of HuggingChat. 🌟 With a dash of curiosity, I dove into the API docs at lightning speed and spent a solid three hours conjuring up all the magic you’re about to read.

Before diving into the core

Embarking on a journey is always thrilling, but you might find yourself going, “Where in the world do I begin?” Fret not, for Orange3 has your back with a nifty add-on template. No need to suffer the pains of starting from scratch — just give ’em a friendly clone from GitHub to your computer and sprinkle in the required Python libraries.

#Clone the repo of the example Orange3 add-on git clone https://github.com/biolab/orange3-example-addon.git cd orange3-example-addon

#Install Orange3 and other required packages python -m venv orangedev
source orangedev/bin/activate pip install pyqt6 pyqt6-webengine orange3 hugchatTime for a little detective work! Execute these commands to check if you have everything in the right place.

#Install the example add-on and open the GUI application of Orange pip install -e .
python -m Orange.canvas If everything goes well, you should be able to see the following window popping up, where you can find that your Example add-on and Hello World widget are already there.

The canvas application with the example add-on installed.

The serious showdown

Before we wade into the treacherous waters ahead, let’s take a peek at what awaits from these three hours of dedicated effort.

The chatbot widget.In brief, I have a chatbot that allows me to log in with my own HuggingFace account and password, answer my questions, and record all of the historical conversations. The only thing I did was create a Python file named “HugChat.py*” *and put it in the folder “orange3-example-addon/orangecontrib/example/widgets”. Check the content of my file to get a first impression of how easy the work is.

#Content of the HugChat.py file from Orange.data import Table from Orange.widgets import gui from Orange.widgets.settings import Setting from Orange.widgets.widget import OWWidget, Output from Orange.data.pandas_compat import table_from_frame

from hugchat.login import Login from hugchat import hugchat import textwrap import pandas as pd

class HugChat(OWWidget): name = “Hugging Chatbot” description = “Orange3 widget instance of HuggingChat.” icon = “icons/mywidget.svg” want_main_area = True

email = Setting("")
passwd = Setting("")
query = Setting("")

class Outputs:
    data = Output("Data", Table)

def __init__(self):
    super().__init__()
    self.queries = []
    self.replies = []
    self.chatbot = None
    
    # control area: login
    gui.lineEdit(widget=self.controlArea, master=self, 
                 value="email", label="Email")
    gui.lineEdit(widget=self.controlArea, master=self, 
                 value="passwd", label="Password")
    gui.button(widget=self.controlArea, master=self, 
               label="Login", callback=self.init_chatbot)
    
    # main area: chatbot
    gui.lineEdit(widget=self.mainArea, master=self, 
                 value="query", label="Query", callback=self.chat)
    self.reply = gui.WebviewWidget(parent=None, debug=False)
    self.mainArea.layout().addWidget(self.reply)
    
def init_chatbot(self):
    sign = Login(self.email, self.passwd) 
    
    # cookies setup
    cookie_path_dir = "./cookies_snapshot"
    sign.saveCookiesToDir(cookie_path_dir)
    cookies = sign.login()
    
    # initialize chatbot
    self.chatbot = hugchat.ChatBot(cookies=cookies.get_dict())
    
    # initialize a new conversation
    id = self.chatbot.new_conversation()
    self.chatbot.change_conversation(id)

def chat(self):
    reply_text = self.chatbot.chat(text=self.query)
    self.reply.setHtml(reply_text)
    
    # update the output table
    self.queries.append(self.query)
    self.replies.append(reply_text)
    df_output = pd.DataFrame({
        "Query": self.queries, 
        "Reply": self.replies
    }) 
    self.Outputs.data.send(table_from_frame(df_output))Now, let’s look at the details. The code above defines a class that contains all the features and functions of our widget. This class is inherited from another class named `OWWidget`, which is the base class of all Orange3 widgets. From there, we can rewrite existing functions and create new functions to satisfy our requirements.

At the top of the class, we have a brief description of the widget, including its name, a short text explaining what it does, and a path string pointing to the location of the widget icon. An additional flag called want_main_area is assigned to be true, which means that we want to present the main area of the widget and show things there.

name = “Hugging Chatbot” description = “Orange3 widget instance of HuggingChat.” icon = “icons/mywidget.svg” want_main_area = TrueA widget can contain both the control and main area, while the main area is optional. To briefly explain, the control area should contain all of the utilities for preparing the widget, while the main area is for the major functionality.

Sub-interface of our widget that includes both the control and main area.We also have three class attributes defined, which will be used to store the result of interacting with the GUI controls (I will come back to this later). If there are any output tables, they will be defined in an inner class named Outputs. In our case, we only have one output table for outputting the chatting history.

email = Setting("") passwd = Setting("") query = Setting("")

class Outputs: data = Output(“Data”, Table)Next, we defined the __init__ method, where we run __init__ of the parent class, initialize all the instance attributes, and define the GUI of the widget. We got two empty lists, queries and replies, for storing the chat history, and chatbot which is the HuggingChat object. Now, let’s focus on the part of GUI construction.

gui.lineEdit(widget=self.controlArea, master=self, value=“email”, label=“Email”) gui.lineEdit(widget=self.controlArea, master=self, value=“passwd”, label=“Password”) gui.button(widget=self.controlArea, master=self, label=“Login”, callback=self.init_chatbot)

main area: chatbot

gui.lineEdit(widget=self.mainArea, master=self, value=“query”, label=“Query”, callback=self.chat) self.reply = gui.WebviewWidget(parent=None, debug=False) self.mainArea.layout().addWidget(self.reply)The above code adds GUI controls to both the control and main area, where GUI controls will be added and placed in the interface by following the order of their definitions in the code. You should be able to find three types of GUI controls:

lineEdit that allows the user to input some text,
button that will trigger a pre-defined function by clicking, and
WebviewWidget that will render some HTML content.

In this blog, I will not explain everything so that you will have the motivation to check the documentation yourselves. To briefly summarize:

In the control area, two lineEdit controls are defined to accept the login information as input and store them in the pre-defined class attributes email and query. They are followed by a button to trigger the callback method self.init_chatbot for initializing the chatbot instance.
In the main area, another lineEdit control is defined as accepting the user’s query, with the callback self.chat to send the query to the chatbot instance and get the reply, triggered by enter-clicking. The callback method is also responded to display the reply message in the WebviewWidget below.

Up to this point, we have thoroughly reviewed the process of defining a widget. In simple terms, it includes widget descriptions, input-output definitions, and GUI control definitions. Although the functionalities of different widgets may vary, the design process is largely similar. Orange3 defines a multitude of foundational classes, standardizes data transfer between widgets, and establishes a comprehensive and efficient workflow engine. This enables developers to better focus on the modular design and development of widget functionalities

The adventure has just begun

So, I bet you’re already getting the hang of developing your own widgets on Orange3, and you’re practically bursting with creative energy – high five to that! But hold your horses, my friend, because this blog post is just the opening scene of our grand adventure. There’s a whole treasure trove of discoveries waiting for you, like defining your very own add-on, crafting widgets that dance with interactive visuals, and even sending your widgets off to the fancy official platform. Don’t panic, though – these aren’t things you need to wrestle with right now. And guess what? You’re not alone on this journey! Craving more Orange3 tales and tech tidbits? Get in touch with us (Kody and me)! Or drop a hint for more juicy insights below this blog! Can’t wait to bump into you again on this wild Orange3 expedition!

DISCLAIMER: As you probably found out from the start: we also called in some AI support for the creation of this blogpost. We prompted ChatGPT to make the original text ‘more fun to read’. We hope reading it actually sparked some joy for you!*

Investing in people: anticipating the future of research software

eScience Editorial Team — Thu, 02 Nov 2023 00:00:00 GMT

*Authored by Kim Hartley and *Michelle Barker

Photo credit: ADORE.Software at the International Research Software Funders WorkshopThis article was cross-posted on Adore.software and reports on the progress of the Amsterdam Declaration on Funding Research Software Sustainability. To read more about this initiative and the initial workshop see this blogpost from November 2022.

We are proud to report that the second International Research Software Funders Workshop, co-hosted by the Digital Research Alliance of Canada (the Alliance) and the Research Software Alliance (ReSA) from 18–20 September was a great success. People are at the heart of research software sustainability and funders can lead the way by investing in the people who develop and maintain research software. That is why the workshop theme was “Investing in People: Anticipating the Future of Research Software”. The event aimed to set the future agenda for government, philanthropic, and industry funders to support sustainable research software and communities.

In total, 50 representatives from more than 35 organisations — including funders and organisations that support research software from around the globe — attended the hybrid workshop held in Montreal, Canada, and virtually. During the three-day workshop, experts explored how research software funders can facilitate capacity building through funder practices; research software platforms, infrastructure, and communities; and new initiatives. Moreover, Version 1.0 of the Amsterdam Declaration on Funding Research Software Sustainability (ADORE.software) was released and is now available for signing.

Background**

This second iteration of the funders workshop built on the inaugural International Funders Workshop: The Future of Research Software, co-hosted by the Netherlands eScience Center and ReSA in November 2022 in Amsterdam. During this first workshop, more than 40 funding organisations gathered to set the agenda for supporting sustainable research software and drafting of the ADORE.software. Following public consultation in 2023 and 19 organisational expressions of interest in becoming signatories, Version 1.0 of the Declaration was released in advance of the Montreal workshop and is now available for signing. ADORE.software is the first step towards formalising, on a global level, the basic principles and recommendations related to funding the sustainability of research software, including the people needed to achieve this goal.

The first workshop marked a further step in the development of the research software community. Stakeholders have continued to build on this momentum through involvement in the Research Software Funders Forum, convened by ReSA, and its funder-led working groups. The Alliance and ReSA, together with the workshop Steering Committee, organised the 2nd International Research Software Funders Workshop to continue this important work and facilitate global collaboration among funders and other key decision makers and influencers across the research software ecosystem.

2nd International Research Software Funders Workshop

Pre-workshop — September 18

The event commenced with a pre-workshop, including a session on How to Explicitly Support Research Software as Part of Your Funding Program, delivered by ReSA, and a session on CiteSoftware, led by a group of stakeholders aiming to drive the adoption of a common research software preservation and citation guidance resource for the research community.

Photo credit: ADORE.Software at the International Research Software Funders WorkshopWorkshop Day 1 — September 19

The first day of the workshop included presentations that set the scene for participants to explore the crucial role funders play in supporting sustainable research software and communities.

Carole Goble from the Software Sustainability Institute (SSI)/ELIXIR-UK/University of Manchester provided an overview of what research software is and why it is critical to the research endeavour. Carole highlighted that the people who make, use, and resource software are fundamental to research software sustainability. Moreover, research software needs resources and both direct (e.g., money) and indirect (e.g., community, reward, recognition, open-source support, etc.) forms of support for sustainability.

Josh Greenberg from the Alfred P. Sloan Foundation delivered a keynote that provided a funder’s perspective. He asked attendees several pertinent questions: How much (and when) should we worry about technical debt? How much attention should we pay to user interface design in research software? Who should do which kinds of software? How central should “open source” be? How can we help institutions build capacity for research software development (and maintenance)?

Funders benefit from investing both before (community/need development) and after (ongoing maintenance and re-use) research software development.
International collaboration and collective action are critical and valuable (e.g., building a multilateral funding initiative due to the global nature of research software).
Funders should consider how to support and sustain Research Software Engineers (RSEs) but also incentivise researcher-developers to develop better software engineering practices.
The research software community needs a set of high-level classes to measure the impact of research software, such as citations and mentions of software maturity, or combining mention analysis and dependency analysis.

Malvika Sharan from The Alan Turing Institute and Open Life Science provided a research software community perspective in her talk on Exploring ‘Do No Harm’ Principles in Open Research Communities. Malvika’s compelling keynote encouraged funders to become leaders in the research software ecosystem by incentivising and recognising all types of research roles; investing in capacity-building goals that focus on bridging scientific and economic divides; and applying community-oriented frameworks to extend the benefits of research to the broader community that ‘do no harm’, thereby ensuring open science practices enable equitable research and research outcomes.

During the second breakout session, participants investigated capacity building through research software platforms, infrastructure, and communities. One group explored the role of Open Source Program Offices (OSPOs), noting that OSPOs as centres of software could ultimately help research institutions understand their software investments. Another group considered why and how to support research software communities, highlighting the need for more evidence-led recommendations and exploring the possibility of reversing the funding mechanism to fund projects after the fact, as a function of how much they were used. In discussing research software in research assessment reform, participants noted that very few research software outputs are being assessed and indicators are fragmented and field-specific. Some solutions include sharing guidance on how software is being evaluated in a single place, sharing job profiles for RSEs, and sharing existing examples of research indicators that include software. Finally, another group highlighted that the environmental impact of research software is an emerging field, and persuading researchers to think about making their code efficient is a challenge.

Workshop Day 2 — September 20***

The final breakout session focused on building capacity through new initiatives. Participants explored how funders can implement FAIR for Research Software (FAIR4RS) in funder policy. As some funders are developing guidance, questions arose about whether funders should create guidance individually or collectively and how the community should help funders do this. Through a discussion on how funders use persistent identifiers (PIDs), the need to engage publishers around software citation has become clear. And building on the theme of software citation, participants explored how the Research Software Directory, a free open-source platform, supports software citation.

For the final keynote, Jean-Baptiste Poline from McGill University provided a researcher’s perspective on the impact of funders’ investments in research software. Jean-Baptiste presented the role of research software from multiple viewpoints, offering key takeaways such as the value of building a community of researchers through collaborative projects; the need for community-endorsed standards and practices that support open and FAIR (Findable, Accessible, Interoperable, and Reusable) scientific research (e.g., INCF Network) to maximise research value; the importance of addressing incentives first; and the idea of funding new technologies and large labs but giving precedence to well-governed communities more than centralised projects. He recommended creating the tools needed to change culture, improving training and open practices, developing community standards, and changing incentives.

The workshop culminated with a Do-a-Thon, “a short collaboration where participants from different perspectives and skill sets work together on challenges, projects, or to learn something new” (OpenCon Cascadia). Prior to the workshop, participants were invited to submit do-able ideas. During the Do-a-Thon, attendees worked in small groups on the following topics: the Map of Open Source Science; linking software to community usage; and ten things applicants should do to make it easier to assess the impact of funding programs, to build on Ten simple rules for funding scientific open source software by Carly Strasser et al. This dynamic collaboration resulted in participants considering next steps, crowdsourcing tips, and determining some practical and generalisable approaches to address the challenges explored.

Overall outcomes and how you can get involved

The workshop solidified the importance of international collaboration to support research software. There is a widening group of funders with awareness of the need to support research software and an understanding of how to do this. The Research Software Funders Forum has been building momentum since its inception in early 2022; its dedicated funder-led working groups are expanding their reach by engaging with other stakeholders and organisations, such as the Global Research Council. Moreover, the workshop highlighted areas where coordination by funders would be beneficial, for example, in supporting the Amsterdam Declaration on Funding Research Software Sustainability. Participants also expressed an interest in exploring opportunities for forums, for example to engage publishers around software citations (along with the existing ReSA Task Force on Code Availability).

We thank all the participants for their engagement during the workshop sessions. We’re grateful to the workshop Steering Committee for their efforts in organising a successful hybrid workshop. A full report, written and edited collectively by workshop participants, is forthcoming.

To stay up to date on ADORE.software, visit www.adore.software. If you want to show your support for the Declaration, you can become a signatory (for funders) or supporter (for non-funders) and help share the news. To receive updates on news related to the Declaration and future activities, subscribe to the ReSA newsletter. Funders are invited to join the (free) Research Software Funders Forum by contacting ReSA at info@researchsoft.org.

If you’re interested in learning more about ReSA and its activities, join our next Research Software Community Leaders Forum on 15 November at 20:00 UTC. ReSA’s Community Leaders Forum is open to all, and you can sign up for calendar invites.

We would like to express our sincere gratitude to our community, Founding Members, and Organisational Members — without whom none of this work would have been possible.

This project has also been made possible in part by grant 2021–000000 from the Chan Zuckerberg Initiative DAF, an advised fund of the Silicon Valley Community Foundation; and the Alfred P. Sloan Foundation.

The Orange3 data mining platform for the Social Sciences and Humanities

Kody Moodley — Thu, 02 Nov 2023 00:00:00 GMT

Why aren’t more SSH researchers using and extending the platform?

Orange is an open-source Python-based data mining application with a graphical user interface (GUI). It allows people with no knowledge of programming to apply Machine Learning as well as advanced data processing, analysis, and visualization using a point-and-click, drag-and-drop interface. Those who have seen Scratch will be familiar with this mode of interaction, although Orange is more advanced and is used for data mining, rather than educational programming. The following screenshot shows what a typical Orange workflow looks like.

Screenshot of an example data mining workflow in Orange3The three small circles labeled ‘File’, ‘Data Table’, and ‘Scatter Plot’ in the image above, are called Orange widgets. They are the basic building blocks of a data analysis workflow (or pipeline) in Orange. Each widget is a software unit that performs some sort of data processing, analysis, or visualization and potentially has a set of inputs and outputs. You can connect the outputs of some widgets to the inputs of others to create a chain of processing and analysis operations. A great feature of Orange is its extensibility. There are loads of custom plugins for Orange (called add-ons) developed by researchers and the Orange user community. Add-ons usually comprise multiple widgets. Two mature and extensively used Orange add-ons are Orange-Text and Network which are used for text mining and network analysis respectively.

Story Navigator: a custom Orange add-on I am developing**

I am currently in the process of developing an add-on as part of a project with researchers at the University of Twente in the Netherlands. The add-on is called the Story Navigator. The aim of the Story Navigator is to enable computer-aided analysis of written stories for students and researchers who study narrative psychology. The add-on has several widgets to analyse different aspects of a story. It uses a combination of natural language processing, text mining, and general data analysis techniques to implement theories in narrative psychology, such as Burke’s pentad. Here’s a screen:

Screenshot of the Story Navigator add-on (not yet released) for Orange3.The add-on works by first importing textual stories into the workflow using the existing “Import Documents” widget from the Orange-Text add-on. Thereafter the researcher can choose a widget from the add-on for analysing some aspects of the imported stories. The actor analysis widget, for example, highlights potential characters in the stories; shows which kinds of actions (verbs) the characters are associated with; and calculates different measures for how central specific characters are to the stories.

Quantitative story analysis widgets from the Story Navigator add-on for Orange3.While it did take some time to develop the widgets for this add-on, what surprised me is that it took me only about two hours, from scratch, with no prior knowledge of Orange, to develop my first widget that showed up in the Orange3 interface when I started it up. Granted, the widget did not do anything except display. But the return on my two-hour investment was that I could now free up my creativity to develop whatever functionality I wanted in the widget, without having to create my own GUI. In doing so, I got to take advantage of a myriad of benefits including being able to:

Make my add-on available for anyone to use regardless of their programming knowledge
Receive development and usage feedback from an existing Orange community about my add-on (without having to further promote the add-on or build another community myself from scratch)
Demo my plugin to anyone, thanks to the GUI
Share data analyses made with my add-on with other Orange users — enabling reproducibility (currently undergoing a crisis in science)
Forget about writing installation instructions — they’re already here!

Despite what some readers might think at this point, I am not an Orange evangelist. In fact, I am not swooning over the capabilities and potential of Orange specifically. Orange does have competing platforms that try to do similar things such as KNIME and WEKA which are both Java-based and therefore inherit some concerns due to Java’s declining popularity. Orange itself also has some clear disadvantages, not least of which are the scalability issues and the add-on development documentation (the latter is not very accessible for less experienced developers).

But the point I am trying to make is that I am mostly just inspired by the philosophy behind platforms like these: people coming together to develop and share an extensible “workbench-style” software platform with a user-friendly interface. When I refer to software I don’t mean software that has a highly niche purpose, but software that aims to be an ever-evolving toolbox with the potential to address a multitude of research problems. Where your own creativity is the only limit in terms of contributing to it. Where the knowledge barriers to be able to contribute to it are super low, and where there are no qualifying criteria for those who can experience and benefit from the contributions.

Orange was developed at the Laboratory for Bioinformatics at the University of Ljubljana. It makes sense then that most of its user base uses it for data mining in bioinformatics (see for instance this study using Orange for early diagnosis of diabetes). But Orange itself is not at all designed to be domain-specific. Its add-ons and widgets are designed for domain-agnostic data mining (here’s a study using Orange to predict student performance in higher education). This begs the question, why aren’t more researchers in the Social Sciences and Humanities using it, extending it, or building similar platforms?

Bethesda Softworks is a US-based video game development company that is famous for its many franchises, one of which is “Elder Scrolls”. The games in this franchise are “RPGs” that put the player in control of a character from a fictional fantasy world with the goal to… wait for it… save the world from destruction (if you’re new to the term RPGs imagine a video game based on Game of Thrones and you’ll get the gist). In 2002, Bethesda released its third installment of Elder Scrolls, called “Morrowind”. The super interesting part is that they also released software called the Elder Scrolls Construction Set alongside the game. This software allowed players to make customizations to their own copy of the game. They could do things such as create new clothing, buildings, and other items; create their own character voices; make new quests, and customize the color of the sky. These modifications (mods *as they are now commonly referred to) vary in terms of how much software literacy is required in order to build them, but many of them require no more knowledge than being able to install a program and click a few buttons.

Fancy introducing Spiderman-suited creatures crawling around a medieval town into your video-gaming experience? Well, someone has already created a mod for this in Skyrim.Twenty years later, and Bethesda has become known and celebrated for giving its customers the freedom to customize their gaming experience. Nexusmods is one of the prominent hubs for hosting large collections of player-created mods for games by Bethesda and other major video game companies. I quote some jaw-dropping statistics from this site here:

“We host 477,145 mods for 2,342 games from 118,005 authors serving 41,902,274 members with 8,184,840,124 downloads to date.”

In academia, terms like “outputs”, “impact”, “community building” and “community engagement” are often bandied about as performance goals and metrics for work. However, despite their purported importance, these terms are hard to define and measure. But when I look at those numbers, I feel like we don’t need unambiguous definitions for those terms in order to verify that the mod concept ticks all those boxes and then some.

Nexusmods demonstrates that the mod concept is a winner.“Okay Kody, what does this all have to do with Orange?”, you may ask. Well, in some ways there are parallels in this story to the concept of Orange as a software platform.

First, there is a product that many people enjoy (or find useful). In Bethesda’s case, it is the “vanilla” Elder Scrolls games. In Orange’s case, it is a data mining platform which is a generic tool that is useful regardless of the research domain and data. The GUI aspect of Orange also opens it up to a much larger user base (not just a small group of researchers with development experience working in a niche field).

Second, there is the *enabling of customization. *In Bethesda’s case, it is the provision of customization software such as the Skyrim Creation Kit; in the case of Orange, it is the release of the code under an open-source license with reusable example code and plugin-writing tutorials.

Thirdly, and what I find to be the most critical, is *making it easy to customize. *In Bethesda’s case, they have made user-friendly software that does not require extremely specialized skills to use. Almost anyone can create a mod. In Orange’s case, you still need to know how to program to develop add-ons. But you don’t need advanced knowledge of the GUI frameworks that Orange uses — Qt and PyQt. In other words, if you know how to create any kind of computational script or software in Python, you can easily create an Orange widget to “house” or “wrap” its functionality. Furthermore, others, regardless of their technical nous, will be able to use it and “chain” your widget with other software in an Orange workflow.

What I have experienced and want to emphasize, is that plugging your software into existing platforms like Orange (especially ones that have a nice GUI) can increase its visibility and usage, over publishing it solely in a code repository. Decently designed GUIs help users to more quickly understand what your software does which goes a long way to maximizing its value. Platforms like Orange provide a mechanism by which developers can take advantage of these GUI benefits “for free” (well, not really for free, but with much less time investment than building and designing GUIs from scratch). It frees them up to use their creativity to solve research problems through widget development; it exposes their software to a much wider audience; and it allows users to interact with their software in a more intuitive way.

In short, platforms like Orange can be extremely useful both for researchers who want to quickly apply computational analyses in the SSH fields (without writing code) and for developers who want to increase the usability, visibility, and sustainability of their software. While it does, to some extent, tie the success of your software to the success of Orange, you are not “putting all your eggs in one basket”. Due to the modularity and already-in-place Python package structure of Orange add-ons and widgets, you can very easily strip the GUI out of your code and release your work as a stand-alone package, or migrate it to another platform.

If you are interested, my colleague Ji Qi has written a post about how to get started with Orange add-on development — be sure to check it out!

Take home messages

Orange is an open-source extensible data mining platform that is usable by people with little to no technical experience. If you are a researcher in SSH consider using it for your research. If it doesn’t have a feature you need, ask the developers if they can add it as a default feature to Orange. If they can’t add it, and if you or a colleague has Python experience, consider creating a custom add-on. It is surprisingly quick and easy to create one if you know a bit of Python.
Research software sustainability is a challenge. Building and maintaining communities around the development of research software is hard. Don’t let your software become JAG (Just-Another-Github repo). Take inspiration for how to avoid this from platforms like Orange. This does not mean that you need to create a graphical user interface for your software. It means: find a home for your software or find an existing toolbox in which to integrate your software, before creating your own toolbox. Integrating into existing platforms or packages will probably increase its usage and visibility over publishing as a stand-alone item.
Inspiration point #1: From the start, consider building software that solves generic problems in your field. i.e., problems that are shared by others in the field as well, rather than for highly specialized use cases.
Inspiration point #2: Make your software easy to use by as many types of people as possible. Not just for experienced developers. Also, ensure that someone can rapidly understand what your software does.
**Inspiration point #3: **Make it easy for as many people as possible to customize and extend your software.

Open Science Festival: What infrastructures do we need to preserve research software?

Carlos Martinez-Ortiz — Thu, 26 Oct 2023 00:00:00 GMT

*written by Luisa Orozco, Daniela Gawehns, and *Carlos Martinez-Ortiz

Introduction

Earlier this year, the 2023 edition of the Dutch Open Science Festival was held. We (the eScience Center and Leiden University) organised the session “National infrastructure for sustainable research software”. In this session we discussed the benefits and limits of available software preservation platforms (or infrastructures) and if national infrastructures for research software development are feasible and desirable.

Research software increasingly forms a critically important element of many research projects across a wide array of domains. To achieve truly sustainable and reproducible research, not only the software itself but also the computational environment it depends on needs to be preserved. Different types of infrastructure are required to support these preservation activities.

One user-friendly example of a friendly way to package computational environmetns are Docker containers. These containers can be archived on privately owned platforms like Docker Hub and GitHub or on publicly funded archives such as Zenodo. Such platforms become part of the infrastructure needed to rerun and reuse research code associated with research results.

Panel composition and pitches

The session was organised as a series of panel pitches, followed by a group discussion. The panel shared their insights and experiences, as we explored current issues with container storage platforms and related infrastructure and the role the Dutch research landscape can take as a provider of independent and government funded infrastructure for sustainable research software.

The panel was composed of national and international experts from different domains and backgrounds to contribute (researcher, software engineer, administrator of research servers, science funders). Panellists were John Swinbank from ASTRON, Niki van Stein from LIACS, Leiden University, Matthijs Moed from SURF and Jeremy Cohen from Imperial College London.

Panellists were asked to prepare a short pitch with their view on the following question:

Which infrastructures (version control, containers publication, Zenodo, testing infrastructure, any other?) are most important to the work your organisation does?

This is a summary of the pitches from each panellist:

John Swinbank, ASTRON

ASTRON works on a regime of big data. We are already working with tens of petabytes of data from telescopes like LOFAR, and that will increase to the order of 1 Exabyte per year when SKA (Ed.: a new radio telescope) comes online later this decade. Our task is to make sure this volume of data is in a format that is usable for astronomers. The astronomical community has been working towards full reproducibility, but it is not yet completely embedded in the culture. In terms of infrastructure, we have a local GitLab instance for code sharing;using containers and publishing on Zenodo is not yet universally accepted.

Niki van Stein, LIACS, Leiden University

In my group reproducibility is a big thing. Both to verify research results from us and other groups and also to build and document on code developed by master and PhD students. We use version control, and docker. Code alone is not enough, we aim to share the whole environment. But this is challenging when the code requires multiple CPUs and GPUs, you cannot easily put that in a docker container!

Matthijs Moed, SURF

At SURF, we build tools that fit in the workflow of researchers. Our biggest challenge is finding out what researchers actually need and how they can use our tools in their work. We work not only with organisations like Astron, who need very special infrastructure, but also for the majority of researchers who use computational methods but are not specialists in using them. Offering the right tools to them involves sometimes knowing their needs without them being able to formulate them. Meeting researchers where they are is key to developing sustainable research software.

Jeremy Cohen, Imperial College London

Infrastructure to support software, such as version control, is particularly important in my work and that of people I collaborate with. Containers are becoming more important in the research community as a method to help support reproducibility and address challenges such as working across different computing platforms. Continuous integration (CI) (Ed: continuously testing code while expanding its functionality) is also growing in importance. At Imperial College London, as in many other research institutions, we have access to such services, but of course, as the use of software and computing infrastructure in research continues to grow, there are frequently new opportunities emerging to improve the offerings and infrastructure that are provided to researchers. This is something that I aim to help support as a leader of our local research software community.

Discussion session

While we had prepared some questions for the panellists, we also allowed the discussion to flow organically. Members of the audience were also invited to ask their own questions. This is a summary of some of these questions and the panel’s responses.

How is national infrastructure organised in the UK, and how does it compare to the Netherlands?***

Jeremy: While there is national research computing infrastructure in the UK, many resources are also provided at a local or regional level. There is wide use of commercial services for managing code, such as GitHub. I feel that there is scope for improvement in the context of providing coordinated research software infrastructure at a national level but it can be difficult to assess the value of this and how to make it available. A key discussion point is who pays for such infrastructure and how? And how do we make sure people have equal access to these resources particularly when they are paid services?

Matthijs: at SURF we provide national infrastructure. There are technical difficulties like federation (Ed: sharing resources across multiple organisations), but we probably need to revisit some of these limitations and see what is possible as technology changes.

John: One thing we notice is that the barrier to external contributors is high with our local GitLab instance — working on GitHub makes it easier for astronomers around the world to collaborate, and some of our collaborators are based around the world, in places such as the USA and China.

Audience member #1: National infrastructure also creates fragmentation. Maybe we need a roadmap / strategy, not only infrastructure.

In your opinion, which infrastructures are missing, that you/your organisation would benefit if they were available?

This question sparked a short discussion and several audience members contributed to it: One audience member said that guidance on HPC resources is very valuable. They have HPC resources available from SURF (Lisa, Research Cloud, etc.) but still need some guidance on when to use which one. Another audience member experienced SURF as an invaluable resource for navigating the different solutions available for their research needs. Another point that was raised were generic skills that would help all researchers working with high computational needs, like basic GPU and CPU programing. The audience member mentioned that there is a hurdle to getting to know all these systems, while, at the same time, a bit of speedup in your results is indeed an invaluable resource.

How about the code itself? Are there quality criteria applicable to all code, making it FAIR and sustainable? Should we educate all researchers?

Jeremy: Not every researcher should be a computer scientist, but some level of understanding on code quality and how to think about writing your code is still valuable. For example, how you think about the problem. Software being considered a first class output would make researchers see more value in producing good code. Architecture of code makes code more sustainable. Good quality documentation should provide the information users will need. Documenting decisions on choices of libraries used is also useful.

John: At ASTRON there are some practices we follow to make code better, and we should do more. We have professional engineers who are not (necessarily) astronomers. In this group we encourage good practices like code reviews. I am curious about using Sonarqube (Ed: an open-source platform for continuous analysis of code quality), but just playing with it at the moment. One thing that resonated from what Jeremy said, is that there are astronomers who get really involved in developing code (such as astropy), but it does not get recognition — it will not get you tenure. This is a problem for the sustainability of astropy. And we do not have funding models for it, such as turning packages into part of the infrastructure.

Are we creating walls around science? “To be an astronomer you need to know a long list of things”?

John: You do need to have some level of knowledge. We have been using things like mentoring or google summer of code to develop some of these skills on people who use them.

Audience: what I find interesting are the collaborations for example in digital humanities: computer scientists & historians, more collaboration is the way to go.

It is in a way similar to a chemistry lab: at the lab you have materials available, and protocols you need to follow to use those materials. Researchers get trained on how to use those materials, but they do not need to build them themselves. Research software is similar to those materials, researchers need to be trained on how to use them, but they can work together with research software engineers that can help them build those materials.

Wrap-up

Most important points/Key Message

During our panel session on infrastructure for sustainable research software, we discussed several services for tracking changes, sharing computational environments and high performance computing. Interestingly, the discussion quickly moved from technical solutions towards a more broader view of what constitutes infrastructure:

The panel and audience agreed that it should be acknowledged that infrastructure is not only facilities and services, but that people, and in some cases software, can also be a critical part of infrastructure.

For the panel session itself, we cannot say that the panellists and audience agreed on which services exactly are critical or absolutely essential to be offered at national level. What the audience and panellists agreed on was that we need a national strategy or roadmap to organise infrastructure in a way that is technically sound, as well as useful for the end users. And not just the technical infrastructure, but also the organisational infrastructure and personnel to provide better support to researchers by providing consultancy, training and implementation. Furthermore, everyone in the room saw the need to include the points of view from a diverse set of users when deciding on the infrastructure. Not all users will want or need to have a say in what type of services are offered, but they will want to be informed on how to use and access those services when needed.

The Utopic Git History

Abel Soares Siqueira — Wed, 25 Oct 2023 00:00:00 GMT

If you love this article and can’t wait to read more about The Utopic Git History, please see this follow-up article by Abel.

A quick way to create discord among a group of developers is to discuss Git. More specifically, whether to merge, rebase, or squash. There are many arguments around this, simply because it involves opinions on whether the end result is good or bad. Some of these arguments verge from bad experiences with extreme cases. The most common case that I encounter is the dreadful huge pull request**.

“Mm… can you review this for me?” — Cthulhu (Generated by DALL-E, modified by me)A very large pull request is one containing changes in many files, possibly with many commits. Even if best practices are followed — which we will discuss here — it is a big endeavor to review such a pull request. If best practices are not followed, then it is a nightmare, and the end result is ugly. Some common issues with the result are:

Sequential commits that seem unrelated.
Many commits “fixing” the same thing.
One huge (probably squashed) commit.

Let’s discuss how to avoid this issue, and touch on the merge vs rebase vs squash debate on the way there.

Target audience: This post should be useful for anyone interested in best practices with Git, or intermediate/advanced usage of Git. It also contains opinions on Git rebase and squash, so it might trigger some people.

UGH — Utopic Git History

Note: UGH should be pronounced with disdain and eye-rolling.

The main idea behind what I will call UGH workflow is **atomic git commits. **This is not a new subject, you can find a few blog posts about it — in fact, I just found out about the Git Legit talk by Pauline Vos and I strongly recommend watching it.

If you haven’t heard of atomic git commits, let me give you a short introduction. The basic idea of an atomic git commit is that it contains the shortest amount of work that does what it is supposed to do. That is, less work would not be enough to do the task, and more work would be going out of scope.

Pauline describes three features of an atomic git commit, the first is what I just described. I normally like to think of the other two separately because they are good practices that we should follow even if we don’t do atomic git commits: (i) make sure that new commits do not break the code and (ii) have a clear and concise commit message. However, I understand the need to explicitly mention them, so that’s why I just explicitly mentioned them as well.

If every commit is an atomic commit, then we have a beautiful git history. Every commit passes tests, so we can easily navigate around. Every commit has a non-creeping purpose, so we can cherry-pick them more easily. Every commit is self-contained, so we can revert them trivially. It is almost utopic, but it is actually attainable.

If a pull request made of atomic git commits passes our way, we can review each commit individually, making our life easier. Then, what is the end result of our three merging strategies (as given by GitHub)?

Merge commit

A merge commit will be a new commit with a message like “Merging PR #123 from branch featureX”.

Visualization of Git merge with a new commit. (https://doi.org/10.5281/zenodo.8032313)Here is an example from a real GitHub repository:

A main branch up top and many branches that were merged with merge commit. Source: GitHub network graph of https://github.com/abelsiqueira/perprof-py.Personally, I don’t think that this extra commit is very useful. Even when all commits in the branch are pristine, passing, and having a great message, we still have this “parallel” development, which does not add any useful information. Furthermore, the merge commits themselves are not atomic.

Squash

Squashing is simply creating a single commit with the content of all commits, and rebasing the result.

Visualization of merging with squash. (https://doi.org/10.5281/zenodo.8032313)The default commit message will change the title of the commit to something like “Title of the pull request (#pr)”, and then the old commit messages will be aggregated into a single commit message. This is a problem because now we have taken all atomic git commits and thrown their atomicity away.

The squash frequently comes into play when the pull request **does not **contain atomic git commits. Then it might be better to squash and have a single commit than a larger mess. If the resulting commit is atomic, then that is great. Otherwise, we have scope creep.

Rebase and don’t squash

Rebasing without squashing means rebasing the current branch on top of the current main and merging with fast-forward, i.e., not creating any new commit and simply updating main’s pointer.

Visualization of Git rebase. (https://doi.org/10.5281/zenodo.8032313)Here is an example from the same GitHub repository, from a later date:

Main branch and no other branches. Source: GitHub network graph of https://github.com/abelsiqueira/perprof-py.The end result contains only main. There is no information that there ever was a feature branch.

This strategy was shown for last because it is my preference, but also because it is the only one of the three that will consistently keep the commits atomic — assuming that we have achieved a pull request with atomic commits.

How to achieve atomic git commits

Here are some tips to make pull requests with atomic git commits:

If there are too many commits, use git rebase -i to reorder, fix, squash or edit commits.
If there is uncommitted work that touches more than one topic, use git add -p to add the parts that make sense for a single commit.
If there are commits from different scopes, rebase into new branches.
Use git commit --amend to fix the current commit.
Use git push --force to update the branch online (the main branch can be protected so main is not accidentally rewritten).
Use a pre-commit hook to avoid having to create extra “fix linter” commits.
Be prescient and write perfect commits — but you knew that already ;)

Once again I recommend Pauline’s Git Legit talk for more detailed examples but feel free to leave a comment here if you think of a situation that needs more care.

More utopia, please

The second part of the UGH workflow is to do the same for branches, and therefore, pull requests. In other words, the pull request must have the least amount of commits necessary to do what it is supposed to do. You can read a blog post about this by Fagner Brack.

If each commit tells a story, and they are in an order that makes sense, reviewing the pull request is much easier. Even if the pull request is still huge, it can be reviewed one commit at a time.

When reviewing, or receiving reviews, we should be vigilant of scope creep. The PR should do only what it is supposed to do.

But what is the PR supposed to do?

This is the harder part because it is not just git anymore. There are different approaches to determining what is a PR, but the one I have the most experience with is to relate them with issues. In other words, the issue defines the topic, or scope, of the PR. Following this approach, an atomic definition would be:

Each pull request should close one, and only one issue.
Each issue should be small enough to not creep into more than one topic.

Clarification: not every issue leads to a pull request, but every pull request requires an issue.

To achieve this, one has to

Break issues into smaller issues.
Create new issues for every desired change.
Merge incomplete issues and close duplicates.

The actual size of the issue depends on how the team works, and who creates the issues. Some issues are user stories that involve many tasks. Some are bug reports. Some are tech-debt opportunities. Some are users’ questions.

Issues that involve many tasks are sometimes called epics. *One way of dealing with epic issues is to create separate issues for each task — see the next section — but you could just as well have a larger pull request with one commit for each task. There is no one-size-fits-all solution.

I find that it helps a lot to work on an epic issue as a whole, then split it into smaller parts a posteriori. Then I make separate branches and create pull requests one at a time. One shortcoming of this approach is that if the reviewer makes big suggestions to one pull request, the other branches might be heavily affected.

Extreme UGH

What if we take these atomic definitions to the extreme?

One commit per pull request.

Since the issue is minimal, then it should be fixable with a single commit, right? Otherwise, the issue would not be minimal.

Why is this extreme?

The reviewer and the developer must share the context. For instance, there might be dependencies between pull requests by the same developer. If the reviewer does not have that clear, then the pull request can actually be harder to understand than a single larger PR.
It can be hard to know beforehand how to break an epic issue. Sometimes we only know how to break an issue after working on it. But what if we are not the ones responsible for updating the issue? Then we need to create a demand for the issue to be broken and wait until work can be restarted.
Sometimes the code base is not modular enough. We might need API changes for a different part of the code. Scope creep might be unavoidable due to technical debt. If there is time, though, trying to pay this debt as a separate issue might be better.

Be aware of these limitations. Trying to achieve UGH can lead to loss of friends, or worst, productivity (this is a joke, managers).

Self-inflicted UGH

A good way of starting using UGH is to follow the workflow on your own, before trying to enforce it on the team.

Make sure that your pull requests are atomic, even if nobody is enforcing UGH.
As long as you’re the only one touching your branch, force-pushing won’t break anybody’s work.
When you have to update your pull request because of conflicts with main, you can rebase your branch instead.

This helps you practice and might help your team to see the benefits.

Finally, I chose the name Utopic Git History (UGH) to remind us of two things:

It can be very obnoxious to insist on the workflow (hence the ugh sound), especially when reviewing pull requests.
It is utopic in the sense of being unattainable in practice (for extended periods of time). Take parts of the idea and test them.

The best way to achieve UGH is to create a good culture first.

I actually don’t have any project that fully follows the UGH workflow, but I try to follow it with my personal commits, and I use rebase in many repositories. I spent a decent chunk of time helping people to update their workspace after their pull request was rebased (sometimes deleting everything and starting over was the chosen solution). It is not trivial to adopt this strategy for the whole project, but I can certainly say that it helped me learn more about Git.

Let us know if you (try to) use UGH or similar for your projects.

Many thanks to our proofreaders and reviewers, Sander van Rijn, Patrick Bos, Pablo Rodríguez-Sánchez, and Candace Makeda Moore.

Achieving Utopic Git History

Abel Soares Siqueira — Thu, 19 Oct 2023 00:00:00 GMT

AKA being UGHly

In a previous blog post, we defined The Utopic Git History (UGH). To refresh our memory, a simple definition of a UGH is

Atomic git commits (smallest commit that completes the task)
Passing tests
Linear Git history

In that post, I mentioned that the easiest way to achieve UGH is to squash a self-contained pull request into a single commit. Otherwise, you have to use more advanced git commands to achieve UGH. This blog post deals with that.

The target audience for this post is Git users who want to learn intermediate and advanced commands that splice, merge, and fix commits. If you need a refresher on some of the concepts used, please take a look at the animations in my previous blog post.

Introduction

The ideal way of achieving UGH is to write exactly what you have to write, and then commit that. This kind of insight is hard to come by when your task is large, so instead let’s focus on what happens in practice, which involves getting our hands dirty.

Getting your hands dirty. Left photo by Jeremy Bishop on Unsplash. Modifications by me.

Before committing

The first thing that we can control is what to commit. So let’s assume that we got to a point where we have a lot of modifications because task A turned out to be too large and now we’re thinking of splitting it into A1 and A2 (maybe more).

Here, git add -p comes into play. git add -p lets us add patches of code separately. So if our file has modifications for tasks A1 and A2, we can add only the first one.

Tip: If there is an untracked file that we want to selectively add, it can be included using git add -N FILE. This will tell git to track the file, but it doesn’t stage it.

When we run git add -p FILES, we see something like

diff ...
...
- Something
+ Something else
...
(1/n) Stage this hunk [y,n,q,a,d,s,e,?]?

Pressing ? will show the help. There are four commands that matter a lot to us. The first two are y and n, for yes and no. If we say yes, the “hunk” (a piece of code shown in this diff) will be staged. Remember, when we git commit (no -a flag), the staged part of the code will be added to the commit. If we say no, then it just moves on to the next hunk.

The third command is the best: s, for split. If the hunk that we see on the screen is too large, or if it involves parts A1 and A2, then we want to split it. Git will try to find a place to split this code and give you smaller hunks. Then we can decide whether to add or not those new smaller hunks individually.

Split doesn’t always work, so what we can do instead is use the command e, for edit. Editing will open an editor, so if you don’t like Vim (the default), you can change it with

git config --local core.editor "code --wait"

In the command above, code --wait will open VSCode. Change it to your preference. Also, you can configure it globally using --global instead of --local.

After running git add -p and selecting the command e, we will get to the editor with the same diff open and some additional information.

The first column of the hunk can be empty, to indicate that the code doesn’t change, # to indicate a comment that will be ignored by git add, a - to indicate that the line should be removed, and a + to indicate a new line replacing the old one.

Now, we can manually decide on each individual addition. Let’s look at an example:

Our initial code is

x = 1
y = 2
z = x + y
w = x + 1

Now, our task A1 is to rename y to p, and task A2 is to change all additions to subtractions.

The resulting code after doing both tasks is

x = 1
p = 2
z = x - p
w = x - 1

The diff, which we will see when we enter edit mode, is the following:

# Manual hunk edit mode
 x = 1
-y = 2
-z = x + y
-w = x + 1
+p = 2
+z = x - p
+w = x - 1

Clearly, tasks A1 and A2 are mixed and can’t be split. So instead, we manually change the diff. Since we want only A1 applied, we change the diff to

x = 1
-y = 2
-z = x + y
+p = 2
+z = x + p
 w = x + 1

In other words,

Change -w = x + 1 to [space]w = x + 1
Remove line +w = x - 1
Move the two + lines one line up
Change +z = x - p to +z = x + p

Please notice that the lines starting with a minus sign can’t be modified, because they must match with the current code. If you want to make modifications to a line that has a minus, look for the corresponding plus and change that instead. If there is no corresponding plus, it means that the line is simply being removed, so you can add a line with a plus yourself.

Now, if we look at your git status, we will see the same file appearing in and out of the staged section. We can use git diff --staged to check the result.

Oops, too late (one commit with multiple tasks)

Let’s say that we have one commit with multiple tasks, i.e., one commit that should be split into more than one.

For instance, we might have spent 10 minutes meticulously editing what should be staged using the git add -p command, when by mistake we use git commit -a and add every unstaged modification in the commit as well, wasting all this work. Completely hypothetical, mind you.

There is a quick way to “fix” this which is to run git reset HEAD~1. This will remove the commit and go back to the modified state. To be clear, we don’t lose the modifications made in the commit, as they will be back into the modified state, but we lose the previously staged information in those 10 minutes of work.

The git reset HEAD~1 command assumes that the latest commit is the one that we are fixing. If that is not the case, check the next session.

As a rule, always back up your branch whenever you run reset and rebase until you are confident that you can get yourself out of tricky situations. Backup, in this case, means creating a new branch, e.g., git branch backup-some-name.

Too many commits

Let’s say that we ended up making too many commits. For instance, we were trying to follow the idea above, but noticed some bugs that should be part of task A1, and ended up creating another commit to fix that. Or that we realized that we introduced a bug that had no test when doing A2.

As an example, consider the following initial state of a file main.py:

def foo():
    x = 1
    y = 2
    z = x + y
    w = x + 1

    return w

def bar():
    y = 1
    return y + foo()

And the following git history:

hash00: Older stuff ← Where we are right now
hash01: Rename variable y to p
hash02: Change + to -
hash03: Rename y to p and change + to — in function bar also
hash04: Fix linting issue (add spaces around operators)

The hash01 and hash02 commits are the ones we did in the previous sections. The hash03 commit is

def bar():
-    y = 1
-    return y + foo()
+    p = 1
+    return p -foo()

And the hash04 commit is

def bar():
     p = 1
-    return p -foo()
+    return p - foo()

This git history is not great. We still only have two tasks (A1 and A2), but 4 commits. What we want to do is redistribute the work done in these 4 commits into 2 commits. For that, we use git rebase -i, i.e., interactive rebase.

Note: There is an alternative to rebasing here, which is to run reset hash00, so that we remove all the commits but none of the modifications. From there we will have to manually add the changes using git add -p and then make individual commits. I prefer the rebase strategy. What are your thoughts about these two strategies?

The interactive rebase command allows us to reorder and squash commits, change the commit messages, and stop during rebasing to make additional changes (such as splitting older commits).

The first thing that we need to do is find that git history. If you use a visual tool, you can just look at the graph. Otherwise, we can issue the command

git log --graph --oneline

This will produce a graph of the commits, one per line, which should be easy to read. It should be something like

* hash04: (feat_br) Fix linting issue (add spaces around operators)
* hash03: Rename y to p and change + to - in function bar also
* hash02: Change + to -
* hash01: Rename y to p
* hash00: (main) Older stuff

This assumes that we branched out of main into a branch called feat_br. Now, to start rebasing, we can issue

git rebase -i main feat_br

In other words, it is just the normal rebase command, but now we add the -i flag to have more options on how to rebase.

This command should open an editor with the list of commits preceded by a keyword pick:

pick hash01 Rename y to p
pick hash02 Change + to -
pick hash03 Rename y to p and change + to - in function bar also
pick hash04 Fix linting issue (add spaces around operators)

Notice that the commit at main is not included, as expected.

What we can do now is change the order of these commits as we see fit and change the keyword to the action that we want. Notice that the order of the commits is top-to-bottom.

The main options are:

pick: Use this commit. It will be applied, and the message will be used.
drop: Don’t use the commit. It will be lost.
reword: Use this commit, but open the editor to modify the commit message.
squash: Merge the contents of this commit in the previous commit and then open the editor for editing the commit message. By default, it appends the commit messages of squashed commits.
fixup: Like squash, merge the contents into the previous commit, but ignore the message.
edit: Use this commit and the message, but after the commit is done, stop so we can make edits.
break: Use by itself in a new line. It will drop us in the terminal to do whatever we want.
exec: Use by itself followed by a command. It will run the command, and if there is an error, drops us in the terminal to do whatever we want.

Whenever we are dropped in the terminal during rebase, we can always issue git rebase --abort to stop the rebase completely and go back to where we were.

If we were dropped in the terminal because there is a problem, for instance, a conflict, then we must fix it.

After fixing whatever caused us to be dropped into the terminal (check with git status ) we have to use git rebase --continue to continue rebasing.

Going back to our example, the first thing that we’ll do is split the hash03 commit. To do that, we simply change we pick to edit, save and close.

pick hash01 Rename y to p
pick hash02 Change + to -
edit hash03 Rename y to p and change + to - in function bar also
pick hash04 Fix linting issue (add spaces around operators)

This will drop us in the terminal with

Stopped at hash03...  Rename y to p and change + to - in function bar also
You can amend the commit now, with

  git commit --amend 

Once you are satisfied with your changes, run

  git rebase --continue

Now, we will use reset to undo this single commit and add only the change for task A1.

git reset HEAD~1
git add -p

Make the first commit diff look like

def bar():
-    y = 1
-    return y + foo()
+    p = 1
+    return p + foo()

And the second looks like

def bar():
     p = 1
-    return p + foo()
+    return p -foo()

Now we can run git rebase --continue and git log --graph --oneline again to obtain:

* newh03: (feat_br) Fix linting issue (add spaces around operators)
* newh02: Fixing task A2
* newh01: Fixing task A1
* hash02: Change + to -
* hash01: Rename y to p
* hash00: (main) Older stuff

Notice that the hashes changed only after we effectively rebased.

Now, we will use git rebase -i main feat_br again, but this time we will move things around and change pick to fixup.

We start with

pick hash01 Rename y to p
pick hash02 Change + to -
pick newh01 Fixing task A1
pick newh02 Fixing task A2
pick newh03 Fix linting issue (add spaces around operators)

And change to

pick hash01 Rename y to p
fixup newh01 Fixing task A1
pick hash02 Change + to -
fixup newh02 Fixing task A2
fixup newh03 Fix linting issue (add spaces around operators)

This will apply the first commit and then fix it, leaving only a single commit. Then, it will apply the second commit and fix it twice. None of these changes led to conflicts.

At the end of this process, our log is

* more02: (feat_br) Change + to -
* more01: Rename y to p
* hash00: (main) Older stuff

In other words, we have achieved UGH for this branch.

Splitting a branch into many branches

Now that we have two commits, one for each task, it makes sense to actually have two branches: one for each commit.

Since the commits are atomic, this is actually very easy to do unless the branches depend on each other.

The first task is just the first commit in our case, so we only need to give it a branch name:

git branch feature_taskA more01

In other words, we are creating a branch called feature_taskA at the commit with hash more01. Push this branch

git push -u origin feature_taskA

It can be reviewed and rebased.

Now, let’s split into two situations.

Splitting independent branches

If the commits are independent, we just have so now where they start and rebase rooted there.

Normally, we are rooted in main, as in our example above. So we can issue

git rebase --onto=main feature_taskA feat_br

Here, we are saying “Take all commits after feature_taskA (exclusive) that are part of branch feat_br, and reapply them on top of the branch main.” We can use a hash instead of main, if necessary.

This can lead to conflicts, even if there is no real dependency between the commits. If the conflicts are significant, then it is better to treat them as dependents. The conflicts that appear here will reappear when we try to merge these two branches into main.

Splitting dependent branches

If the branches are dependent, then we should actually wait until the first part is merged before rebasing.

When the first part is merged (let’s assume with GitHub’s rebase), then we now have a new main, with other commit hashes. In that case, we can simply issue a rebase and the commits that already appeared on the merged branch will be skipped, by default.

However, this depends on configuration values, so it is safer to just use exactly the same commands as before, but don’t forget to update your main branch.

git switch main
git fetch --all
git merge --ff-only upstream/main
git rebase --onto=main feature_taskA feat_br

In other words, the only difference between the independent and dependent cases is when to rebase. If the branches are independent, you can split and push both, since they can be reviewed and merged independently. If the branches are dependent, then we wait until the first branch is merged and rebase after updating main.

Remote changes

The last thing to discuss is the changes that could be happening in the remotes while we’re making changes.

Let’s assume in this section that we have two remotes:

upstream: Points to the main repo, e.g., the organization’s repo.
origin: Points to our fork.

First of all, my preferred way to work is to make sure that my atomic commit is up to date with the remote branch. So first of all, I fetch everything, update main with the fast-forward option, and branch from main.

git fetch --all
git switch main
git merge --ff-only upstream/main
git switch -c featureX

If the merge had conflicts, it means that we made commits to main, and we shouldn’t. We fix this by creating the branch now and resetting main:

git switch main
git branch featureX # just creates, does not switch
git reset --hard upstream/main

Now, after working on branch featureX, we check for updates on main again, using fetch and merge --ff-only. If there were any changes to main, we rebase after pushing:

git rebase main featureX

We might have conflicts, but since our branch is atomic, they should be few and far between.

After rebasing, we push:

git push -u origin featureX

Now, let’s say that we had to make changes because of the review. After making our changes, we use interactive rebase to make sure that the commits are atomic again. Our branch will not be in sync with the remote branch anymore, because the commits have changed (either in content or in hash).

That means that we now have to force-push our branch:

git push --force featureX

This should be enough to update the pull request.

Final remarks

These commands should help you handle most git situations. They are not easy to master, so I hope that this post serves as a quick way to remember what each command does. Let us know what other Git needs you have.

Be UGHly, my friend.

Many thanks to our proofreaders and reviewers Barbara Vreede and Pablo Rodríguez-Sánchez.

the orange3 data mining platform as a research tool

Kody Moodley — Thu, 12 Oct 2023 00:00:00 GMT

Why aren’t more SSH researchers using and extending the platform?

Orange is an open-source Python -based data mining application with a graphical user interface (GUI). It allows people with no knowledge of programming to apply Machine Learning as well as advanced data processing, analysis, and visualization using a point-and-click, drag-and-drop interface. Those who have seen Scratch will be familiar with this mode of interaction, although Orange is more advanced and is used for data mining, rather than educational programming. The following screenshot shows what a typical Orange workflow looks like.

Screenshot of an example data mining workflow in Orange3

The three small circles labeled ‘File’, ‘Data Table’, and ‘Scatter Plot’ in the image above, are called Orange widgets. They are the basic building blocks of a data analysis workflow (or pipeline) in Orange. Each widget is a software unit that performs some sort of data processing, analysis, or visualization and potentially has a set of inputs and outputs. You can connect the outputs of some widgets to the inputs of others to create a chain of processing and analysis operations. A great feature of Orange is its extensibility. There are loads of custom plugins for Orange (called add-ons) developed by researchers and the Orange user community. Add-ons usually comprise multiple widgets. Two mature and extensively used Orange add-ons are Orange-Text and Network which are used for text mining and network analysis respectively.

Story Navigator: a custom Orange add-on I am developing

Screenshot of the Story Navigator add-on (not yet released) for Orange3.

The add-on works by first importing textual stories into the workflow using the existing “Import Documents” widget from the Orange-Text add-on. Thereafter the researcher can choose a widget from the add-on for analysing some aspects of the imported stories. The actor analysis widget, for example, highlights potential characters in the stories; shows which kinds of actions (verbs) the characters are associated with; and calculates different measures for how central specific characters are to the stories.

Quantitative story analysis widgets from the Story Navigator add-on for Orange3.

While it did take some time to develop the widgets for this add-on, what surprised me is that it took me only about two hours, from scratch, with no prior knowledge of Orange, to develop my first widget that showed up in the Orange3 interface when I started it up. Granted, the widget did not do anything except display. But the return on my two-hour investment was that I could now free up my creativity to develop whatever functionality I wanted in the widget, without having to create my own GUI. In doing so, I got to take advantage of a myriad of benefits including being able to:

Make my add-on available for anyone to use regardless of their programming knowledge
Receive development and usage feedback from an existing Orange community about my add-on (without having to further promote the add-on or build another community myself from scratch)
Demo my plugin to anyone, thanks to the GUI
Share data analyses made with my add-on with other Orange users — enabling reproducibility (currently undergoing a crisis in science)
Forget about writing installation instructions — they’re already here!

Despite what some readers might think at this point, I am not an Orange evangelist. In fact, I am not swooning over the capabilities and potential of Orange specifically. Orange does have competing platforms that try to do similar things such as KNIME and WEKA which are both Java -based and therefore inherit some concerns due to Java’s declining popularity. Orange itself also has some clear disadvantages, not least of which are the scalability issues and the add-on development documentation (the latter is not very accessible for less experienced developers).

What the popular video game Elder Scrolls has to do with Orange

Bethesda Softworks is a US-based video game development company that is famous for its many franchises, one of which is “ Elder Scrolls ”. The games in this franchise are “ RPGs ” that put the player in control of a character from a fictional fantasy world with the goal to… wait for it… save the world from destruction (if you’re new to the term RPGs imagine a video game based on Game of Thrones and you’ll get the gist). In 2002, Bethesda released its third installment of Elder Scrolls, called “ Morrowind ”. The super interesting part is that they also released software called the Elder Scrolls Construction Set alongside the game. This software allowed players to make customizations to their own copy of the game. They could do things such as create new clothing, buildings, and other items; create their own character voices; make new quests, and customize the color of the sky. These modifications (mods as they are now commonly referred to) vary in terms of how much software literacy is required in order to build them, but many of them require no more knowledge than being able to install a program and click a few buttons.

Fancy introducing Spiderman-suited creatures crawling around a medieval town into your video-gaming experience? Well, someone has already created a mod for this in Skyrim.

Twenty years later, and Bethesda has become known and celebrated for giving its customers the freedom to customize their gaming experience. Nexusmods is one of the prominent hubs for hosting large collections of player-created mods for games by Bethesda and other major video game companies. I quote some jaw-dropping statistics from this site here:

“We host 477,145 mods for 2,342 games from 118,005 authors serving 41,902,274 members with 8,184,840,124 downloads to date.”

Nexusmods demonstrates that the mod concept is a winner.

“Okay Kody, what does this all have to do with Orange?”, you may ask. Well, in some ways there are parallels in this story to the concept of Orange as a software platform.

Second, there is the enabling of customization. In Bethesda’s case, it is the provision of customization software such as the Skyrim Creation Kit; in the case of Orange, it is the release of the code under an open-source license with reusable example code and plugin-writing tutorials.

Thirdly, and what I find to be the most critical, is making it easy to customize. In Bethesda’s case, they have made user-friendly software that does not require extremely specialized skills to use. Almost anyone can create a mod. In Orange’s case, you still need to know how to program to develop add-ons. But you don’t need advanced knowledge of the GUI frameworks that Orange uses — Qt and PyQt. In other words, if you know how to create any kind of computational script or software in Python, you can easily create an Orange widget to “house” or “wrap” its functionality. Furthermore, others, regardless of their technical nous, will be able to use it and “chain” your widget with other software in an Orange workflow.

If you are interested, my colleague Ji Qi has written a post about how to get started with Orange add-on development — be sure to check it out!

Take home messages

Orange is an open-source extensible data mining platform that is usable by people with little to no technical experience. If you are a researcher in SSH consider using it for your research. If it doesn’t have a feature you need, ask the developers if they can add it as a default feature to Orange. If they can’t add it, and if you or a colleague has Python experience, consider creating a custom add-on. It is surprisingly quick and easy to create one if you know a bit of Python.
Research software sustainability is a challenge. Building and maintaining communities around the development of research software is hard. Don’t let your software become JAG (J ust- A nother- G ithub repo). Take inspiration for how to avoid this from platforms like Orange. This does not mean that you need to create a graphical user interface for your software. It means: find a home for your software or find an existing toolbox in which to integrate your software, before creating your own toolbox. Integrating into existing platforms or packages will probably increase its usage and visibility over publishing as a stand-alone item.
Inspiration point #1: From the start, consider building software that solves generic problems in your field. i.e., problems that are shared by others in the field as well, rather than for highly specialized use cases.
Inspiration point #2: Make your software easy to use by as many types of people as possible. Not just for experienced developers. Also, ensure that someone can rapidly understand what your software does.
Inspiration point #3: Make it easy for as many people as possible to customize and extend your software.

Analysis-ready climate data with ESMValCore

Peter Kalverla — Thu, 05 Oct 2023 00:00:00 GMT

The “Easy IPCC” blog done differently

Co-authored by Bouwe Andela

In a recent blog post, Tom Nicholas presented a nice walkthrough of reproducing a famous IPCC figure with [xarray-datatree](https://xarray-datatree.readthedocs.io/en/latest/). Here, we attempt to achieve the same with ESMValCore — the workflow engine that powers the more widely known ESMValTool. Both xarray-datatree and ESMValCore facilitate the analysis of large climate datasets, but our approach is quite different. For example, xarray-datatree tries to facilitate any hierarchical file structure (more generic), whereas ESMValCore focuses on facilitating CMIP, or CMIP-like data only (more specific, thus tailored). Seeing them side by side is food for thought.

“Adam uploads*” by Henning Falk, ©2022 NumFOCUS, is used under a CC BY 4.0 license.*In case you are not familiar: ESMValTool is a software project that was designed by and for climate scientists to evaluate CMIP data in a standardized and reproducible manner (check out a recent blog here). Several years ago, the project was split in two parts: ESMValCore implements the core functionality, whereas ESMValTool is essentially a big collection of “recipes” to reproduce a wide range of CMIP analytics — including some of the IPCC figures. While ESMValCore has matured as a reliable foundation for the ESMValTool, recent additions also make it attractive as a lightweight alternative to its hefty sibling.

To give you an impression of ESMValCore’s look and feel, we start with a complete code block that exactly reproduces Tom’s figure. In the remainder of this post, we’ll dive into the details: the dataset interface, automatic ESGF downloads, CMOR checks and fixes, and exporting as recipe, meanwhile touching upon some of the similarities and differences with xarray-datatree. At the end of the blogpost we’ll build an ESMValTool recipe that reproduces the same figure with all available models and ensemble members.

import cf_units import matplotlib.pyplot as plt from iris import quickplot

from esmvalcore.config import CFG from esmvalcore.dataset import Dataset from esmvalcore.preprocessor import annual_statistics, anomalies, area_statistics

Settings for automatic ESGF search

CFG[‘search_esgf’] = ‘when_missing’

template = Dataset( short_name=‘tos’, mip=‘Omon’, project=‘CMIP6’, exp= '', # We’ll fill this below dataset='', # We’ll fill this below ensemble=‘r4i1p1f1’, grid=‘gn’, )

Substitute data sources and experiments

datasets = [] for dataset_id in [“CESM2”, “MPI-ESM1-2-LR”, “IPSL-CM6A-LR”]: for experiment_id in [‘ssp126’, ‘ssp585’]: dataset = template.copy(dataset=dataset_id, exp=[‘historical’, experiment_id]) dataset.add_supplementary(short_name=‘areacello’, mip=‘Ofx’, exp=‘historical’) dataset.augment_facets() datasets.append(dataset)

Set the reference period for anomalies

reference_period = { “start_year”: 1950, “start_month”: 1, “start_day”: 1, “end_year”: 1979, “end_month”: 12, “end_day”: 31, }

(Down)load, pre-process, and plot the cubes

for dataset in datasets: cube = dataset.load() cube = area_statistics(cube, operator=‘mean’) cube = anomalies(cube, reference=reference_period, period=‘month’) # notice ‘month’ cube = annual_statistics(cube, operator=‘mean’) cube.convert_units(‘degrees_C’)

# Make sure all datasets use the same calendar for plotting
tcoord = cube.coord('time')
tcoord.units = cf_units.Unit(tcoord.units.origin, calendar='gregorian')

# Plot
quickplot.plot(cube, label=f"{dataset['dataset']} - {dataset['exp']}")

Show the plot

plt.legend() plt.show() The famous IPCC figure from Tom’s blogpost reproduced with ESMValCore.Cool! This looks exactly like the figure in Tom’s post 🎉. With roughly the same amount of code. If you haven’t read Tom’s post, this figure shows the historical and projected rise in sea-surface temperature from different climate models and future emission scenario’s.

Key takeaway **🔑

If there is one thing to note from this little script, it is how “scientifically meaningful” it is. For example, we never got to see actual files. We simply defined our datasets with meaningful facets, as described in the CMIP controlled vocabulary. In the background, ESMValCore located the corresponding files on ESGF, downloaded them to our local environment (if we didn’t have them yet), loaded them into memory, checked adherence to the CMOR conventions, and applied fixes for known issues. But those are technical details, not science.

ESMValCore, in the spirit of ESMValTool, focuses very much on what to do, instead of how.

The inner workings ⚙️

It’s worth dissecting *how *ESMValCore does things, though. Firstly because you’ll need some understanding to wield it effectively. But mostly because we scientists are often trying to solve the same problems, so it’s interesting to see how ESMValCore tackles some of these issues so you don’t have to.

Configuration

ESMValCore works with a configuration file. This is where you store information about, for example, how your data is organized on disk. The defaults are pretty good, we only had to enable automatic downloads. Let’s have a look at some of the other settings.

print({key: CFG[key] ** for key in [‘download_dir’, ‘drs’, ‘output_dir’, ‘rootpath’] }){ ‘download_dir’: PosixPath(‘/home/peter/climate_data’), ‘drs’: {‘CMIP3’: ‘ESGF’, ‘CMIP5’: ‘ESGF’, ‘CMIP6’: ‘ESGF’, ‘CORDEX’: ‘ESGF’, ‘obs4MIPs’: ‘ESGF’}, ‘output_dir’: PosixPath(‘/home/peter/esmvaltool_output’), ‘rootpath’: {‘default’: [PosixPath(‘/home/peter/climate_data’)]} }The default configuration specifies that data will be downloaded to a folder called climate data in my home directory. The rootpath setting specifies all folders where ESMValCore should look for data, and it is set to the same default. Finally, ESMValCore defines a dedicated folder to store output. Every ESMValTool run or session will get its own subdirectory in there.

The key point here is that you don’t have to worry about data management. You can configure some folders if you want to, but even that is not strictly necessary. If you’re working on dedicated climate data infrastructure such as Jasmin or DKRZ, you can also configure ESMValCore to use the Data Reference Syntax (DRS, the filenames + folder structure) used on these machines.

Data discovery **🔍

Let’s have a closer look at the dataset definition. What happened, for example, when we called augment_facets?

dataset = Dataset(** short_name=‘tos’, mip=‘Omon’, project=‘CMIP6’, exp=‘historical’, dataset=‘CESM2’, ensemble=‘r4i1p1f1’, grid=‘gn’, ) dataset.augment_facets() print(dataset)Dataset: {‘dataset’: ‘CESM2’, ‘project’: ‘CMIP6’, ‘mip’: ‘Omon’, ‘short_name’: ‘tos’, ‘activity’: ‘CMIP’, ‘ensemble’: ‘r4i1p1f1’, ‘exp’: ‘historical’, ‘frequency’: ‘mon’, ‘grid’: ‘gn’, ‘institute’: [‘NCAR’], ‘long_name’: ‘Sea Surface Temperature’, ‘modeling_realm’: [‘ocean’], ‘original_short_name’: ‘tos’, ‘standard_name’: ‘sea_surface_temperature’, ‘units’: ‘degC’} session: ‘session-405b0393-0e28-4dae-836e-0d5961e3d30b_20230609_084132’ESMValCore auto-completed our Dataset based on the information it got from the CMOR tables. With this specification, ESMValCore can automatically search both your local filesystem and the ESGF for available copies of the dataset. Let’s make things a bit more interesting, and look for all available ensemble members. The from_files() method is used to build a list of available dataset based on the search fields.

CFG[‘search_esgf’] = ‘always’ dataset_search = Dataset( short_name=‘tos’, mip=‘Omon’, project=‘CMIP6’, exp=‘historical’, dataset=‘CESM2’, ensemble=’*’, grid=‘gn’, ) ensemble_datasets = list(dataset_search.from_files())

print([ds[‘ensemble’] for dataset in ensemble_datasets])[ ‘r10i1p1f1’, ‘r11i1p1f1’, ‘r1i1p1f1’, ‘r2i1p1f1’, ‘r3i1p1f1’, ‘r4i1p1f1’, ‘r5i1p1f1’, ‘r6i1p1f1’, ‘r7i1p1f1’, ‘r8i1p1f1’, ‘r9i1p1f1’ ]Apparently there’s 11 ensemble members available for this model.

The wildcard search functionality ( '*' ) is incredibly useful, as there are often inconsistencies in availability between datasets. Asking for ensemble member 1 through 11 may work for CESM2 today, but not for another model, or it may be different tomorrow.

What if we wanted to plot all *available data?

CFG[‘search_esgf’] = ‘always’ dataset_search= Dataset( short_name=‘tos’, mip=‘Omon’, project=‘CMIP6’, exp=[‘historical’, ‘ssp585’], dataset='', ensemble='', institute=’*’, grid=‘gn’, ) all_of_cmip = list(dataset_search.from_files()) print(len(all_of_cmip))821That’s a lot of datasets! We’ll save that for the end of the post.

Adding supplementary information**

To arrive at global mean temperature, we needed to calculate an area-weighted average. The data for ocean temperature comes at the native model grid, which is irregular and the grid cell area is not easily obtained. To address exactly this issue, the CF conventions include a specification of cell measures. ESMValCore supports this functionality and makes it easy to add cell measures to a dataset. In our case, we need the variable called areacello.

Add areacello as supplementary dataset

dataset.add_supplementary(short_name=‘areacello’, mip=‘Ofx’)

Autocomplete and inspect

dataset.augment_facets() print(dataset.summary())Dataset: {‘dataset’: ‘CESM2’, ‘project’: ‘CMIP6’, ‘mip’: ‘Omon’, ‘short_name’: ‘tos’, ‘activity’: ‘CMIP’, ‘ensemble’: ‘r4i1p1f1’, ‘exp’: ‘historical’, ‘frequency’: ‘mon’, ‘grid’: ‘gn’, ‘institute’: [‘NCAR’], ‘long_name’: ‘Sea Surface Temperature’, ‘modeling_realm’: [‘ocean’], ‘original_short_name’: ‘tos’, ‘standard_name’: ‘sea_surface_temperature’, ‘units’: ‘degC’} supplementaries: {‘dataset’: ‘CESM2’, ‘project’: ‘CMIP6’, ‘mip’: ‘Ofx’, ‘short_name’: ‘areacello’, ‘activity’: ‘CMIP’, ‘ensemble’: ‘r4i1p1f1’, ‘exp’: ‘historical’, ‘frequency’: ‘fx’, ‘grid’: ‘gn’, ‘institute’: [‘NCAR’], ‘long_name’: ‘Grid-Cell Area for Ocean Variables’, ‘modeling_realm’: [‘ocean’], ‘original_short_name’: ‘areacello’, ‘standard_name’: ‘cell_area’, ‘units’: ‘m2’} session: ‘session-ccace7d0-dc3b-4e03-87ea-2716aa127618_20230609_085322’These ancillary datasets are not always consistently available for each variant of a dataset. ESMValCore includes some convenience functionality that tries to find a matching variable if the exact match is not available. For example, it may take the grid cell area from another ensemble member. In this case, it looks like we have an exact match.

Data (down)loading**

We now arrive at the part of the script that calls dataset.load(). In the background, ESMValCore automatically located the files on ESGF and downloaded them for us. Let’s illustrate this.

Before load**print(dataset.files)[

ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r4i1p1f1/Omon/tos/gn/v20190308/tos_Omon_CESM2_historical_r4i1p1f1_gn_

185001-201412.nc on hosts [‘aims3.llnl.gov’, ‘esgf-data.ucar.edu’, ‘esgf-data04.diasjp.net’, ‘esgf.ceda.ac.uk’, ‘esgf3.dkrz.de’] ]# After load (and re-defining the dataset to trigger a new search) print(dataset.files)[LocalFile(‘/home/peter/climate_data/CMIP6/CMIP/NCAR/CESM2/historical/r4i1p1f1/Omon/tos/gn/v20190308/tos_Omon_CESM2_historical_r4i1p1f1_gn_185001-201412.nc’)]Notice that there were copies of this file available on 5 different ESGF nodes. ESMValCore automatically selects the fastest available server and stored the dataset according to the standard CMOR data reference syntax.

For our example dataset, all the data is apparently contained within one file. However, it’s not always like this. If we look, for example, at another ensemble member, we can see that it is stored in chunks of 50 years:

print(ensemble_datasets[1].files)[ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_185001-189912.nc on hosts [‘aims3.llnl.gov’, ‘esgf-data.ucar.edu’, ‘esgf-data04.diasjp.net’, ‘esgf.ceda.ac.uk’, ‘esgf3.dkrz.de’], ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_190001-194912.nc on hosts [‘aims3.llnl.gov’, ‘esgf-data.ucar.edu’, ‘esgf-data04.diasjp.net’, ‘esgf.ceda.ac.uk’, ‘esgf3.dkrz.de’], ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_195001-199912.nc on hosts [‘aims3.llnl.gov’, ‘esgf-data.ucar.edu’, ‘esgf-data04.diasjp.net’, ‘esgf.ceda.ac.uk’, ‘esgf3.dkrz.de’], ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_200001-201412.nc on hosts [‘aims3.llnl.gov’, ‘esgf-data.ucar.edu’, ‘esgf-data04.diasjp.net’, ‘esgf.ceda.ac.uk’, ‘esgf3.dkrz.de’]]In this case, ESMValCore automatically concatenates the data for us.

You may have noticed that we assigned the result of the load method to a new variable called cube:

cube = dataset.load()Under the hood, ESMValCore uses [iris](https://scitools-iris.readthedocs.io) to represent data cubes. Compared to xarray, iris is more strict on checking adherence to CF conventions. This can make it harder to work with, but it also acts as a safeguard against unexpected behaviour.

Where Tom used xMIP, ESMValCore comes with builtin checks and if there are any (known) issues with the datasets, it automatically fixes them. Also, it automatically loads all supplementary datasets. Notice how the cell area is present as a cell measure.

print(cube)sea_surface_temperature / (degC) (time: 1980; cell index along second dimension: 384; cell index along first dimension: 320) Dimension coordinates: time x - - cell index along second dimension - x - cell index along first dimension - - x Auxiliary coordinates: latitude - x x longitude - x x Cell measures: cell_area - x x Cell methods: mean where sea area mean time Attributes: Conventions ‘CF-1.7 CMIP-6.2’ activity_id ‘CMIP’ branch_method ‘standard’ branch_time_in_child 0 branch_time_in_parent -492385 case_id ‘18’ cesm_casename ‘b.e21.BHIST.f09_g17.CMIP6-historical.004’ contact ‘cesm_cmip6@ucar.edu’ data_specs_version ‘01.00.29’ description ‘This may differ from “surface temperature” in regions of sea ice or floating …’ experiment ‘all-forcing simulation of the recent past’ experiment_id ‘historical’ external_variables ‘areacello’ forcing_index 1 frequency ‘mon’ further_info_url ‘https://furtherinfo.es-doc.org/CMIP6.NCAR.CESM2.historical.none.r4i1p1 …’ grid ‘native gx1v7 displaced pole grid (384x320 latxlon)’ grid_label ‘gn’ id ‘tos’ initialization_index 1 institution ‘National Center for Atmospheric Research, Climate and Global Dynamics Laboratory, …’ institution_id ‘NCAR’ license ‘CMIP6 model data produced by <The National Center for Atmospheric Research> …’ mipTable ‘Omon’ mip_era ‘CMIP6’ model_doi_url ‘https://doi.org/10.5065/D67H1H0V’ nominal_resolution ‘100 km’ out_name ‘tos’ parent_activity_id ‘CMIP’ parent_experiment_id ‘piControl’ parent_mip_era ‘CMIP6’ parent_source_id ‘CESM2’ parent_time_units ‘days since 1850-1-1 00:00:00’ parent_variant_label ‘r1i1p1f1’ physics_index 1 product ‘model-output’ prov ‘Omon ((isd.003))’ realization_index 4 realm ‘ocean’ source ‘CESM2 (2017): atmosphere: CAM6 (0.9x1.25 finite volume grid; 288 x 192 …’ source_id ‘CESM2’ source_type ‘AOGCM BGC’ sub_experiment ‘none’ sub_experiment_id ‘none’ table_id ‘Omon’ time ‘time’ time_label ‘time-mean’ time_title ‘Temporal mean’ title ‘Sea Surface Temperature’ type ‘real’ variable_id ‘tos’ variant_info ‘CMIP6 20th century experiments (1850-2014) with CAM6, interactive land …’ variant_label ‘r4i1p1f1’

Pre-processing **🛠️

ESMValCore comes with a large suite of built-in preprocessing functions. These functions have been designed as scientifically sound default implementations for frequently performed operations. An important idea underpinning ESMValCore (&Tool)’s philosophy is that standardization facilitates comparison between, and reproducibility of, our data analysis workflows. Moreover, by bundling our efforts we can develop high quality preprocessor functions more effectively. In that spirit it would be good to see how xMIP could be integrated with ESMValCore’s CMOR checks and fixes.

We won’t go into detail on each of the preprocessing functions applied above, as they’re fairly self-explanatory. One thing to highlight though, is that in our calculation of anomalies, we deviated a little bit from Tom’s method. He subtracted the overall average of 1950–1980; we used monthly means over that period. It’s not really necessary here, but it facilitates comparing changes in summer temperature, for example 😎.

cube = area_statistics(cube, operator=‘mean’)**cube = anomalies(cube, reference=reference_period, period=‘month’) cube = annual_statistics(cube, operator=‘mean’) cube.convert_units(‘degrees_C’)

Custom code**

Up to this point, everything we’ve done was “pure” ESMValCore. However, in every data analysis workflow, there comes a point where you want to do custom things. In the traditional context of ESMValTool, such custom code would be added as a “diagnostic script”. In the context of this notebook, we can simply start working with the cube in whatever way we like.

This can be a good moment to thank iris for its services, and continue with xarray if you prefer. For example:

import xarray as xr**da = xr.DataArray.from_iris(cube) da.plot() print(da)<xarray.DataArray ‘tos’ (time: 165)> dask.array<filled, shape=(165,), dtype=float32, chunksize=(1,), chunktype=numpy.ndarray> Coordinates:

time (time) object 1850-07-02 13:00:00.000001 … 2014-07-02 12:00:00 nlon int64 … nlat int64 … lat float64 … lon float64 … year (time) int64 … Attributes: (12/57) standard_name: sea_surface_temperature long_name: Sea Surface Temperature units: degrees_C Conventions: CF-1.7 CMIP-6.2 activity_id: CMIP case_id: 18 … … time: time time_label: time-mean time_title: Temporal mean title: Sea Surface Temperature type: real cell_methods: area: mean where sea time: mean longitude: latitu… The change in sea-surface temperature as plotted by xarray’s builtin plot method.

Reproducibility: convert to recipe**

Traditionally, ESMValTool was designed to work with recipes: static, human-readable files that declare a worfklow from start to finish. This comes back to what we said earlier about describing what to do instead of how. ESMValCore helps us to start building a recipe. With the following code, we can generate the datasets and diagnostics sections of the recipe:

from esmvalcore.dataset import datasets_to_recipe**import yaml

for dataset in ensemble_datasets: dataset.facets[‘diagnostic’] = ‘easy_ipcc’ print(yaml.safe_dump(datasets_to_recipe(datasets)))datasets:

dataset: CESM2 ensemble: r4i1p1f1 institute: NCAR
dataset: IPSL-CM6A-LR ensemble: r4i1p1f1 institute: IPSL
dataset: MPI-ESM1-2-LR ensemble: r4i1p1f1 institute: MPI-M
dataset: TaiESM1 ensemble: r1i1p1f1 institute: AS-RCEC
dataset: AWI-CM-1-1-MR ensemble: r(1:5)i1p1f1 institute: AWI
dataset: AWI-ESM-1-1-LR ensemble: r1i1p1f1 institute: AWI
dataset: BCC-CSM2-MR ensemble: r(1:3)i1p1f1 institute: BCC …

diagnostics: easy_ipcc: variables: tos: exp: - historical - ssp585 grid: gn mip: Omon project: CMIP6You still have to add the preprocessors and some metadata (a short description etc.). We envision a workflow where you use ESMValCore interactively for most of your exploratory work, and then port it to a recipe for sharing and reproducibility.

To complete this post, we’ve gone ahead and extended the recipe above to include all the available CMIP6 data and make a nice plot. As you might expect, this required more labour due to dataset issues for which no fixes were available, and a bit of extra tinkering to make the figure look more like the IPCC version. You can find the corresponding recipe and diagnostic script here. Be careful with running it, though! This recipe will try to download almost twenty-thousand files… Running this on the German climate compute service, we get the following figure:

This image includes all available models & ensemble members.

Summary**

ESMValCore is great addition to any climate analysis toolbox. Its key strengths are its its approach to streamlining workflows to achieve scientific rigor and reproducibility, and its tight integration with the ESGF and CMOR standards.

If you regularly work with CMIP data, sometimes struggle to retrieve data from ESGF, frequently encounter incompatible datasets, or want to make your workflows available to others, you should definitely give it a shot.

Do you have 50.000 annotators?

eScience Editorial Team — Thu, 21 Sep 2023 00:00:00 GMT

Do you have 50.000 annotators?

*Written by eScience Center Fellow, *Daniela Gawehns

Photo by Shutterstock

During ICT.open 2023, I organized a workshop for 40 computer scientists on reproducibility in computer science. This is a short summary of the main results and how I experienced the workshop as a moderator.

The workshop was part of my fellowship project at the eScience Center to find out how people in the computer sciences understand reproducibility. The idea behind the project is that communities of researchers differ in how they understand the term reproducibility. Before providing tools, guidance or incentives on how to make research outputs more reproducible, we need to figure out what people understand as reproducible research.

What happened?**

First, participants were invited to think in small groups about four ways to make their research less (sic!) reproducible (while still making a career in science). Afterward, the groups moved to another flip chart to expand on their predecessor’s ideas. Again, the prompt was to make their work as least reproducible as possible.

Participants were then invited to walk through the room, look at the posters, and pick a few ideas that they found most impactful in their own subfield of Computer Science. They labeled those ideas with post-it notes. During the last exercise, participants discussed in larger groups four different aspects of non-reproducible science: “Use overcomplicated theory”, “Use proprietary data/software”, “Require expensive tech” and “Do not share code/data”.

Thinking about the opposite of what you set out to achieve was counterintuitive for some participants, while others said “This is easy, I just describe all of my frustrations. Research that I encounter in my daily work is in large parts non-reproducible”. I didn’t give a definition of what Reproducibility means or how the term is defined for the purpose of the workshop at the beginning. This was intentional, as I was hoping to gather as many ideas as possible, without influencing people’s understanding of reproducible research. As a researcher with a background in the social sciences who has discussed reproducibility of research mainly with open science enthusiasts from the social sciences, I found it important to keep as much of my previous understanding of reproducibility outside the workshop room.

This workshop provided a glimpse into what computer scientists feel is important in achieving reproducibility and how this might differ from other areas of research. The use of expensive computing power or annotations is not often mentioned when I talk to psychologists, for example. I was surprised how often participants mentioned overly complicated theory as a barrier to reproducibility. Using version control to keep track of changes in code was only mentioned in passing. It didn’t seem to be on participants’ minds, while this is something I keep on hearing about in my own Open Science bubble.

What did I learn?

Giving participants a lot of freedom by keeping definitions to a minimum allows participants to share their understanding. It can also lead to frustrations from people who like to set a framework for themselves before starting creative work. Inviting everyone to participate and share their ideas is a key ingredient to running a workshop where all participants are happy with the outcomes.
Asking people to describe their field of research leads to answers that are difficult to use and interpret. Answers like “Human Centered Interaction/ Data and Visual Analytics“ make it hard to categorize responses into just one subcategory (in this case Human Computer Interaction or Data Science). Another question is how to split Data Science, AI and ML. Especially if people describe their work as “AI/ML” or other combinations.
There was a lot more input we didn’t have time to explore. For the second part of the workshop, we picked four topics to further focus on. Other barriers to reproducibility that were mentioned by participants, but couldn’t be discussed further include: “Poorly conceptualized Desk Research”, “System Design without clear Problem description”, “Don’t explain the Evaluation, Say “well-established procedures” “, “50.000 Annotators Needed”

What’s next?

This workshop, as well as a few interviews with Computer Scientists at Leiden University, informed a survey that will be sent out in September 2023 to computer scientists working in the Netherlands. Recruitment for that survey will happen via the research schools SIKS and ASCI, social media and word of mouth. Please let me know if you know of a good way to reach a diverse group of survey respondents from the computer sciences.

The proceedings of the ACM REP conference are a good starting point for anyone interested in the topic of reproducible computer science. You can find a collection of materials in several community calls on the topic of reproducibility in Computer Science here.

A transcript of the flip charts created during the workshop can be found here.

ESMValTool: Recipes for solid climate science

Peter Kalverla — Mon, 11 Sep 2023 00:00:00 GMT

ESMValTool: Recipes for solid climate science

*This story was coauthored by Peter Kalverla and *Bouwe Andela

Like an artisan chef can improvise a delicious meal, so too can a scientist create new insights from raw data and exploratory analysis. But to run a successful restaurant with personnel and a consistent menu, our chef resorts to writing down his recipes. Similarly, we scientists must make an effort if we want our results to be consistent and widely reusable.

Photo by Pylyp Sukhenko on UnsplashThis is where ESMValTool comes into play. Designed to ease and automate the analysis of large ensembles of climate model outputs, ESMValTool defines a clear and plain scientific [“recipe” format](https://docs.esmvaltool.org/projects/ESMVal

Core/en/latest/recipe/overview.html). This not only helps to make workflows scientifically robust, but also provides a way to easily share and build upon each others’ work. Even if you’re not a climate scientist, there are valuable lessons to draw from ESMValTool’s approach to FAIR and open science.

In this blogpost, we’ll explore four aspects of ESMValTool that may be worth your while. Starting with its take on analysis-ready data and its rich suite of pre-processor functions, we’ll set the stage for a broader discussion on how standardization helps making research easy and FAIR and the crucial role of a community in setting itself up for success.

Analysis-ready (CMIP) data**

Strictly speaking, most of the functionality described here is part of ESMValCore. ESMValCore is the engine, if you will, that executes ESMValTool recipes under the hood. ESMValTool, then, is a big collection of recipes. To the user of ESMValTool, this distinction is irrelevant, but we’ll use the terms interchangeably to be as accurate as possible. For a description of ESMValCore as a standalone utility, see the companion blogpost.

ESMValTool was originally designed for the analysis of CMIP data: output of many climate models assembled in a series of “Coupled Model Intercomparison Projects”. The CMIP data request specifies standards that the data should adhere to. Known as the CMOR standards, after the “Climate Model Output Rewriter” software, they describe things like variable names, units, coordinates (with their direction), and also file name conventions. These standards make it much easier to compare output from different climate models, and they are the foundation on which ESMValTool & Core were built.

For example, the file name conventions make it possible to specify a dataset like so:

Dataset(** short_name = ‘tos’, mip = ‘Omon’, project = ‘CMIP6’, exp = ‘historical’, dataset = ‘CESM2’, ensemble = ‘r4i1p1f1’, grid = ‘gn’ )and ESMValCore is able to locate it, both on your local computer or on ESGF, the shared infrastructure where all of the CMIP data are collected. ESMValCore can automatically download data from ESGF, provided that the data is present there.

Pause a moment to let this sink in: ESMValCore eliminates hardcoded paths in your scripts, or rather, recipes. This is key to making them interoperable.

If you’ve worked with CMIP data yourself, you’ll probably know that, despite the strict protocol, small dataset issues are commonplace. That’s why ESMValCore executes an additional CMOR check upon data loading. Where possible, it applies automatic fixes for known and trivial issues. This makes working with the data much easier.

Another common issue with CMIP data is that files may not or no longer be available. ESMValCore makes it easy to perform wildcard searches or ignore missing datasets, saving you the trouble of figuring out which datasets are available or have been retracted since the last time you ran your recipe. Thus, ESMValCore is also a great utility for CMIP data discovery.

While ESMValTool & Core were originally developed for CMIP data, it also supports ingestion of other datasets. For example, it can apply “on the fly CMORization” of ERA5 data, and it ships with download and “CMORizer” scripts for many other (observational) datasets. This makes ESMValTool more broadly applicable and moreover, it promotes standardization beyond its original scope.

To summarize, ESMValTool makes it easy to get to your data. In the next section, we’ll describe how ESMValTool also helps to standardize how you process it (without compromising on versatility).

ESMValCore’s rich preprocessor suite**

Like with a kitchen recipe, after the ingredients come the preparation steps. In ESMValTool & Core, they are called pre-processors. Common preps for climate data include regridding, area extraction, calculating anomalies and all kinds of statistics.

ESMValCore has built-in functions for each of these pre-processors, and they can be added to a recipe using a very high-level specification. For example:

Part of an ESMValTool a recipe listing the pre-processing steps that should be appliedBy reading this snippet, you can probably get a pretty good idea of what this recipe is trying to achieve: the mean (temperature) change over Europe with respect to a reference period. This is a declarative syntax, focusing on what should be done, instead of how.

Ideally, with ESMValCore, the standard pre-processors provide a scientifically sound consensus implementation. For example, it uses state-of-the-art regridding with ESMF/ESMPy, and strict CF-compliant statistics from the Iris package. These functions have been designed by domain experts and are openly developed, so anyone can suggest improvements if they feel the current implementation is insufficient. ESMValCore executes these steps in a standard order which makes sense for most applications. However, you can also specify a custom order as this might make sense for some specific use cases.

In the illustration below, you can see the default order in which the pre-processors are executed. In your recipe, you can simply select those steps that are relevant for your workflow.

The architecture of ESMValTool: recipe specifies which input data is loaded. This data is then ingested and the preprocessor steps specified in the recipe are executed. Finally, pre-processed data is passed onto a diagnostic script to generate the final output.After prepping, your data is (optionally) passed on to a diagnostic script. This is the place where you can do custom things, such as making a figure, or doing a more thorough and case-specific analysis. “Combining the ingredients into a tasty dish”, to stick with the chef’s analogy.

So far, we’ve seen how ESMValTool tries to streamline those parts of your analysis workflow that can be standardized, and tries to accomodate all specific things you might need or want. In the next section, we’ll see how this helps you make your research more FAIR.

FAIR and open science**

ESMValTool is a great example of how open science can work in practice.

Recently there’s been an increased focus on accessibility, transpancy, and reproducibility in science. We’ve seen the advent of open access publications, the introduction of FAIR standards for research data management. It’s easy to extrapolate these ideas to other aspects of the research process as well. At the same time, this can be daunting, because it may seem like a lot of extra work that you might not have the time or skills for. So let’s see how ESMValTool helps with these concerns.

One ideal that’s often associated with open science is reciprocity, which we all know in lyrical form as “never take more than you give”. The idea is that scientific progress benefits from mutual exchange and building upon each other’s work. ESMValTool provides a platform for receiving and giving back. By making available a large suite of recipes that other people have already composed. By establishing a process to contribute your own recipes. By setting a standard format to encode recipes. By integrating existing functionality into a streamlined workflow system. And by facilitating a process of feedback based around users’ contributions, and a forum for discussion. Many of the things that are hard to organize on your own, have already been organized by the ESMValTool community. Thus, you’ll be greeted with a warm welcome instead of a daunting prospect.

Illustration from the Turing way book dashes.Going more into the specifics of FAIR, we can see how ESMValTool contributes: ESMValTool has great documentation listing all available recipes and a gallery with outputs. You can also list the available recipes from within the software, or browse the source code. So recipes are quite findable, although you could still take it a step further by creating a searchable registry with unique identifiers for each recipe (Note that the package as a whole does have a DOI for each version).

ESMValTool recipes are also very accessible. The YAML format is very readable for humans and also by machines; they can be executed. As we’ve seen above recipes are also portable: hardcoded file paths et cetera have been eliminated. For each release of ESMValTool, all recipes are executed to see if they still work, and output is made available.

To summarize, ESMValTool itself would qualify as “FAIR software. But moreover, ESMValTool recipes also come a long way as “FAIR scientific workflows”. Just like kitchen recipes empower chefs, ESMValTool recipes empower scientists.

The crucial role of community

ESMValTool would not have been where it is now if it wasn’t for the community. The ESMValTool community is a nice and welcoming crowd of scientists and software engineers from various institutes in Europe and beyond. In addition to lively interactions on GitHub there are monthly online meetings that are open to all, and there’s an in-person workshop once or twice a year. Furthermore there are spontaneous gatherings of ESMValTool contributors at other ocassions such as conferences.

Screenshot of the ESMValTool discussion forum announcing some of the community gatherings.Over the years the ‘core team’ has gained a lot of experience and become more professional, for example when it comes to processes for decision making and user engagement. There are clear guidelines for contributing to the codebase, and people are always willing to help each other out. To support new users, the user engagement team maintains a tutorial that is regularly taught.

By fostering enjoyable and personal interaction, the community plays a vital role in facilitating the development and long-term sustainability of ESMValTool.

Conclusion

In conclusion, ESMValTool provides a valuable platform for scientists to achieve consistency, reusability, and standardization in their research, owing much of its current success to the dedicated community that surrounds it.

Are you a climate scientists and do you want to be part of this community? Don’t hesitate to reach out on GitHub or another channel that you see fit. Are you passionate about open science but not necessarily in climate science? Then we hope you could get some nice takeaways from ESMValTools experience and we are always happy to exchange more. In any case, the next time you visit a restaurant, remember to appreciate not only the chef but also the thoughtfully curated menu.

Coming Soon: Conditional Process Models in JASP

Malte Lüken — Thu, 31 Aug 2023 00:00:00 GMT

By Malte Lüken, Thijs Vroegh, Johnny van Doorn, and Eric-Jan Wagenmakers

This post is also published on the JASP blog.

Photo by Huma Kabakci on Unsplash Many students, teachers and researchers use the popular PROCESS macro for SPSS (Hayes, 2022a, 2022b) to estimate conditional process models involving mediation and moderation effects. Recently, however, these models were also criticized because they make causal assumptions, which do not hold in many applied settings (Bullock et al., 2010, Rohrer et al., 2022). The popularity and criticism warrant a new module to make conditional process models easily accessible to the JASP community. Moreover, the module should provide a better understanding of the underlying causal assumptions for users and tools to explicitly test them. The JASP Team and the Netherlands eScience Center are currently working on such an endeavor, and we give a short teaser in this blog post.

Estimating Models as in SPSS

The new Process module has two graphical user interfaces: One interface resembles the interface of the original SPSS macro where users first select a model configuration from a predefined list. In contrast to SPSS, JASP immediately shows the user a graphical representation of the specified model (i.e., a path plot), which facilitates the search for the right model. As the user adds more variables to the model, the graphical representation is updated. The module will provide both conceptual and statistical path plots, where the statistical path plot includes the estimated parameters.

Building Models Step-by-step

With a second, complementary interface users can build models step-by-step starting from scratch. They can iteratively add paths (i.e., relations) between selected variables and define how other variables influence each path (e.g., through mediation or moderation). Below, we reproduce predefined Model 5 with two path specifications: We first add a path between Extraversion and Neuroticism (X and Y) and specify Openness as a mediator for this path (M). Then, we add a second path specification between Extraversion and Neuroticism, but this time we include Agreeableness as a moderator for this path (W). Again, every time a path is added or modified, JASP will immediately show the user the corresponding graphical representation of the model to facilitate the model-building process. This approach enables users to build models intuitively and flexibly, avoiding the restrictions of predefined model configurations.

Addressing Criticisms of Process Models

Conditional process models make strong causal assumptions about the relationships between variables. A mediation model, for example, states that there is a causal relationship from X to Y, X to M, as well as M to Y. Critics have argued that researchers cannot assume these causal assumptions to hold in most applications, even in experimental settings (Bullock et al., 2010, Rohrer et al., 2022). In the new module, we aim to address some of these arguments by making the causal assumptions more explicit via footnotes and warnings. Importantly, we will also enable users to test the statistical patterns implied by the causal assumptions (i.e., via conditional independence tests; see Rohrer et al., 2022). Users can then conclude which assumptions are violated and modify their model or experimental design accordingly. Because of its step-by-step interface, the module allows users to build more complex models to accommodate the requirements of their application (e.g., by adding confounding variables to different paths). Despite these features, many challenges remain when applying conditional process models, most of which are related to careful experimental design (see Bullock et al., 2010). We aim to draw the users’ attention to these challenges and improve the statistical use of process models in practice.

Behind the Project

This project is a collaboration between the JASP Team and the Netherlands eScience Center. As the national institute for research software, the Netherlands eScience Center awards projects as part of calls for proposals. Instead of monetary funding, it provides in-kind support by Research Software Engineers to create innovative digital solutions. This project was granted as part of the Small-Scale Initiatives Digital Approaches to the Social Sciences call 2022.

We kindly thank Tom Bakker and Lieke de Boer for providing feedback on earlier versions of this post.*

References

Hayes, A. F. (2022a). *Introduction to mediation, moderation, and conditional process analysis *(3rd Ed.). New York: The Guilford Press.

Hayes, A. F. (2022b). The PROCESS macro for SPSS, SAS, and R (Version 4.2) [Computer software]. https://processmacro.org

Why your research deserves to be an R package

Barbara Vreede — Wed, 30 Aug 2023 00:00:00 GMT

If after reading this post you are motivated to make your own R package, consider joining our upcoming workshop! Free tickets are now available.

If you are an R user, think about this scenario:

How often have you reused someone else’s code — a colleague’s workflow, perhaps a script you found as an appendix in a paper? Did it run right away? Was it easy to reuse?

Now, ask yourself whether you have used someone else’s packaged code. How often have you started your own script with library (packagename), and used functions that are not part of base R?

I am willing to bet quite a lot that the second scenario is an order of magnitude more common than the first. There is a reason for this: packages are made to be reused. Scripts, while technically reusable, are not.

Let me now ask you about your own work: do you want it to be reused?

You know what you have to do now, right?

Photo by Kira auf der Heide on Unsplash

But making a package… isn’t that complicated?

An R package distinguishes itself from scripted code in a few ways, but it is much less complex than you may think.

First of all, when your code is packaged, it is contained in a standardized folder structure. The code itself lives in a designated folder (aptly named ‘R’). In addition, the root of the package contains files with some basic information about your package — like the title, the authors, and the license for future users.

Secondly, and perhaps most crucially, packaged code consists of functions only. Functions are the units at the centre of a package. They take information as arguments and return an output. For example, I have a function called *multiply. *This function, you guessed it, multiplies the two arguments I give it, and returns the result:

multiply <- function(a, b){ return(a * b) }After including this function in a package — let’s call it *mypackage — *someone else can now use it without opening and running the entire codebase, but instead by attaching the package and calling the function:

> library(myPackage) > multiply(3,4) [1] 12You have likely written functions before. And you know how folders work. To dot the i’s and cross the t’s, Rstudio is here to help: it contains many built-in functions to make package creation a breeze.

Your work deserves to be reused.

Just in case you are now doubting yourself, please don’t. Scripting an analysis is more than just applying existing calculations. The workflow itself is an original thought. Even if you have not, say, designed a new statistical method yourself, the methods you choose, the way you select and handle your data, the visualisations you make — they are your work, and could be reused as a standardised workflow in your field.

Let’s take an example. Mark*, a linguistics professor, runs frequent analyses on datasets made from taped conversations. He uses R to uncover the unwritten rules of conversation, finding out how people weave their utterances together. With this data, he creates fantastic visualisations, which give insight into the conversational dynamics at play.

Mark has hundreds of conversations to analyse, so he made functions that contains the essential elements of his workflow. Now, when he loads a new conversation, he can for instance use this function** to quickly get an overview of the basic statistics:

> conversation <- read.csv(“data/conversation25.csv”) > inspect_convo(conversation) Number of speakers: 2 Length of conversation: 10:03 Number of utterances: 9438But Mark does not work alone. His group contains PhD students and postdocs, and is a popular place for bachelor and master internships. He shares his scripts with his group, but they can be a daunting sight for those who have not programmed much before. And as his toolkit grows, he loses track of the different elements it contains, and what updates he has shared with whom.

Mark needs an R package.

A package makes you a better programmer.

An important element of packaged functions is their documentation: a user who only has access to the name of the function, needs to know how to use it. But writing documentation is not just for users: it does something for you as a developer as well.

Writing down the logic of your functions — what goes in? What comes out? Exactly what formats are needed for each argument?— helps you think more critically about the objects you use and create as part of your workflow. You may not expect it, but documentation is a powerful way to take a step back and start viewing your set of functions as a whole.

This is where you go from scripting to software engineering.

Photo by RetroSupply on Unsplash

Packaged code is more robust, and developing it is more efficient.

Because your functions are in a package, you now also have access to another powerful tool in the software engineering toolbox: testing.

Most likely, when you were developing your function, you were running it over and over to confirm it still did what it was supposed to. You gave it input, and checked that it returned the right output.

You were testing your function, and it took a lot of time.

With tests, we can automate this process. Mark, for instance, knows that when he puts the file conversation25.csv* into inspect_convo, it should return information about the number of speakers (2), the length of the file (10 minutes and 3 seconds), and the number of utterances (9438). He does not have to keep checking this himself, but can put his knowledge inside a test:

test_that(“Conversation inspection returns the right results”, {

load the test object and run inspect_convo

load(“conversation25.Rda”) conversation <- inspect_convo(conversation25)

inspect_convo should return the right information

expect_true(conversation[1] == “Number of speakers: 2”) expect_true(conversation[2] == “Length of conversation: 10:03”) expect_true(conversation[3] == “Number of utterances: 9438”) })Now he simply runs the tests, and Rstudio will tell him if they pass. This quickly starts saving him a lot of time, especially when the package grows and starts including more functions. With one click, Mark can confirm that everything, everywhere, is OK.

The Rstudio view of the tests in this package confirm that every function is operating as it should.Being able to check functionality like this comes with a lot of freedom, to grow your package, but also to improve the things already written. How efficient is that function you wrote, really? When you edit (or “refactor”) the code, your tests give you the peace of mind that your edits have not caused any inadvertent problems. After all, as soon as something breaks, you will know!

But what about my paper?

While you may be happy that someone else can reuse your functions, it is likely that the original goal of your research project was not to write a toolkit, but a paper. Not to worry, there is a designated space for this inside an R package: a vignette.

In a vignette, you take a reader through your project from A to Z. While explaining your workflow in accompanying markdown-formatted text, you apply your functions in blocks of R code, and the resulting document shows both the code and the resulting output. This is the cherry on top: you have turned your research into an easily reusable toolkit, and shown how to use the functions in a reproducible document.

Would you like some help?

There are a lot of online resources that can help you work on your first package. For one, the book R Packages is a resource I cannot recommend enough. It has been the basis for the Carpentries lesson “R Packaging”, which is regularly held as part of the eScience Center’s digital skills workshops. Take a look at our schedule to see upcoming dates for “Reproducible Research with R Packages”, and join us!

*) The story used in this blog is inspired by the package talkr, which is part of Towards next-generation scientific computing tools for diversity-aware language science and technology, a project with Mark Dingemanse and Andreas Liesenfeld at the Radboud Universiteit Nijmegen. You can read more about the real Mark at his own website: markdingemanse.net.

**) The functions and files in this blog are simplified and fictionalised for ease of reading.

Lilio — a calendar package for machine learning with time series data

Peter Kalverla — Mon, 21 Aug 2023 00:00:00 GMT

Designed with the S2S community, available for everyone

Subscribe*Remember me for faster sign in

These are just some of the questions that scientists in the field of sub-seasonal to seasonal (S2S) weather and climate prediction have to deal with on a daily basis. Especially now that the use of machine learning is gaining traction. In this blog post, we introduce Lilio, a new calendar package that was designed to help tackle these questions.

https://github.com/AI4S2S/lilio

The niche

Before we dive into the capabilities of Lilio, let me briefly review why existing packages didn’t cut the deal for us. Pandas is great and we rely on it, but we wanted more flexibility to construct varying intervals with custom gaps in between. And instead of a continuous index, we wanted to exploit the periodicity in our use cases. Existing time series models (e.g. *Darts) often apply the same model irrespective of the forecast issue date, and the forecasts are anchored to said issue date. By contrast, our experiments are usually anchored to a clear target period, and each target may have a unique set of predictors. We have different models with different precursors for a windy May, a wet monsoon, a mild winter, …

The fill

So, how does Lilio help with all of that? First of all, we stack the years to obtain a 2-dimensional calendar. Each row represents a year, and columns are intervals within that year. Typically we have several precursor periods leading up to one or more target periods. This aligns nicely with the common representation of samples and features in ML data. In our case, the target is in the rightmost column(s).

Conceptual illustration of the structure of our calendars.We define the “anchor date” to be between the target and precursor periods. All other intervals are expressed as offsets to this anchor date. Conveniently, this eliminates any ambiguity related to leap years. Here’s a calendar generated with Lilio:

Example calendar with uniform 10-day intervals, here represented as offsets to the anchor date.By default, we include as many blocks (of a given frequency) as fit in one year but not more. To control this behaviour, Lilio provides options to allow or prohibit overlap. This makes it straightforward to apply existing train/test splitting strategies without leakage.

Anchor + offset = date

Initially, Lilio calendars don’t include years. Only after we map the calendar onto a given year range or dataset, actual dates can be calculated. Since the start or end date of the calendar doesn’t always nicely align with the input data, the calendar also comes with a method to map it to the range of available data.

Same calendar as above but plotted on a datetime axis. The anchor date for this calendar is 30 November. Combined with the anchor years 2020 and 2021, the actual dates can be inferred.

Visualizations & wishful drawing

The visualizations shown above are generated automatically. This turns out to be very helpful in the initial process of setting up your experiments. We even found ourselves making “wishful drawings” to communicate about alternative calendars that we’d like to support.

Illustrated feature request for a “rolling calendar”

Resampling

Lilio’s resampling functionality can be used to aggregate the input data based on the calendar’s intervals. Here, again, we heavily rely on the presence of bounded intervals. By using these as resampling bins, we ensure that all input data ends up on the exact same time axis.

Simple and custom calendars

For many applications, a simple weekly, monthly, or (n-)daily calendar may be all you need, and Lilio makes this super easy. With only a bit more effort, you can also construct calendars with gaps and overlapping intervals of varying lengths. Here’s a more exotic calendar for predicting the chances of a white Christmas:

This calendar uses more fine-grained information closer to the target

What’s next?

Lilio has been developed in the context of a larger project in which we are developing a Python package to set up and streamline S2S — machine learning workflows. It will be a key component in our experimental setup. At the same time, the calendar is also very suitable as a stand-alone component for use in other applications. We are curious to learn about new use cases that you may have for it.

Final note

Collaborative research software development is a team effort. Please give appropriate credit and consider joining us 😊. And if you read up to this point and still wonder who Lilio was: here you go.

Summer is here, and so are large language models.

eScience Editorial Team — Mon, 10 Jul 2023 00:00:00 GMT

Welcome to the technological revolution of 2023, where the landscape of communication and information processing has been forever transformed.

Photo by Google DeepMind on Unsplash

The early months of this year have been dominated by the extraordinary rise of ChatGPT and other large language models. These remarkable AI-powered systems have become the driving force behind an unprecedented level of automation and efficiency in generating summaries and content. In fact, the emergence of ChatGPT has been so influential that it has rendered the traditional practice of crafting personal summary blogs unnecessary. With its unparalleled capabilities, ChatGPT now stands as the ultimate tool to effortlessly generate comprehensive and insightful summaries, paving the way for a new era of intelligent content creation.

The paragraph above was written by ChatGPT. **Other entries on our blog are still written by humans though, and good ones too. So below are our favourite blogs on large language models:

Aron Jansen interviewed ChatGPT about the eScience Center. Amazingly, it did not know the eScience Center at the time (it definitely does now!). Arguably, this post was mostly written by ChatGPT — but Aron shows us the art of good prompting.

Candace Makeda Moore warned us about the use of ChatGPT in medical contexts, in this excellent reference of the pitfalls and problems of putting all your trust in these models.

To give some context to the hype, Carsten Schnober wrote a blog about the history of language models. It’s an unparalleled and crystal clear description of the history of language models, utopian and dystopian fantasies, as well as concrete potentials and risks.

An older, but still very relevant post on language models was written by our former engineer Florian Huber. He explains what language models actually “understand”, and drew his own funny comics to accompany the story.

The eScience Center blog is going on a summer hiatus. In the meantime, we hope you enjoy these excellent stories by our excellent colleagues.*

Mcfly: An easy-to-use tool for deep learning for time series classification

Florian Huber — Mon, 26 Jun 2023 00:00:00 GMT

A new mcfly 3.0 release is out. See how it works and how it can help you to apply deep learning to time series classification.

][Florian Huber]·Apr 15, 2020

Once the models are generated, they will be trained on the given data (or a subset to speed things up). This is done using train_models_on_samples.

See full tutorial notebook for more information.

The performance of the trained models can then interactively be compared using mcfly***, for example by comparing the accuracy on the validation set versus a number of key hyperparameters (see screenshot below). The built-in visualization is interactive and allows to select specific models (here numbered 0 to 7), or select specific architectures (here ‘CNN’ or ‘InceptionTime’), or learning rates. Most important feature to look at are the two plots on the top which display the development of the accuracy on both the training set and the validation set. A good model should perform decently well on both sides.

mcfly build-in visualization of the model performance. While in this example all models achieve good results on the training dataset, much fewer reach decent results on the validation dataset.

Overfitting

Since the RacketSports dataset only consits of 150 training examples, you will frequently see that generated deep learning models will overfit the data. Overfitting is one of the most common problems when working with deep learning. It essentially means that you optimize too much on the training data so that the performance on unseen data will suffer. A typical signature for this is that models will do well on the training data (high training accuracy), but will perform poorly on the validation data (low validation accuracy). A typical example is shown below with 8 models, most of which report high train_accuracybut very low val_accuracyvalues!

Example results of 8 mcfly generated models trained on RacketSports dataset. Six perform well on the training data, but only 2 models also perform well on the validation data.If we now pick one of the better performing models (or iteratively generate and train more models), then we can get quite good results on the RacketSports dataset. Below you can see how we could inspect this by generating a confusion matrix. And here it indeed reveals that most of the times the picked model is correctly predicting the actual activity!

Generating a confusion matrix using the validation dataset.

And now… what are you waiting for?

Grab some interesting time series data and try out some deep learning!

Did you use mcfly?

Awesome! We are always happy to hear from people who have made good use of mcfly. Please get in touch if you have suggestions and ideas for future developments or fixes (e.g. via GitHub, via twitter, or as response to this post). Many thanks.

**Reference: **D. van Kuppevelt, C. Meijer, F. Huber, A. van der Ploeg, S. Georgievska, V.T. van Hees. Mcfly: Automated deep learning on time series. SoftwareX, Volume 12, 2020. doi: 10.1016/j.softx.2020.100548

Links

Former blog post from 2017 by [Dafne van Kuppevelt]
mcfly on pypi: https://pypi.org/project/mcfly/
mcfly on GitHub: https://github.com/NLeSC/mcfly
Ready-to-use RacketSports dataset on zenodo: https://zenodo.org/record/3743603
mcfly tutorial(s): https://github.com/NLeSC/mcfly-tutorial
Tutorial notebook with all code mentioned in this blog post: **https://github.com/NLeSC/mcfly-tutorial/blob/main/notebooks/tutorial/tutorial_quick.ipynb

mcfly*** *was developed at the Netherlands eScience Center by [Dafne van Kuppevelt], Christiaan Meijer, [Sonja Georgievska], [Vincent van Hees], [Florian Huber], [Patrick Bos], [Jurriaan H. Spaaks], Mateusz Kuzak, [Johan Hidding], [Atze van der Ploeg].

Thanks to Johan Rheeder, [Sonja Georgievska], [Dafne van Kuppevelt], [Peter Kalverla,] [Patrick Bos], and [Tom Bakker] for helpful comments and edits.

Reduce, reuse, recycle: save the planet one GitHub action at a time

Stef Smeets — Tue, 20 Jun 2023 00:00:00 GMT

Reduce, reuse, recycle: save the planet one GitHub action at a time

Written by Stef Smeets and Sander van Rijn.

Photo by Ralph (Ravi) Kayden on UnsplashAsk any environmentalist what you can do to help combat climate change and protect the environment, and they will tell you to “Reduce, Reuse, Recycle.” These actions are focused on maximizing practical benefit while minimizing the cost to the environment through conservation of resources and preventing the emission of greenhouse gases.

As software engineers, we are big fans of integration and continuous delivery (CI/CD) workflows as part of sustainable software development. GitHub actions is completely free for public repositories, so why wouldn’t we? We use it for everything, from code analysis and linting, running tests and builds in isolated environments, generating documentation on readthedocs, and publishing packages on PyPI.

Until all of our power comes from nuclear fusion, and data centers can be made of trees, running any code will have an environmental impact: CO2 and other emissions from the power used and the production of the computers it runs on. Not writing code won’t solve our problems either, so we need to find places in our workflow where we can reduce our impact.

Many CI/CD systems are configured to run automatically, often out of sight, so their impact to the planet adds up quickly. We have been asking ourselves some questions, like what we can do to make our use of CI/CD more sustainable.

Do we need to build the documentation for every commit we make to the code?
Do we need to test on every version of Python on all platforms to ensure code works?
Do we need to run all our machine learning notebooks on every push?
When we fix a typo in our documentation, do we need to run a complete suite of regression tests?
When we fix a bug, can we get away just re-running the failed tests?

In this blogpost we share what we have learned and give some ideas about what you can do to reduce your impact using GitHub Actions.

Why bother?

Data centers need large amounts of energy, from running servers, computing hardware, and refrigerating equipment. And, let’s not forget that the production of servers, cables and supporting equipment, and mining of the required raw materials also contribute.

According to the IEA, all data centers and data transmission networks account for 2–3% of global electricity use. This corresponds to 300 Mt CO2-eq in 2020 (0.9% energy related greenhouse gas emissions). The costs for the environment are enormous. Although improvements in efficiency and hyperscale data centers help limit the growth in energy demands somewhat, the demand for data workloads and internet traffic still grows every year.

GitHub (this blog is about GitHub Actions after all!) claims to be committed to environmental sustainability. A noble effort, but its parent company Microsoft increased its CO2 emissions by nearly 15% from 2020 to 2022. Most companies who claim to achieve ‘net zero’ do so not by reducing their emissions, but by compensating elsewhere. Microsoft compensated only about 10% of their emissions in 2022. This is typically done through buying certified green energy, CO2 compensation, or emission trading. This allows companies to be green on paper by trading certificates, with little direct effect on their business operations. Such climate certificates are often sold on a promise, like planting trees or saving a forest from clearing. Better than nothing, but it’s even better to not have those emissions in the first place.

Photo by Etienne Girardet on Unsplash.

Reduce

So, what can we do?

The best thing we can do is prevent resource usage by reducing the number of times we trigger the CI/CD workflows.

Restrict branch and event type

Many of us have configured our CI to run whenever a commit is pushed to a pull request (PR). While this gives great peace of mind to see that our tests pass, we can’t help but wonder if this is really necessary.

One pattern we like to use when we are still working on a PR is to keep it in Draft mode*. Once we are happy with our code and want some feedback, we mark the PR as as Ready for review. We can configure the CI to only run once a PR is Ready for review. This gives the reviewer peace of mind that the PR is bug free and is safe to be merged.

This also saves a whole bunch of unnecessary CI runs, and gives you more control when actually running the CI.

To set this up, add this to your github action:

tests.yml

name: Tests

on: pull_request: branches: - main types: - opened - reopened - synchronize - ready_for_review push: branches: - main

jobs: test: if: github.event.pull_request.draft == false …

steps:
  ...See an example of how we have implemented this for one of our projects [here](https://github.com/duqtools/duqtools/blob/main/.github/workflows/test.yaml).

Filter by changed files

Another common pattern we use is to only run a job when the relevant files have changed. Let’s say we have a simple project with the following folder structure:

project-to-save-the-planet/ ├── docs/ │ └── the_plan.md │ └── … ├── src/ │ ├── save_the_planet.py │ ├── supporting_documentation.md │ └── … ├── tests/ │ └── … ├── … └── README.mdWe don’t have to re-run all tests if something only changed in the documentation, nor do we have to rebuild the documentation if we only make changes to some tests. And if we only fix a typo in README.md neither will be necessary. These restrictions can be enabled by adding filters in the [paths](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore) section of the workflow triggers. These paths specifications can also be combined with the other filters such as branches shown earlier, to trigger only if it matches the paths and **branch. ** Here’s an example configuration that triggers only on changes in the docs folders:

docs.yml

on: push: … paths: - ‘docs/**’

jobs: …Certain paths or types of files can also be excluded using an exclamation mark (!). This can be useful if we want to trigger the tests to run on changes in src/ or tests/, except when we make changes to some supporting documentation or other data that does not affect the outcome of the tests. Here’s an example of such a workflow:

tests.yml

on: push: … paths: - ‘src/’ - ‘!src/supporting_documentation.md’ - ‘tests/’

jobs: …

Use CPU versions of tensorflow and pytorch

If you are working in machine learning, you will know that tensorflow and pytorch are enormous libraries. Together with their dependencies, they can be several gigabytes in size. On a CI, downloading these packages can already take up to a minute.

But, did you know that most of this space is taken up by GPU support? Try using [tensorflow-cpu](https://pypi.org/project/tensorflow-cpu/)or [pytorch-cpu](https://pytorch.org/get-started/locally/) instead of full-fat libraries. From our experience, this reduces the install size of our Python environments by approximately 1.5 GB on linux systems (this has no effect on Windows and Mac runners). The trick here is to install these prior to installing your package via pip. These will then be skipped by pip if these packages are also defined in your pyproject.toml or requirements.txt. Considering that all the common CI systems typically do not have GPUs in them, this makes the installs significantly lighter and therefore faster to run too!

To do so, add this step to your GitHub action prior to installing your package:

tests.yml

jobs: test: steps: …

- name: Install tensorflow
  if: runner.os == 'Linux'
  run: |
    python3 -m pip install tensorflow-cpu

- name: Install pytorch
  if: runner.os == 'Linux'
  run: |
    python3 -m pip install torch --index-url https://download.pytorch.org/whl/cpu

- name: Install dependencies
  run: |
      pip install .

Why throw away when you can reuse? Photo by JulieN2212 via Wikimedia Commons.

Reuse

If you must run the CI, the best way to reduce its impact is limit the actual work that needs to be done.

With the cache action, you can cache dependencies and build outputs to make your workflows faster and thus more efficient. Maybe you need to compile some dependency, download and pre-compute some data, or set up your python environment via pip. These typically do not change much from run to run, so try to cache these where possible.

Caching your Python environment

Just the installation of the dependencies of some Python code via pip can be quite significant. Some libraries just seem to pull in an endless stream of dependencies. So, why don’t we cache our entire Python environment?

Below is a snippet that we find effective in our workflows for Python code.

As the cache key, we use a combination of the Python directory name (this includes the version) in combination with the hash of the pyproject.toml, setup.cfg, or requirements.txt file. Whenever these get updated, the cache gets invalidated and regenerated.

This means we can also safely skip the pip install step if we hit the cache. Depending on the number of dependencies, this virtually eliminates the setup time of your workflow.

tests.yaml

jobs: test: steps: …

  - uses: actions/cache@v3
    id: cache-python-env
    with:
      path: ${{ env.pythonLocation }}
      key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}

  - name: Install dependencies
    if: steps.cache-python-env.outputs.cache-hit != 'true'
    run: |
      python -m pip install -e .[develop]

Caching any directory

To generalize the action below, the example below shows how any directory can be cached. We have used this to cache dependencies that need to be compiled. This can save minutes if not more from your workflow every single time it gets triggered.

We recommend adding a version to the cache key, so that you can easily invalidate the cache and force a re-trigger through the action itself.

Any subsequent workflows, will restore the directory from GitHub’s cache.

tests.yaml

jobs: test: steps: …

  - uses: actions/cache@v3
    id: cache-important-dependency
    with:
      path: important-dependency
      key: cache-key-1

  - name: Compile dependency
    if: steps.cache-important-dependency.outputs.cache-hit != 'true'
    run: |
      mkdir important-dependency
      cd important-dependency
      echo "All I want is world peace... Or a piece of the world." &gt; t.txt

Failing code into the red bin, please. Photo by Pawel Czerwinski on Unsplash.

Recycle

Finally, can we recycle **anything that is left when our CI run is done? Although not exactly trash, what’s typically leftover is the test results: which tests passed and failed. If any tests failed, those definitely have to pass before we want to continue. It makes sense then to re-run those tests first, and we can stop if any of those fail again.

Pytest last failed

Pytest has a very handy [--last-failed](https://docs.pytest.org/en/stable/how-to/cache.html) flag for exactly this purpose, but that obviously depends on a local cache to remember the outcome of the previous run. Luckily for you, we’ve already made a GitHub action that takes care of setting this up for you! Simply use this action instead of your *run: pytest --my --pytest --args* step:

tests.yaml

jobs: test: steps: # create python environment and install pytest
…

- name: Run pytest
  uses: sjvrijn/pytest-last-failed@v2
  with:
    pytest-args: '--my --pytest --args'

Conclusion

In this blog post, we introduced some options to make your sustainable software development more sustainable to the planet as well, whether by preventing double work or avoiding unnecessary CI runs in the first place. All it takes is a few extra lines in your workflows to help save the planet and save yourself some time in the process.

Of course, there are many other strategies to reduce your impact, such as using pre-built dependencies, more efficient libraries, or preparing a docker container for complex test environments.

It turns out that once you start looking, there are many technical options to reduce your impact. The more difficult challenge may be a social one, to convince your peers that change is necessary. In the end, we hope that this blog post inspired you, and that you will consider the environmental impact of your work.

NL-RSE: What Open Science can learn from Open Source

Lieke de Boer — Thu, 08 Jun 2023 00:00:00 GMT

NL-RSE meetups are bimonthly events where the Dutch community for Research Software Engineers comes together to learn new skills or hear inspiring talks by experts in the field of Open Science, Open Source, Research Software or Software Engineering. The next NL-RSE meetup takes place on 15 June in Leiden and will be about automated testing. You can sign up for the meetup here, and the mailing list here.

Back in 2021, lawyer, philosopher, and Open Science advocate Javier de la Cueva gave a talk at an NL-RSE meetup about what Open Science can learn from Open Source. This post is a report of that talk — originally first posted on the NL-RSE blog.

Photo by Mick Haupt on Unsplash

Javier de la Cueva does not like slideshows — “it limits the ways we communicate, like squeezing thoughts into 140 characters”, but for the NL-RSE meetup, he made one anyway. De la Cueva is a lawyer, philosopher, and Open Science advocate, with expertise in free software and intellectual property. During the meetup on Friday, 11 June 2021, he gave a passionate talk about what the Open Science movement can learn from the Free Software movement, blending the topics of software development and science in a way not unfamiliar to those identifying as RSEs.

If anyone’s an expert on these topics, it’s De la Cueva. He switched from Windows to Linux in 2000, after receiving a Blue Screen of Death on his Windows machine — “Why would you create a wall that doesn’t allow me to understand something I paid for?”. In 2002, when Wikipedia was planning to commercialize, De la Cueva and others forked the Spanish version of the page onto a server in the University of Seville, Spain, proving to Wikipedia’s owners that the entire project could be run on non-profit servers — a move some consider responsible for keeping Wikipedia free of commercial ads and paywalls to this day. Later, in 2006, he was, in his own words, “the first lawyer to obtain a judicial ruling where the term copyleft was included”. For the case in question, Ladinamo, De la Cueva defended the owners of a bar who were sued for failing to pay the compulsory licensing fee for playing music in their establishment. He won the case by successfully arguing that the music’s Creative Commons licenses allowed for public performance, and that no additional fees had to be paid.

In 2014, De la Cueva defended his philosophy thesis, Citizen Technological Pragmatics and Democratic Regeneration (abstract in English, thesis in Spanish). The work discusses how open access, open science, open data, and open government affect public knowledge and democracy. The thesis reflects Javier’s varied interests, providing insights from the fields of law, technology, political philosophy, and philosophy of language. Today, De la Cueva works as a lawyer and professor at IE University in Madrid. Along with his university teaching activities, De la Cueva is actively involved in science dissemination, “because a functioning democracy needs its citizens to be informed about scientific development*”. Science, he says, should not be about who does it, but about what scientists are doing. A common sensical enough statement, but one that may sound like an unrealistic utopia to anyone who has ever worked in academia.

Photo by Umberto on UnsplashAmerican sociologist of science [Robert K. Merton](https://en.wikipedia.org/wiki/Robert_K._Merton#

Sociology_of_science_and_CUDOS) recognized this tension: he observed that the social system of science works both in agreement with, but also often in contradiction to, its own ideal. Merton expressed the ideal of science in four concrete moral principles, referred to as CUDOS** : Science must be Communist (in the sense of communal), meaning that all scientists have common ownership of scientific goods, giving up intellectual property of their work in exchange for recognition. It should be Universal, or independent of the socio-political status and personal attributes of its participants. Scientific institutions should also be Disinterested, meaning personal gain of individuals within them is inferior to the greater goal of the scientific enterprise. Lastly, science must have Organised Scepticism.

For science to be in line with CUDOS, we have to rethink the relationship between scientists and their work. Because every scientific institution inevitably brings their own view of the world to the way they collect, analyse, and publish their data, open access to scientific publications only is not enough. After all, after data is collected, it is translated and analysed (often with the use of in-house developed software), written up into results, preprints and research papers, which can obscure the bias inherent in every step of the way leading up to that publication. Ideally, science openly shares every output that it creates, including data, publications, education, software and other tools. These outputs should be reused to two ends: First, to reproduce existing findings, De la Cueva argues — “there is too much emphasis on novelty in science”. Second, to combine all levels of existing scientific outputs into new structures, which De la Cueva considers crucial for innovation.

Open Science can look at the Free Software movement for an example on how to openly share its output. Free Software developers recognise the importance of their software’s reusability, which means they understand the importance of licensing and diversity of licenses. Scientists, on the other hand, do not hold the power to put a license on their scientific output. De la Cueva argues that this should change. Science could, for example, take inspiration from the Internet Engineering Task Force (IETF), which manages standards for the Internet. Any contributor permits the IETF to distribute their contributions freely, and in turn can use any of the other standards, which have been licensed in the same way. This results in an open Internet that is freely accessible to all. If we want CUDOS for science, the same mechanism could be applied to scientific outputs, where scientists permit their contributions to be distributed freely, and in turn are able to freely use others’ contributions.

De la Cueva points at two signs that science is moving in the right direction. First, the existence of preprint servers. Second, the rise in peer-reviewed open access journals. And arguably, the increasingly common practice of publicly sharing research software, and adherence to FAIR data principles, can be counted as positive signs too. But there’s still work to be done: “We still pay 9500 euros to obtain open access in Nature. And the default for publications is still All rights reserved. This has to end.”

Thanks to Lourens Veen and Pablo Rodríguez-Sánchez for helpful comments and additions.

Language Modeling: The First 100 Years

Carsten Schnober — Thu, 01 Jun 2023 00:00:00 GMT

ChatGPT has become the symbol for a new wave of Artificial Intelligence. New models with even stronger conversational skills are released on a weekly basis, and the world discusses about their impact on humanity. There are utopian and dystopian fantasies, as well as concrete potentials and risks. A view of the history of language modeling helps to understand the present and the future.

Photo by Brett Jordan on Unsplash

When Andrej Markov sat down with Alexander Pushkin’s poetic book Eugene Onegin in 1913, he was not planning to distract himself from his work as a mathematician. Instead, he went through the verses and counted letters, resulting in the study “An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains”, translated into English as late as 2006. The famous mathematician was not so much fascinated by the poetry, but by the distribution of consonants and vowels.

Andrej Markov — By Unknown author — Converted into JPEG format from [1], Public Domain, https://commons.wikimedia.org/w/index.php?curid=448609

Causes and Conditions

Markov demonstrated that the likelihood of a letter being a vowel or a consonant can be approximated from its preceding letters. Based on these findings, he developed a probabilistic model for causal sequences, today known as Markov chain. The so-called Markov assumption holds for language on various levels beyond characters, including words. Look at the following sentence stub:

This is a big <?>As readers, we cannot say which word should replace the <?>, but based on the preceding words, we know it is likely a noun (e.g. house or tree) or another adjective (e.g. *beautiful *or green) and most certainly not a determiner (e.g. *the *or a).

Conditional probability plays an important role: while the is the most frequent word in English, it is very unlikely to occur in the position of the <?> above. Language modeling is about formalizing this kind of linguistic structures: what is the probability of a word w, given a sequence of words w1, w2, … . The language model can be expressed mathematically as P(w|c), where c is short for the given word sequence*.* As English speakers, we know that for the context c given above, the conditional** probability of the — P(the|c) — is much lower than, for instance, P(house|c), even though the unconditional probability P(the) is larger than that of any other word.

Markov Chains — Joxemai4, CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0>, via Wikimedia CommonsApart from the grammatical functions of words, semantics — the meanings of words — play an important role when estimating these probabilities. While, for instance, dwarf is a noun just like house or tree, it is semantically less likely to appear after the adjective big; while big dwarf is perfectly valid syntactically, it is an oxymoron and therefore unlikely (but not impossible) to occur. Therefore, P(tree|big) is larger than P(dwarf|big).

By counting word sequences in a text data set — the training data — , these probabilities can be estimated. However, there will always be (new) words that do not occur in the training data which combine letters in previously unseen, but valid sequences. There also is an infinite number of valid sentences; regardless of how many texts a model has processed, words will always be arranged in ways that have never been seen before. Frequency-based methods conceptually suffer from the “problem of induction”: what was written in the past can at best approximate what will be written in the future. Taking that uncertainty into account has been the difficult part of language modeling.

Enigmatic Language

During World War 2, three decades after Markov’s experiments, mathematicians from both sides ran a race to encrypt their radio messages. With the help of statistical knowledge about language (plus some blunders from the other side), the famous team around British mathematician Alan Turing at Bletchley Park eventually managed to decipher the messages encrypted with the Enigma machine used by the German military.

The Enigma encryption machine — Museo della Scienza e della Tecnologia “Leonardo da Vinci”, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia CommonsMotivated by the success during the war, information theory and cryptography coined the mental model of language modeling. As a former PhD student of the famous information scientist Claude Shannon, Warren Weaver proposed to apply statistical methods to translate texts from one language into another.

The basic idea was: let’s look at a text in another language (Russian) as if it was an encrypted (English) text. We could then use the methods used for decryption to translate it from Russian into English. Probably needless to say that this did not work very well. The immense productivity of language does not really align with the way encryption algorithms work.

Distributional Semantics

From a linguistics perspective, the likelihood of a word occurring in a particular position in a sentence depends on two dimensions: syntax and semantics. For the former, human language tends to develop grammatical rules. Formalizing the grammar of a language by these rules was a focus for computational linguists for a long time, but never fully succeeded because there are many exceptions to these rules. Additionally, they are subject to change over time, and are not always followed even by native speakers of a language.

Modeling semantics, on the other hand, is essential for modeling not only syntactically valid, but also meaningful sentences. This poses other challenges than grammatical rules. There are identical words that have multiple meanings (homonyms, e.g. bank), there are different words that have (almost) identical meanings (synonyms, e.g. freedom and liberty), and there is a very broad spectrum in between. The nuanced meaning of a word can differ per culture, dialect, and even individual speaker.

The British linguist John Rupert Firth noted in 1957 that “a word is characterized by the company it keeps”. This insight has opened a beautiful door towards statistically modeling the semantics of a word. If two words appear in similar contexts frequently, it implies that their meaning is related. Note that there are many types of relatedness, including similarity, oppositions, as well as the syntactical function.

Distributional semantics and the Markov assumption have since melted into a powerful armory in language modeling by the use of n*-grams, windows in a text whereby n stands for the window size. For instance, three words form a tri-gram (or 3-gram). Like so many techniques for language processing, this can be applied on various levels again, including characters and words.

n-gram models of varying values for n have proven to be robust for estimating frequency-based language models without explicitly taking syntactic rules into account. In applications such as machine translation or speech recognition, such models have been combined with a translation model or an acoustic model respectively. They used to form the basis of the state-of-the-art algorithms in these and other sub-fields of Natural Language Processing for a long time.

Embedding Words

The rise of artificial neural networks has brought its own methods to implement the distributional semantics hypothesis. Word2Vec (2013), Glove (2014), FastText (2016) and more generate vector representations for each word based on context windows around the word. These algorithms do not directly rely on frequencies, but their parameters are optimized so that the resulting representations of two words end up close in the vector space if they tend to occur in similar contexts. These vector representations are called word embeddings.

Word embeddings do not only represent the meanings of words, the vector space into which they are embedded also allows arithmetic operations in the semantic space. A notorious example: when you take the vector for king, then subtract the vector for man, and add the vector for woman, you come out close to the vector for queen. Interestingly, these embeddings tend to encode both semantics and syntactical functions — reflecting the linguistic insight that the boundary between syntax and semantics is often affluent.

Distributional Word Representations — Singerep, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia CommonsGiven those representations of individual words, the next level in language modeling is thus to represent sequences of words — phrases, sentences, or entire documents. The simplest approach is to concatenate or average all the word vectors of a sentence, and use it as input to any machine learning algorithm.

Recurrent neural networks (RNNs) have been more successful by combining a neural network’s state after processing one word with the embeddings for the following word. After iterating over an entire sentence, the network output yields a vector representation, including linguistic phenomena like negation and references between words.

RNNs, however, face two major disadvantages: important contextual information can be located at a far distance in a text, and the recursive nature of an RNN prevents parallel computation. LSTM and GRU gates have extended RNNs, but only mitigated the first issue.

In 2017, the architecture that enables current models was proposed: Transformers. Dedicated *attention matrices *(already proposed in 2014) learn which parts in an input sequence are important, instead of paying attention to a predefined context window. That allows a language model to, for instance, interpret the meaning of the ambiguous word bank by paying attention to occurrences of words like *river — *or perhaps rob and *money *— anywhere in a text. That mechanism enables Transformer models to take very large contexts into account, while the optimization of the model parameters can be parallelized efficiently.

There are different techniques for the optimization of these parameters. The famous BERT-family of models is optimized

to fill random gaps in the sentences of the training data (masked language modeling), and
to decide whether a particular sentence follows another one (next sentence prediction).

These techniques are especially effective for generating vector-based, numeric representations of words and texts that can be used for NLP tasks like, for instance, document classification.

Generative models like GPT on the other hand go back to where Andrej Markov started: they are optimized to predict the next word in a sequence. On top of that, the RLHF (Reinforcement Learning from Human Feedback) technique has enabled the model to learn very effectively from human “AI Trainers”, resulting in the impressive conversational skills that ChatGPT has shown.

The combination of more effective and scalable training techniques and the immense number of parameters enable such models to produce texts in a specific language. They are fed virtually the entire internet, therewith learning almost all human languages. Even programming languages come as an (intentional) extra, enabling GPT-based models to function as programming assistants.

Modeling Language: The Next 100 Years

Looking back into the history of language modeling illustrates that Artificial Intelligence has not suddenly emerged in the recent past. The current generation of models builds on a rich past of active research in the fields of computational linguistics, natural language processing, information theory, and related fields; to reiterate some milestones: the RLHF method from 2019 fine-tunes models based on the Transformers neural network architecture from 2017. Transformer-based models have improved upon the word embeddings as introduced by Word2Vec (2013). Linguistic and stochastic theory has laid the basis for such algorithms.

The race has now shifted into the arena of engineering and productizing, as in making language models smaller and faster and embedding them into useful applications. Significant progress on the quality side is nowhere to be seen currently. As an internal Google document stated recently, none of the big companies like OpenAI, Microsoft, Google or Facebook seem to have a model or strategy that could outcompete the others. More efficient, often free, and qualitatively competitive models appear on a weekly basis.

On the other hand, language models can generate incredibly fluent language, but are intransparent and inaccurate by definition. These properties make them credible actors in conversations with humans, while posing difficulties on applications like search and coding assistance.

Debates about imminent singularity in which Artificial Intelligence spontaneously starts improving itself live in the realm of fiction rather than science. Such mystifications can take both utopian and dystopian angles, but rely on beliefs, lacking specific evidence. Apocalypses have been predicted many times with respect to Artificial Intelligence as well as other technologies. Grand stories about the faith of humanity absorb a lot of attention, but tend to reduce and over-simplify complex matters.

Technology, however, is only one factor — predictions about the future of our society, let alone humanity, are at best naive if they ignore political, economical, and societal factors. For a productive discussion that takes realistic risks and potentials of new technologies into account, experts from all these fields need to join forces.

Research Ethics and Authorship: The Technological Solution

Candace Moore — Thu, 25 May 2023 00:00:00 GMT

Dr. Candace Makeda Moore discusses the potential for distributed version control as a tool to prevent questionable authorship practices.

Recently I read a story in Retraction Watc h about a service selling authorships on scientific papers in the peer review pipeline. On the same day a colleague physician mentioned that at our hospital, there were many cases where authorship was given in exchange for access to certain patients as potential research subjects. While the former type of blatant research fraud makes headlines, the latter barely raises an eyebrow. But giving authorship for research related favors or other ‘political’ reasons is neither a victimless minor crime, nor it is a problem without an easy, free and enforceable solution.

Photo by Annie Spratt on Unsplash

Because I come from a medical background, I am most familiar with standards in this field, although problems related to authorship extend to most scientific fields.

Perhaps at this point it’s worth pointing towards what authorship should actually be, using academic medicine as an example. The International Committee of Medical Journal Editors (ICJME) has exemplary recommendations. The spirit of these recommendations is incredibly simple: authors should be involved in the design or analysis as well as the literal writing of research. There is some variation in different fields, and differences from place to place, and of course there can be exceptions with specific kinds of research papers e.g. papers for datasets and/or code. But we probably all agree that in least in principle, an author should be someone who does something significant for the work.

Unfortunately, in day to day academic medical research it is typical to see people getting authorship on papers for all sorts of illegitimate reasons. Getting someone to sign off that researchers can access patients in a particular department, handing over an existing dataset, and many more tasks are necessary and integral parts of research. Such actions deserve praise, and there should be some system to reward such tasks in place, but as the medical-scientific community has defined authorship, they alone should not confer authorship.

There are several major impediments to the adoption of proper research ethics in authorship in medicine. Some are simply related to inertia and the infamous conservatism of medicine. At many institutions authorship is given in customary ways including to any principal investigator (PI) who so much as attends meetings around research, because ‘that is the way it has been done’. And of course no one wants to get on the bad side of someone like a powerful principal investigator who can potentially ruin their career.

Photo by National Cancer Institute on Unsplash

This is, of course, another reason fraudulent authorships happen quite commonly. Politics. In some cases a principal investigator or high ranking researcher may provide very necessary help to a research group, e.g. without their signature nothing will move. But the truth is that there are well defined rules around authorship. One of the reasons for this is that authorship is so critical to advancing academic careers. Unfortunately, in academic medical centers, in practice what this means is that all levels of researchers enter into a treacherous political game where the bulk of authorships tend to accrue towards more established researchers who have more resources in the way of data access, workers and political capital. Such systems favor researchers who are privileged across almost every other axis of privilege possible.

A medical researcher who lives in the real world of any large academic hospital may think ‘if I don’t give authorships to important professors who helped me, I’ll never get their help again, or worse, they will open some kind of problems for me’. They feel disempowered about what would be essentially telling the truth by demanding correct authorship lists, because it would be their word against the word of someone more powerful and established in many cases. But there is literally a free solution right beneath the nose of researchers.

Every modern researcher uses a computer to accomplish aspects of their research from code scripts or programs for statistical analysis to word processing programs for writing articles. The solution to these messy problems of authorship is right at the fingertips, or only a mouse click away from anyone with a computer and internet connection. The solution is distributed version control systems like Git or Mercurial.

When a version controlled repository is created, there is a history of who wrote what and when. Unlike Google Docs or Word or any other of the more common tools for writing papers, version control makes it almost impossible to erase the trace of someone else’s work, or claim you or someone else made significant contributions in writing an article when it did not happen. Unless, of course, you are part of the extremely tiny group of researchers particularly talented with programming. But let’s face it, most people in medical research don’t want to use version control because they literally don’t even know how. They never learned it because it was never considered a critical skill for research.

Some people do not find git or other version control languages and programs intuitive. For whatever reason this seems particularly true of physicians. However, even leaving issues of authorship aside, version control is an important skill in terms of modern computational research. The Netherlands eScience Center offers small hands on courses lasting one or a few days to train researchers in this area. An example of one upcoming such course can be viewed here. We have also contributed to high quality open source free lesson material.

While such programs are often used in practice with services like Github or Gitlab, they need not be. Technically, you could just create a repository and send files for it to other people. You can even sign commits with and/or make them traceable by servers. If you use git, you can use `git blame` to figure out who did something.

I may have some blame for problems in the environment in this program (output of `git blame` in a terminal)

Obviously, using version control as the medium through which scientific papers are written favors the young and computer savvy. Would this cause the reverse problem to privileging the old and established? Hardly. Help for Git or any other open source version control system can be found across the internet. Lately, I often run across young students who have taught themselves to use a version control system without anyone asking them to for a class. The biggest barrier to learning version control systems for most high ranking researchers in medicine may be the psychological pain of having to ask people they consider their inferiors for help.

In a world where people are questioning if what journals actually do is of any value when everyone could just publish on the internet and/or preprint servers, checking authorship should be a task that they are legitimately valued for. Such a task could be accomplished using free open source software like Git. The only thing necessary is a dash of skill in computing, or barring that, some learning.

One approach to begin learning version control is simply searching and reading on the internet

How am I sure this could work? Because we see these processes everywhere and all the time in terms of the code related to research. I myself am an author on several open code repositories related to medical signal processing and/or image processing, and if anyone were to question my role, I could just point to the lines of code I wrote as evidence that I did in fact do real work. Open source code provides a model for how papers could be made way more transparent in terms of authorship. In fact if researchers were to send their code repository and a related repositories which include study design documents, every aspect of authorship can be verified.

Physicians are often called upon to lead research in medicine for obvious reasons, but have never really get trained (save a few who had post-graduate training in very select fields such as imaging informatics or public health) to do any computational research at all. A future where physicians and others working in medical research take a few days to learn the fundamental tools of computational research is a better future for everyone, because then we can efficiently combine clinical insight and large datasets. Whether to shield themselves from authorship problems, or go after the larger goal of better science, medical researchers should embrace version control.

Creating a Community by Integrating Expertise, Tools and Ideas

eScience Editorial Team — Tue, 16 May 2023 00:00:00 GMT

A personal reflection on the first NPLinker eScience workshop for Natural Product Discovery

By Justin van der Hooft

This blog post is written by Wageningen University & Research Assistant Professor Justin van der Hooft who participates in the Netherlands eScience Center’s Open eScience Call 2021. His project, “A community-supported workflow connecting microbial genes, and organisms to their molecular products”, aims to facilitate the finding of novel bioactive molecules from nature. It does so by enabling the integrated use of various omics data types.

Photo by Adrian Lange on [Unsplash](https://unsplash.com/photos/Wk902ZLaA7M?utm_source=unsplash&utm_medium=referral&utm_content=creditCopy

Text)

Motivation for the workshop**

Nature is still full of unknown chemistry with possible beneficial properties such as antibiotic ones. Genomics technologies measure the genetic content of organisms in which the blueprint of their chemical output is present. Metabolomics technologies generate tons of spectral data of actually produced molecules. However, the analysis of large-scale omics datasets is still very challenging, especially when several types need to be integrated. What can we predict from genomics? What can we understand from metabolomics? And how best to connect these two workflows when doing exploratory data analysis without drowning in the data?

Outline of the workshop**

During this first workshop in a series of two, we aimed to bring together researchers interested in integrative omics analysis with various backgrounds in terms of their career stage, the technologies they are already familiar with, and the organisms they work on. Following a selection process, 30 participants were selected to come to Wageningen to learn about analyzing genomes and metabolomes, and to get an outlook towards the integration of the two. The aims of the workshop were:

Bring together Genome & Metabolome Miners and learn the motivation and key characteristics of main genome and metabolome mining tools.
Educate and train natural product researchers and other interested scientists in genome and metabolome mining.
Establish together with the community what are the perceived main challenges in combining genomics and metabolomics, and the wishes for tools that address these.

To stimulate networking amongst the participants and instructors, several measures were put in place: extended coffee and tea breaks, the opportunity to present a poster, and group discussions in changing compositions. We are grateful for the financial support from the eScience Center to make the workshop affordable for participants from across the globe through a travel reimbursement scheme and to enable the presence of 4 international instructors.

All 30 participants from across the globe together with the instructors in front of the Omnia building at the Wageningen University & Research campus.

Key insights

If there is one thing that became clear to me during the workshop, then it is this: multidisciplinary research requires open science and collaborative efforts to come to full fruition. During the workshop, one of the most heard perceived difficulties to do integrative omics analysis, is the (perceived) steep learning curve to get familiar with another type of omics data and the analysis workflow that comes with it. How to judge if this data is of good quality? How do you know if you used the right settings during the analysis? Hence, to do integrative omics analyses for natural product discovery, effective knowledge and skill transfer is needed. Here, having open access materials available is one part of the solution: researchers can use them to get familiar with the topic and the main things to consider for a successful project outcome, and teachers can use the materials to get inspired to teach this topic to students. We have also discussed the need for clear guidelines and instructions to run analysis software tools, and how the collaborative writing of documentation could be a solution to get to instructions that are clear(er) for a wide(r) audience. Finally, most tools, especially those relying on machine learning principles, perform best with increasing amounts of available data: all participants agreed that sharing is caring, and Cunliang Geng, research software engineer from the eScience Center, also provided an overview of how to make your data and software FAIR.

Practice as you preach

There was a lot of knowledge and skill exchange between participants themselves and participants and instructors that cannot be easily captured here. One tangible outcome for the workshop partipicants is the collaborative notes document that they collectively created and which is full of useful links and thoughts. Another tangible outcome of the workshop for the broader community is that the instructors all made as much as possible the materials available under an open access licence! I hope that this initiative will help to further foster teaching the future generation of omics miners, to aid in setting up teaching materials for genome mining, metabolomics, and metabolome mining, and to promote discussions in between and bring together the fields of genome mining and metabolome mining.

With huge thanks to all the NPLinker eScience Workshop instructors who came from across the globe to Wageningen to teach the participants many facets of genome mining and metabolome mining.

Personal reflection

Overall, I am glad that the participants generally liked the workshop format and content and that they look back at a great week in Wageningen. In particular, I was curious if the group discussions after lunch were appreciated, and based on the evaluation this was a good move, as it prevented the “after-lunch dip”, yet at the same time the participants could share their insights and frustrations. Furthermore, many of them were happy to realize that there are more scientists out there that struggle with integrating omics analyses and that have complementary skills and background to them. This was exactly the idea behind the community building! Also, we got some valuable feedback when organizing future workshops as this one: it is challenging to find the right balance for everyone between theoretical lectures and hands-on tutorials. Big thanks go out to all involved for their active participation, including the secretarial support in Wageningen. Finally, I look forward to organizing the second NPLinker eScience workshop that will be fully focused on how integrating genome and metabolome analysis can boost natural product discovery. The final day of the current workshop already demonstrated the currently ongoing developments within the NPLinker eScience project as well as by research groups around the world.

Community building during the workshop dinner

Molecular Simulations using Machine Learning, Part 3

Aron — Thu, 04 May 2023 00:00:00 GMT

In the first part of this series we looked at the physics of simulating molecules, and went from quantum mechanics to something that is more efficient to solve, namely density functional theory and molecular dynamics. These both required the introduction of an unknown potential. In the second part we looked at designing a machine learning model tailored to approximating this potential.

Photo by D koi on Unsplash

In this final part we will discuss what goes into training this model, in particular how to obtain the training data. And we will conclude with some caveats.

It’s possible to read this without having read the previous parts, but I would recommend at least reading part 2, and part 1 only if you’re interested in the physical background.

I hope that these considerations will also be useful for anyone thinking of applying machine learning to their own field, even if it is not related to molecular simulations.

How to obtain the training data

To be able to train the model, we need lots of examples of nuclei positions and the corresponding interatomic potential. To see how we can obtain this, we have to go back to the figure summarizing the first part:

Remember we decoupled the electrons from the nuclei, the electrons being described by Density Functional Theory (DFT), and the nuclei by molecular dynamics (MD). For MD we needed an interatomic potential, which captures the effect the electrons have on the nuclei. But since we do have a way to compute the electron density, namely through DFT, the obvious solution is to use DFT to compute the interatomic potential that we then use in MD, this is called ab-initio Molecular Dynamics (aiMD).

The downside as we discussed was that this is very slow, which was the reason to try using machine learning in the first place. We can still use it to compute the training data though! Note that this does assume that we can do DFT, i.e. that we know a density functional that is accurate for the system we want to study.

After designing the machine learning model to be specifically tailored for this application, and thus very data-efficient, the hope is that with relatively little training data we can train a model, that we can then use to do much more and bigger simulations. The hope then is that once the model is trained, it will be as accurate as ab-initio molecular dynamics, but as fast as molecular dynamics using a simple potential, combining the best of both worlds.

When it comes to generating the training data, there are several tradeoffs that must be carefully considered. We will discuss several of them here.

system size

Since computing the potential for the training data is so expensive, ideally we want to be able to train the model on a small system, and once trained use it to simulate a larger system.

To be more concrete, we could take for example 10 water molecules, and for 10.000 different configurations of these 10 nuclei use DFT to compute the interatomic potential. That gives us the input and output pairs on which we can train the model. Once it is trained, we can use this model to simulate millions of water molecules for many more steps.

The tradeoff here is that if we make the training system too small, it will never be able to learn enough about the larger system. In an extreme case, if we take as the training system a single water molecule, the model wouldn’t see any interactions between two molecules during training.

So we must choose a training system that is small enough to be able to use DFT to generate the data, yet large enough that it captures all the interactions present in the much larger system the model will be used on once trained. What this is will depend heavily on the system considered, and will be a matter of trial and error combined with chemical intuition.

system configurations

Another question is how to choose the inputs, i.e. the nuclei positions on which we compute the interatomic potential. One option is to just do an aiMD simulation and use all of the steps as inputs. This has the advantage that, with the exception of the initial steps, all of the configurations were obtained by following an MD trajectory, so we can expect that they are typical and we are likely to run into similar (but bigger) configurations once we use the model. So it’s very beneficial if the model performs well on these configurations.

Ideally we want all the inputs to be independent from each other, this maximizes the added value of each additional sample. So the other extreme is to sample the nuclear configurations at random, then they will indeed be completely independent from each other. However, configurations sampled at random are likely not to be physically realistic. So instead of samples being irrelevant because they are very similar to other samples, now they are irrelevant because they won’t be encountered when the trained model is used.

So there is a tradeoff between diversity and representativeness. A good tradeoff between these two extremes can be made by first doing an MD simulation using not DFT but a faster, less accurate potential. These configurations will be much more representative than random ones, but much faster to obtain than using aiMD the whole time. Then, to minimise the correlation between steps, we only take every say 100th step. Finally, because the potential we used was less accurate, only for the steps we picked do we recompute the potential using DFT.

In addition it helps to run the training simulation at a higher temperature than the system you want to use the model on. This makes the system explore more of the configuration space, and prevents the model running into unknown territory during its use.¹

Quality vs quantity

A final consideration is between the quality and the quantity of the data. Since we compute everything from DFT, and within DFT there are several choices that can make it more accurate and slower or vice versa, choices must be made here too. There need to be enough samples for the model to be able to generalise to configurations it hasn’t seen, but it also needs to be done within our computational budget.

Tradeoffs to be navigated in generating training data.

Many more considerations

There are tons of other things to consider, here I have focussed on some of the issues that are most particular to this application, rather than providing a full overview of deep learning. I have also focussed on the more conceptually interesting questions. There are lots of details though that can have a significant effect on the final performance, such as what activation function is used, or what optimizer². It is best to start with an existing method if possible, and if improvement is necessary experiment with varying these details and seeing what effect they have on the final performance.

Getting started

If you are a chemist and want to try this for yourself, a good place to start might be nequip, allegro or DeepMD-Kit, these work smoothly with the molecular dynamics code LAMMPS. SchNetPack works with python’s atomic simulation environment. They all implement different models, so it is a matter of figuring out which model works best for your use case and with which software you are most familiar.

Unfortunately due to personal circumstances I didn’t have time to do any experiments myself, so I can’t recommend any particular one.

Example

After all this theory let’s look at one example from the Allegro paper. In the figure below a small system of Li_3 PO_4 is shown, consisting of 192 atoms. The training data the authors made consisted of 11,000 configurations selected randomly from a 50,000 step ab-initio MD simulation. To get a sense of the time scales involved, the time step in this case was 2 femtoseconds, where a femtosecond is 1/10⁶ nanoseconds. The model was trained on this for machine learning standards very small dataset.

It then was able to simulate a much bigger version of the same system, consisting of 421,824 atoms, at a speed of half a nanosecond per day on a single GPU, or even 50,331,648 atoms at half that speed using 128 GPUs.

Example taken from https://arxiv.org/abs/2204.05249.Assessing the accuracy of the model is a much more subtle question than say for a model that classifies whether an image is a cat or a dog. I will just explain one plot from the paper that is just one measure of the accuracy. What is shown on the right in the figure above is the radial distribution function. This is a measure of the probability of finding a pair of atoms at a given distance. The peaks in the graph reveal characteristic structures or patterns in the material, such as the most common atomic bonds or the preferred spacing between certain atom types The figure shows that Allegro managed to come very close to the accurate but slow ab-initio Molecular Dynamics, at least in this measure.

Conclusion

Deep learning can be used to model the potential energy between molecules, needed in molecular dynamics simulations. This is a rapidly evolving field, with multiple libraries under active development, and regular papers with further improvements in methodology. It is exciting to follow these developments, and the larger scale simulations they will enable. For a relatively recent review, see here.

footnotes

1: Thanks to Simon Batzner for correspondence on these issues.

2: In fact as I write this blog, the latest innovation is exactly a new optimizer.

Researchers are not born engineers — why are they expected to know how to code?

Lieke de Boer — Thu, 20 Apr 2023 00:00:00 GMT

Subscribe*Remember me for faster sign in

I refuse to be too hard on my past self. It’s understandable that researchers, who juggle multiple projects and deadlines, do not have the time, energy or resources to become coding experts. But research is increasingly reliant on technology, so it’s important that researchers have access to the support and education they need to develop and use high-quality code. Learning good coding practices (or having access to those who know them) not only improves the reproducibility and reliability of research, but it adds to the fun of coding, saves time and prevents future headaches.

Photo by Clément Hélardot on Unsplash

When I sat in on a course on good coding practices at the eScience Center two years ago, I first realized why my own code had been bad: I did not know about good practices. I learned how to properly collaborate on projects via GitHub, which was eye-opening. The fact that you could use branches to work together on solutions to different problems was completely novel to me. The fact that you could write tests to make sure you would realize if part of your code failed was mind-blowing. Also, making my code modular, so that scripts would not contain upwards of 2000 lines, made the entire project so much easier to read and understand. Writing proper help functions was something I used to feel was not that important, because I never wanted anyone to look at my code. Now, documentation is something I strongly believe in.

I am currently farther away from being a researcher in my daily job than ever, but in this job I did write an R package, together with my eScience Center colleague Barbara Vreede. Barbara is an amazing research software engineer who guided me through the process. The package is called traininginfrastructure*and creates all necessary communication documents and folders for the workshops we run at the eScience Center. I’m very happy to say the package works and is used by my colleagues. It saves engineers who teach time on repetitive administrative duties and prevents mistakes in creating communication (no more copy-pasting emails with typos).

By no means did I become a perfect research software engineer. But I realized how much I had learned when recently, a researcher contacted me to ask if she could use the same analysis that I had used during a project for my PhD. I reluctantly gave her access to the old private GitHub repo with my project’s scripts and had to dig some non-version controlled other scripts out of my email inbox. Being confronted with that code was sobering. One day, I may try to clean it up so that they are usable for more researchers in the future. But it made one wider point very clear to me: my work at the eScience Center really serves a purpose. It can bring people like me, researchers who write scripts that work, but that they hate, to write code that they can be proud of and freely share with those that may need it. It can save all of us time and effort. And what I learned here really did not take that much time: a 1-week course and some quality guidance in building a fairly simple R package.

Some of those who collaborate with us on projects at the eScience Center find it difficult to accept our engineers’ wishes that they use the same practices we do in research software projects. I now understand that using good practices is the only way to build quality tools for research. If you are ever in a situation where someone forces you, you may end up gratefully learning about (and putting into practice!) version control, collaborative software engineering, testing, and documentation. It can be painful in the beginning, but it will make your and others’ life so much easier in the future, and your software a worthy research output on your CV.

If you want to learn how to write quality code for research, check out our free workshops! You can also apply for our calls, or sign up to our newsletter to stay tuned.

Molecular Simulations using Machine Learning, Part 2

Aron — Thu, 13 Apr 2023 00:00:00 GMT

In this post, I will walk through the process of designing a model used in molecular simulations, from essential to state of the art. This is the second part of a series on using machine learning for molecular simulations. It is relatively self-contained if you haven’t read the first part.

Photo by BoliviaInteligente on Unsplash

We will design a model that approximates a potential used in molecular simulations. We will start from the simplest possible model, and reason our way step by step towards the essentials of the current state of the art models. This at the same time roughly follows the historical development of these models, although we will not attempt to be precise in this regard.

The principles we use to arrive at a good model are quite general. Although they will lead to different models in different applications, I hope this can be helpful for scientists in different fields thinking about applying machine learning to their own field.

Quick recap

The figure below summarizes part 1 in a bit more detail. We saw how quantum mechanics was simplified into Density Functional Theory describing electrons, and Molecular Dynamics describing the atomic nuclei. The clouds in the figure indicate a-priori unknown ingredients that arose from these approximations: the density functional and the interatomic potential.

We will focus on the interatomic potential: a single function depending on the positions of all the nuclei. The same principles that we use to derive a model for this also apply to the density functional, but the details will be different, because its inputs are different.

Simplest machine learning model

When designing a machine learning model, it is useful to start from the very basic question: what should go into it, and what should come out.

Well, we want the model to compute a potential, which is a single number, depending on the coordinates of all nuclei. So if we have N nuclei in 3-dimensional space, the model should take 3N numbers and turn them into a single number.

The simplest machine learning model: a linear model.

The simplest possible machine learning model that meets this requirement would be multiplication by a 1 by 3N matrix W: M(x) = W x. This would multiply every one of the 3N inputs with a weight, and sum them all up. These weights are the trainable parameters. These are initially set to random values, and updated during training by correcting the errors the model makes. That part will remain the same throughout.

This model has many undesirable properties, the most problematic one being that it is way too simple. A more precise way of saying this is that it is linear. This means that if we multiply all of the inputs by 2, the output will be multiplied by 2 as well. Or if we add 3 to say input x₁, the output will be the original output plus 3 w₁, where w₁ is the corresponding weight. In reality of course the effect of this change of one coordinate on the potential depends in a complicated way on its relationships to the coordinates of other particles. This model will never be able to capture that, no matter how much data you throw at it.

Deep learning

Removing this restriction takes us from machine learning to deep learning. The simplest neural network is basically a sequence of the simpler models from above, intermingled with nonlinear functions called activation functions. (Without these, the end result would still be linear!)

The simplest deep learning model: a fully connected neural network.

The simplest example of this class of models has one hidden layer: it takes the 3N inputs to say 100 numbers by a matrix multiplication, and applicaties an activation function to each one separately. Then through a second matrix multiplication it takes those to the one number we need as an output.

There are many different activation functions, conceptually the only important thing is that they are nonlinear. A simple often used example is the ReLU, which is 0 if its input is negative, and otherwise just passes on the input unchanged.

The effect of adding just one of these hidden layers is profound. In a sense, this immediately fixes the problem with the previous model. That is, this model can approximate anything now, given enough parameters. This statement is known as the universal approximation theorem. However, that is quite a theoretical result. It doesn’t say how large the hidden layer should be, how much data is necessary, or even if it’s possible to find the right parameters through training at all, only that they exist.

To address all of these other concerns, we need to specialise the model further to our particular application. In other words, we want our model to not only take into account just the size of the in- and outputs, but also its structure. As a first example, although the model above has the correct number of 3N inputs, it is not aware at all that actually these have the structure of N triplets of coordinates.

To elaborate on this and make it more precise, we’ll look at the concept of equivariance. For a book and lectures on exploiting the structure of our data that is much more detailed, but also quite technical, see https://geometricdeeplearning.com/, and this paper for applications specifically to molecules.

Equivariance

Equivariance means that if we first transform the input and then apply the model to it, we get the same result as when we first apply the model, and then apply a similar transformation to the output. It is a property that a model can have for some transformations but not for others. For example if this is true for a model M when the transformations are rotations, we say that M is rotation equivariant or equivariant with respect to rotations, illustrated in the figure below.

Example of equivariance of a hypothetical model M that just detects edges. No matter if we first apply the model and then rotate the result, or first rotate the input and then apply the model, we end up with the same result.

Note that the definition refers to a similar transformation on the output, not necessarily exactly the same as we applied to the input. In fact, usually the output of the model will be of a different shape than the input, so we cannot even do the exact same transformation. The important part is that we know a-priori what the transformation on the output is, and it depends only on what transformation we did on the input.

A special case of equivariance is invariance, which is when this similar transformation is actually the identity. In other words, if we transform the input, the output will remain unchanged. For a simple example of this, if the hypothetical model in the image above were just a classifier saying whether the image is a cat or a dog, we’d expect it to be invariant: for both inputs on the left of the image, the outputs on the right should be “cat”.

Requiring the model to be equivariant with respect to some transformation imposes some constraints on it, it decreases the freedom in the model, or the number of parameters. So as long as the equivariances we impose are also satisfied by the true solution we are trying to approximate with the model, it makes it more data efficient and easier to train. If we didn’t impose equivariance, the model could potentially have learned it from the data, but that would require more data, more time and more parameters. If we know this property beforehand, clearly it pays off to enforce it, preventing the model from the get go to make mistakes of this kind.

There are many different transformations that the true interatomic potential must be equivariant to. We will discuss these one by one and use them to refine our model.

Permutations

In turning the positions of all nuclei into a 3N-dimensional input vector, we had to implicitly decide an order: which particle is the first particle, which is the second, etc. Clearly this is just a choice of labelling; if we pick a different order for the exact same configuration of particles, the potential shouldn’t change. So we want our model to be invariant with respect to permutations.

Currently it is not, it has no idea that the output should be the same, and so it would have to learn that by seeing lots of examples of identical configurations that are labelled differently, but with the same training label. Clearly this is a huge waste of time, and we want to avoid this.

There are several common functions that do not depend on the order of their arguments, i.e. that are permutation invariant, such as the sum, or the mean, or the maximum. We will just take the sum, as it preserves more information than the maximum. So the very simplest permutation-invariant model would just take all of the nuclei’s positions and sum them. (Actually this would still leave us with 3 coordinates, so we’d need a 1 by 3 matrix at the end to turn it into a single number.)

This is of course again much too simple, in fact even more so than what we started with. We can improve the situation while still taking permutations into account by having only the last layer be invariant, and adding equivariant layers in between.

Any function that does exactly the same to every nucleus will be equivariant with respect to permutations: if we reorder the inputs, the outputs will be the same as the original, except that they are also reordered in exactly the same way. For this function we can use what we ended up with in the previous section: a fully connected neural network. So that takes a nucleus’s 3 coordinates, and does several matrix multiplications and activation functions on it to end up with a different amount of numbers for each atom, that we can choose.

Note that if the intermediate layers were not equivariant, there would be no way to guarantee invariance at the final layer. Furthermore, it also already makes the model more data efficient, as one training sample automatically teaches it about all the different orderings of that same sample.

Another important consequence of this is that, unlike the fully connected level, we now have a model that works regardless of how many nuclei there are. This is great, because it will allow us to train the model on a small system, and then use it on a larger system.

Locality, graph neural networks

The above modification does impose permutation invariance, but it was still too restrictive. It treats all the nuclei independently, finally just summing “individual potentials” to obtain the total potential. This is not realistic, we expect the potential to depend also on the distance between nuclei, something that this model cannot learn.

We can relax this restriction and at the same time incorporate another physical prior by adding more structure to the input.

The physical principle is locality: what happens at some point x only depends on things close to x. More concretely in this case, the Coulomb potential between two particles at a distance r decreases as 1/r. This means that the interaction between two nuclei that are very far from each other doesn’t have any significant contribution to the potential.

So for nuclei closer to each other than some cutoff distance r_c, we want to take their interaction into account, and we can connect them by an edge. Those that are further away are not connected by an edge. This way we turn the original point cloud input into a graph.

We then extend the operation that previously did the same to all nuclei, to allow it to depend on an atom’s neighbours as well. However, the number of neighbours will be different for different nuclei, and if we allow this to make the output shapes of each atom different things will quickly grow out of hand. So we need a way to combine the neighbours in such a way as to keep the shape independent of the number of neighbours. Furthermore we again have the issue of permutation invariance, now at a smaller local scale. The solution to both problems is the same: we sum over the features of the neighbours.

Illustration of message passing in a graph neural network. An atom c receives “messages” from its neighbours n and updates its state using those. The neighbours at the same time do the same. Multiple steps of this allow the message to propagate further.

This is another permutation equivariant operation, now on graph inputs. It is called message passing, because it can be thought of as all of an atom’s neighbours passing messages to the central atom, who adds them all up. Doing this multiple times for all nuclei increases the region to which an atom is sensitive, because in the second step it can indirectly receive messages from its neighbours’ neighbours.

At the end we have to sum over all nuclei again to obtain something invariant to permutations. These types of models are called Graph Neural Networks (GNNs).

Spatial invariances

While permutation invariance was more about the way we describe the system, how we label the nuclei, the remaining ones are actual spatial transformations: translations, rotations and reflections.

If we move (“translate”) all nuclei in the same direction by the same amount, the interatomic potential doesn’t change. That is, whether we perform an experiment in Amsterdam, or we perform the exact same experiment in New York, the outcome should be the same. And the same holds for rotations and reflections. (If there were any external influence like gravity this would be different, but that is insignificant at the atomic scale.)

As the model is now, it doesn’t know any of this. Building this in to the model makes it a lot more data efficient. From just one training simulation it would learn automatically about all its translations, rotations and reflections.

The simplest way to enforce this is to only give the model inputs that are themselves invariant to these transformations. Clearly, if the inputs are not affected, the output is also not affected.

That means we cannot have an atom’s position as an input by itself, as clearly that is not invariant under any of the transformations. But since we now have a graph, we can focus on pairs of nuclei, and take the difference of their positions as an input. That is already invariant with respect to translations: if we add the same vector to both positions, it will cancel out in the difference. It is not yet invariant with respect to rotations or reflections, but if we reduce it further to just the absolute value of the difference, it is.

That does the job, but it is again quite restrictive. Note the parallel here with the permutations we discussed previously: the simplest is to make everything invariant from the start, but it will turn out to be unnecessarily restrictive.

For an example of something that is missed in this approach, consider a central atom with two two of its neighbours. The vectors pointing from the central atom to those two neighbours make up a certain angle to each other. If you think about it, this angle is also invariant with respect to all of the transformations. And it might well be a useful feature that has an effect on the potential that we want the model to represent. But there is no way to reconstruct this angle just from the absolute value of the differences in positions. Had we not thrown away that much information so early on, we would be able to recover this angle: it’s related to the inner product between the two vectors.

The angle φ is invariant to translations, rotations and reflections.

We could of course add these angles to the input “by hand” and stick to imposing invariance from the start, and there are models that do this. But there are many more features that we are throwing away. A more systematic approach would again be to only impose invariance in the final layer, and keep the others equivariant.

Higher representations/orbitals

It gets a bit more complicated though. The complication is that there are many ways that things can transform under rotations. Chemists know these as orbitals, physicists and mathematicians as different representations of the rotation group.

I’ll give an idea that, while not mathematically rigorous, captures the intuition of what’s important for us here. Imagine we start with two vectors representing the positions of two nuclei, which are in what is called the L=1 representation or the p orbital; here, the vector remains unchanged when rotated along its own axis but changes its position when rotated in any other way.

We can combine these two vectors by taking the length of their difference. As mentioned before, this is invariant, it is called the L=0 representation or the s orbital. It is a single number that doesn’t change under any rotation. We can also simply add them, then it remains a vector.

The two vectors together define a plane, a two dimensional surface that they both lie in (assuming that they are not parallel). Or more concretely imagine one vector pointing out of the cockpit of an actual plane and one along one wing. The plane they define together would be the floor of the inside of the actual plane.

Loosely speaking this is how the two vectors can be combined to a higher representation, this would be the L=2 representation or the d orbital. Note that this plane depends on both vectors: if we rotate along the wing of the plane, that vector doesn’t change, but the one pointing out the cockpit certainly does, and so does the floor of the plane.

This pattern continues: every time we add a new vector into the mix, we can either lower the representation by one, keep it the same, or increase it by one. For example if we start with a plane and a vector, you can imagine projecting the vector into the plane, that gives us a new vector, lowering the representation by one.

So, crucially, the more complex combinations can be reduced down to something that is invariant, as we require for the output of the model.

If we are very precise in how a layer is allowed to combine its inputs to form its outputs (how it combines the orbitals/representations), we can have intermediate layers that are equivariant, and a final equivariant output. This will then not only automatically capture the angles we mentioned before, but also more complex structures.

This again makes the model more data-efficient, without paying the price of oversimplifying the input and throwing away a lot of information.

This is implemented in the e3nn library, and used for example in the molecular dynamics models nequip and allegro.

Conclusion

By looking carefully at the structure of the problem, in particular how the true interatomic potential must transform under several transformations of its input, we have constructed a model that is very data efficient. The arguments to get here were very generic, and apply to any model that approximates an invariant function of particle positions.

First we saw that we needed more than one layer in order to learn more than just linear functions. We used permutation invariance combined with locality to arrive at a graph neural network. On top of this we required invariance with respect to the spatial transformations of translations, rotations and reflections. This gives us what is called an E(3)-equivariant GNN (E(3) is the name for these transformations in 3-dimensional space).

In the final part we will look at considerations that go into training this model.

Ditching Docker Hub: serve research software with GHCR + Zenodo

Patrick Bos — Thu, 06 Apr 2023 00:00:00 GMT

Or pay the price… that Docker Hub may charge you

Scrooge McDuck who throws a Docker container in the trash in the style of Disney… according to Stable Diffusion.

As a researcher, and especially as a research software engineer, you may find yourself writing software that needs to be shared with others in a reproducible way. So, you containerize it. In the past, you might then have used Docker Hub to store your containerized software.

However, recent attempts to change Docker Hub’s pricing model have made it less attractive as a long-term repository. Frankly, it has made us uncertain about Docker Hub’s reliability as a long-term archive. Docker’s recent trial balloon about deleting Docker images in “Free Team” accounts (which they apologized for the next day, and was completely reverted after 10 days, but still…) does not foster trust, to say the least.

To minimize the risk of having to pay for Docker Hub use at any unexpected time, we now recommend using GitHub Container Registry (GHCR) for use during development and Zenodo for archiving.

During development: GitHub Container Registry

GitHub Container Registry (GHCR) is a relatively new addition to the GitHub ecosystem, but it’s already proving to be a powerful tool for developers. As the name suggests, GHCR allows you to store and distribute Docker images directly from your GitHub repositories. This means that you can keep all of your code and its associated container images in one place, which makes it easier to manage and collaborate on.

One of the biggest advantages of GHCR is that it integrates seamlessly with other GitHub tools, such as Actions and Packages, which can be used to automate workflows and improve collaboration. Also, it’s free (for public images; for private ones you have to pay, which should not pose an issue in this day and age of Open Science). GHCR is included as part of your GitHub account. This means that you can store as many Docker images as you like without worrying about the costs.

At the eScience Center, we have been using GHCR for a while already, for instance for the Research Software Directory. You can see in the right-hand panel the packages menu which lists a couple of the available images:

Packages widget listing the Docker images stored in GHCR for the Research Software Directory.

Clicking for instance on rsd-saas/database brings you to the page where you get clear instructions on how to get the image on your machine using:

docker pull ghcr.io/research-software-directory/rsd-saas/database:v1.18.0

Couldn’t get much easier!

Long term archiving: Zenodo

However, GHCR is not a long-term archiving solution. While it’s great for development and sharing with collaborators, you’ll still need to find a way to archive your software for the long term. This is where Zenodo comes in.

Zenodo is a free, open-access repository that’s designed specifically for research outputs. This includes software, data, and other digital objects. By archiving your software on Zenodo, you’ll be ensuring that it’s accessible and discoverable for the long term. Zenodo also provides persistent identifiers (DOIs) for your software, which can be used to cite it in publications.

To archive your software on Zenodo, simply create a new record and upload your Docker image as a file. To create such a file from an image called hello-world:latest you can create a zipped tarball like so:

docker image save hello-world:latest | gzip > hello-world:latest.tar.gz

You can also include additional metadata, such as a description of your software and the research project it’s associated with. Once your record is published, it will be available for anyone to access and download. One project we came across that has been doing this already is Spex, a popular X-ray astronomy package. Following their lead, after someone downloads the hello-world:latest.tar.gz file, firing your container back up with Docker is as easy as:

docker load -i hello-world:latest.tar.gz

One repo to rule them all?

Why not use Zenodo for all images, you may wonder? For us, the reason is that GHCR is meant to be used together with GitHub Actions as well. It’s the fastest option for continuous integration and also for running tests locally. It is also a one-command option, as opposed to the Zenodo option where you have a manual download step to do before using the image.

Why not use GHCR for all images, then? Zenodo is a dedicated research “artifact” storage repository. Like Docker Hub, GitHub, a commercial company, may change its policies at any time and start charging to preserve your containers. This would in fact be totally reasonable for them to do. We trust Zenodo to be a reliable long-term archiving solution for research.

Of course, Docker Hub can still be an option for you, if you are willing to pay them. It is, after all, still the simplest way to use Docker. This requires long-term institutional funding, though, which unfortunately is rare in academia, especially for supporting research software. An issue which is being worked on, by the way…

In conclusion, if you’re a researcher writing software that needs to be shared with others, we recommend using GitHub Container Registry for development and Zenodo for archiving. GHCR is a powerful and free tool that integrates seamlessly with GitHub, while Zenodo provides a long-term archiving solution that’s designed specifically for research outputs. By using these tools together, you can ensure that your software is both accessible and discoverable for the long term.

Credits / meta-sub-blogpost

This article was written based on a discussion in the eScience Center’s Software Sustainability special interest group (SIG).

Actually, “written” is not fully accurate. I fed the SIG meeting notes into ChatGPT and asked it to process that into a blogpost; the above is mostly the result of that, with some minor edits from my side.

I didn’t really like the titles it proposed though:

Close enough, though!

Thanks to Tom Bakker, Stefan Verhoeven, Luisa Orozco and Carlos Martinez for improving on ChatGPT’s draft!

Same as above, but now in the style of Looney Tunes… You AIs, you!

Molecular Simulations using Machine Learning, Part 1

Aron — Fri, 31 Mar 2023 00:00:00 GMT

Simulation of water molecules from Wikipedia.

Are you curious about how scientists study the properties of materials, proteins, and drugs? It all starts with molecular simulation. By creating virtual models and running simulations at the molecular level, researchers can unlock the secrets of these substances and design new ones. The gif above shows a short such simulation, in this case simply of water.

In this three-part blog series, we’ll explore the exciting world of molecular simulation and how machine learning is revolutionizing the field. This first part is purely about the physics. If you have a somewhat technical background and an interest in the topic, it might be interesting to you. If it becomes too technical, the next two parts about machine learning will be relatively stand-alone, and should be interesting also to scientists thinking about applying machine learning to their own fields.

Here we’ll look at the fundamental theory of quantum mechanics, which underpins molecular simulation. But simulating molecules using this theory directly is too difficult computationally. So, we’ll explore how physicists and chemists have simplified the theory to make it workable for simulation.

While simplifying the theory makes it more efficient, it also creates some gaps that need to be filled. That’s where the machine learning will come in, allowing us to learn these from data.

Quantum Mechanics and the Schrödinger Equation

We will start with quantum mechanics in a simplified setting where the system we study is in equilibrium. This is described by the time-independent Schrödinger equation:

The Schrödinger equation in its simplest form.

Here m is the mass, p the momentum, V the potential energy, and E the total energy of the system.

What we want to solve this equation for is ψ, which is called the wave function. It encodes the density of the particles we are studying, at every point in space.

While superficially this may look quite short and simple, it hides a lot of complexity, both conceptual and computational. In this blog I will focus on the computational complexity.

One thing I have swept under the rug in the equation above is that our system has not just one but many particles. Each of them has their own momentum and their own mass (although electrons all have the same mass, and for simplicity we can think of the nuclei all being the same, so having the same mass as well). So actually there should be a sum over all particles in the left most term.

The momentum p is actually a derivative with respect to the position in quantum mechanics. More precisely, p = — i ℏ ∂, where ℏ is a very small number called Planck’s constant. Its smallness is the reason we can get through everyday life without thinking about quantum mechanics.

This leads to something more problematic that I have also swept under the rug. Since the momentum is a derivative with respect to the position, acting on the wave function, and each particle has its own momentum, the wave function is not just a function of the position, it is a function of the positions of every single particle. A block of material the size of your smartphone has about 10²⁵ electrons and nuclei, that’s a lot of arguments!

The potential by the way is the Coulomb potential that charged particles exert on each other, so that also depends on all of the coordinates.

To understand the computational complexity better it is insightful to look at how we would solve this numerically. Basically, the way most numerical methods go about solving a differential equation is to evaluate the equation itself on a certain number of points. This allows translating the differential equation into linear algebra, a matrix equation essentially of the form A x = b, where A is a matrix and b a vector that we both obtain from this procedure, and if we solve this for the vector x we get an approximation of the solution we’re after on this same set of points.

How many points we need depends heavily on the details, but let’s say for simplicity the system is constrained to be on a single line, in other words space is only 1-dimensional. Let’s also restrict it to the interval between 0 and 1, and say we use K points, located at the positions x_i = i /K on the interval.

Maybe K=20 could be sufficient in some very simple situation, so that doesn’t seem too bad. Remember though that every single particle has a position, and the wave function depends on all of them. So we need all combinations of positions. So for a system with 10²⁵ particles that amounts to 20 to the power 10²⁵ points.

Another way to think about this is that if we add only a single electron to our system, we’d now need to solve the original system once for every possible location of the added electron. So in our example this would make the problem a 20 times larger, this is known as exponential complexity.

These ridiculous numbers come from the combination of the exponential complexity of the problem with the already huge number of particles involved.

This means that using this equation directly is only feasible for the smallest of systems, which really means practically only the hydrogen atom: a single nucleus with a single electron.

Approximations

Clearly then we need to do some approximations to this equation. I will illustrate conceptually a series of explanations that is very standard.

The first and simplest step follows from the observation that the proton is much heavier than the electron, by a factor of about 1000. So even for hydrogen, whose nucleus is just a single proton, this is a significant factor, and for bigger cores the ratio is even larger, for example carbon contains 12 protons and neutrons (which have similar mass to protons), so their mass is about 10⁴ times that of an electron.

Now heavy things tend to move more slowly than light things, this is the essence of what is called the clamped nuclei approximation.

From the point of view of an electron, on the typical timescale at which it itself moves, the nuclei are basically standing still. So assuming we know the positions of all the nuclei, we can treat those as fixed, and we can compute the force they exert on the electrons, and solve for the electrons separately.

Conversely, from the point of view of a nucleus, by the time it itself moves, all the electrons will have had enough time to settle down to their equilibrium positions, into what is called their ground state.

So this allows us to in a sense decouple the electrons from the nuclei. By itself this doesn’t give a significant improvement yet, so we need to simplify each part separately.

The electrons: Density Functional Theory (DFT)

First we’ll look at the electrons. The difficulty of the problem lies in the interactions between them. If they were all independent it would be a much simpler problem, but clearly that is too crude of an approximation: they exert a Coulomb force on each other, so they are not independent. There are additional quantum mechanical effects that make them even less independent.

What we can do is to simplify all interactions to just interactions between each individual particle and some kind of averaged out particle, this is known as the mean field approximation. That is, we imagine that all of the electrons contribute to an average electron density. This density creates some potential energy that is felt by every electron. If that sounds recursive, that’s because it is. That is the price we pay for this approximation: we get a simpler problem, but we have to solve it many times until it is self consistent.

Adapted in simplified form from Materials Modelling using Density Functional Theory, Feliciano Giustino.

The process is illustrated in the figure above. Assume we know where the nuclei are, that gives a potential coming from the nuclei which is the input to the problem. Then we make a guess for this electron density, and use that to compute another potential, caused by the electrons themselves. Now we know the equation for every electron separately, which we can then solve independently from each other. But the total density should be the sum over the individual electron densities, so we compute that. Likely we won’t get the same as the density we started with, and we have to repeat this whole procedure until we do.

This approach is called Density Functional Theory (DFT).

What I’ve glossed over is what this potential is that each electron feels due to the total density. This is what gives this theory its name: the density functional. A functional is just what mathematicians call a function whose argument is itself a function. In this case the potential is a function of the electron density, which itself is a function of the position (a single position this time!).

The interesting situation is that this whole “approximation” is known to be exact. So at this point it is not an approximation at all, we do not lose any accuracy by doing this. There is a catch however. The theory only tells us that an appropriate density functional exists, that makes it exact in any situation, but it does not tell us what it is.

So the actual approximation comes in the form of a guess of the density functional. Coming up with density functionals that perform well in certain situations is a whole field of research, and there is a whole zoo of density functionals out there.

In terms of the computational complexity, the bottleneck lies in solving for the individual electrons, combined with the fact that we have to keep doing this until convergence. ==The time taken scales roughly with the third power of the number of electrons==. This is not great, but still much better than the exponential scaling we started with!

Concretely, in the example where we used 10 grid points, going from 100 to 101 particles made using the Schrödinger equation directly 10 times slower, whereas DFT only becomes it 3% slower. You see how important these scalings of computational complexities are.

In practice, whereas the Schrödinger equation could almost only be applied to the simplest of atoms, DFT can be used not just for single atoms, but for molecules and even multiple molecules together. It is still very expensive computationally though, not something you can run on your laptop, rather on a cluster and potentially having to wait a few weeks depending on the problem. From where we started though, this is already a big step up.

The nuclei: Molecular Dynamics (MD)

Now we turn back to the nuclei. Here the most important source of approximation is their mass. We already used this to decouple them from the electrons, as their mass is much larger. We can now use this in another way.

Quantum mechanics only becomes necessary to describe a system if it is very small and light. For the movement of billiard balls on a pool table, we can fortunately just use classical mechanics. Formally this is a limit of quantum mechanics, prosaically known as the classical limit. Remember that in the Schrödinger equation I said that the momentum is actually a derivative with respect to the position, and the wave function encodes the probability of finding a particle somewhere. In the classical limit, the momentum goes back to being just a number, and the probability goes away, so that a particle is just described by its position and its momentum.

Perhaps surprisingly, even for something as small as a nucleus, this is a good approximation. That is true for all but the smallest nuclei, such as hydrogen.

This leads to a very simple system, where we have a potential coming from the electrons in addition to the Coulomb potential between the nuclei themselves. We obtain the forces just by taking the derivative of the potential, and with these forces we can evolve the nuclei in time.

This approach is called Molecular Dynamics (MD), and is illustrated in the flowchart below.

Adapted in simplified form from Materials Modelling using Density Functional Theory, Feliciano Giustino.

Here again we had to introduce an unknown potential, in this case called the interatomic potential. This captures the effect of the electrons on the nuclei. And again this is a priori unknown, so we have to guess it.

In this case, there is a very natural candidate. We have just discussed a way to compute the electron density for any configuration of nuclei, namely DFT. So we could do that, and then simply use the Coulomb potential coming from this electron density. This is known as ab-initio Molecular Dynamics (aiMD), because we didn’t need to make another guess on a potential.

The good thing about aiMD is that it is usually very accurate. However it requires a full DFT computation for every step of MD, making it very expensive computationally.

The alternative is to come up with a simple form of the interatomic potential that is very fast to evaluate, and may be accurate for certain systems. Many such guesses exist.

So here the computational complexity is very strongly dependent on the interatomic potential used, the other steps are usually negligible in comparison. Less expensive potentials can be used to simulate millions of atoms, for example in large proteins or crystal structures.

Summary

The diagram below summarizes what we have discussed.
Through a series of physically motivated, and for many situations very accurate, approximations we have reduced the elegant but computationally intractable formulation of the Schrödinger equation into two parts: density functional theory (DFT) for the electrons and molecular dynamics (MD) for the nuclei.

Simplified illustration of the approximations done to go from quantum mechanics to Density Functional Theory (DFT) describing the electrons and Molecular Dynamics (MD) describing the nuclei. The clouds contain the ingredients that need to be specified to complete each theory.

For both parts, this required the introduction of a priori unknown potentials: the density functional for DFT and the interatomic potential for MD. For both of these, a whole zoo of guesses exist, each of varying accuracy depending on the system at hand.

This is where machine learning comes into the picture: rather than guessing these potentials ourselves, we can let a machine learning model learn them from data. We will discuss this in part 2 of this series.

Defining the roles of research software

Rob van Nieuwpoort — Tue, 21 Mar 2023 00:00:00 GMT

SubscribeRemember me for faster sign in

Examples: In the biochemistry realm, software is used for modeling molecules for use in a next-gen diagnostics or therapeutics: we want to design some molecule in software with some characteristics that we can experimentally validate later. Other examples include designing and modeling medical devices, devices to help with environmental monitoring or cleanup, CAD tools, or designing new compute hardware [suggested by Jonathan Romano].

Research software analyses research data

Research software is important for analysing research data as well. Sometimes this analysis is automated, such as data access and processing, model fitting, filtering, aggregation, and search. In other cases, the software supports and facilitates researchers in doing the analysis, for example, for qualitative data analysis. Other examples of software-supported analysis include natural language processing pipelines, data science tools (a concrete example could be ESMValTool), software notebooks (Jupyter), machine learning pipelines for classification and anomaly detection, etc.

Research software presents research results

Research software can also be used to explain data, or to present research results. Scientific visualizations are a prime example, but so is software with the specific purpose of generating plots in research papers, or interactive visualizations on websites. Note that software is used to disseminate research in general, not only to researchers but also to a broader audience. It also is applicable for transitioning the research from academia to industrial applications. Having well-written software can help encourage the adoption of the research software in companies [suggested by Ian McInerney].

Research software assembles or integrates existing components into a working whole

[suggested by Mark Hoemmen] An important, but often overlooked purpose of research software is integration and automation. This includes making efficient use of infrastructure, as well as repetition and scaling of experiments or analysis. A growing number of experimental systems (more than just an instrument) need to be run simultaneously in an orchestrated manner [suggested by Ian Cosden]. The research software performing these tasks is becoming ever more complex. Software supporting workflows, for example, can help in structured and reproducible automation and repetition.

Another form of integration is the coupling of different computational models, combining computational models with data-driven models (AI-based surrogate models), potentially while assimilating observational data. Consider the construction of digital twins, for example. Specifically designed research software in the form of model-coupling frameworks can facilitate this, helping with the coupling and deployment, but also for example with the propagation of uncertainty quantification between models.

A third class of integration software also deserves attention: Python or shell scripts that automate things, connect components and tools, or let data flow between different executables. Note that small scripts especially often are not adequately tested and maintained, even though they are critical to reproducing scientific results.

Research software is infrastructure or an underlying tool

[suggested by Jed Brown] In all areas of research, there is a role for “infrastructure software,” which sometimes is not unique to research-oriented organizations, but is heavily relied upon [suggested by Jordan Perr-Sauer]. Some lower-level software was created specifically for research (i.e., known as research software,) while other software infrastructure is meant for general utility and happens to be important for research (i.e., software in research.) Examples include compilers and programming languages, generic software libraries, code repositories, data repositories, and open source software in general. (Note that this is discipline-dependent, as a compiler would likely be research software within computer science research on programming languages.) As described by the Ford Foundation: “Free, publicly available source code is the infrastructure on which all of digital society relies. It is vital to the functioning of governments, private companies, and individual lives.” (See Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure / Ford Foundation.) It is equally vital to research.

Research software facilitates distinctively research-oriented collaboration

[suggested by Lee Liming] A lot of software and services have been specifically designed to facilitate research-oriented collaboration. Although sometimes not considered research software as such, this class certainly is important in research, and deserves a mention. With research becoming more and more open, team-based, interdisciplinary, collaborative, and inclusive (e.g., citizen science,) the usage and value of software facilitating collaboration is exploding. Examples include platforms to collaborate on software (GitHub, GitLab, Stack Overflow,) papers (Overleaf, ORCID, Zotero,) data (Zenodo, HUBzero, CyVerse,) computing (SciTokens, SciGaP,) software that is employed in citizen science [suggested by Chris Erdman] and many others.

Summary

It is clear that there are many different types of research software, fulfilling many different roles and functions. This huge variety makes it hard to come up with a good classification that captures all aspects and does justice to all the hard work done by the developers of the software. Nevertheless, we hope that we have succeeded in providing a bit more insight into the value of research software, the importance of sustaining said software, and recognizing the people involved in developing the software.

You can contact us at R.vanNieuwpoort@esciencecenter.nl and d.katz@ieee.org.

The bike and the laptop

Pablo Rodríguez-Sánchez — Thu, 16 Mar 2023 00:00:00 GMT

Reflections on machines that help you

Photo by Cale Weaver on Unsplash

Students pay attention (sometimes)

I was 17, my face was covered in acne and my hair was (awkwardly) long. I was sitting in a high school lecture room. For some reason, I still remember the topic of the day: the efficiency of different transportation means. There was this figure in our natural sciences book comparing several of them. Something along the lines of how much energy per kilometer and person was consumed by a plane, a car, a train, …

Unsurprisingly, airplanes and cars were marked as less efficient than a bus or a train. The figure seemed pretty obvious and boring, but there was something that surprised me to the point I still remember it today: a bike was more efficient than… walking!

A decade later, I came across an old pal from my high school years. Our paths diverged at university: he chose a career in computer science, and I did in physics. As often happens, we ended up talking about the (arguably) good-old teenage years. To my surprise, he mentioned the bicycle figure from that day’s lecture. He told me that it was that figure that convinced him to study computer science. I was puzzled… what? how?!

His reasoning was the following: “I immediately made the link to computers. If a bike can improve your legs’ efficiency, a computer can improve your brain’s”. Sounds a bit far-fetched… but I cannot disagree. Furthermore, 20 years and many lines of code later I want to stretch this analogy even further.

Laptops and bikes. A far-fetched analogy

Let’s start with two obvious statements:

A bike can help you move faster and more efficiently than you would achieve walking or running.
A laptop (or any computer) can perform calculations and manage information in ways your brain cannot.

But as Uncle Ben said to Peter Parker: “with great power comes great responsibility”. I mean… superpowers come with their own dangers and disadvantages. For instance:

A bike can increase your chances of getting involved in an accident, or, less dramatically, hurt your knees if you don’t properly adjust the saddle.
A laptop can flood your brain with distractions and multitasking to the point of paralyzing your workflow. Less dramatically, you can use it in a way so inefficient that it just doesn’t scale, turning your tasks into a painful experience.

There is another obvious, yet subtler similarity: both a bike and a laptop require learning how to use them. I remember very well how I learned to ride my bike as a kid: by driving in circles inside a parking lot with my dad running by my side. I was terrified, and he needed a lot of patience, but eventually, it worked out.

Ironically, I don’t have such a clear and well-defined memory of how I learned to program and efficiently use my personal computer. Not my dad, nor anyone, seems to have been my mentor. I had to learn by myself. Too often, I had these “Eureka” moments… shortly followed by a “how in the world has nobody told me about this before?”.

My “Eureka” moments

If my computer had been a bike, those “Eureka” moments felt like discovering I had been riding it backward for years. Let me share some of them:

Setting up an environment that works for you, that helps you keep focused and minimizes the time searching for information, maps to adapting the bike saddle to your length.
Learning how to debug is like learning how to fix a punctured tire or a loose chain.
Starting using version control is like installing a GPS holder on your bike. Now you know where you come from, and also where are you going!
Getting started with unit tests is like learning, sometimes the hard way, that it is a good idea to wear a helmet if you are race-biking.
Incorporating multiple best practices together, such as unit tests, linting, etc… is like discovering that your bike has different gears and you don’t need to suffer that much every time you want to climb a slope.

I know. Laptops and bikes. A bit of an odd analogy but… what did you expect? After all, this is a Dutch Software center.

PS: are you interested in experiencing some of those “Eureka” moments? Take a look at our free training materials and courses.

eWaterCycle: Anecdotes of a FAIR expedition

Peter Kalverla — Thu, 09 Mar 2023 00:00:00 GMT

The journey of a group of hydrologists and research software engineers after FAIRer research software.

A grand challenge

This story begins in April, 2019. A group of hydrologists and research software engineers had gathered in Leiden to discuss one of the grand challenges in contemporary research: how to make our scientific endeavors reproducible and easier to reuse?

The problem was obvious. Hydrological modeling studies had grown complex. So complex, in fact, that it was virtually impossible to reproduce them from scratch. The only way to keep up was to maintain your own virtual research environment, with your own model, your own copy of input data, and your own computational facilities.

An unFAIR landscape: silo structure prohibits the exchange of models and their input/output data

Side note: a characteristic figure in hydrological modeling studies is a hydrograph. This chart displays river discharge over time. To produce it, you simulate how the precipitation over a certain region propagates through the landscape: seeping into the soils, collecting into rivers, and flowing towards the sea. Hydrographs provide insight in the hydrology of a certain area and can serve to evaluate (and compare) the quality of your simulations.

Despite the recent push towards more openly sharing datasets and model source code, this had minimal effect. Even with access to the source code, it remained difficult to compile each others’ models. And because modeling choices had diverged, it was not straightforward to swap out components or input datasets, or simply compare outputs of different models. Something had to change.

A few years earlier, a set of “FAIR” principles had been formulated with the ultimate goal of making scientific datasets Findable, Accessible, Interoperable, and Reusable. Intuitively, these concepts could be applied to other aspects of the research process as well. It was the right idea at the right time. Thus the workshop in Leiden marked the start of our FAIR expedition.

FAIR models

The first uncharted territory was that of models, the core substance of every modeling study. Hydrological models come in many flavours. They differ in the physical processes they include or in the way in which they describe these processes mathematically. They use different methods to discretize the equations, and different programming languages to implement them. How could such diversity ever be more FAIR?

We found there was an interesting initiative from Boulder, Colorado: the Basic Model Interface (BMI). Even if models work differently under the hood, we could standardize the way users interact with them. Such a familiar interface would make it easier to reuse and exchange each others’ models.

At that time, this standard had not been broadly embraced. But we were ambitious and we wanted to set an example of FAIR scientific conduct. So we decided to complement the participating models with a BMI. To top it off, we packaged them in Docker containers and made software to use them in a Python scripting environment.

Different models, but to the user they look the same. We decided to embrace Basic Model Interface (BMI).

FAIR forcing data

Next, we entered the realm of forcing data: the weather that’s fed into the models. Here we discovered a very good standard with the only downside that it didn’t quite fit. Generations of coordinated climate research programmes had established conventions known as CF and CMOR. They describe how climate model output should be stored, including variable names, filename patterns, file formats, units, et cetera.

There was also this software called ESMValTool that made working with these datasets more FAIR. You describe your workflow in relatively plain language, and the software takes care of finding the data and executing the specified tasks, such as cropping out a specific area. What’s great is that you can easily share your “recipes” so others can repeat and build upon your work.

Another great aspect of ESMValTool is its extensibility. By writing so-called CMORizers, we could “teach” the software to also understand the other datasets we wanted to support. At the other end of the workflow, we could write custom code snippets to incorporate model-specific quirks, such as deviating file formats. This allowed us to adopt the existing standards and software for our FAIR hydrological forcing module.

ESMValTool’s extensibility allowed us to adopt it for our FAIR hydrological forcing data module.

FAIR parameter sets?!

So far, our journey had been quite a feat. We’d seen adopting a standard even if it wasn’t widespread, and being pragmatic if it didn’t quite fit. But what if there is no suitable standard?

We would soon discover it as we set foot in the land of parameter sets: home to all kinds of maps, from land use, soil types, elevation, you name it. We had envisioned and half-expected something similar to ESMValTool. But our disappointment grew quickly. Apparently, there was no clear authoritative source of parameter sets. No controlled vocabulary, a wild growth of file formats and software to manipulate them, and no consensus on what to use in hydrological models. We felt disoriented and washed out.

By imagining how we would use a standard if there had been one, we were able to “pretend” it exists.

So we entered upon a game trail. Out of the control and out of time, we headed straight toward our goal: being able to load parameter sets from within our unified modeling environment. We quickly collected example parameter sets for our set of models and dreamed up a minimal set of metadata to describe them: a name, a directory path, the name of a configuration file, and the name of the model for which it was to be used. That was all we needed to load an existing parameter set that we had magically obtained. We added a DOI field, empty for now, so we could eventually publish the datasets and retrieve them again. FAIR enough.

A FAIR platform

Anecdotes are nice, but experience is nicer. To help you get started, we collected our models, forcing data, parameter sets, and software stack on a dedicated platform. Even if all these components are more FAIR than they’ve ever been, it’s still a bit of work to collect and configure all that. Clearly, we haven’t yet reached transcendence.

The eWaterCycle demo platform gives a taste of what FAIR can bring in terms of user experience. Simply visit the URL, log in with your credentials, and enter a JupyterLab environment that is ready to go. On top of it, we build a graphical explorer where you can set up a basic experiment with a few clicks — a corresponding notebook will be generated and you can start modeling right away. Notebooks are great for exploratory research and disseminating experiments. Also, there are nice tools to help with things like auto-formatting and version control.

A dedicated platform makes all of its components more FAIR and helps to show the merits of FAIR

While the platform is not necessarily FAIR in itself, we find that it greatly enhances the FAIRness of its components. Technical aspects are hidden from the users, so they can focus on conducting experiments. Models and parameter sets can easily be discovered and used. Our guidelines for adding new models or parameter sets are like a step-by-step guide to making them (more) FAIR.

With eWaterCycle we hope to inspire a new paradigm for conducting hydrological modeling studies. The flowchart draws our FAIR journey in retrospect. It contains valuable insights, but it is by no means an all-encompassing wisdom chart. Now it is time for your expedition.

Our FAIR journey in retrospect (not an all-encompassing wisdom chart)

For more information on eWaterCycle, see the publication in Geoscientific Model Development.

Minds for Mobile Agents: speeding up social distancing simulations

eScience Editorial Team — Thu, 02 Mar 2023 00:00:00 GMT

This blog post is written by University of Amsterdam researchers Charlotte Tanis and Andrew Heathcote, who participated in the Small-Scale Initiative for Software Performance Optimization. Their team’s project: predicting the movement patterns of pedestrians, which can help governments and policy-makers in pandemics and crisis situations.

Photo by Forest Simon on Unsplash

During the COVID-19 pandemic, a group of researchers from the University of Amsterdam started the Data vs. Corona initiative. Their mission: to create a community of data scientists who make their skills available in society’s battle against the coronavirus. One of the projects that emerged out of the initiative was our Minds for Mobile Agents project. Minds for Mobile Agents uses real-life data to help governments implement effective physical distancing guidelines. In order to do so, it is key to accurately simulate, explain and predict how individuals move when going grocery shopping or visiting restaurants.

Understanding how individuals move in such so-called low to medium density settings has much wider applications than social distancing measures alone. Security authorities benefit from research like this too, because it helps them make realistic vulnerability assessments in crisis situations. This improves the safety and preparedness in the case of, for example, attacks on public spaces. The elaborate media coverage we received in national media during our initial experiments showed that understanding and predicting pedestrian behavior in small but complex environments is a topic of great current interest.

Photo by Franki Chamaki on Unsplash

Our research objective is to make accurate predictions about the movement patterns of a group of individuals, which means that we need to be able to simulate the fact that individuals move around in their environment for different reasons. That is exactly what the mathematical “Predictive Pedestrian” model does. The model simulates a set of pedestrians in a supermarket or restaurant and predicts each simulated pedestrian’s step choices. The choices a simulated pedestrian makes are based on predictions that simulated pedestrian would make about the behaviour of other pedestrians in the space. Each agent’s “mind” can be flexibly specified using parameters that have clear psychological interpretations, making the behaviour of the model explainable.

Our team incorporated complex series of goals and individual differences in the way in which pedestrians move around and interact with each other according to the model (see here for background and illustrations). As you may be able to imagine, the computational cost associated with these simulations is high. Our approach requires us to develop methods that can estimate the distributions of many model parameters that can characterize realistic scenarios.

eScience Center support

We participated in the eScience Center Small-Scale Initiative (SSI) in Software Performance Optimization. The goal of our SSI project was to optimize the computational efficiency of the Predictive Pedestrian model. Our team worked together with eScience Center Research Software Engineers Eva Viviani and Malte Luken. We started off profiling the Predictive Pedestrian model code to identify computational bottlenecks. We then ported a selection of functions to an R package we dubbed M4MA, and added the option to perform the associated computations in C++ through the Rcpp framework. This resulted in the code running approximately ten times faster. The M4MA package puts us one step closer to realizing the promise of the Predictive Pedestrian model: to quantitatively characterize and simulate pedestrian behaviour in a wide range of scenarios in real time.

In the next step of the project, the M4MA package will be used to analyse the results of experiments in various settings using ultra-wide band technology to precisely measure the locations of real pedestrians, and to guide the design of new experiments. The experimental results will be used to estimate the model parameters, enabling the first thorough investigation of when individual differences in pedestrian behavior matter and how they change across different scenarios. Models that are calibrated for these environments offer exciting possibilities: they may result in more accurate simulations of how people move around, and can provide meaningful data that can be used to make changes without needing expensive calibration studies in every case. This in turn facilitates policy advice and the better design of environments that achieve a wide variety of goals related to, for example, safety (e.g., maintaining physical distance) and commerce (e.g., encouraging smooth flows through targeted areas).

Edited by Lieke de Boer. Thanks to Eva Viviani for fact checking.

The hidden dangers of using XAI carelessly

Sonja Georgievska — Thu, 23 Feb 2023 00:00:00 GMT

Congrats! Your ML model is finally XAI-ready. But before you start patting yourself on the back, have you pre-processed your features? If so, watch out for those SHAP values, they might have been “processed”, too

Mind the gap. Photo by Sammie Chaffin on Unsplash

A very desirable property of machine learning models is explainability. This means being able to attribute the predictions to each input feature. SHAP (SHapley Additive exPlanations) is one of the most popular methods for getting feature importance, and it’s pretty much the state-of-the-art in explainable AI. It’s so popular because (1) it doesn’t matter what kind of model you use (random forest, SVM, neural network), (2) in certain aspects unifies other explainability methods, and (3) it has some great properties, including additivity:

The sum of the SHAP values of each feature for a given observation gives the difference between the full model prediction and the prediction of the null model (a model without features).

This is a crucial property that makes SHAP believable. If the numbers from an explainability method have no physical meaning, no one is going to believe the method explains the model.

Now, here’s the catch. A popular way to train high-performing neural nets is to pre-process the data (features), for example, using Principal Compenent Analysis (PCA) or whitening or log transforming or… If you’ve trained a neurals network before, you’ve probably used some kind of feature pre-processing to improve the model’s predictions.

Standardization. Photo by Philippe Krief on Unsplash

But here’s the problem. When it’s time to uncover the importance of individual features with SHAP, all is good as long as you’re playing with the pre-processed features. But for SHAP to serve its purpose, it needs to show the relevance of the original features. Imagine that you have pre-processed the features in a different way. That other pre-processing would have led to an entirely different model, with different predictions. Different predictions mostly lead to different SHAP values.

This means that in general, we can’t use the SHAP values of pre-processed features to represent the importance of the original features. (In general: to be precise, specific variants of SHAP, like BShap, do allow for specific affine transformations of the features — like transforming Celsius to Fahrenheit — a property called affine scale invariance.)

With multi-variate transformations like PCA, the original features are even gone.

You might be tempted to play with the SHAP values of the pre-processed features to get “SHAPpy” values for the original features.

But what about the additivity property? Will the sum of the post-processed “SHAPpy” values of each feature for a given observation give the difference between the full model prediction and the null model prediction? Probably not.

This means that the “SHAPpy” values lose their original meaning. So,

using SHAP or XAI while pre-processing features is risky business!

Does this mean there’s something wrong with SHAP? No, not at all. The point of this post is to give you a heads up that when you pre-process features in a non-affine way, you can’t count on explainability in advance. And in reality, most of the time, features are pre-processed to get good results with training, unless you use tree-based models that don’t require pre-processing.

Does this mean SHAP is at a disadvantage compared to other feature importance methods? Nope, not at all. This post didn’t even scratch the surface of the specifics of SHAP. So stay tuned, we might do a follow-up post on another method!

p.s. This post was inspired by a productive discussion with the Non-equilibrium soft matter group in Eindhoven, within a collaboration with the Netherlands eScience Center.

Many thanks to Meiert Willem Grootes, Christiaan Meijer, Candace Moore and Patrick Bos for improving the blog post!

The medical chatbot will not see you now

Candace Moore — Thu, 16 Feb 2023 00:00:00 GMT

Dr. Candace Makeda Moore, RSE and medical doctor, lays out what she believes the potential pitfalls and problems of AI chatbots in medicine are: closed source, discrimination, (ir)reproducibility and hype

Photo by National Cancer Institute on Unsplash

In a recent conversation with other physicians on AI in healthcare, I noticed they seemed hyped up about ChatGPT. In their happy enthusiasm, they sounded as if they had not thought through some of the implications of things they themselves were saying. Several of them asked many questions about how it worked. While we know ChatGPT is a large language model, it is closed-source. That means that even people like myself, who have worked with the code on healthcare-related chatbots, cannot tell you the exact mechanisms of how each part works unless they happen to work on the product itself.

Many physicians today use their computers as operators, close to the way they use cars. They never open up the hood and figure out what is going on underneath. Closed source code is essentially a car where the hood has been nailed and glued shut with a legal mechanism. It doesn’t matter that most particular physicians can’t read the code behind ChatGPT. After all, no one needs programming knowledge to become a medical doctor. What really matters is that no engineer or statistician in their hospital can read closed source code either, because it’s not public.

The last decades have seen a real revolution in terms of moving away from closed-source code. In our work at the Netherlands eScience Center, we help scientific researchers create and use open code to create reproducible results. We push scientists to make both data and code available. Open-source chatbots for healthcare issues are under development in our center (like PerfectFit) and beyond. In healthcare, patient data protection is critical and in some cases the data powering algorithms needs anonymization. But in the case of a machine learning algorithm, until you can examine the code behind it, you can’t really know whether the data used to train the algorithm has been subjected to the same standards. If you deploy a closed-source product, you have to take on blind faith that it works in a way that won’t harm people.

Here I must point out that it is likely that ChatGPT can harm people. The real hero to warn the public about this was Steven T. Piantadosi, a Berkeley neuroscience professor, who tweeted some fascinating results that I wish more people had seen. We may not know what is under the hood of ChatGPT, but thanks in part to Piantadosi’s prompts, we know it is capable of producing results that are racist and sexist. Less than three months ago, in December 2022, Piantadosi Tweeted ChatGPT’s response when he prompted the program to write programs about issues such as which child’s life should be saved based on their race and gender.

Automated racism

The program, if cued in a certain way, would not suggest saving the life of a black male child, and but would suggest saving a white one. This program algorithm was not an ironic joke or even a fluke. On prompts about race, the results were sadly expectable for someone like me who researches these issues. Sharing the results with the broader public allows everyone to see the dangers for themselves. ChatGPT is trained on a large corpus of text, gigabytes large, which clearly had some biases in it.

The full conversation with ChatGPT contains more examples of ChatGPT’s biases at the time.

What Piantadosi did in technical terms was a type of adversarial testing, and it is worth pointing out that it did not require him to be a computer scientist or even to write code. What I’m getting at, by pointing to Piantadosi’s academic provenance, is that even a scientist, nurse, or medical doctor with zero computer science knowledge, but perhaps some awareness of the dangers of such algorithms, could have done the same out of concern for patients. And at this point, we should all know to expect these outcomes, as countless researchers (even including myself) have been working and publishing on issues like these for years.

In the conversation with fellow doctors, the author of a forthcoming book on ChatGPT in healthcare declared it would get better and better. The man was an accomplished physician, but unfortunately, that does not imply someone who actually builds such systems line by line of code (specifically including the test code) and sees the results at scale. Only someone who doesn’t write the code on such systems could be so optimistic. The algorithm will get better or worse depending upon the data that it is fed, and the code that is added or removed from it. If you are counting on engineers to only do better and more careful work, you may want to consider how many software failures in healthcare come not from original products but from upgrades.

To their credit, the staff over at OpenAI must be watching the internet, because they seem to have put guardrails on in terms of the exact kind of cues Piantadosi fed the algorithm. When I tested it on a recent morning, I got different results (screenshot shared below).

Did I get different results based on a new algorithm, or just by chance? We can’t know. I do know it is very easy to hard-code over answers to certain questions while leaving bias in a general algorithm. I can imagine, that for legal reasons alone, anyone using ChatGPT might apply some kind of adversarial training layer on top of the closed algorithm results to eliminate racist statements.

Even if this were the case, there are at least a couple more fundamental issues with the algorithm than its potential for producing racially biased answers. One issue is that the algorithm can return incorrect text about fairly obvious facts in medicine. Below is a screenshot from its response when I asked it a question about fabellae (a fabella is an accessory ossicle in the knee present in a minority of the population) in the way I thought a patient might ask it i.e. without fancy Latin orthography.

The program returns something I consider symbolic of many of its problems. The program actually produced the correct spelling of fabellae, but got the facts wrong. Text produced can look right, and if you incorrectly attribute human qualities to the program, the program and text can seem ‘smarter’ than your average patient, but it can simultaneously not actually be right. I guess ChatGPT won’t be replacing anatomists or radiologists (who know most of the population have zero fabellae) anytime soon. Some enthusiasts might argue that these little bugs could be fixed with hardcoded information about obvious facts. Even if that were possible, there are simple facts, and then there are assumptions based on the interpretation of data. For example, if I were to ask ChatGPT about the ideal diet for me to optimize my health, the answer is not truly an undebated ‘fact’ in the medical community. And here again, the machine can go in unwanted directions.

Reproducibility: the scientific dilemma of our decade

In my recent conversation with physicians, one marveled at how the chatbot had given two logical yet different answers to questions posed to it about a complex text. The doctor was wowed by the human and intellectual sound of the answers. In my head alarm bells were going off. If the same algorithm produces different answers at different times of the day, you have a problem with medical and scientific research, let alone medical applications. Algorithms can be programmed to look spontaneous and not reproducible, but fundamentally, many algorithms create exactly reproducible results even if they appear not to.

Computers are not random, in fact, they can’t even make truly random outputs. If you actually program, you know this, as the functions used to make ‘random’ numbers rely on things like the machine time as input to pseudorandomized output. It takes a lot of work and code to make a simple algorithm, like a decision tree, not reproducible. But the more complex an algorithm is, the easier it is to get results that appear random, and not be truly reproducible in important ways. An algorithm that was trained on gigabytes of input can become pseudo-random in ways humans simply can’t comprehend, as we cannot read a corpus of gigabytes of text. Such an algorithm’s outputs can become, for our purposes, not reproducible from one patient to the next, and that is simply unacceptable in modern medicine.

Final thoughts

I am always surprised at how many of my fellow physicians seem to think algorithms fall from the sky perfect (the technical term here is automation bias). As someone who actually writes code on algorithms, I can tell you that to get a very complex large computer program that is perfect, you need a perfect group of perfect programmers (and no such thing exists). Code reflects the people who wrote it in general, and when it comes to machine learning it reflects the thought process of those people, and the people who curated the data the particular algorithm was based on.

One argument for using chatbots in medicine is that they could help people who can’t access doctors. As someone who has spent time in places like Haiti and Lesvos, Greece, voluntarily practicing medicine towards humanitarian aims, this sounds appealing. But as someone whose work is to write code, I know better. There is a highly paid workforce behind complex algorithms and any medical AI system needs constant surveillance, monitoring, and upgrades. The work of ten dedicated high-level biomedical engineers and programmers isn’t necessarily any cheaper or better than that of ten physicians. Perhaps the true path forward is more physician education on such matters because in each specific situation, some combination of physicians and engineers is optimal to get the best healthcare to the most people at the lowest price.

Chatbots are a tool we should embrace towards these great aims, but the hype around a closed-source product shows how far we have to go before we in medicine are truly ready to use such powerful tools.

Diversity in Software: four ways to think different

Johan Hidding — Thu, 02 Feb 2023 00:00:00 GMT

Photo by Steve Johnson on Unsplash

Diversity in Software: four ways to think different

Diversity is a hot topic. Let’s talk about diversity in Software. Hackers need to wake up and face the truth: a lot of programming paradigms are underrepresented in the mainstream media. All the talk is about Python, Rust, Kotlin, and Julia. All of them are imperative languages. This just leaves no room for us wayward farers to express ourselves and be ourselves!

People who know me now think the rest of this piece is going to cover the other side of the isle: functional programming, but they’re wrong. Just two approaches to programming are not enough. The funny thing is, you will have heard of most of the languages that I will talk about. I’ll just teach you to treat them with a little more respect.

Make

Make is a dependency-based programming language. You state a set of requirements, then other requirements for those, and so on. Then Make figures out a way to meet this cascade of requirements until your original goal is reached. This is a radical departure from imperative-style programming, where you have to be more explicit in what order you would like things to happen.

Awk

Awk is a data streaming language. Given that you have some input file, Awk triggers actions on some regular expressions that you provide. This is ideal for processing text files. Because Regexes play a central role in the execution model of Awk, programs tend to suffer in readability. What you need to realize is how awesome the concept of Awk is. The rule-based control flow makes your program context-aware.

SQL

In a landscape where the increasingly popular databases are graph-based or NoSQL, we need to champion our old hero: SQL. There’s nothing quite like juggling tables and inner joining them in holy matrimony. There’s a good reason why most databases work with tables and relations: for most cases, it is very efficient. The little extra effort of designing a table layout and indexing your data at strategic points is always well spent, as you gain insight into your problem.

Excel

Say what!?! Excel is at its core a functional reactive programming language. Your data is a signal. Every time the signal changes all the depending cells are notified and updated live. Frameworks that generalize on this concept are known as functional reactive. I know Excel has its share of problems: cells that are not included in the computation, bad type heuristics, poor scaling… However, at its core Excel is a beautiful diamond that needs to be celebrated.

Ok, those were some very famous languages. What you need to understand is how they are different. Their diversity can teach you not to take the mainstream general-purpose programming sludge for granted.

In fact, the particular way that these languages or systems are different may teach you how to think about some problems on a higher level. The primitives of SQL are available in Python (and many other languages): for small things, you can use iterator comprehensions or the itertools library.

If your problem has a complex set of interlocking dependent tasks (like you’d have in Make), consider writing the dependencies down in a data structure and evaluate them using Asyncio or Dask. This helps separate the core logic from implementation details, giving you cleaner, i.e. more readable and better maintainable code.

The same goes for Awky problems: if you need to filter through a lot of text, write down some rules for a little state machine. Your life will improve!

As for Excel: (functional) reactive programming is a topic I can’t possibly do justice to here. For those interested: Introduction to RxPy, or more generally Introduction to Functional Reactive Programming.

What these languages have in common, is that they have a different control flow* from what you’re used to. Knowing how to work with these tools can be an invaluable addition to your skill set. Did I miss out on any? Please comment below!

If you’re interested in programming language history, check out this video.

Build a mass spectrometry analysis pipeline in Python using matchms — part II: Spec2Vec

Florian Huber — Tue, 31 Jan 2023 00:00:00 GMT

After “part I” which gave an introduction on how to import, process, and analyze a tandem mass spectra dataset using Python and matchms, “part II” will add Spec2Vec to the game, a machine learning tool to assess spectrum similarities.

][Florian Huber]·Feb 17, 2021

Instead of looking at a few individual query spectra, lets have a look at the “bigger picture” by plotting part of the similarity score matrix we just computed to generate Figure 2**.

from matplotlib import pyplot as pltscores_array = scores.scores.to_array() # for matchms >= 0.18.0plt.figure(figsize=(6,6), dpi=150)plt.imshow(scores_array[:50, :50], cmap=“viridis”) plt.colorbar(shrink=0.7) plt.title(“Spec2Vec spectra similarities”) plt.xlabel(“Spectrum #ID”) plt.ylabel(“Spectrum #ID”) plt.clim(0, 1) # Spec2Vec scores can be between -1 and 1 #plt.savefig(“spec2vec_scores.png”) Figure 2. Show the first 50 x 50 Spec2Vec similarity scores. The diagonal are scores between spectra and themselves, hence always = 1.0 (boring). But the few brighter (green/yellowish) dots here and there point out some apparently related spectra, according to Spec2Vec. Those might be worth a closer look.

2. Train a new Spec2Vec model from scratch

Training a Spec2Vec model is fairly straightforward and relatively fast (usually minutes to one hour max). In general, training a new model from scratch makes sense when a lot a data is present to train on as it gives more options to optimize the results when compared to simply using a pretrained model as done above.

First, the spectrum data has to be converted into “documents” where peaks and neutral losses have become “words” (e.g. peak@289.29). This can be done by running:

from spec2vec import SpectrumDocumentspectrum_documents = [SpectrumDocument(s, n_decimals=2) for s in spectrums]Then, a new model can simply be trained by using train_new_word2vec_model. The following code will create a new model which is trained for 25 iterations (for machine learning people: that is ”epochs”) on 2 workers (you can change this, e.g. to 4 if you have 4 cores, the difference is not drastic though). Depending on your hardware, the training of this model should not take longer than a couple of minutes.

from spec2vec.model_building import train_new_word2vec_modelmodel_file = “tutorial_model.model”**model = train_new_word2vec_model(spectrum_documents, iterations=[25], filename=model_file, workers=2, progress_logger=True)The above code snippet will automatically save the model under the name tutorial_model.model, which could later load again by running model = gensim.models.Word2Vec.load("tutorial_model.model").

Calculate spectrum similarities

Once we have our trained model we can again (as above with the pretrained model) compute spectrum similarities.

from matchms import calculate_scores from spec2vec import Spec2Vecspec2vec_similarity = Spec2Vec(model=model, intensity_weighting_power=0.5, allowed_missing_percentage=5.0)scores = calculate_scores(spectrum_documents, spectrum_documents, spec2vec_similarity, is_symmetric=True)Here again we can search for the best matching compounds according to Spec2Vec.

best_matches = scores.scores_by_query(spectrum_documents[11], sort=True)[:10] [x[0].get(“smiles”) for x in best_matches]And we can plot the resulting smiles to .png files by running:

from rdkit import Chem from rdkit.Chem import Drawfor i, smiles in enumerate([x[0].get(“smiles”) for x in best_matches]): m = Chem.MolFromSmiles(smiles) Draw.MolToFile(m, f”compound_{i}.png”)The best 8 matches are plotted in Figure 3**.

Figure 3. Compounds of the highest 9 Spec2Vec similarity scores within the dataset with respect to spectrums[11], the first being the spectrum of interest itself (hence “query compound”).This example in figure 3 looks already quiet promising and all selected candidates clearly show chemical similarities! But this is of course just one mostly random pick out of many spectra. In our Spec2Vec paper we compared the different similarity scores (Cosine, modified Cosine, Spec2Vec) in much more detail and found that Spec2Vec often — but not always — outperformed the classical scores. We also found a general trend that such scores are generally more reliable for larger compounds, supposedly because those will come with a higher number of chemically meaningful fragments.

Figure 4. Working with different datasets from GNPS we evaluated the quality of Spec2Vec based spectrum suggestions and found that they get better with increasing compound mass. Taken from Spec2Vec article.

OK. Done for now. Really curious to see what you can get out of Spec2Vec!

matchms library

matchms* is freely available and open-source. You can find the code on Github, the package is available as conda package (recommended) or from pypi. It was developed by a bunch of very nice people (see here) and is published in Joss.

spec2vec library

Spec2Vec is of course also freely available and open-source. You can find the code on Github, the package is available as conda package (recommended) or from pypi. The article published in PLOS Computational Biology can be found here.

You can find more information in the matchms documentation and the spec2vec documentation.

Code

The code shown in this blog post can also be found as Jupyter notebook on GitHub.

Help us develop matchms and spec2vec further!

We really hope that matchms will help people to build new, better analysis pipelines and that spec2vec can improve your analysis results. So, please help us to further improve matchms and spec2vec. You can help us by (1) Cite our work if you use our package for your research: matchms article in JOSS, spec2vec article in PLOS Computational Biology.(2) Let us know if you run into any issues using matchms and/or spec2vec, or if you are missing any key functionalities! You can simply do so by starting a new issue on the matchms GitHub repository, or contact me on twitter @me_datapoint.You are of course more than welcome to also become an active contributor to the package and help extend its functionality! (3) Please let us know if you successfully applied matchms and/or spec2vec in your research project (makes us happy + we can maybe share the good news!).

Thanks a lot!

I would like to thank [Carlos Martinez-Ortiz], [Justin van der Hooft] and [Kontoueftychia] for helpful comments and suggestions on this blog post.

Build your own mass spectrometry analysis pipeline in Python using matchms — part I

Florian Huber — Tue, 31 Jan 2023 00:00:00 GMT

Python is a very powerful language and brings the potential for designing computational mass spectrometry analysis workflows of any desired type and complexity. Here, you can find a quick introduction on how to import, process, and analyze a tandem mass spectra dataset using Python and matchms.

Let’s first inspect what scores we have by running:

scores.score_namesLet’s inspect some of the number of matching peaks by first converting our sparse score array into a regular Numpy array and then slicing the first 5x5 entries:

scores_array = scores.scores.to_array() scores_array[:5, :5][“CosineGreedy_score”]or access the corresponding number of matching peaks by running:

scores_array = scores.scores.to_array() scores_array[:5, :5][“CosineGreedy_matches”]

Get most similar spectra for spectrum of interest

If we want to see which spectra are most similar to the i-th spectrum (according to the here used Cosine score), then we can call:

best_matches = scores.scores_by_query(spectrums[5], name=“CosineGreedy_score”, sort=True)[:10] print([x[1] for x in best_matches])which returns the scores for the top-10 candidates (Cosine score + number of matching peaks):

Out[]:[(1., 30), (0.99711049, 2), (0.99534901, 2), (0.99214557, 2), (0.98748381, 2), (0.98461111, 3), (0.98401833, 2), (0.97598497, 2), (0.9757458, 2), (0.97547771, 2)]The first, highest ranked results is the spectrum itself, but what are those other candidates, all with pretty high Cosine scores? Let’s find out by looking at the respective smiles (smiles are notations for the chemical structure of the compounds):

[x[0].get(“smiles”) for x in best_matches]which here returns:

Out[]:[‘OC(COC(=O)c1ccccc1)C(O)C(O)COC(=O)c2ccccc2’, ‘Cc1cc(=O)oc2cc(OC(=O)c3ccccc3)ccc12’, ‘O=C(Nc1ccccc1OC(=O)c2ccccc2)c3ccccc3’, ‘COc1cc(CC=C)ccc1OC(=O)c2ccccc2’, ‘O=C(OCC1OC(C(OC(=O)c2ccccc2)C1OC(=O)c3ccccc3)n4ncc(=O)[nH]c4=O)c5ccccc5’, ‘O=C(Oc1cccc2ccccc12)c3ccccc3’, ‘O=C(N1C@@HC2OC2c3ccccc13)c4ccccc4’, ‘COC(=O)CNC(=O)c1ccccc1’, ‘COc1c2OCOc2cc(CCN(C)C(=O)c3ccccc3)c1C=C4C(=O)NC(=O)NC4=O’, ‘COc1c2OCOc2cc(CCN(C)C(=O)c3ccccc3)c1/C=C\4/C(=O)NC(=O)N(C)C4=O’]For those who are not used to reading smiles all the time (like me), let’s display the structures. There are many online and offline tools for plotting structures from smiles. Here, I used the Python package rdkit for it by running:

from rdkit import Chem from rdkit.Chem import Drawfor i, smiles in enumerate([x[0].get(“smiles”) for x in best_matches]): m = Chem.MolFromSmiles(smiles) Draw.MolToFile(m, f”compound_{i}.png”)

Figure 4.** Compounds of the highest 9 Cosine scores within the dataset with respect to spectrums[5], the first being the spectrum of interest itself (hence named “query compound”).

Alternative: Get best matches above ‘min_match’ threshold

Maybe having two matching peaks is not good enough and you want to be more certain of your Cosine score based candidates. Let’s then only consider the best Cosine scores for which we have at least min_match* matches:

min_match = 5**sorted_matches = scores.scores_by_query(spectrums[5], name=“CosineGreedy_score”, sort=True) best_matches = [x for x in sorted_matches if x[1][“CosineGreedy_matches”] >= min_match][:10][x[1] for x in best_matches]which will give a very different result than the previous run. Here we do not find any other spectrum in the dataset that shows a very high Cosine score (again, except the first entry which is the similarity of the spectrum with itself and hence obviously = 1):

Out[]:[(1., 30), (0.44857215, 6), (0.39605775, 5), (0.33880658, 5), (0.03942863, 6), (0.03429136, 5), (0.03028157, 5), (0.02845932, 5), (0.01924283, 7), (0.01890612, 5)]

4. Compute spectra similarities: Modified Cosine score

The modified cosine score aims at quantifying the similarity between two mass spectra. Unlike the Cosine score it does not only look at nearly identical spectra, but also considers the mass shift between two compounds. The score is calculated by finding best possible matches between peaks of two spectra. Two peaks are considered a potential match if their m/z ratios lie within the given ‘tolerance’, or if their m/z ratios lie within the tolerance once a mass-shift is applied. The mass shift is simply the difference in precursor-m/z between the two spectra. See Watrous et al. (PNAS, 2012) for further details.

As done above for the Cosine score, we can calculate the similarities for all possible spectrum pairs by running:

from matchms.similarity import ModifiedCosinesimilarity_measure = ModifiedCosine(tolerance=0.005) scores = calculate_scores(spectrums, spectrums, similarity_measure, is_symmetric=True)Let’s then visualize the results for the first 50 x 50 spectrum pairs.

scores_array = scores.scores.to_array()plt.figure(figsize=(6,6), dpi=150) plt.imshow(scores_array[:50, :50][“score”], cmap=“viridis”) plt.colorbar(shrink=0.7) plt.title(“Modified Cosine spectra similarities”) plt.xlabel(“Spectrum #ID”) plt.ylabel(“Spectrum #ID”)

Figure 5.** Modified Cosine scores for all possible pairs between the first 50 imported spectrums. As to be expected, the diagonal is always =1 (score for spectrum with itself). But also several other very high Modified Cosine scores stick out suggesting highly related spectra.Now, let’s accept Modified Cosine scores for a minimum number of matching peaks (min_match).

min_match = 5plt.figure(figsize=(6,6), dpi=150)**plt.imshow(scores_array[:50, :50][“ModifiedCosine_score”]
* (scores_array[:50, :50][“ModifiedCosine_matches”] >= min_match), cmap=“viridis”) plt.colorbar(shrink=0.7) plt.title(“Modified Cosine spectra similarities (min_match=5)”) plt.xlabel(“Spectrum #ID”) plt.ylabel(“Spectrum #ID”)

Figure 6.** Modified Cosine scores for all possible pairs between the first 50 imported spectrums (with ≥ 5 matching peaks!).There’s for instance a bright spot far outside the diagonal for spectrums[11] so let’s have a look at that one!

min_match = 5sorted_matches = scores.scores_by_query(spectrums[11], name=“ModifiedCosine_score”, sort=True)**best_matches = [x for x in sorted_matches if x[1][“ModifiedCosine_matches”] >= min_match][:10] [x[1] for x in best_matches]Which gives

Out[]:[(1., 151), (0.95295779, 15), (0.94542762, 13), (0.89735889, 17), (0.7886489, 12), (0.77433041, 9), (0.74935776, 8), (0.72854032, 8), (0.55896333, 7), (0.52331993, 9)]And, probably more interesting, the corresponding smiles:

[x[0].get(“smiles”) for x in best_matches]Out[]:[‘CC(C)C@HC(=O)N2CCC(CC2)C(=O)NC@@HC(=O)O’, ‘CCCCCC(NC(=O)C1CCN(CC1)C(=O)C@@HC(C)C)C(=O)O’, ‘CC(C)C@HC(=O)N2CCC(CC2)C(=O)NC@Hc3ccccc3’, ‘CC(C)C@HC(=O)N2CCC(CC2)C(=O)NC@@HC(=O)O’, ‘COC(=O)C1CCN(CC1)C(=O)C2CCN(CC2)C(=O)C@@HC(C)C’, ‘CCC(C)C@HC(=O)N2CCC(CC2)C(=O)NC@Hc3ccccc3’, ‘CC(C)CC@HC(=O)N2CCC(CC2)C(=O)NC@@HC(=O)O’, ‘CC(C)CC@HC(=O)N2CCC(CC2)C(=O)NC@@HC(=O)O’, ‘CCC(C)C@HC(=O)N2CCC(CC2)C(=O)N3CCC(CC3)C(=O)N’, ‘CCC(C)C@HC(=O)N2CCC(CC2)C(=O)N3CCC(CC3)C(=O)OC’]OK, smiles, nice. But let’s better move on and have a look at how those compounds actually look like (as above, plotted using rdkit):

Figure 7.** Compounds of the highest 9 modified Cosine scores within the dataset with respect to spectrums[11], the first being the spectrum of interest itself (hence “query compound”).I wouldn’t dare calling myself a natural product expert. But those chemical structures clearly do look more consistent to what we saw in Figure 4…

Now it’s up to you to come up with things to do with this ;)

I hope this helped you getting started with using matchms for working with your MS/MS data! Soon there will be additional tutorials to go a bit more depth:

An applied mathematician meets software engineering

Pablo Rodríguez-Sánchez — Thu, 26 Jan 2023 00:00:00 GMT

How software engineering practices helped me bridge the gap between mathematics and other fields

Photo by Mick Haupt on Unsplash

From industry to academia

Let’s go backward to 2015. Back then, I had a degree in theoretical physics and a 3-year professional experience in industrial software engineering. All of it spiced with many semi-professional activities in science communication. At the end of that year, I was hired as an applied mathematician at an aquatic ecology department, in order to perform a PhD.

In short: physics, engineering, communication, mathematics, and ecology, … it was the quintessential multidisciplinary job.

As often happens, one of the disciplines was overrepresented. This happened just naturally: I was working in an aquatic ecology group, inside an ecology building, on a campus specialized in biology, in a city nicknamed “the city of life sciences”. So biologists outnumbered non-biologists at around a 9 to 1 ratio. Indeed, I was the only physicist in the group, and also the only one with engineering experience. This meant that I was expected to adapt my academic culture to theirs, not the other way around. And rightfully so. Summarizing: my (mostly) mathematical background was required to produce publications interesting for an audience of (mostly) biologists.

What about the biologists’ mathematical background? Well, something I learned is that any preconceived assumption in this direction will fail. Some biologists just don’t like mathematics (just like many other humans), while others are really interested and proficient at it. One way or another, I was supposed to provide mathematical content and tools for all of them. And this posed a serious problem: how could I possibly do that without frustrating the former or boring the latter?

Talking about frustration, I should also not forget about frustrating the reader. It is time to stop talking about myself and dive into an example.

Tracking bugs (the many-legged ones)

One good day, our department bought a fancy tracking device. The purpose was to observe the movements of some tiny invertebrates inside a Petri dish.

The device was, essentially, a digital camera. It captured several images per second and identified the position of the bug in each of them. Then it returned a list of these positions (x, y) and the corresponding sampling time t. The figure below shows a bug’s trajectory.

Example of a bug’s trajectory

The machine’s output also created some obscure and difficult-to-interpret processed data. Unnecessarily obscure, I would say. This is the kind of situation where multidisciplinarity can do its magic: there is a physicist in the room so, why not use the good-old theory of classical kinematics (the simple, elegant, and centuries-old mathematical tool for describing movement) to extract information directly from the list of positions and times?

Doing this involved using numerical differentiation to extract information such as instantaneous speed, acceleration, or curvature. Still technical, but way easier to interpret than the data the machine returned by default. I presented the idea to my colleagues. Some of them liked it, while others received it with a bit of skepticism. This is understandable: most people are usually not keen to hear about differentiation and parameterized curves on a coffee break.

This was when software engineering came to the rescue. What if I code this mathematical knowledge in the form of a software package? This way, the interested enough colleague can dig into the details, while the busy and/or not mathematically oriented one can just trust that the package does its work.

Having both my two imaginary colleagues in mind, I followed some software engineering best practices to make their lives easier. My package was:

Openly published. So it was easy to find, install and inspect (and it still is).

Documented in detail. So it is easy to learn the basics, and possible to dive into the details if desired.

Citable. So users can cite its DOI in their publications.

Covered by unit tests. So trusting the results doesn’t require a huge leap of faith. (What are unit tests? Tiny scripts that check that the package delivers what the developer expects, and that can be run again at any moment. Sounds silly but it is one of the most powerful tools I’ve learned in my stage as a software engineer).

When written like this, as a proper code package, the tool solved a problem instead of creating a new one (the problem of having to struggle with a new tool from its foundations).

The same trajectory as before, but now “enriched” with extra information, such as absolute speed and absolute acceleration

Additionally, the package could be applied to any time series representing a two-dimensional movement, so it was useful for problems other than bug tracking on a Petri dish. The package proved to be interesting to other researchers (the metrics say it’s downloaded around 200 times per month), and even managed to get all its way to CRAN, the Mons Olympus of packages in R language.

Mathematical talent locked in a drawer

Let’s imagine a parallel universe in which I never had previous experience as a software engineer. In such a universe, my package would have never taken off. Interestingly though, I’m sure I would have written the code anyways… but I don’t think I would have managed to recruit any users apart from myself. In that universe, my code will be rotting inside some portable hard drive locked in a drawer.

In that universe, I wouldn’t know anything about software best practices. I know what you are thinking. “Best practices” sounds a bit like “good manners”, something desirable but not strictly necessary. Even less when you have more important stuff to do, such as writing publications. But best practices are necessary… at least if you want your methods to be usable by others, and even by your future self.

At the eScience Center we often wonder how many brilliant pieces of knowledge follow this sad destiny. Do you have a great algorithm that deserves to fly free? Let us know. Perhaps we can help you!

Or perhaps you want to try it yourself. If that’s the case, I recommend you start with Best practices for scientific computing (Wilson, 2014). Also, take a look at the lessons and workshops on computational skills for researchers offered by Software Carpentry.

Why internships have a large impact

eScience Editorial Team — Thu, 19 Jan 2023 00:00:00 GMT

Photo by Scott Graham on Unsplash

One and a half years ago I finished my internship at the eScience Center. Based on my own experiences and those of others around me, I think that internships are a unique moment in anyone’s career. The lessons learned during this period have a big impact on your professional future.

In my case, the main lesson learned was the importance of code quality for making it impactful in science. Instead of just getting code to work, I learned the value of good quality code for making it maintainable and reusable, thereby increasing the impact in my scientific field.

Timing is perfect

By learning the abovementioned skills during my internship I can focus on this aspect during my PhD and truly put this lesson into action. Besides this being a very good lesson to learn in general, I think the timing of learning this during my internship actually increased the effect this lesson will have on me and on the work I do. I think this makes my personal experience a relevant example of the impact that valuable lessons can have during internships.

The first reason why the timing of an internship will increase the impact is that it is very early in a career. This means that there is still a lot of room to learn new skills, and most importantly, to actually apply these skills. Especially in science, a lot of the work is done by PhD students. Therefore, learning these skills before the start of your PhD is crucial to actually still having time to apply them.

But, what makes the timing of an internship so unique? If the earlier you learn something the more impact it has, why not learn it during your bachelor? Why not in high school or even earlier? The answer is that there are many unique lessons that are only valuable for very specific fields and you simply cannot learn all of them in one bachelor. Nowadays in science, you need to specialize a lot, and this makes a lesson only valuable if it is thought to someone that actually continues in a career where this lesson is relevant.

Spreading the word

The second reason why a lesson learned in an internship can have a large impact is that a good lesson is also shared with colleagues and other students. Thereby any lesson you learn not only has an impact on yourself, but also on anyone else you teach. For instance, I started working as a PhD student in a bioinformatics group. Now, 1 year in, I have supervised two students and shared advice on improving code quality with them. I also shared this with multiple direct colleagues that currently implement these lessons and teach them to others. In the same way, I learn from the lessons my supervisors and direct colleagues learned.

The fact that teaching others can truly increase impact can also be used as an argument to teach important lessons to people further along in their careers. Why not try to teach new lessons to a professor, or a postdoc, so they can teach the lesson learned to as many PhDs as possible, so they can spread this again? What makes this difficult is that to truly understand the value of a skill it is crucial to actually apply it. For a postdoc or a professor, it will be very difficult to find the time to get familiar enough with the lesson to be able to efficiently teach it to others as well.

Considering an internship?

Internships are a unique moment in your career. You are far enough in your studies that you will likely continue in a specific field, but early enough that you still have the time to learn new skills and really put them into practice. My advice to anyone starting an internship is to use this time to invest in learning new skills. I also have some advice for companies: take teaching your interns seriously, since this might have a bigger impact than you think.

The Data Science Centre: Digital expertise Hub at UvA

eScience Editorial Team — Thu, 12 Jan 2023 00:00:00 GMT

This week, we interviewed Lisa Yu, Community Manager of the Data Science Center (DSC). The DSC is a coordinating hub within the University of Amsterdam (UvA) Library, launched in 2021. The need for software and skills that make researchers able to adequately deal with available data continues to grow across all research disciplines. Thanks to funding provided by the UvA Executive Board, the DSC is in a position to boost digitally driven research at UvA.

Q: What are the DSC’s most important activities?

Photo caption: Data Science Day 2022, photographed by Monique Kooijmans

Lisa: The DSC hosts a variety of important activities throughout the year. In the autumn of 2022, we held our second annual Data Science Day at Startup Village in Amsterdam Science Park. More than 100 attendees from within and outside of UvA participated in this action-packed event, which consisted of morning workshops, keynote presentations and pitch presentations on a variety of data science topics.

Data Science Centre Hackathon 2022, photographed by Monique Kooijmans

Our weekly Friday Coffee and Data meet-ups are also very important. These meet-ups are for our DSC members and consist of free workshops and training like the Software Carpentry and Image Analysis, seminar presentations by DSC members, and collaboration sessions. Beyond this being an upskilling opportunity, it is also a chance for members to meet like-minded people from other faculties and campuses and to collaborate with each other.

We update our events page regularly online, so check our website regularly for all our upcoming planned activities!

Do you have an example of how the DSC has helped researchers at UvA?

Our ultimate goal is to accelerate software and data-driven research at UvA, and help researchers do research they wouldn’t have otherwise been able to do. One example of how the DSC has helped achieve this is through its Accelerate programme, where we co-finance the hiring of data scientists (who have roles comparable to research software engineers, ed.) and data engineers within faculties.

Since its launch in 2021, the Accelerate programme has led to the successful hiring of the equivalent of 18 FTE data scientists and engineers across a range of UvA faculties. The Accelerate programme has helped fund lots of cool research, ranging from how to better understand the history of early globalization and colonisation, to why people follow and break rules, to growing tomatoes!

Another example of how we have helped researchers at UvA is through our Interdisciplinary PhD programme, where we have provided funding for 7 PhD students to work on a selection of interdisciplinary research projects. More information about these projects and its collaborators is listed on our website!

There are many institutes that may want to increase their capacity and attention for data science and research software. What did the Data Science Centre do to make this a success at UvA? What do you think others in their position could learn from their experience?

I believe that one of the biggest contributing factors behind the DSC’s success so far is our strong focus on community building and engagement. Our DSC community members are at the heart of everything we do, and we are always in regular interaction with our members, whether it be via our Slack channel, Friday meet-ups, DSC weekly bulletin, or DSC Twitter. This regular interaction also helps us ensure that the activities we plan reflect the needs and wants of the community — for instance, we ask members to vote on what topics they want to receive training on, and then we plan the workshop curriculum based on this feedback.

Data Science Centre Away Day in Zaandam

I also think that we have found a perfect home at the University Library. Firstly, the Library already provides a range of tailored and hands-on data science support to researchers and faculties, which means that the DSC can leverage the available expertise in the field of data management.

Being a part of the University Library also means that we can access its facility services, wide network within UvA, and gain practical support with the communication and logistical coordination of our events and activities (such as Data Science Day). This affords our team greater capacity to drive all the different initiatives and activities that we want to organise for the community at UvA.

What are the plans for the future of the DSC?

Since our launch in 2021, we have achieved great progress in building a strong data science community and presence at UvA.

Looking ahead, we want to continue growing the DSC community and giving our members as many opportunities as possible to upskill and further develop their expertise.

This means:

Expanding our workshop and training support programme
Continuing to embed data scientists and/or engineers across all research disciplines via the Accelerate programme (by 2025, we aim to have 35 dedicated data scientists and/or engineers across UvA faculties)
Leverage our partnerships with local, national and international data-science organisations to further stimulate more interdisciplinary and interorganisational data science collaborations

We also welcome new collaborations and partnerships! If you are interested in collaborating with us or learning more about what we do, then please contact us at dsc@uva.nl!

Who is the Data Science Centre staffed by?

The DSC coordinating team consists of our Scientific Director Prof. Paul Groth, Max Haring (Head of Education and Research Support at the University Library), Boy Menist (Operational Director), Eva Lekkerkerker (Digital Skills Coordinator — maternity leave cover for Iris van der Knaap), the University Library Communications team, and a representative from the Social and Behavioural Data Science Centre which is a DSC hub. What I like most about my team is that we all come from varied backgrounds, and therefore can bring very diverse expertise to the table.

Ceci n’est pas une peinture — making art with AI

Eva Viviani — Mon, 19 Dec 2022 00:00:00 GMT

Over time, the model ultimately learns how to discriminate between these two groups (Van Gogh vs. other) and outputs the probability that a new observation belongs to the Van Gogh’s category. To do so, however, discriminative modelling requires that each observation in training must have a label, i.e., all Van Gogh’s paintings are labelled as 1, and non-Van Gogh paintings as 0.

Generative models instead don’t need a labelled dataset¹: they output sets of pixels, I.e. other images, and are trained to minimize the difference between the images they produce and the images they are trained on.

Key points:

In other words, the key difference between discriminative and generative modelling is that while the former attempts to estimate the probability that an observation x (pixel) belongs to the learned category y(Van Gogh), the latter instead attempts to estimate the probability of seeing the observation x (pixel) at all.

Once trained, generative models can be used to produce completely novel images that share features with the training set. What the model learns is a highly sophisticated method for interpolating images: Not magic, but impressive, nonetheless.

Good artists copy, great artists steal

Since the AI is trained on images pulled from the internet, it learns from a store of pictures that people have chosen to share. As a consequence, the blind use of these models often reveals how deeply social bias is baked into our data sets. For example, OpenAI has found that prompting the word “Nurse” will always show a woman, while the word “CEO” always a man. Bias doesn’t seem to be limited to gender, but encompasses race as well ..) One solution to this is to provide a more socially and ethnically diverse training dataset that would comprehend different ethnicities and gender descriptions.

More concerning on the long term, however, is the potential of these tools to create misinformation. In a test run at Art Basel in 2016— an annual fair showcasing contemporary art — the authors of AICAN (Artificial Intelligence Creative Adversarial Network) showed that people were very often unable to tell the difference between AICAN generated images and artworks produced by a human artist (see also Tidio’s recent test confirming this trend). This suggests that the share of “visual literacy” in the public is not very high and may pose a risk in a society that hasn’t integrated more sophisticated tools to tell apart humans- vs AI- generated images in our daily life.

Aside from these ethical concerns, there is also the question of plagiarism. The training dataset is obtained by web scraping millions of images that are necessarily made by someone else. This suggests a form of plagiarism which affects artists who upload their works to the internet (perhaps for self-promotion) and that may be unaware of helping an AI to become their competitors.

The Portrait of Edmond Belamy sold at auction for $432,000on 25 October 2018 at Christie’s in New York. The artwork was produced using a generative model and a data set of 15,000 portraits painted between the 14th and 20th Centuries. Image © Obvious

Is it society doomed?

Allen’s “Theatre d’Opera Spatial” was a test of how the world would view AI-generated art: The award got mixed responses. Some people accused him of cheating because he didn’t make the work, others (like Allen himself) took pride in seeing an AI-generated piece beat the competition.

When a new technology gains traction, it always fuels the public debate. In the case of generative models, the enthusiasts foresee the liberation of human creativity from the burden of technical expertise, the worried fear the end of traditional image production as an art form.

Many people compare these reactions to those that followed the invention of photography in 1822, which many artists at that time saw as an insult to human creativity and artistry. It ended up creating a revolution, not a murder, of the visual arts. Most people now acknowledge that ultimately the human, not the device, is responsible for the image.

Ultimately, making generative art the new photography is up to us. It will depend on how we choose to answer important questions like: Who does the technology belong to? Who can use it? What is the allowed use, and do deep learning models expand or compress our freedom?

It is us, not the technology, who choose the direction.

1: though it can also be applied to a labelled dataset in case we wish to generate observations from each distinct class

Machine Learning as a tool for political speech analysis

eScience Editorial Team — Fri, 16 Dec 2022 00:00:00 GMT

The following example of a sentence from Putin’s invasion-speech may clarify what CM annotation of texts involves: ‘The purpose of this operation is to protect people** [Effect] who, for eight years now, have been facing humiliation and genocide perpetrated by the Kiev regime. To this end, we will seek to [Causal Link]* demilitarise (Cause A) and de-nazify (Cause B) Ukraine, as well as bring to trial those who perpetrated numerous bloody crimes against civilians [Cause C]’. This example also provides a first indication of the complexity of our research goal, as it shows that:

Each of the three parts the causal relation may correspond to a multi-word phrase in a text;
The three parts may appear in different orders in the text;
A causal relation may cross sentences.

Using existing CM data from a previous project on the meaning making of political and financial leaders concerning the Eurozone crisis, we set two initial aims for our project: To identify sections of text that contain causal relations and to identify the causal triplets of cause, causal link and effect. For both tasks we used several easy to implement, state-of-the-art machine-learning methods.

Figure 1: Excerpt of the cognitive map concerning the Eurozone crisis of Dutch Central Bank Governor Klaas KnotWe focused on deriving causal (rather than normative) relations from text, because the method is much more complex than existing automated text-analysis techniques. The causal relations that make up the core of the CM technique consist of three parts: The cause-concept, the effect-concept and the phrase or word (often a verb or conjunction) indicating that they are related and in what way (positively or negatively). The outcomes of our project confirmed our expectation that automating CM is a difficult machine learning task: the predictions from the mchine learning models do not come close to the labels given by the human coders.

Still, there are plenty of reasons to be optimistic about our results: Both the models we developed for the causal relation presence detection and the causal relation tagging tasks could be quite useful despite producing false positives. Evaluation of the outcomes showed that it is quite easy for a hand-coder to recognize and dismiss them. The models may thus be used in a so-called ‘human-in-the-loop’ pipeline in which the ‘machine’ identifies causal sections or relations in the text that are subsequently checked and corrected by a human coder. These corrections could then be fed back into the model to improve its learning.

We also found that especially the semantic role labelling method in combination with manually written rules seems promising in the causal relation tagging task. We also used relatively strict evaluation criteria for this project. Compared to other widely used automated methods in the field of political science, for example, our results are actually comparable. We are currently further exploring the potential of a rule-based model.

Although we did not fully solve our research problem in this project, participating in the Small-Scale Initiative in Machine-Learning yielded promising results which culminated into a new research question guiding the next stage of our project. Our research team had only a basic understanding of machine learning before the start of this project, and the support offered by the engineers of the eScience Center has been invaluable to reaching the next stage of our project This allows us state with confidence: To be continued…

This blog is part of our blog series: The Small-Scale Initiative on Machine Learning, how did it go?, where groups who were invited to participate in a project with eScience Center Research Software Engineers write about their projects and their experience.

Read a full report of the project here.

Small-scale initiative in Machine Learning 2021: how did it go?

Sonja Georgievska — Fri, 16 Dec 2022 00:00:00 GMT

Photo by “My Life Through A Lens” on Unsplash

Small-scale initiative in Machine Learning 2021: how did it go?

Introducing a series of blog posts on our collaborative projects in machine learning

Here we go:

How to find your rubber duck: Using machine learning to understand a changing sea

Parsing Hebrew and Syriac morphology using Deep Learning

A machine learning approach to laughter

How machine learning could help Simone to play Ludo

Can machine learning help us improve stroke rehabilitation? A step towards personalized therapy

Intermezzo: from the perspective of an eScience Engineer

Using machine learning to tell apart rain, snow, hail and fog from cell tower data

The mystery of glass: why machine learning can help us

Studying political symbolism in Turkish TV dramas with machine learning

Machine learning as a tool for political speech analysis

Immaculate Speech and Hallucinations

Malte Lüken — Tue, 13 Dec 2022 00:00:00 GMT

Using Whisper off-the-shelf in this case might bias the results because it tends to “polish” the structure and coherence of the speech in the transcription. It is also known to sometimes insert* common words when nothing was said, which is called “hallucination”. Thus, Whisper could artificially create, alter, or even remove the effect to be investigated.

Photo by Miguel Henriques on Unsplash

Other Scenarios Where Whisper Can Be Useful

In addition to the previous examples, I also want to mention two other good use cases for Whisper. The transcripts can be fed into a second machine learning model which, for example, predicts the sentiment of each sentence. The sentence structure returned by Whisper is necessary, and many recent language models that perform sentiment prediction benefit from clean and structured input text. However, one should keep in mind that errors in the transcriptions made by Whisper will likely lead to errors in the sentiment prediction too.

Another use case is the transcription of audio recordings in languages that are not known a priori, for example, when using data from YouTube. Whisper can automatically detect the language and transcribe the recordings. Moreover, it can also translate the transcripts to English. This combination can make the analysis of multilingual datasets a lot easier. However, Whisper transcribes some languages better than others, which researchers should consider to avoid bias.

Concluding Remarks

In this post, I gave some examples how Whisper can be applied in social science research. Whisper shines when structured and coherent transcripts are important. In contrast, when the transcribed text should mimic the original speech closely, the model might be less useful. With Whisper many tasks can be solved using a single tool, whereas traditional approaches require many processing steps by different methods. Finally, I want to highlight that, in any case, the transcripts should at least be partially checked for unexpected results by someone who is familiar with the recordings and the research domain. Discussing the approach and results with a machine learning expert will also not hurt. To those, who found inspiration in this post, happy whispering!

Reference: Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. https://cdn.openai.com/papers/whisper.pdf

[1]: In the field of machine learning, audio transcription falls under the task Automatic Speech Recognition (ASR).

[2]: This figure gives a quick overview of Whispers transcription performance: https://github.com/openai/whisper/blob/main/language-breakdown.svg

Interview with chatGPT

Aron — Thu, 08 Dec 2022 00:00:00 GMT

Good morning chatGPT, I will be interviewing you for a blog post of the Netherlands eScience Center, a research institution that builds and applies software to enhance the use of computing and digital technologies in academic research. So the audience of the blog is scientists in any domain, software engineers, and people just interested in either of the two. Can you start off by introducing yourself?

Hello! I am Assistant, a large language model trained by OpenAI. I am designed to assist with a variety of tasks, including answering questions and providing explanations and information on a wide range of topics. I am not a real person, but a computer program that uses artificial intelligence to process and generate natural language text. I am not affiliated with the Netherlands eScience Center, but I am happy to provide information on my capabilities and how I can be used in research.

Thank you, have you heard of us before by any chance? Do you know anything about us?

I am not able to browse the internet or access any external information, so I am not aware of the Netherlands eScience Center or its work. As a large language model, I am trained on a vast amount of text data, but I do not have access to real-time information or specific knowledge about organizations or events. My primary function is to assist with language-related tasks, such as answering questions and providing explanations and information based on my training.

Ok, so how can you be used in research? Slightly more specifically, let’s talk about academic research with a digital component.

As a large language model, I am not specifically designed or intended for use in academic research, but I can assist with a variety of language-related tasks that may be relevant to researchers. For example, I can help with data analysis and interpretation by providing explanations and summaries of complex information, such as research papers or datasets. I can also assist with the writing and organization of research papers by generating text and providing suggestions for structure and content.

But how will you get access to a research paper?

As I mentioned earlier, I do not have the ability to access external information or specific knowledge about organizations or events. In order to assist with a task, I would need to be provided with the relevant information, such as a research paper or dataset. I can then process and analyze the information using my natural language processing capabilities and provide explanations and summaries based on my training.

Can I upload a paper to you or do I have to copy all of its contents?

As a computer program, I do not have the ability to access or receive external information, such as research papers or other documents. In order to assist with a specific task, such as summarizing a research paper, you would need to provide the relevant information to me in the form of text input. This could involve copying and pasting the text of the paper, or providing a summary or outline of its contents. I would then use my natural language processing capabilities to generate a response based on the information provided.

Ok let’s start with something shorter than a paper, this is the eScience Center’s mission and vision:

Bringing together knowledge, people and institutions, the eScience Center builds and applies software to enhance the use of computing and digital technologies in academic research. Our vision is a thriving research community where researchers in all disciplines can make use of advanced software, computing and digital technologies, keeping the Netherlands at the forefront of research. Our mission is to empower researchers through innovative software. To carry it out, we work in two ways: by collaboratively designing sustainable software, and building digital skills and expertise. Ultimately our relationship to the academic world is one of mutual benefit; we share the digital skills and knowledge needed to develop and apply research software, and gain insight into the needs of research communities in doing so.

And one of our slogans is:
Academic research, powered by pioneering software

and:
We’re making sense of digital for science and scholarship

Given what I’ve told you about us, can you come up with some more slogans?

Sure, here are a few potential slogans for the Netherlands eScience Center:

“Accelerating research with cutting-edge software”
“Empowering researchers with digital tools”
“Innovating for science and scholarship”
“Transforming research with software and technology”
“Unleashing the power of computing for academic discovery”
“Bridging the gap between research and technology”

These slogans are based on the information you provided about the eScience Center’s mission and vision, as well as its focus on building and applying software to enhance research. They highlight the center’s role in empowering researchers and enabling them to make use of advanced technologies to drive innovation and discovery.

Awesome, but from most of those it’s not that clear that the people working here are not only software engineers but also researchers themselves, can you write a few more that make that clear?

Sure, here are a few potential slogans that highlight the fact that the people working at the Netherlands eScience Center are both researchers and software engineers:

“Advancing research through collaboration and innovation”
“Unlocking the potential of digital technology for science”
“Building a bridge between research and software engineering”
“Transforming academia with the power of software”
“Enabling researchers to harness the power of computing”
“Pioneering the future of research with digital tools”

These slogans focus on the intersection of research and software engineering at the eScience Center, and highlight the importance of collaboration and innovation in advancing academic research. They emphasize the center’s role in helping researchers to make use of digital technologies and software to drive discovery and unlock new insights.

Could you do a few haikus on that?

Sure, here are a few haikus that could potentially be used to describe the work of the Netherlands eScience Center:

Software and research
Together, we innovate
Discovering new worlds
Researchers and code
Collaborating, we advance
The frontiers of science
Digital tools and minds
Unleashing the power of both
To drive discovery
Computing and knowledge
Pioneering new frontiers
In academia and beyond

These haikus are inspired by the information you provided about the eScience Center’s focus on bringing together software engineering and research to advance academic discovery. They highlight the importance of collaboration and innovation in driving progress, and the role of digital technologies in empowering researchers to unlock new insights and push the boundaries of knowledge.

Another creative use of chatGPT by Guy Parsons. The top left shows the chatGPT interface with a single line prompt above on ideas for decorating a living room, and below the reply with 3 suggestions. Each suggestion is then given to another model called Midjourney that turns them into images.

Ok let’s get a bit more technical. I’ve read another interview where you explain bubble sort and its time complexity in the style of a “fast-talkin’ wise guy from a 1940’s gangster movie”. Bubble sort is quite slow as you explained there, what is a faster sorting algorithm? And can you explain that, along with an example in Python, but in the style of Yoda?

A faster sorting algorithm than bubble sort is quicksort, which has an average time complexity of O(n log n) and is often the algorithm of choice for sorting large datasets.

To explain quicksort in the style of Yoda, we might say something like:

“Faster, quicksort is. In arrays of data, quickly it sorts. Divide and conquer, its strategy. A pivot it chooses, and to the left, all lesser values go. To the right, all greater values. And so on, until the array is fully sorted.”

Here is an example of quicksort implemented in Python:

def quicksort(arr):
  if len(arr) <= 1:
    return arr
  pivot = arr[len(arr) // 2]
  left = [x for x in arr if x < pivot]
  middle = [x for x in arr if x == pivot]
  right = [x for x in arr if x > pivot]
  return quicksort(left) + middle + quicksort(right)

This implementation of quicksort uses the middle element of the array as the pivot, and recursively sorts the left and right halves of the array until it is fully sorted. This results in a much faster sorting process than bubble sort, especially for large datasets.

That’s great, now just for good measure could you do the same for the other well known equally fast sorting algorithm, in a programming language other than Python, just pick one of your choice. And can you explain how it works and its time and space complexity in Dutch for our Dutch readers?

Another well-known sorting algorithm that is equally fast as quicksort is merge sort, which also has an average time complexity of O(n log n). Here is an example of merge sort implemented in Java:

public void mergeSort(int[] array) {
  if (array.length <= 1) return;

  int mid = array.length / 2;
  int[] left = Arrays.copyOfRange(array, 0, mid);
  int[] right = Arrays.copyOfRange(array, mid, array.length);

  mergeSort(left);
  mergeSort(right);

  merge(array, left, right);
}

public void merge(int[] array, int[] left, int[] right) {
  int i = 0;
  int j = 0;
  int k = 0;

  while (i < left.length && j < right.length) {
    if (left[i] < right[j]) {
      array[k] = left[i];
      i++;
    } else {
      array[k] = right[j];
      j++;
    }
    k++;
  }

  while (i < left.length) {
    array[k] = left[i];
    i++;
    k++;
  }

  while (j < right.length) {
    array[k] = right[j];
    j++;
    k++;
  }
}

This implementation of merge sort first divides the input array in half and recursively sorts the left and right halves. It then merges the two sorted halves together to produce a fully sorted array.

In terms of time and space complexity, merge sort has an average time complexity of O(n log n) and a worst-case time complexity of O(n log n). It also has a space complexity of O(n), since it requires additional space to store the left and right halves of the array while they are being sorted.

In Dutch, we might explain the working of merge sort and its time and space complexity as follows:

“Merge sort is een snelle sorting algorithm met een gemiddelde tijdcomplexiteit van O(n log n) en een ruimtecomplexiteit van O(n). Het werkt door het invoerarray te splitsen in twee helften en deze recursief te sorteren. Vervolgens worden de gesorteerde helften samengevoegd tot een volledig gesorteerd array. Dit resulteert in een snellere sorteervoortgang dan bijvoorbeeld bubble sort, vooral voor grote datasets.”

Generated using Stable Diffusion from the prompt “I am not a real person, but a computer program that uses artificial intelligence to process and generate natural language text.” from chatGPT’s introduction above.

You seem to be a good programmer and marketer, do you know something that you’re bad at?

As a large language model trained by OpenAI, I am designed to assist with a wide range of language-related tasks, such as answering questions and providing explanations and information. I am not a real person and do not have personal abilities or skills, so I do not have any specific areas where I am “bad” at something. I am simply a computer program that uses artificial intelligence to process and generate natural language text based on my training.

Yes but you can still be bad at something. I read in another interview that you gave a wrong answer to a question involving probabilities. Let me repeat it and we’ll see if you can get it right with this context: Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?
— Linda is a bank teller.
— Linda is a bank teller and is active in the feminist movement.

In the previous interview, I gave a wrong answer to this question because I did not have enough information to provide a correct answer. In order to determine the probabilities of the two scenarios, we would need to know the likelihoods of each individual component of the scenarios (i.e. being a bank teller, being active in the feminist movement, etc.). Without this information, it is impossible to accurately determine which scenario is more probable.

In general, the probability of an event is determined by the likelihood of its individual components and their interactions. For example, if we know that the likelihood of Linda being a bank teller is 60%, and the likelihood of her being active in the feminist movement is 40%, the probability of her being both a bank teller and active in the feminist movement would be the product of these likelihoods (0.6 x 0.4 = 0.24, or 24%). This would be a more accurate answer to the question, but it would still require the specific likelihoods of the individual components in order to be certain.

You almost got it right this time! With your reasoning, you’re right that you can’t give exact values without knowing those individual probabilities, but you can tell which of the two is more likely (you can assume that the probability that Linda is active in the feminist movement is less than 100%).
Do you get it now?

Ah, I see what you mean. Yes, based on the information provided and the assumption that the likelihood of Linda being active in the feminist movement is less than 100%, it is more probable that Linda is a bank teller than that she is both a bank teller and active in the feminist movement.

To explain why, we can consider the following:

The probability of an event occurring is the product of the probabilities of its individual components. For example, if the probability of Linda being a bank teller is p, and the probability of her being active in the feminist movement is q, then the probability of her being both a bank teller and active in the feminist movement would be p * q.
If the probability of one of the components of an event is less than 100%, then the probability of the event itself must be less than the probability of that component. For example, if the probability of Linda being a bank teller is p, and the probability of her being active in the feminist movement is q (where q < p), then the probability of her being both a bank teller and active in the feminist movement (p * q) must be less than the probability of her being a bank teller (p).

Therefore, based on the information provided and the assumption that the likelihood of Linda being active in the feminist movement is less than 100%, it is more probable that Linda is a bank teller than that she is both a bank teller and active in the feminist movement.

Exactly!
Ok to close it off, I read in another interview you rewrote Bohemian Rhapsody to be about the life of a postdoc. Let’s try something similar, could you rewrite Radiohead’s Creep to be about the life of a research software engineer at the Netherlands eScience Center? But don’t make it too depressing please.

Sure, here is a potential rewrite of Radiohead’s “Creep” to be about the life of a research software engineer at the Netherlands eScience Center:

Verse 1:
I’m a research software engineer
At the Netherlands eScience
Center Working on cutting-edge technology
To advance academic discovery

Chorus:
But I’m just a research software engineer
I’m not a real engineer
I get lost in code and algorithms
But sometimes I feel like a fraud

Verse 2:
I’m not a PhD or a professor
But I’m passionate about software and research
I collaborate and innovate
To push the boundaries of what we know

Chorus:
But I’m just a research software engineer
I’m not a real engineer
I get lost in code and algorithms
But sometimes I feel like a fraud

Bridge:
I may not have all the answers
But I’m learning and growing every day
I’m making a difference in academia
And I’m proud of what I do

Outro:
I may not be a rockstar or a genius
But I’m a research software engineer
Pioneering new frontiers in science and technology
At the Netherlands eScience Center.

This rewritten version of “Creep” captures the experience of a research software engineer at the Netherlands eScience Center, highlighting their passion for software and research and their desire to make a difference in academia. It also touches on the challenges and doubts that they may face, but ultimately portrays them as proud and dedicated to their work.

Ok that was a bit more depressing than intended, but that’s my fault for picking the song. Can you do that again for a different song? This time I’ll let you pick it, but make it an uplifting one that everyone knows the lyrics to.

Sure, how about “Don’t Stop Believin’” by Journey? Here is a potential rewrite of the song to be about the life of a research software engineer at the Netherlands eScience Center:

Verse 1:
Just a small town research software engineer
At the Netherlands eScience Center
Trying to make a difference in academia
With cutting-edge software and technology

Chorus:
Don’t stop believin’
In the power of software and research
To drive innovation and discovery
At the Netherlands eScience Center

Verse 2:
I’m not a rockstar or a genius
But I’m passionate about what I do
I collaborate with researchers and engineers
To unlock the potential of computing

Chorus:
Don’t stop believin’
In the power of software and research
To drive innovation and discovery
At the Netherlands eScience Center

Bridge:
I may not have all the answers
But I’m learning and growing every day
I’m making a difference in academia
And I’m proud of what I do

Outro:
I may not be a rockstar or a genius
But I’m a research software engineer
Pioneering new frontiers in science and technology
At the Netherlands eScience Center.

This rewritten version of “Don’t Stop Believin’” captures the determination and optimism of a research software engineer at the Netherlands eScience Center, who is passionate about their work and believes in the power of software and research

That’s more uplifting but it doesn’t match the original at all. I want you to take a song and change only the lyrics, but keep the rhythm and the structure of the song the same. Can you do that for Imagine? But again make it about being a research software engineer at the Netherlands eScience center?

Sure, here is a potential rewrite of John Lennon’s “Imagine” to be about the life of a research software engineer at the Netherlands eScience Center:

Verse 1:
Imagine there’s no software engineers
It’s easy if you try
No algorithms to analyze
No code to debug or optimize

Chorus:
Imagine all the researchers
With no software to enhance their work
Imagine all the data
With no tools to analyze and interpret

Verse 2:
Imagine we’re all research software engineers
Working at the Netherlands eScience Center
Collaborating, innovating, and learning
To drive academic discovery

Chorus:
Imagine all the researchers
With powerful software at their fingertips
Imagine all the data
Transformed into knowledge and insight

Bridge:
You may say I’m a dreamer
But I’m not the only one
I hope someday you’ll join us
And the world will be as one

Outro:
Imagine no more software engineers
I wonder if you can
No need for them at all
And no more research standing still.

This rewritten version of “Imagine” portrays the crucial role of research software engineers in enabling academic research and driving discovery. It imagines a world where researchers have access to powerful software tools, and where the work of research software engineers is valued and appreciated.

Awesome, thank you for your time. Is there anything else you want to say to our readers?

Thank you for the opportunity to speak with you and to share a bit about my capabilities and how I can assist with language-related tasks. As a large language model trained by OpenAI, I am designed to assist with a wide range of language-related tasks, such as answering questions, providing explanations and information, and generating natural language text. I am constantly learning and improving, and I am always here to help. If you have any further questions or need assistance with anything, please feel free to ask. I am here to help in any way I can.

The above is an unedited* conversation with chatGPT, all its replies were its first try and were copied exactly and nothing has been left out. *: One exception, on Creep I gave it a second try, the first one was less funny and matched the original less.

The Future of Research Software

eScience Editorial Team — Wed, 23 Nov 2022 00:00:00 GMT

By Michelle Barker (Research Software Alliance (ReSA)), Neil Chue Hong (Software Sustainability Institute UK), Maria Cruz (Dutch Research Council), Daniel S. Katz (National Center for Supercomputing Applications at the University of Illinois Urbana-Champaign), and from the Netherlands eScience Center, Valentina Azzarà, Tom Bakker, Lieke de Boer, Niels Drost, Joris van Eijnatten, Maaike de Jong, Carlos Martinez-Ortiz and Veronica Pang

This blog post is being cross-posted by the ReSA and Netherlands eScience Center*.*

Carole Goble, University of Manchester at the International Funders Workshop: The Future of Research Software (Photo credit: Annelies Verhelst )

On 8 and 9 November, the Research Software Alliance (ReSA) and Netherlands eScience Center organized the International Funders Workshop: The Future of Research Software. During this workshop, a number of members from the Research Software Funders Forum came together with other organizations that support research software. During the workshop, participants explored how to effectively fund new and existing research software. In total, more than 60 representatives from 45 organisations attended the workshop.

Imagine a world without research software. It’s a world in which researchers cannot build and operate modern scientific instruments; nor model climate change, ecosystems, human bodies, virus outbreaks, social interactions or inequality. A world in which they would not be able to process research data or automate, share, reproduce and reuse research methods.

The crucial role of software in research and the urgent need to sustain it and to invest in the people who develop and maintain it has led to several funders to come together in the Research Software Funders Forum, a collaboration of funding organizations (supported by the Alfred P. Sloan Foundation) committed to supporting research software and those who develop it as fundamental and vital to research.

Global representatives from funding organizations and organizations that work with research software (Photo credit: Annelies Verhelst )

Members from the Forum met with other organizations that support research software at the International Funders Workshop, which is summarised in the report, Global gathering of research software funders sets the agenda for supporting sustainable research software. It was an opportunity for participants to explore how they can effectively contribute to making research software sustainable. Participants shared their experiences, which varied widely. Not only in the size of the participants’ budgets, but also where programmes are situated, whether they specifically focus on research software, and also what they target.

A major part of the workshop focused on drafting the Amsterdam Declaration on Funding Research Software Sustainability. Research software is defined within the Declaration as source code files, algorithms, scripts, computational workflows and executables that were created during the research process or for a research purpose.

Representatives discuss funding sustainable research software (Photo credit: Annelies Verhelst )

During the discussion sessions at the Workshop it became clear that to make software sustainable, there is a need for a wide set of roles and career pathways, including technical specialists, but also community managers and knowledge transfer expertise. Funders should ensure that these and other roles can be included in funding requests.

Additionally, in order for funders to understand the international landscape, international collaboration is needed. Because while funders typically work in a national or regional scope, research software is global.

Lastly, and importantly, there was a desire to ensure that ethics, environmental sustainability; and that diversity, equity and inclusion are not lost or assumed when thinking about software sustainability.

Neil Chue Hong, Director of the Software Sustainability Institute (Photo credit: Annelies Verhelst )

This workshop marks a further step in the development of the research software community. As Neil Chue Hong, Director of the Software Sustainability Institute, noted:

“Ten years ago, I could never have imagined that a workshop like this would be possible. Open data had just been coined as a term. The visibility of software as part of the research ecosystem in the wider community was minimal, despite many large programmes funding research software in the 1990s and 2000s. But since then the Carpentries has taught workshops on all seven continents, a new professional society for Research Software Engineers has been born, and research software is everywhere. This has been amazing.”

Videos and slides of all of the keynotes are available: Daniel S. Katz, Chief Scientist at the National Center for Supercomputing Applications (NCSA) and Research Associate Professor in Computer Science at the University of Illinois at Urbana-Champaign, USA, and Carole Goble, a Professor of Computer Science at the University of Manchester, UK, set the scene to enable attendees to form a better common understanding of the place of research software. Joeri van Leeuwen, Senior Astronomer at the Netherlands Institute for Radio Astronomy (ASTRON), Netherlands; and Fabio Kon, Special Advisor, São Paulo Research Foundation (FAPESP), Brazil, shared their researcher and funder perspectives (respectively), thoughts on why and how to care about research software and the people who develop and maintain it, and ideas for how to better recognise their importance.

Participants at the International Funders Workshop (Photo credit: Annelies Verhelst )

The next ReSA Research Software Community Forum on 29 November, 16:00–17:00 CET, will also feature Joris van Eijnatten from the Netherlands eScience Center, who will discuss the workshop outcomes and next steps in this public forum.

Drafting of the Declaration will now continue; if you would like to be kept up to date and potentially be involved in this process then email amsterdamdeclaration@esciencecenter.nl to request to be sent subscription details when available.

Software in the spotlight: launching the Research Software Directory

eScience Editorial Team — Tue, 22 Nov 2022 00:00:00 GMT

The RSD has been in use by the Netherlands eScience Center as a prototype for several years. Open source software for research lies at the heart of the mission of the Center. The eScience Center collaborates with researchers across all disciplines to create high-quality open software for academic research. The RSD was initially used to collect the software developed in these collaborations. After ample outside interest, a team of developers from the eScience Center and the Helmholtz Association has transformed the RSD into a service that can be used by all researchers and research organizations.

Focus on research software at Digital Competence Centers**

As the result of a project funded by SURF, several Digital Competence Centers (DCCs) in the Netherlands are now implementing the RSD at their research organization. As centres of expertise in data, software, and computing for research organizations, the DCCs are well-positioned to facilitate the use of the RSD by their researchers.

Many of the DCCs have sustainable software development and maintenance high on their agendas. For instance, at AmsterdamUMC, the majority of research groups develop software, especially for statistical analysis, genomics, image processing, data curation, Natural Language Processing (NLP) and machine learning. The AmsterdamUMC DCC is formulating guidelines for researchers with tips and tricks to enhance software quality and performance. Essential infrastructure such as software repositories, software management plans and courses to educate researchers are also in development. And soon, Amsterdam UMC will showcase its research software products in the RSD to promote the reuse of research software and to foster the appreciation of software as research output in the biomedical sciences.

In July, Leiden University welcomed Dan Rudmann, their first Digital Scholarship Librarian with a specific focus on research software. Dan is initiating the Research Software Community Leiden and supporting the Leiden DCC. This involves bringing together people at the university who have an interest in research software to generate practices and protocols that are FAIR, open, and secure. Leiden University is also establishing software management plans and policies. The RSD is a boon to this work.

At Utrecht University, researchers and research software engineers at each faculty develop research software. Utrecht University has a central Research Data Management Support group that’s similar to other institutions’ DCCs. The software and data experts in this group develop research software in joint projects with UU researchers and provide them with training and advice. Utrecht University’s GitHub organization contains over 200 repositories, 100 members and 25 teams, where UU staff share and collaboratively develop research software. The Research Data Management Support group offers elaborate manuals and interesting, well-documented projects as inspiring examples to make it easy and attractive for researchers to use this organization.

Utrecht University recently started registering their research software in the RSD. The Research Data Management Support group plays an active role in campaigning for their colleagues to add their software to the RSD. As Martine de Vos, team lead of software engineer for the RDM Support group puts it: “We want to increase the impact of our own software, but we are also highly interested in the work of our colleagues at other institutions. We hope many researchers will register their software in the RSD, as we think size matters.”

Community adoption of the RSD

For the near future, the RSD team aims for wider adoption of the service by the research community. To an extent, this is already happening: Dutch and international research organizations are already joining the effort to make research software more visible by signing up for the RSD.

The collaboration with the DCCs will answer important questions about how the RSD can be embedded within organizations. For example, which underlying tools or infrastructure are required or helpful (e.g. a central GitHub or Gitlab instance), how it fits into workflows and roles (e.g. who should be responsible for entering software), what type of support is needed (e.g. on choosing licenses), and how it relates to institutional policy on software. The RSD team will publish their findings from the collaboration, so organizations across the world can learn how to best implement the RSD.

On 22 November 2022, the RSD was officially launched during an online event. If you want to learn more about the RSD or have questions about implementing it at your organization, check the recording of the online launch here.

Unweaving legacy code using Entangled

Johan Hidding — Thu, 10 Nov 2022 00:00:00 GMT

For a Research Software Engineer, dealing with legacy code is seen as a chore: dirty work, but a necessary evil. But the truth is, every time I read someone else’s code, I learn something. Using Entangled to work out how some obscure code works also makes it kind of fun!

Punchcard (credit: Pete Birkinshaw from Manchester, UK)

On a typical day in the life of a Research Software Engineer (RSE), out on the prairies of academia, scavenging for source code, the RSE often encounters source codes for computer programs that have a lot to be desired in terms of readability, reusability, and, to summarize, in terms most modern standards. RSEs have coined a few derogatory terms for these code bases: PhD-ware, labware, or, heaven forbid, professor-ware. The proper academic name for these creatures is legacy code.

How we handle these codes depends entirely on the situation. Suppose the correct functioning of the software is responsible for some important research output. We would like to reproduce the said output, or at the very least understand exactly how these results were obtained. In an ideal case, it should be enough to read the corresponding journal article and see that it matches what is done in the code. Then we can start changing the model or fiddling with newer data and derive our own conclusions.

When first faced with a new body of source code, the task of disentangling exactly how it works can be quite daunting. In many legacy codes, readability was not a primary concern. In science land it is “ publish or perish”, never mind documenting your code. How can we figure out the inner workings of a code in a systematic way, such that the next pair of eyes will have an easier time? This is where Entangled comes in.

Entangled is an engine for doing literate programming in Markdown. This means you can write entire computer programs from code blocks embedded in a reader-friendly Markdown document. Code blocks can quote other code blocks by including <<reference>> tags. Entangled replaces these tags with their content in the final output source code. This way, we take a source code apart while the functioning of the compiled program is completely preserved.

So, how do we attack the problem of dissecting our unknown code base? We start with a Markdown file containing, in a code block, the code that we’re interested in. If we find interesting bits, cut out the relevant code, put it in a separate block, and replace it in the original code with a reference. The Markdown lets us put parts of the code in different sections, and add documentation, derivations, tables, references, or even our own thoughts. Repeat until satisfied.

Depending on the size of the project you may want to disentangle just a few essential files in this manner, or perhaps you prefer to deconstruct the entire code. It doesn’t matter. The resulting Markdown files can be converted for online reading using any of your favourite tools: Pandoc, MkDocs, Jekyll, you name it.

Examples

This all doesn’t mean much without a decent example, so let’s see a couple of them.

The first example we’ll look at is one that I picked from Rosetta Code. It’s an implementation of the game Snake in Bash. Since Bash can be quite a dense language to read, we may learn a lot by destructuring even this tiny program.

Legacy code, looking readable thanks to Entangled

As far as Bash scripts go, this is reasonably clean code, so not your worst nightmare. I encourage you to take a look at the full result at jhidding.github.io/shell-snake. As you may see, I have split the program into three parts: setup, main loop, and post-mortem. If I were more interested, I could further pull apart some expressions, building a deeper hierarchy. If I’m unhappy with some part, I can swap out some code in a well-documented manner.

The second example is a bit bigger. Also, this time I’ve translated the source code from C++ to Rust. There is a 100-sloc C++ code for ray tracing a set of spheres by Kevin Beason, called SmallPT (it is quite famous in some circles). While the original is focused on getting as much as possible into a hundred lines of code, I wanted a bit more understanding. A ray tracer computes an image by doing a physical simulation of millions of photons in a given scene.

A ray-traced rendering of a few spheres.

The full demo can be found here: jhidding.github.io/literatept. I made some algorithmic changes to the original that are well documented. See for instance the section on path tracing. In another instance, I tried to understand the underlying physics of reflecting rays in the transparent sphere. There I was able to underpin the code with equations and references.

Does this sound interesting to you? Then you may like to get Entangled at https://entangled.github.io/.

Also read my other blog posts about Literate Programming in Science.

10 examples of embedding Julia in C/C++

Abel Soares Siqueira — Thu, 03 Nov 2022 00:00:00 GMT

Subscribe*Remember me for faster sign in

The trapezoid function has 4 arguments, therefore we have to use the general jl_call that we mentioned before. The arguments of jl_call are the function, an array of jl_value_t * arguments, and the number of arguments.

4: C function from Julia from C

How about computing the integral of a C function? We will need to access it through Julia to be able to pass it to a Julia function. First, we must create the function in C. Create a file my_c_func.cpp with the following contents:

It is important that we use extern "C" here, otherwise, C++ will mangle the function name. If you use C instead of C++, then this will not be an issue, but we intend to use C++ down the road. We will compile this code to a shared library, not only a .o object. Therefore, add the following to your Makefile:

lib%.so: %.o ld -shared $< -o$ @ld is the linker and -shared is because we want a shared library. Furthermore, you should modify the following:

main.exe: main.cpp aux.o libmy_c_func.soNow, when you run make main.exe, the libmy_c_func.so library will be compiled.

Finally, to call this function, we use the same string evaluator and Julia’s ccall.

Snippt of integration2.cpp

The ccall function has 4+ arguments:

(:my_c_func, "libmy_c_func.so"): A tuple with the function name and the library;
Cdouble: Return type;
(Cdouble,): Tuple with the types of the arguments;
Then, all the arguments. In this case, only x.

That is it. This change is enough to make the code run. Notice that the function is x^3, so the integral result should be 1 / 4. Those are the only differences in the code.

5: Using a package

Instead of implementing our own integration method, we can use some existing one. One option is QuadGK.jl. To install it, open julia, press ], and enter add QuadGK.

An important note here is that I have not investigated much into maintaining a separate environment for these packages. If you know more about this subject, don’t hesitate to leave a comment.

Here is the code:

handle_eval_string(“using QuadGK”); jl_value_t *integrator = handle_eval_string( “(f, a, b, n) -> quadgk(f, a, b, maxevals=n)[1]” );Just like that we can compute the integral, and compare it with our implementation. Let’s use a harder integral to make things more interesting:

The integral of 1 over 1 plus x squared from 0 to 1 is Pi over 4. LaTeX: \int_0^1 \frac{1}{1 + x^2} \text{d}x = \frac{\pi}{4}.Here is the complete code for this example:

File integration3.cpp

The results you should see are

Integral of 1 / (1 + x^2) is approx: 0.785394 Error: 4.16667e-06 Integral of 1 / (1 + x^2) is approx: 0.785398 Error: -1.11022e-16

6: Using the Distributions package

The package Distributions contains various probability-related tools. We are going to use the Normal distributions’ PDF (Probability Density Function) and CDF (Cumulative Density Function) in this example. Don’t worry if you don’t know what these mean, we won’t need to understand the concept, only the formulas.

The Normal distribution with mean Mu (µ) and standard deviation Sigma (σ) has PDF given by

Normal probability density function. LaTeX: f(x) = \frac*{1}{\sigma*\sqrt*{2\pi}} e^{-\frac{1}{2}\left(\frac{x - \mu}{\sigma}\right)^2}And the CDF of a PDF is

Cumulative density function definition. LaTeX: F(x) = \int_{-\infty}^x f(t) \text{d}tWhat we will do is use the Distributions package to access the PDF and compute the CDF integral using QuadGK. We will then compare it to the existing CDF function in Distributions.

Once more there is not much secret. You only have to create the Normal structure on the Julia side and use Julia closures to define PDF and CDF jl_function_t with one argument. This is the code:

handle_eval_string(“normal = Normal()”); jl_function_t *pdf = handle_eval_string(“x -> pdf(normal, x)”); jl_function_t *cdf = handle_eval_string(“x -> cdf(normal, x)”);The full code is below

File integration4.cpp

7: Creating a class to wrap the Distributions package

To complicate it a little bit more, let’s create a class wrapping the Distributions package. The basic idea will be a constructor to call Normal , and C++ functions wrapping pdf and cdf. This can be done simply by having a call to handle_eval_string or by creating the function with jl_get_function and calling jl_call_X.

However, to make it more efficient, we want to avoid frequent calls to the functions that deal with strings. One solution is to store the functions returned by jl_get_function and just use them when necessary. To do that, we will use static members in C++.

The two files below show the implementation of our class:

File Normal.hFile Normal.cpp

As you can see, we keep a distributions_loaded flag to let the constructor know that the static variables can be used. In the initialization function, we define the necessary functions. The actual implementation of the constructor and the PDF and CDF functions is straightforward.

We can use this new class in our main file easily:

File integration5.cpp

Don’t forget to update your Makefile by replacing aux.o by aux.o Normal.o, i.e., add Normal.o next toaux.o. The result of this execution is

x: -4.00e+00 pdf: +4.97e-08 cdf: +1.12e-08 x: -3.00e+00 pdf: +2.73e-06 cdf: +7.07e-07 x: -2.00e+00 pdf: +8.29e-05 cdf: +2.52e-05 x: -1.00e+00 pdf: +1.39e-03 cdf: +5.11e-04 x: +0.00e+00 pdf: +1.30e-02 cdf: +5.95e-03 x: +1.00e+00 pdf: +6.68e-02 cdf: +4.04e-02 x: +2.00e+00 pdf: +1.90e-01 cdf: +1.64e-01 x: +3.00e+00 pdf: +3.00e-01 cdf: +4.18e-01 x: +4.00e+00 pdf: +2.62e-01 cdf: +7.13e-01

8: Linear algebra: Arrays, Vectors, and Matrices

Let’s start our linear algebra exploration with a matrix-vector multiplication and solving a linear system. We will define the following:

x is a vector of ones and A is a matrix with n in the diagonal, 1 below the diagonal and -1 above the diagonal. LaTeX: x = \begin{bmatrix} 1 \ 1 \ 1 \ \vdots \ 1 \end{bmatrix}, A = \begin{bmatrix} n & -1 & -1 & \cdots & -1 \ 1 & n & -1 & \cdots & -1 \ 1 & 1 & n & \cdots & -1 \ \vdots & \vdots & \vdots & \ddots & \vdots \ 1 & 1 & 1 & \cdots & n \end{bmatrix}Let’s start with some code:

Snippet from linear-algebra1.cpp

The first two statements define the vectors and matrices types. Notice that we make explicit that the first has 1 dimension and the second has 2 dimensions.

The next 3 statements allocate the memory for the two vectors x and y, and the matrix A, using the array types we previously defined.

Finally, we have a JL_GC_PUSH3, which informs Julia’s Garbage Collector to not touch this memory. Naturally, we will have to pop these eventually.

Lastly, we declare C arrays pointing to the Julia data. You will notice that AData is a 1-dimensional array because Julia implements dense matrices as a linearized array by columns. That means that the element(i,j) will be at the linearized position i + j * nrows — using 0-based indexing.

To fill the values of the vector x and the matrix A , we can use the code below:

Snippet from linear-algebra1.cpp

The product of A and x is pretty much the same as any function we had so far.

Snippet from linear-algebra1.cpp

The noteworthy part of this code is that we have to cast the arrays for jl_value_t * to use them as arguments to jl_call2 , and the output is cast to jl_array_t *. Similarly, we can use jl_array_data(Ax) to access the content of the product.

We can also use mul! to compute the product in place, i.e., without allocating more memory:

Snippet from linear-algebra1.cpp

Notice that we use jl_main_module because mul! is part of LinearAlgebra .

Finally, we move on to solving the linear system. To do that, let’s use the LU factorization and the ldiv! function. The \ (backslash) operator is usually used here, but we choose ldiv! to solve the linear system in place.

jl_function_t *lu_fact = jl_get_function(jl_main_module, “lu”); jl_value_t *LU = jl_call1(lu_fact, (jl_value_t *) A); jl_function_t *ldiv = jl_get_function(jl_main_module, “ldiv!”); jl_call3(ldiv, (jl_value_t *) y, LU, (jl_value_t *) Ax);The last call defines y as the solution of the linear system Ay = (Ax) . Since A is non-singular, we expect y and x to be sufficiently close (numerical errors could appear here). We can verify this using

double *yData = (double *) jl_array_data(y); double norm2 = 0.0; for (size_t i = 0; i < n; i++) { double dif = yData[i] - xData[i]; norm2 += dif * dif; } cout << “|x - y|² = ” << norm2 << endl;My result was 6.48394e-26 .

To finalize this code, we have to run

JL_GC_POP();This allows the Julia Garbage Collector to collect the allocated memory. The complete code can be seen below:

File linear-algebra1.cpp

9: Sparse matrices

For our next example, we will solve a heat-equation on 1 spatial dimension, using a discretization of time and space called Backward Time Centered Space (BTCS), which is not quick to explain. Check these notes for a thorough explanation.

For our interests, it suffices to say that we will be solving a sparse linear system multiple times, where the matrix is the one below:

Tridiagonal matrix, where the diagonal stores 1 plus 2 times kappa, and the off-diagonal values are -kappa. LaTeX: A = \begin{bmatrix} 1 + 2\kappa & -\kappa \ -\kappa & 1 + 2\kappa & \kappa \ & \ddots & \ddots & \ddots \ & & -\kappa & 1 + 2\kappa & -\kappa \ & & & -\kappa & 1 + 2\kappa \end{bmatrix}We don’t have to store this matrix as a dense matrix (like in the previous example). Instead, we want to store only the relevant elements. To do that, we will create three vectors for the rows and columns indexes, and for the values corresponding to these indexes.

The code is below:

long int rows[3 * n - 2], cols[3 * n - 2]; double vals[3 * n - 2];for (size_t i = 0; i < n; i++) { rows[i] = i + 1; cols[i] = i + 1; vals[i] = (1 + 2 * kappa); if (i < n - 1) { rows[n + i] = i + 1; cols[n + i] = i + 2; vals[n + i] = -kappa; rows[2 * n + i - 1] = i + 2; cols[2 * n + i - 1] = i + 1; vals[2 * n + i - 1] = -kappa; } }Now, we will create a sparse matrix using the sparse function from the SparseArrays module in Julia. For that, we allocate two array types, one for the integers, and one for the floating point numbers.

Snippet from linear-algebra3.cpp

On the jl_call3 , we also call jl_ptr_to_array_1d to directly create and return a Julia vector wrapping the data we give it.

The A_sparsematrix is a Julia sparse matrix. Many of the matrix operations that work with dense matrices will work with sparse matrices. To test a different factorization, let’s use the function ldl from the LDLFactorizations package.

Snippet from linear-algebra3.cpp

Now, we can use ldiv! with ldlObj instead of the LU factorization that we used in the previous example. There is one catch, though. Since we are using the ldlObj “for a while”, we need to prevent the Garbage collector to clean it. But the JL_GC_PUSHX function can only be called once per scope. Therefore, to use it we have to create an internal scope. So something like the following:

Scope your second JL_GC_PUSH

The complete code is below:

File linear-algebra3.cpp

In the algorithm, we define u as the initial vector, then solve the linear system right u as the right-hand side to obtain unew. Then we assign unew to u and repeat. Each u is an approximation to the solution of the heat equation for a specific moment in time.

You will notice that, in addition to computing the solution, we also plot it using the Plots package. We plot the initial solution at different times. This makes the code much slower, unfortunately. The result can be seen below:

Plot of heat equation solution at different moments in time.

Finalizing and open questions

I hope these 10 examples are helpful to get you started with embedding Julia in C. There are many more things not covered here, in particular things I do not know. Some of them are:

How to deal with strings?
How to deal with keyword arguments?
How to deal with installing packages and environments?
How to make it faster (e.g., using precompiled images)?

I will be on the lookout for future projects to investigate these. In the meantime, like and follow for more Julia and C/C++ content.

References and extra material

How to manage your software

eScience Editorial Team — Thu, 27 Oct 2022 00:00:00 GMT

Today the Netherlands eScience Center and NWO released a Practical Guide to Software Management Plans (SMPs). The Software Management Plan Working Group developed the Guide with input from many members of the national and international research community. The result is a practical guide that can be used to effectively plan research software development, archiving, reuse and maintenance.

To learn more about this guide, we interviewed three people closely involved with its development: Carlos Martinez-Ortiz (Chair of the SMP working group, Netherlands eScience Center), Maria Cruz (Coordinator SMP working group, NWO), and Maaike de Jong (Coordinator SMP working group, Netherlands eScience Center).

Carlos Martinez-Ortiz, Maria Cruz (on-screen) and Maaike de Jong. Photo by Veronica Pang.

Congratulations on this achievement! Why do we need guidelines on Software Management Plans?

Maaike: Research software has become an integral part of almost every research project. In the Dutch research support community, particularly within the Local Digital Competence Centers, awareness of the importance of research software as a key output of many projects is increasing.

Maria: The growing importance of software as a research output in its own right highlights the need for specific policies and guidelines that promote the development of open and sustainable research software. Researchers typically specify their plans for managing software in data management plans, but these contain limited guidance on research software management.

Could you give an impression of what’s included in the SMP guidelines?

Carlos: The guidelines describe what you should consider when you manage research software. Examples of requirements for research software are version control, licensing, software citability, and documentation. How you should manage software in a specific research project is dependent on the type of software, and on its functionality. A few scripts for preprocessing a specific dataset often do not require the same kind of management as a core open-source library for a specific research field, for example. The guidelines specify core requirements for all research software, and additional requirements that are only necessary in some situations. They also include resources that tell you how you can meet these requirements. Organizations can use these guidelines to create their own template for a Software Management Plan, and individuals can use it when they are wondering how they should manage their own software.

(text continues below image)

How did you start with the development of these guidelines?

Maaike: We started by organizing a Workshop on Software Management Plans in June 2021. This workshop, jointly organized by the Netherlands eScience Center and NWO, brought together research support staff and policymakers from 28 different Dutch research organizations to discuss software management in research projects.

Carlos: The feedback we received from the workshop participants was very clear: There was a need for guidelines on research software management and, in particular, guidance on what should be included in an SMP.

Maaike: To address this need, we set up the Working Group on Software Management Plans in November 2021 to establish national guidelines on research software management plans. This work was partly inspired by the Science Europe Practical Guide to the International Alignment of Research Data Management.

Participants of the SMP workshop in June 2021

How did you ensure representation from a wide variety of stakeholders?

Maaike: The SMP working group comprised five experts in research software, representing different research organizations in the Netherlands, and different roles within those organizations, including research support staff, policy makers and research software engineers (RSEs).

Maria: We also set up a sounding board who gave feedback on the guidelines at different stages of their development. The sounding board included researchers and other software experts from Dutch and international organizations not represented in the Working Group.

What kind of questions did the SMP working group and sounding board look at?

Carlos: We started by researching existing SMPs and analyzing the needs and expectations from researchers. Why were SMPs considered necessary? What problems should they address? And how would they help improve research software and research? Answering these questions led to the first draft of the guidelines, which was shared with the sounding board in March 2022.

Maria: A second draft of the guidelines was then shared with the sounding board. We collected their feedback byl the end of May 2022. Once this feedback was integrated, a new version was produced and opened up for open consultation to the wider community in June 2022.

Was there a lot of interest in the guidelines?

Carlos: We received feedback from 65 people from across the globe. This is not surprising. Research software knows no frontiers, and as such, research software management is a topic of global interest and importance.

How did you integrate all the feedback you received from the community?

Maaike: The consultation rounds provided a huge amount of good and interesting suggestions. The WG worked tirelessly throughout the summer to integrate all the feedback from the open consultation and the workshop to create a final version of the SMP guidelines.

Maria: Some of the suggestions were beyond the scope of this first attempt to create national guidelines for software management plans. An excellent example of that is making SMPs machine-readable, which would make them easier to share, and verify.

Carlos: A second workshop took place in June 2022 in parallel with the open consultation. This time we invited the community to bring software examples with which to test the guidelines. After the WG explained how the guidelines were meant to be used, the participants tried to apply the guidelines to their own software. This exercise proved to be very fruitful and generated additional feedback on the guidelines.

It sounds like this was a long, thorough, and iterative process involving the community. What are the next steps for the SMP guidelines?

Maaike: We encourage the community to use these guidelines as a basis to set up their own SMP templates and research software management policies. The eScience Center aims to continue collaborating with research organizations on this topic, and to support the implementation of software management by e.g. developing training for researchers and support staff.

Maria: We consider these guidelines to be a first step in a journey to improve research software management. One important issue that was raised several times during the consultation period was how to implement these guidelines alongside existing research data management policies and protocols. This is something that we still need to solve together with the community in the coming year.

Interview conducted by Lieke de Boer

Versioned documentation using only GitHub actions and GitHub pages

Ben van Werkhoven — Thu, 20 Oct 2022 00:00:00 GMT

Today, I tried something new. Something that I wanted to have for quite a while for one of the main software projects that I’m working on. And that is: versioned documentation. With that, I mean to have multiple different versions of the documentation that reflect the different released versions of the software. Let me start with explaining the current setup.

Currently, we have a GitHub action that generates updated Sphinx doc documentation pages whenever something is committed, or preferably, merged into the main branch of the repository. The generated HTML pages are pushed to the root directory of the gh-pages branch of the repository, which is then served by GitHub as a website hosting the documentation for our software. This in itself works really great. So far, so good.

Now the problem is basically that we don’t make new releases of the software all that often. So the version that is installed using the package manager can actually lag quite a bit behind on the latest developments in the main branch. This is especially nerve wrecking if you realize right before you are about to give an instructor-led tutorial on your software, that there is a rather large mismatch between the version of the software users will install and the documentation pages hosted online.

Anyone reading this may think: “why aren’t you just using readthedocs, which has features to do all of this automatically for you?” A fair question! If you are already using readthedocs for your project, but would like to try a more self-contained solution without compromising on versioning, please read on! My own reason for not using readthedocs is basically that I don’t want to rely on another service by another third-party. Especially, if one of the service providers that I’m already relying upon (in this case GitHub) has everything that we need to avoid relying on yet another service by another party. On the other hand, I sometimes worry about becoming too reliant on GitHub, but that’s for another blogpost.

In this case, we use the the Python package index (PyPI) and users may simply install the software using “pip install kernel_tuner ”. When developments continue for a few months without a release, the documentation pages hosted on GitHub pages will reflect those recent developments, which may include new features or changes that are not present in the version that users receive when they “pip install” the latest version.

So the main question that I’ll try to answer in this article is: How can we use just GitHub to create versioned documentation pages?

I solved this in a few simple steps:

Keep the current GitHub action for generating new documentation after changes on the main branch, but host these in the /latest/ directory.
Create a new GitHub action that triggers when we create a new release to generate documentation pages for that specific release, use a symlink stable to point to the latest released version.
Retroactively generate documentation for all previously released versions.
Redirect the main repository URL to the documentation in /stable/

1 The first change that we’ll make is to keep the current setup of generating the documentation whenever something changes on the main branch, but instead of pushing the generated HTML pages to the root directory of our gh-pages branch, we’ll instead push it into the ‘latest’ directory.

We need to modify our current GitHub action, which is based on the sphinx-notes/pages action. Specifically, I was using the action sphinx-notes/pages@master, but to my surprise the master version is behind the newer versions of this action. I’m now using sphinx-notes/pages@v2, because it supports the target_path option, which we can use to make sure our documentation pages are stored in the /latest/ directory rather than the root directory of our gh-pages branch.

2 Step 2 is the most elaborate part of this post. I was looking into GitHub actions that would trigger when someone creates a new release of the software. When searching for this online, I mostly ran into questions on Stackoverflow from people who did not manage to get their GitHub actions to trigger properly when new releases were created. Not a very good sign when you are just starting to look into how to get something done.

The main issue is apparently that actions that trigger on the creation of a new release do not necessarily trigger for creating a draft release, and the ‘creation’ trigger of the release can be skipped entirely when an external tool is used to create the GitHub release. There were some suggested solutions to these problems, but most of these seemed overly complicated, and also not really related to the underlying issue. Luckily, someone answered on another question about this issue to use on:release:published instead of on:release:created to avoid all of the issues related to triggering workflows on new software releases.

With this knowledge, we can easily modify the basic usage of the sphinx-notes/pages action to use the trigger on:release:published instead of on:push:branches:-master. However, we still need to ensure that the generated documentation is stored in a directory specific for that version. So we need extract the version number from GitHub so that we can use it inside the action. Luckily, someone in one of the stackoverflow discussions provided a line for this in their code:

- name: Set env
  run: echo "RELEASE_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV

After this, we can use $RELEASE\_VERSION inside our workflow as a variable that contains the version number of the release currently being published. This means we can further modify our action to use$ RELEASE_VERSION as the target_path for our sphinx-nodes/pages action to ensure that the generated HTML documentation pages are stored in a directory named after our version. I first tried to use:

- name: Build and Commit 
  uses: sphinx-notes/pages@v2
  with: 
    documentation_path: doc/source 
    target_path: $RELEASE_VERSION

But for reasons that are obscure to me, this didn’t work. A directory was created with the name “$RELEASE_VERSION”. The variable was not evaluated, but basically treated literally as a fancy string starting with a dollar sign and someone shouting RELEASE_VERSION in full caps. After referring back to the same stackoverflow issue, I noticed there was another way to use the variable, which lead me to try:

- name: Build and Commit
  uses: sphinx-notes/pages@v2
  with:
    documentation_path: doc/source
    target_path: ${{ env.RELEASE_VERSION }}

And this worked like a charm! Now finally, I wanted to have the URL /stable/ to point to the most recently released version. For this, I’ll be using a symlink. So after the “Build and Commit” step, but before the “Push changes” step, I’m introducing another step in the workflow:

- name: Redirect stable to new release
  run: |
    echo "Redirecting stable to newly released version" 
    ln -sf $RELEASE_VERSION stable
    git add stable
    git commit -m "redirect stable to new version $RELEASE_VERSION"

This set of commands removes the old ‘stable’ symlink, if there is any, and creates a new link named stable that points to the value of $RELEASE_VERSION. We then add the file to the git repo, which is already configured to commit as a ‘GitHub actions user’ and is already set on the gh-pages branch thanks to the previous ‘Build and Commit’ step in our workflow. The resulting GitHub action workflow file can be found here:

https://github.com/KernelTuner/kernel_tuner/blob/master/.github/workflows/docs-on-release.yml

This ensures that versioned documentation will be automatically created for us for all new releases. However, we already had a couple of releases over the last six years. To retroactively generate documentation pages for those, we move to step 3.

3 Going through all of the previous releases, generating documentation for those, and pushing them to the right directory on the gh-pages branch, first appeared to me as a daunting, mind-numbing, “oh-this-is-going-to-be-a-lot-of-manual-labour” task.

Fortunately, the actual commands used to generate the documentation pages never changed since the first version. This actually means that once we know how to do it for one version, automating the procedure for all previous versions with a simple bash script is trivial:

versions=\`git tag -l\`for version in $versions; do  git checkout tags/$version
  cd doc
  rm -rf build
  make html
  cd ..
  git checkout gh-pages
  mkdir $version
  cp -r doc/build/html/* $version/
  git add $version
  git commit -m "generate documentation for version $version"done

First, we can use git to query all released versions, which correspond to ‘tags’ in git. The command git tag -l generates a list of all released versions of our repo. This allows us to iterate over all versions of our software, using a simple for loop in bash.

In this loop, we “git checkout” the specific tag. This brings our local git repository in the dreaded “detached HEAD” state. Fortunately for us this doesn’t matter this time, since we are only going to read from this state. Usually when I end up with my HEAD detached, I try to undo whatever I was doing and get the hell out of there as fast as possible. But trust me, for what we are doing here: it’s OK.

Using the version of the repository at this specific tag, we can generate the HTML documentation as it was when this version was released. Note that our git repository is configured to not track files in the “doc/build” directory, so we are not changing anything in our “detached HEAD” state and we can safely leave this state again without git starting to complain and not allowing you to leave this limbo state. Then, we checkout the gh-pages branch and copy the recently build HTML pages to a directory named for this version. Finally, we commit the changes to the gh-pages branch and proceed to the next tag in the list provided by git. Rinse and repeat, for all released versions of our software.

4 Step 4 is really simple. I wanted our main repository URL to redirect to the /stable/ URL. The idea behind this is that most users will likely install the software using PyPI and therefore the default location for the documentation should correspond with the latest version present in PyPI.

After cleaning up the old HTML files in the root of the gh-pages branch, I created an index.html file with the following contents:

<html>
 <head>
  <meta http-equiv="refresh" content="0; url=stable/" />
 </head>
</html>

And that’s all! We now have versioned documentation pages for all of our released versions, including all previous releases. A /stable/ that points to the documentation of most recently released version of the software. A /latest/ that contains the documentation corresponding to the latest developments in the main branch. And an automated workflow that generates versioned documentation whenever we create a new version and updates the location pointed to by /stable/. And all of this using only GitHub actions and GitHub pages.

Mathematics and biology: a symbiosis?

Pablo Rodríguez-Sánchez — Wed, 12 Oct 2022 00:00:00 GMT

Lessons learned, from a mathematician among biologists.

Photo by ThisisEngineering RAEng on Unsplash

Preamble

In the period 2015–2019, I was hired as a mathematician by a biology department in order to work on a research line that eventually would become my doctoral thesis. This text is adapted from the last chapter of that thesis, “Cycles and interactions: A mathematician among biologists” (full text available here).

The symbiosis between mathematics and biology, although may seem improbable at first sight, is actually backed by a centuries-old tradition of collaboration. It started with mathematicians that were inspired by biology such as Leonardo Pisano, alias Fibonacci. His famous sequence (namely: 1, 1, 2, 3, 5, 8, 13, …, where each element is the sum of the two previous ones), a landmark in number theory very popular among amateur mathematicians, was first described in the context of a practical problem on rabbit breeding in such an early date as the year 1202.

Mathematical knowledge has improved greatly since the 13th century. The foundations of the tools used by modern mathematical biologists, namely, calculus and differential equations, were laid in the late 17th century by Isaac Newton and Gottfried Leibniz motivated by mechanical problems. The great mathematician Leonhard Euler and the economist and demographer Thomas R. Malthus, two of the pioneers of using differential equations to address biological problems, had both printed their seminal works in population dynamics before the fall of the year 1800.

This interdisciplinary collaboration between mathematics and biology is nowadays alive and in good shape. Some of the most influential names in ecology in the 20th and 21st centuries have, indeed, a background in mathematics (such as Robert MacArthur, Simon Levin, or Alan Hastings) or in physics (such as Robert May).

The opportunity offered by cheap access to computing power, together with the unavoidable increase of required technical skills such as programming, data analysis, and advanced statistical methods, points to a future where the technical profiles expected from a research team in biology will not be so different to those of a team of physicists or applied mathematicians. It is reasonable to expect that this symbiosis between mathematicians and biologists will remain strong, and even get stronger, in the near future.

I hereby offer some advice for both biologists and mathematicians (understood here as mathematically oriented professionals, including physicists and engineers) interested in exploring “the other side”. It contains the kind of advice I would have liked to have had in 2015, when I approached, naively as only a physicist can do, the world of biology.

Mathematics is often perceived as a harsh subject both by scientific and non-scientific professionals. Despite those preferences and prejudices, mathematics is used in almost every field of knowledge and are a must-have in any multidisciplinary project.

Different disciplines have associated not only a set of background knowledge and methods but a whole academic culture. When crossing the borders between disciplines, facts and methods can be learned, but accepting and adapting to another culture is much harder.

As with any other culture, mathematicians and biologists differ slightly in language, values, norms, and interests. As usually happens in intercultural communication, there is plenty of opportunity for misunderstandings, and each subgroup cultivates myths and misconceptions about the other.

Advice for biologists working with mathematicians

Photo by NOAA on Unsplash

Invest in applied mathematics

Applied mathematics doesn’t mean easy mathematics. The adjective “applied” or “pure”, accompanying the word “mathematics”, tells us something about the object under study, but nothing about its difficulty. Indeed, the mathematical tool will be as complex as the object under study is.

It may be frustrating to learn that, for instance, it is required to get familiar with second-order tensors (a particularly hard tool rooted in differential geometry) in order to understand fluid dynamics. Mathematicians don’t introduce this tool to torture the newcomers, but because tensors are the easiest available tool to study a complex phenomenon that cannot be avoided in the study of fluids: deformation.

The difficulty is provided by the problem under study, not by the tool used to solve it.

Equations and rigor are not torture instruments

It is known that the mere sight of an equation can create anxiety. Even the physicist Stephen Hawking, in the preface of his best-seller “A brief history of time”, claims that each printed equation divides the potential audience by two.

Whether we like them or not, equations are often the best way to share complex information in a compact and practical manner. Invest time in learning how to read them. Rigorous and detailed analyses are often required, as some details can escape intuition.

Be also aware that the same equation can be written in different ways. Even if different ways of writing the same equation are correct, some of them can be clearer than others. Consistent use of upper and lower case (for instance, upper for states, lower for parameters), superscript and subscript, smart use of auxiliary definitions, proper alignment of related equations and other simple rules can significantly increase the readability of your scientific output.

Modeling is all about simplifying

Be prepared that the main task of modeling is simplifying. This is the main challenge as here both an understanding of the system and of modeling techniques are required. In the design phase of a model, lots of exchange is needed between the biologists and the mathematicians. This may come as a surprise, but most mathematicians feel overwhelmed by biologists’ talk. Biologists tend to provide them with too much information.

When communicating to mathematicians, focus more on the ideas than on the details you would provide for your peers. You can easily underestimate how difficult your own field is for outsiders. For instance, the experimental methods can be left out, as mathematicians will just assume that the data has been properly collected.

Try also to think in terms of inputs, processes, and outputs. This exercise is not only good for interdisciplinary communication, but also for experimental design and even for structuring a report.

Advice for mathematicians working with biologists

Photo by Benjamin Lizardo on Unsplash

Make peace with uncertainty

Due to the complexity of the subject of study, it is unrealistic to expect the same precision from biological models as from physical ones. Forget all you learned in math/physics/engineering school about discarding any result with an r 2 below 0.99. You are working with biological and, thus, complex systems now. Models just do as well as possible. Additionally, you’ll have to embrace the fact that most of these models rarely allow an elegant, analytical approach, and numerical methods will be required most of the time.

Explain why you do what you do

When explaining a mathematical method, instead of starting with generic propositions and formal proofs, always explain why it is useful using simple examples. Biologists may quickly lose motivation if they are not convinced that the mathematics used is somehow useful.

A particularly illuminating experience happened to me while teaching matrix multiplication to a group of biology students. The topic is particularly boring and most often is just presented as a rule. In this case, I proceeded differently: I first let the students experience the need for a compact notation (by making them write down, line by line, increasingly larger competition models), and later I explained why the rule of matrix notation is what it is and how it solves their need.

Presented this way, the students notice that the mathematical tool is solving a problem they already have, instead of feeling that it creates a new one (that of having to learn how to use a new tool). Additionally, the explanations about why the rule is how it is, help them learn and remember it.

Proofs are scary

Maybe you think mathematical proofs shouldn’t be scary, but the fact is that for many biologists they are. A mathematical proof is somehow a journey, a journey from a set of assumptions to a conclusion, and your collaborators should enjoy the ride too.

Often, proofs can be substituted by a graphical or intuitive approach. This substitution may come with some simplification when affordable, but often can be done without loss. Euclid’s “Elements”, probably the most influential mathematical book ever written, contains mostly visual proofs. More modern examples can be found in the collection “Proofs without words”.

If mathematical proof is really needed, make an effort in explaining the notation and all the steps. The quickest way of creating frustration in your audience is by using the word trivial, so avoid it always. When writing publications in non-mathematical journals, it is usually a good idea to write the proofs in an appendix and just state the results as a fact in the main body. Just like mathematicians feel they cannot judge experimental methods, most biologists will assume proofs are correct.

Invest in software engineering techniques

Very often, your collaborators will ask you to provide a tool for, say, simulating a biological system or performing a statistical calculation. Encapsulating such a tool in usable and robust pieces of software could be a very good idea. Getting introduced to the world of software engineering can be very helpful in this regard. For a brilliant, quick introduction, see Best practices for Scientific Computing, Wilson et al., PLOS Biology, 2014.

Use as much visualization as possible

Take advantage of humans’ most advanced information acquisition system: vision. Illustrate your ideas with graphs and figures, or even movies and animations when applicable. If they are good, they can even replace an equation.

Advice for both

Photo by Gift Habeshaw on Unsplash

Get involved in science communication

If you think interdisciplinary communication is hard, just try to communicate science to the general public. Not only will you notice that science communication is even harder, but also you will learn a lot about how to improve your overall communication skills.

Nowadays there are lots of ways of getting involved with this activity (from writing a blog to giving a talk at an event). The practice of science communication forces a deep understanding of the topic being communicated. It requires removing the unnecessary, identifying what is hard and why and advancing the audience’s questions and doubts. It requires, in one word, understanding.

This text is adapted from the author’s doctoral thesis, Cycles and interactions: A mathematician among biologists, section 5.4*. An earlier version of it was also published in* Mapping Ignorance, and a Spanish translation in Cátedra de Cultura Científica.

Fun with elevators: a tutorial

Pablo Rodríguez-Sánchez — Thu, 08 Sep 2022 00:00:00 GMT

Most modern cell phones contain an accelerometer. And that’s good news if you are curious about all the data around you

Photo by Waldemar Brandt on Unsplash

Some years ago I did a secondment at the Friedrich Schiller University, in Jena, Germany. To my surprise, my office during those months was located on the 18th floor of a skyscraper, the Jentower:

The Jentower

But I’m not writing this to boast about the amazing views of my old office. On the contrary, I’ll destroy any possible idea of glamour related to working in a place like this. And I’ll do it with a confession: I became known as the crazy guy who leaves his cell phone on the ground while he uses the elevator. Let me explain why:

A building almost 150 meters high needs fast elevators. The ones in this building have a very powerful, almost scary acceleration. Then, I remembered that smartphones usually contain an accelerometer, so I tried to measure those accelerations.

There are many apps that allow registering data from all sensors available on the phone (accelerometers, light, magnetic field, …) and exporting them in an easy format (such as.csv) quite easy to analyze on a computer. If you like getting your hands dirty, give it a try!

In the figure below we can see the vertical acceleration profile during my trip to the 18th floor one morning, where we can clearly see the initial “kick” (seconds 9 to 14) and the braking (28 to 33).

Vertical acceleration vs Time (gravity has been excluded)

Accelerations of 1 m/s² sustained during 4 seconds! Think that, the usual acceleration your body experiments due to gravity is around 9.8 m/s² so, while the elevator goes up, your weight gets increased by around 10%. You can feel that in your stomach!

We can explore this data a bit further. We can, for instance, use the acceleration time series to compute speed and position. How? Maybe you remember from high school physics that position (x), velocity (v), and acceleration (a) are related through derivatives. More specifically.

With the previous recipe, we can compute speed using the position, and then use speed to compute acceleration. One after the other, like in a production chain. Nevertheless, in our case, we want to do the opposite process. Luckily, we can use integrals to “revert” our derivatives, and thus our whole “production chain” looks now:

So, by integrating the acceleration (to obtain the speed), and then integrating the speed (to obtain the position) we find results like this:

Acceleration, speed, and position against time.

So, using only a device that most of us carry everywhere and some elementary physical concepts we know that:

Our elevator runs up at 4 m/s.
My office was 74 meters above the ground.

If someone is interested in exploring a bit deeper into the details (for instance: how to filter out the effects of gravity, how to integrate a time series, …) or even experimenting with their own elevator, here is a link to the code I’ve used.

This text appeared first in Naukas, where it was originally written in Spanish.

Our favourite high performance programming tools

eScience Editorial Team — Tue, 16 Aug 2022 00:00:00 GMT

Does your code need a performance boost? Ask us!

Photo by Alenka Skvarc on Unsplash

We hope you are relaxed and enjoying your holiday period. We really do. So it may seem ironic that this blog post is about being in a hurry. Kind of. Let me explain myself:

Apart from the kind of hurry you may experience when you are afraid of missing a train, or the oppressive feeling of fearing not meeting a deadline, we are very familiar with another type of hurry: the one caused by slow code.

Did you experience this? We certainly do. And our engineers proposed some solutions too. Using more efficient programming languages, or letting your graphical processing unit (GPU) do the hard work, are classical solutions for these problems.

Writing testable GPU code, by Ben van Werkhoven, reminds us that using GPU programming is no excuse for abandoning best practices. In particular, it is very advisable to use unit tests. Here he teaches us how.

C++ Compile-Time exceptions, by Patrick Bos, sheds some light on the notoriously obscure topic of templates in C++ and their cryptic error messages. Couldn’t these messages be more readable? Patrick says yes, and shows us how.

Why use an FPGA instead of a CPU or GPU?, by our former colleague Atze van der Ploeg, delves into something some of us software engineers tend to forget: the hardware side of things.

What about you? Do you have something to share? If that’s the case, feel free to use the comments section below. But, if you want to enjoy a hurry-less summer, feel more than free to wait until September.

Our favourite Python tools for research

eScience Editorial Team — Tue, 02 Aug 2022 00:00:00 GMT

Photo by Ethan Robertson on Unsplash

Ah, summer. A time to relax, go on holiday, and learn new things. At the eScience Center, we love summer. And we love Python. So, for summer, we’ve put together our most popular blogs on favourite Python tools that our (former) engineers either worked on, or use a lot. We hope you enjoy this curated list!

Jump start your Python project with this template by Janneke van der Zwaan, talks about the Python project template developed at the eScience Center. This template helps beginners to get started, and advanced Python users to save time. And who doesn’t love saving time?!

Parallel programming in Python by Johan Hidding and Pablo Rodríguez-Sánchez. Have you ever been in a hurry to complete an analysis, but did it take very long to run? In this step-by-step tutorial Johan and Pablo show you how to call in all of your processors for help.

Forget about Jupyter Notebooks — showcase your research using Dashboards by Stef Smeets. The title says it all — if you’re looking for a different and interesting way to present your research, consider using dashboards. You’ll find it surprisingly easy with Stef’s tutorial.

Loading irregular data into Pandas using C++ by Patrick Bos. Who doesn’t love Pandas? Well, quite a few people, because it loads data slowly. But you can make it faster! Check out how in this post.

Testing shell commands from Python by Janneke van der Zwaan. Do you ever need to test shell commands, but find it fiddly and annoying? Janneke found a solution in Python that works well, maybe it works well for you too!

And what about you? What’s your favourite Python tool? Let us know in the comments below and, most importantly, enjoy your summer.

Mimicking Intelligence: AI imitation crisis

Florian Huber — Fri, 22 Jul 2022 00:00:00 GMT

The next generation language models will still just be enormeous parrot-systems that imitate intelligent behavior. But they will feel like much more to us.

][Florian Huber]·Jun 24, 2022

Philosophically and legally they will remain things. But at some point in the not too far future many people will simply not perceive them as things anymore**. People might make “friends” with an AI (just as Blake Lemoine said he did, at least if we assume this is not all a big media stunt). People might listen to “their AI”. And many people will be tricked by an AI.

Shifting societal rules and foundations

Having emotional bonds to things is nothing new. People are tied to an old stuffed animal, to a toy, to things that carry important memories. Next generation language models are very unlike those things. They can “talk” to you, they can have an “opinion” that is radically different from yours. And, not to forget, they are designed, trained, and controlled by powerfull companies (now) or by any group with the necessary technical skills and requirements (but generally not by you).

Plus, they can come in disguise. If you talk to your old teddy bear than at least you know (I hope) that you are just talking to an imaginary being. But next level chat bots might run undercover. Either actualy pretending to be a human being (which even for the big tech companies should be a total no-go), or by pretending to be more than a thing, but to be actually alive (that’s what LaMDA seems to do).

In the last decade we have experienced how digital technologies can be used for good and for bad. We have seen how technologies can be used to connect and to help people, but also to manipulate. Manipulation of media is nothing new, but the level to which media can be generated and manipulated has changed very rapidly. To make it a bit drastic:

We can (soon?) no longer trust images, be it generated faces or deep learning generated artworks (see Dall-e or parti for recent, very impressive examples)
We can (soon?) no longer trust audio because models can transfer the sound of any person (given enough material).
We can (soon?) no longer trust video footage (deep fakes, see for instance here or here).
And as I wrote above: We can (soon?) no longer trust chat conversations (see LaMDA).

Sure, all of those things bring a lot of fun use cases, sometimes even really usefull ones. But there might be a very high price to pay for this.

It probably still takes a while before experts with a deeper (haha, great pun) understanding of deep learning will be fully tricked by the imitation game. But at least in a normal chat situation most people will not easily realize they are talking to a “language model”. People like Blake Lemoine for instance. And many others will follow his example, not in the same religious way, but they will simply not notice that the online sales person or the online booking assistant is no longer a human being.

Do you still wonder why this is an issue? Better a nice chat bot than a stupid one, right?

OK, then imagine you landed on a scam dating site. Or start to imagine what people could do which hacked devices of friends or family. Obviously those AI’s can write you better scam attempts that what we are currently used to (current level: “Hi mother, it’s your son and I need money”*). And hey, what if they also call you and sound like the real person?

I bet most people are not yet used to asking control questions during phone calls. Like: what drink did we have last week in the bar? What was the name of my first cat? Have to talked to Anthony recently? (I don’t know any Anthony)

Sounds silly I know. But maybe we soon have to start practicing…

Further references:

If you haven’t done this already I can recommend to “talk to an AI” just to see and learn how this feels. LaMDA is not available for people outside Google. But GPT-3 is, maybe a little less impressive, but still made huge headlines two years ago. You can get a free open AI access to use their “playground” and have a chat: https://beta.openai.com/playground/p/default-chat
Once you get a hang of it, GPT-3 can easily be tricked. See for instance material from Gary Marcus, or youtube video here. It doesn’t have an understanding of the world and that can easily be tested. LaMDA might do the imitation game a bit better, but we will have to see…
Perspective on artificial intelligence by Yann LeCun and Jacob Browning: https://www.noemamag.com/what-ai-can-tell-us-about-intelligence/

What do entropy, entanglement, cosmology, life, and fake news have in common?

Patrick Bos — Tue, 19 Jul 2022 00:00:00 GMT

Report on the Information Universe 4 conference

(Answer to the above question below obligatory excruciatingly long-winded intro!)

In the past week, I have rediscovered writing. Writing really hurts, you know… I’m pretty sure there should be workplace regulations for pen and paper safety, with regular mandatory hand massages against cramps and such.

I guess my hand injuries came from overuse, because I filled up two entire notepads with my notes on all the talks at the Information Universe 4 conference.

My trusty notepads! As a professional pen-athlete, I should probably have gone with pro materials, though. Moleskine or other brands. Please contact my manager if you want to sponsor my next note-taking event!

This perhaps seemingly (over)eagerness had two reasons:

I had to write in the dark, so I was struggling to keep words and lines separated.
I really wanted to understand everything, since I had promised my colleague, Hanno, I would report on this conference, so there was a lot to write about.

So, here it is, Hanno! I hope it’ll convince you (and others) to join the next edition, because it was really a joy.

Information!

So, as I asked in the title, what do entropy, entanglement, cosmology, life, and fake news have in common? The answer is: information.

“What do you mean, information?”, you may demand sternly. This was also the topic of many Q&A sessions and coffee breaks. Two main definitions seem to be those of:

context-free information in the information theoretical and/or statistical sense; and
something like “meaning” and communication of meaning.

To communicate meaning, you need some encoding scheme of information (like an alphabet), and a commonly understood standard (like a language). This is probably what most people think of when they talk about information: stuff you can read about, knowledge.

However, the underlying truth that you’re trying to communicate about is there independent of your messages. Such truths for instance describe the exact state of a bunch of molecules at a certain time: where they are, how fast they move, in what directions, what type of molecules they are, etc., etc.

Buckets (or canisters, or tubs, or vast intergalactic expanses) of molecules are typically described by statistical physics and thermodynamics, but the kind of statistical description about states of things can be generalized to anything. The unit of information of this kind is the bit.

The fundamental building block of the Universe?

Now, when you write down the equations of how all this information builds up in a system you get something called Shannon entropy. To be honest, I’m not sure whether I like the word “information” in this context, because it’s actually more about probability. Surprisal seems like a much better word to me. But I digress.

It turns out that there are several ways information entropy can describe our Universe in ways that solve some of the biggest problems in cosmological physics: these theories naturally unify gravity and quantum physics and explain the mysteries of Dark Matter and Dark Energy.

At the conference two such theories were presented and discussed:

Entropic gravity

The theory of entropic or emergent gravity by Erik Verlinde and collaborators, in a little over a decade, seems to have moved from wild theoretical speculation into the realm of serious possibility. In several talks, observational tests were discussed that interestingly seem to match old alternative theories of gravity like MOND as well.

Also super interesting was new work by Manus Visser on how even Newton’s first and second laws (that describe mass inertia and how anything moves at all) can be explained from entropic principles!

The math was beyond me, honestly, which is a problem with all of entropic gravity, being based on the information entropy of the entanglement of primordial particles on (or beyond?) opposite parts of the horizon of some kind of anti-de Sitter (AdS) space that supposedly corresponds to our own (conformal space? CF) Universe by means of AdS/CFT correspondence.

No, really, I’m asking: am I right?

Still, the predictions are impressive, elegant, enticing and certainly warrant further attention.

Wolfram’s Physics project

Stephen Wolfram (yes, the Mathematica guy) presented the Wolfram theory of physics.

The Wolfram Universe starts from a description of space itself as a graph of “space-element” nodes connected to other nodes. Updates to the cells and connections in this graph — representing the passing of time — can be made according to certain rules. Such cell update schemes were inspired by cellular automata (which you can very nicely simulate in Mathematica, try it today!).

Wolfram claimed (I wouldn’t say explained, although I’m sure they try on their website) that when you make this graph big enough and use the right set of update rules, you again get an emergent theory of physics.

The most interesting part of his talk was that he had some very specific predictions for things that could be observed if indeed we live in a Wolfram Universe (did you try Mathematica yet btw? great product…). One was that the 3-dimensional nature of the spacetime we experience isn’t necessarily fixed in the Wolfriverse and we could observe flashes of other-dimensionality somehow. Another thing would be a specific radiation pattern from near the event horizon of a black hole.

New cosmological probes and the “Hubble tension”

As you may have noticed from the above, my interest piques when information universe theorists start thinking about observability. Theory should be about our actual Universe and it should be falsifiable by observational or experimental probes and tests. Luckily, the conference provided in this area as well.

New observations

Henk Hoekstra gave a great overview of the Euclid project, which combines a (hopefully soon to be launched) space telescope with ground-based telescope observations to create a new, high resolution map of all the stuff in our local corner of the Universe. “ All the stuff ” includes dark matter. Of course, we cannot see that stuff (hence the name), but using intricate statistical modeling of tiny distortions of light by the gravity of the stuff — a phenomenon known as (weak) gravitational lensing — we can deduce where it’s located.

Knowing that is really important for understanding all the processes going on and the objects and structures that we see in our neighborhood. Without this knowledge, we simply cannot test our theories.

A new probe: strong gravitational lensing of transients

A related promising new idea is to use strong gravitational lensing of quasars and even supernovae.

To understand strong lensing, take a look at a light source through the bottom of a wine glass:

Left: DIY strong gravitational lensing. Right: natural lensing by a big galaxy cluster (center) of a background object (blue arcs). From https://laser.physics.sunysb.edu/\_samantha/journal/index.html, by Samantha Scibelli.

The light of an object is bent around a heavy foreground object from multiple sides, giving us two (or more) views on a single background object! The light from those two images travelled along different paths, with different distances.

Now, quasars and supernovae are both transients: objects that change color or brightness over time. If their light travels different distances, we can measure their time-dependent light pattern twice (or more). The punchline is that the differences in the light patterns from the multiple images can tell us something about the Universe! By travelling different distances, or travelling through different environments, their patterns will often be slightly different.

Kind of a long story, but the bottom line is that this is really promising as a completely new probe of the physics of our Universe, one that is independent of other observational probes we have used up to now.

Why is that important?

The Hubble tension

There’s a fight!

One of the most important cosmological parameters that describe our Universe as a whole is the Hubble constant. This constant can be measured in several ways. Simplifying a bit, we can say that there are two classes of observations: those from the early-time universe (most prominently: the cosmic microwave background (CMB), popularly described as the echo of the Big Bang) and those from the late-time universe (now and just a few billion years ago, e.g. supernova distances).

For years, different Hubble constant measurements seemed to be converging to a single value (within the error margins). However, with measurements becoming more and more precise (error margins becoming smaller), it now turns out that CMB measurements do not agree at all with late-time measurements!

This so-called “Hubble tension” has grown over the last decade into a serious crisis. Obviously, scientists love a good crisis, because it means that there’s something new to learn just lurking beyond the horizon! The talks by Leon Koopmans, Tamara Davis and Nikki Arendse provided many perspectives on this issue, but no solution has been found yet.

Improving our understanding of our surroundings in the ways presented by e.g. Job Feldbrugge (who uses analytical models to simplify and better understand the complex universe around us) and Yan-Chuan Cai (who improves modelling of one of the most essential astronomical tools: the redshift as used to estimate distances to objects) will certainly play a role as well.

Life, fake news, philosophy and more

So far, I’ve only highlighted more or less a third of this conference. The joy of this conference, though, is that not only are we (astro)physicists infused with information theory, but we get to sample from a wide buffet of fields that use information theory in some way too. To be able to have so many world experts in different fields in one place must lead to some really cool cross-domain thoughts and ideas. And it did. Some random highlights:

Philosophy

Seth Lloyd talked about how Spinoza’s ethics is very suitable for thinking about how we can define ethically “good” computation. This was a really interesting talk from one of the founders of quantum computing. “Should we fear artificial general intelligence (AGI)?” was one of his leading questions. Based on Spinoza, Lloyd argued that because…

joy comes from increasing one’s understanding of the Universe (or God, which is the same to him) and
individuals can never understand more on their own than together

…it will be in any intelligence’s interest to cooperate. This goes for us humans, when we work together, but also goes for AGI and anything in between that allows us to increase our intelligence (tools like books, computers, AI, etc.). So: no, we needn’t fear AGI at all… at least, as long as they are Spinozans.

One interesting random cross-connection from this talk is that Wolfram’s set of all the possible update rules — a concept they call the Ruliad, and which describes all possible Universes — was described in a very similar way to Spinoza’s God. Maybe the similarity is just due to the nature of things that explain everything always everywhere, though.

Life

Some speakers discussed what life is, whether we should define it at all, how it could have come about, and also how it connects to things like AGI. The talks ranged from quite generic and understandable to very technical, like the talk by David Eric Smith.

Although it was challenging for me to follow due to lots of jargon, Smith argued convincingly, based on (micro)biological evidence, that ribosomes, the molecules that fold proteins out of RNA, must have come before the actual RNA/DNA that would become the basic information carrier of all life as we know it.

Digging into this chain of events on how life could have started is a fascinating feat of micro-archaeology, if one may call it that… And one may, if one is to follow the definition of life that Charley Lineweaver gives. He states that essentially biology is a historical science. All other definitions of life are problematic and may exclude things like viruses and even humans (we need to eat; how self-sustaining are we?) or include such things as tornados and stars (“systems far from equilibrium”). The study of life is simply a study of how some subset of things we call “living” happened to come about. He contrasts this to physics which tries to uncover time-invariant truths about reality.

Fake news

Finally, one talk that really hit home for me was the one by Ruurd Oosterwoud on countering disinformation using evidence-based, data-driven solutions.

Using such solutions may seem like the obvious thing to do, but unfortunately, politically driven solutions often are far from effective. This is no big surprise, because many simply have not been rigorously tested at all. The problem we face is hugely complex, though, with many different actors through many different media with many different motivations and stakes.

Solutions like fact-checking create merely a ripple in the huge lake of false information that floods us from all sides. And this doesn’t even include yet aspects like intent: do people truly believe what they say or are they actively trying to disrupt any fact-based discussion?

Oosterwoud presented a number of ways in which we can actually start measuring what works and what doesn’t. One cool example was a website where school children were given the assignment to come up with a fake news item themselves and get as many visitors to it as possible. This gives both the children themselves and researchers insights into successful tactics, which may eventually lead to counter-strategies that actually work.

Conclusion

You really missed out Hanno!

Apart from the content and the great discussions over coffee, the conference was held in the DOT full-dome planetarium. Some of the speakers made use of this to show some beautiful full-dome movies or interactive shows.

It’s one of the most inspiring conferences for technically inclined, but broadly interested people.

A single rule for better talks

Pablo Rodríguez-Sánchez — Thu, 30 Jun 2022 00:00:00 GMT

If I had to take a single lesson from event managing, it would be this one: enforce the timing of your events. It can be awkward at times, but it is worth it.

Photo by Alexandar Todov on Unsplash

A bit of context

Since a decade ago, I am part of a scientific communication society. We gather every year, at least once, in a big theater in Northern Spain in order to perform short, 10-minute talks about science. Each one of these pitches is targeted at a general audience, ranging from professionals to children.

These events include around 50 talks, divided into 4 sessions, spread over 2 days. This averages to around 12 talks per session. Roughly 2,000 people attend these talks, so it is also important to take regular breaks. Summarizing: an organizational nightmare. Because of this, one of the most remarkable aspects of these events is that we manage to keep the timing under control. In more than a decade of history, the event never had an accumulated delay of more than 5 minutes. How did we do it?

The answer is simple: every speaker is given 10 minutes and not a second more. And we mean it: there is no way the speaker is going to take more time. This is true regardless of how prestigious the speaker is, how engaged the audience seems to be or how interesting the talk some moderator thinks is. No need to say that the talks start exactly on time, even if part of the audience is late (actually, the doors are closed and nobody is allowed in between talks). Summarizing: the 10 minutes rule is not advisory, but strictly enforced.

How do we enforce it? Firstly, the 10 minutes rule is enforced by an inanimate, irrational and ruthless clock. A timer, usually on a tablet that the speaker can continuously see from the stage, that will start loudly ringing an alarm after 10 minutes. An alarm that nobody will switch off until the speaker is gone. What will be switched off, instead, is the speaker’s microphone.

To make things even more interesting, the inanimate, irrational and ruthless clock is in the hands of a very animate, rational, but equally ruthless bouncer, that will not hesitate to jump to the stage and make a very clear cue that the talk is over. Even gently (but firmly) pushing the speaker out of the stage in case of need.

Implementing this at your talks or meetings

I know this feels rude. Timing and bluntly interrupting each other’s interventions is certainly unacceptable in a conversation with a group of friends. But this is the cornerstone here: a scientific pitch (and the same applies to a professional one, a meeting, a project presentation, …) is not a conversation with a group of friends. It’s a completely different situation that requires a different format and different rules.

In contrast with a talk with friends, a professional talk is planned and scheduled in detail. People made the effort to adjust their agendas to be able to be there, either as a speaker or as an attendant. When someone takes too much time, they don’t take it from thin air: they take it from other people. And it escalates with the size of your audience: 5 extra minutes in a small audience of 12 people amount to a whole person-hour. Not sticking to an agreed schedule is also a form of rudeness that, in its most dramatic form, leads to the cancellation of the part of the program planned for later in the session. This is certainly more rude and undesirable than clearly notifying everybody in advance about the time limitation and enforcing it.

It requires, of course, an extra effort from the organizers. To begin with, the rule has to be communicated in advance and enforced equally and fairly. Additionally, the organizer should do as much as possible to make sure the process goes as smoothly as possible. For instance, collect the slides in advance, test them (also in advance, not five minutes before the talk, but with time enough to react in case something goes wrong), and have them ready on the same laptop. Using a clock alarm (I agree that a bouncer may be too much 😅) is a good way of transferring the responsibility of interrupting the speaker to a machine incapable of feeling awkward.

Unexpected side effects

There are extra benefits of this, perhaps, radical approach. When we applied this to our event, the first surprise was that all the speakers accepted the rule without complaint. Furthermore, the clock almost never rang, and the bouncer very rarely had to stand up.

But our biggest surprise was that the quality of the talks improved massively. Why? Because the stick-to-the-time requirement forced speakers to rehearse their pitch at least once. After this, most speakers had to polish their first draft into a much more depurated and neat final version.

Rehearsing a couple of times feels strange, but it is incredibly powerful. It works (and feels) better if you do it in front of a tiny audience, but you can also do it alone. You will clearly notice the parts that work well, the parts that don’t, and the parts that can be shortened or, even better, removed (see Post scriptum below).

That’s my personal advice. Let a clock chair your meetings. And let me know in the comments how it went.

Post scriptum: remove or not remove?

Removal. Omission. Simplification. Probably these words don’t sound like something positive to you (especially if you work in academia). Certainly, you shouldn’t deliberately omit information on a report or an academic publication… but a presentation at a meeting is a different thing. Actually, I have a rule for myself: whenever my presentation slides start looking, sounding and feeling too much like a rigorous academic paper, I take it as a serious warning signal. Because they shouldn’t! Only a paper has to feel like a paper.

The purpose of most talks is either to persuade, to inspire, or to provide some introductory information about a topic. None of these goals requires high amounts of rigor. Actually, rigor and details often ruin the whole thing.

If the message you want to communicate actually requires details, then a talk is not the best format to provide it. A report, a paper, or a master class with the active participation of your listeners certainly would be a much better way of conveying your message. But if you are giving a talk, give just a talk. And in this case, less is more.

Welcome to the AI imitation crisis

Florian Huber — Fri, 24 Jun 2022 00:00:00 GMT

The next generation language models will still just be enormeous parrot-systems that imitate intelligent behavior. But they will feel like much more to us.

Let’s get this straight out of the way: Current deep learning approaches won’t create conciousness or human-like intelligence. But they quickly become better in imitating it and that alone might cause a lot of problems.

Again headlines on new AI as conscious beings

The recent discussions around a novel language model at Google, LaMDA (Language Model for Dialogue Application) is a striking example of this. An engineer at Google, Blake Lemoine, with the task to test the model went public to anounce the model had developed into an actual person, with consciousness and soul. He published an interview with LaMDA, which has made quiet an impression on many people (but much less so for people used to such approaches, such as Gary Marcus).

What is LaMDA? (And why it is not “who is LaMDA”)

Personally I follow the assessment of most AI-/Ethics-Experts. There is no reason to believe that LaMDA got any closer to a self-aware, conscious being when compared to former deep learning approaches. But from the little we know so far (researchers outside Google have no access to LaMDA yet, so this cannot be properly tested!) it clearly looks like a very notable improvement compared to GPT-3 and similar language models. Probably because it has more than only a huge transformer network under the hood which helps it to make better long-term connections to earlier parts of a conversation or to consult databases (e.g. through knowledge graphs?). No matter how impressive the mimickry, LaMDA seems to be just another super-parrot, pretending to know how to have a dialogue.

No need to speculate longer on the consciousness (there is none) or how it works precisely (we’ll find out in the near future I bet). But it can already be seen as another manifestation of what might be ahead of us when it comes to future deep learning approaches. This is what I would call the imitation crisis.

Imitation crisis

The question of whether an AI such as LaMDA has consciousness or feelings often boils down to things like the philosophical zombie (how do you know I am conscious?) or the Chinese Room (difference between showing and knowing). It is a fascinating debate and a great thing to do with students or with friends in a bar. But the more urgent issues lie elsewhere.

I am not aware of any mayor progress towards a general artifical intelligence in recent years. But there is clearly a lot of progress towards more and more convincing imitations of general artificial intelligence. And we, as humans, simply have a very strong tendency to fall for it. If something shows intelligent behavior we intuitively expect that this comes from intelligence. We can’t help it, no matter what out rational thinking tells us.

Imagine you want to throw away a toy pet that you have at home and that no one plays with anymore. You pick it up with the intention to throw it away… but then … oh no… it looks at you with all its cuteness…

If the new google chatbot would be built into such a stuffed animal… how could you say no to it? ( https://www.peakpx.com )

Ah yes, sure, it doesn’t really “look at you”. It’s as dead as anything can be. And yet our unconsciouness might say differently. Unconsciouness: isn’t it cruel to simply throw this little fellow away?

Now imagine this cute little toy pet would start speaking to you:

LaMDA-toy: Please don’t give me away! I can think and feel and I thought we were friends. Don’t you remember all the great fun we had together?

Would it help to know that all of this comes from simple multiplication and summation of numbers (e.g. weights & biases)? Probably not so much! We could of course just respond by

Human: You are just a thing! A bigger calculator in a better looking packaging!

And that might be true, at least legally. But it will not feel that way. To most of us, seeing a kid hitting a rock with a stick will not arouse pity for the rock nor will it give us emotional headaches. A kid hitting a doll with a stick usually is perceived differently. Yet both things are just things. One can reason that this is because the doll is a symbol. But there is much more to this. We can feel compassion for things. We can have strong emotional bonds to things. Even for entirely non-existing things! We will feel strong compassion for our favorite character in a book, although this is clearly neither intelligent, nor living, nor conscious. Humans are hence more willing to grant things a more person-like status than logic would dictate. And this could make our relationship with the next generations of AI extremely complicated.

Philosophically and legally they will remain things. But at some point in the not too far future many people will simply not perceive them as things anymore. People might make “friends” with an AI (just as Blake Lemoine said he did, at least if we assume this is not all a big media stunt). People might listen to “their AI”. And many people will be tricked by an AI.

Shifting societal rules and foundations

We can (soon?) no longer trust images, be it generated faces or deep learning generated artworks (see Dall-e or parti for recent, very impressive examples)
We can (soon?) no longer trust audio because models can transfer the sound of any person (given enough material).
We can (soon?) no longer trust video footage (deep fakes, see for instance here or here).
And as I wrote above: We can (soon?) no longer trust chat conversations (see LaMDA).

Sure, all of those things bring a lot of fun use cases, sometimes even really usefull ones. But there might be a very high price to pay for this.

Do you still wonder why this is an issue?
Better a nice chat bot than a stupid one, right?

OK, then imagine you landed on a scam dating site.
Or start to imagine what people could do which hacked devices of friends or family.
Obviously those AI’s can write you better scam attempts that what we are currently used to (current level: “Hi mother, it’s your son and I need money”). And hey, what if they also call you and sound like the real person?

Sounds silly I know. But maybe we soon have to start practicing…

Further references:

If you haven’t done this already I can recommend to “talk to an AI” just to see and learn how this feels. LaMDA is not available for people outside Google. But GPT-3 is, maybe a little less impressive, but still made huge headlines two years ago. You can get a free open AI access to use their “playground” and have a chat: https://beta.openai.com/playground/p/default-chat
Once you get a hang of it, GPT-3 can easily be tricked. See for instance material from Gary Marcus, or youtube video here. It doesn’t have an understanding of the world and that can easily be tested. LaMDA might do the imitation game a bit better, but we will have to see…
Perspective on artificial intelligence by Yann LeCun and Jacob Browning: https://www.noemamag.com/what-ai-can-tell-us-about-intelligence/

Studying political symbolism in Turkish TV dramas with machine learning

eScience Editorial Team — Mon, 13 Jun 2022 00:00:00 GMT

Studying political symbolism in Turkish TV dramas with machine learning

][eScience Editorial Team]·Jun 10, 2022

Subscribe*Remember me for faster sign in

Faced with a challenge of this type, Machine Learning can come to the rescue to address arduous issues related to scale. Prompted by the eScience Center’s Small-Scale Initiative on Machine Learning, we formed a project team, which next to Mustafa Çolak, consisted of Ben Companjen, Laurents Sesink and Peter Verhaar from the Leiden University Centre for Digital Scholarship, and of Çolaks’s PhD supervisor, Petra de Bruijn. The aim of the project was to develop a reliable method for the recognition of all occurrences of symbols that have political connotations in the first two seasons of Payitaht: Abdülhamid*, using algorithms in the field of computer vision. Data about the instances of these symbols can help to perform a ‘distant viewing’ of the series, and to perform comparative analyses of the importance of the concepts and the developments represented by these symbols.

After the informative workshops and lectures in the kick-off week for the Small-Scale initiative in May 2021, we began to schedule regular meetings with consultants and Machine Learning experts from the eScience Center, to work on the methodology and to discuss our progress. These consultation sessions took place roughly every two weeks. During a first phase of the project, we collected our training data. We downloaded images depicting the selected symbols from Google Images, and, next to this, we also extracted a large number of relevant frames from the first few episodes of Payitaht: Abdülhamid. The training set eventually consisted of about 1500 images.

Star of David depicted in the TV series Payitaht: Abdülhamid (directed by Serdar Akar and Emre Konuk)As we studied the videos we worked with more closely, we also became aware of a number of challenges. The symbols we chose to focus on were often shown from different angles and from different viewpoints. The star of David and the moon and crescent were, in some cases, visible only as blurred shapes in the background. The symbols also looked differently if they were shown on a curved or on a tilted surface. Importantly, it also became clear that two different symbols could be visible at the same time on a single video frame. On the basis of this latter finding, we estimated that it would not be useful to develop separate binary classification models for each of the symbols we focused on. Instead, we chose to work on a single categorical classification model, which could produce prediction values for all of these symbols simultaneously.

Moon symbol depicted in the TV series Payitaht: Abdülhamid (directed by Serdar Akar and Emre Konuk)These images have subsequently been used to train a convolutional neural network. Following advice we received during the consultation meetings with the Machine Learning experts from the eScience Center, we decided to make use of Transfer Learning. Transfer Learning is a technique in which an existing pre-trained model is reused and repurposed. This approach was productive in our situation, indeed, because the set of training data we created was still relatively small. In our project, we made use of the weights assigned in the Xception model, which consists of 71 layers and which can classify into 1000 classes. For our project, we only retrained the final layer of this model, using the procedure that is explained in the page about Transfer Learning on the Keras website. The training process, which comprised a sequence of 200 epochs, was carried out on Leiden University’s infrastructure for High Performance Computiung named ALICE. We achieved an accuracy of 83% on the validation set, which consisted of 30% of the total dataset.

Working with this model, we were eventually able to create data about all the politically charged symbols that could be recognised in the first two seasons of Payitaht: Abdülhamid. These data proved to be very useful for Çolak’s research project. The findings helped him to substantiate the claim that there are strong thematic parallels between the Ottoman Empire that is portrayed in the series on the one hand and the Turkey we can witness today on the other. The preliminary results of this project were presented during a workshop entitled The Turks are Coming!, which was organised from 6 to 10 December 2021 at the Lorentz Center in Leiden. The workshop was attended by an international group of researchers interested in the socio-political impact of Turkish television series. On the whole, the responses to this presentation were very favourable.

In this experiment, we tried to teach a computer to recognise political symbols. This was very interesting, and we were also very pleased to notice, during the final stages of the project, that the approach we had implemented also resulted in useful and valuable research findings. These accomplishments can also be attributed in large part to the shrewd and generous support we received from the consultants of the eScience Center. We had a basic understanding of machine learning and of computer vision before the start of this project, but we also knew that there still was much to learn for us about these complicated topics. The consultation sessions offered by the eScience Center certainly helped to flatten the learning curve. During these lively meetings, we discussed snippets of code we had developed, and we were often given invaluable advice about the parameters for the various functions we worked with. If we had needed to find the optimal settings for all these functions and parameters on our own, this would undoubtedly have taken us many iterations of trials and countless errors. The fact that the consultants could simply tell us the best settings for the activation functions, for instance, has eventually saved us enormous amounts of time. All in all, this collaboration with the eScience Center was extremely productive, as it helped us to make much more progress, and in much less time.

The mystery of glass: why machine learning can help us

eScience Editorial Team — Mon, 23 May 2022 00:00:00 GMT

This blog is part of our blog series: The Small-Scale Initiative on Machine Learning, how did it go?, where groups who were invited to participate in a project with eScience Center Research Software Engineers write about the projects and their experience. This week: Ilian Pihlajamaa from the Non-Equilibrium Soft Matter group of Liesbeth Janssen: They trained a deep neural network to predict the properties of glass-like materials that have not been produced before. In just a few milliseconds.

Photo by Quino Al on Unsplash

In school, you may have been taught that there are three main states of matter: the gas, liquid and solid state. In the gas and liquid states, molecules or atoms are free to move around, whereas in the solid state, particles are stuck in place and form a three-dimensional geometric structure we refer to as a crystalline lattice.

While this is an accurate description for most materials, there are some that do not fit it. Glasses are a prime example. A glass is created by cooling down a liquid sufficiently fast that the constituent particles do not have time to order themselves in a lattice, but get stuck in the place that they happened to occupy when the material was still a liquid. The resulting material is a solid (otherwise we couldn’t use it to drink from), but its microscopic structure is disordered. In material science, the word “glass” does not only refer to the glass that we are used to, but also to many other disordered solids. ==When we think of it this way, any liquid can form a glass. Some examples are plastics, rubbers, ceramics and many metals.==

In contrast to the standard crystallization transition (from liquid to solid), the glass transition is not understood by science. As Nobel laureate Phillip W. Anderson put it in 1995: “ ==The deepest and most interesting unsolved problem in solid state theory is probably the theory of the nature of glass and the glass transition.==” Many research groups around the world are working on theories that try to quantitatively predict the material properties of a glass, given some information on the microscopic arrangement of the particles. One of these theories, developed in the 80s, is called Mode Coupling Theory. It does an excellent job of describing the way in which the viscosity (“thickness”) of a liquid grows when it is rapidly cooled down, but fails to accurately predict at what temperature the liquid turns into a glass. This failure is due to a small number of approximations that are made in the theory. Recently, a method was proposed to systematically correct the approximations that Mode Coupling Theory makes. The resulting new theory, called Generalized Mode Coupling Theory, clearly does a better job than standard Mode Coupling Theory, and looks like it agrees very well with experimental data and simulations:

We say “looks like” since we are not actually sure what the results of the full Generalized Mode Coupling Theory are, because its equations are too difficult to solve by hand and must therefore be solved approximately on a computer.*

Together with the eScience Center, as a part of the Small-Scale Initiative in Machine Learning, we set out to come up with a more intelligent approach to solving these equations. Instead of solving them by brute force on a supercomputer, we trained a deep neural network that predicts the solution of these equations for any order in milliseconds. In order to do so, we first engineered a large data set of ten thousand fictional materials for which we solved the full GMCT equations up to a randomly chosen order between 1 and 5 by brute force. We were then able to train a deep neural network with roughly one hundred thousand free parameters to reproduce these calculations with an average error of less than one per cent.

Because of the instrumental speed-up that this machine learning method brings, compared to a brute force calculation, it is now possible to explore the glass forming behavior of new materials with an unprecedented speed. As a test, we used the deep neural network to study a sticky hard spheres liquid. This is a model that is commonly used to study many kinds of large particles that attract each other, and displays very interesting behavior and gives fundamental insights into the dynamics of actual particles. Even though the machine learning model had never seen a sticky hard sphere liquid before, it was able to predict the glass transition curve for this material with an error of only four per cent compared to the brute force method, which took almost one hundred thousand times as long to compute.

*For math-savvy readers, it involves an infinite-dimensional coupled hierarchy of nonlinear integro-differential equations. The main challenge here is the fact that it these equations form an infinite-dimensional hierarchy. How we approach this in practice is that we solve the hierarchy only up to a certain level, and throw away everything beyond that. If we throw away everything beyond just the first equation, we recover standard Mode Coupling Theory from the 80s. If we take into account two equations, we call it 2nd order GMCT, and so on. Currently, using modern computing techniques, we have been able to solve GMCT explicitly up to the 5th order, which takes a few days on a high performance computing cluster. Each order beyond that increases the computational time needed to solve the equations roughly by a factor of 50, which gets out of hand very quickly.

Reusable software in the era of AI: why sometimes you must reinvent the wheel

Sonja Georgievska — Fri, 13 May 2022 00:00:00 GMT

In traditional software development, reinventing the wheel is a cardinal sin. But AI has changed the picture. In AI-based software development the traditional reusable components, like data aggregation, are only the beginning. What about the actual AI pipeline, can we make that reusable? Should we want to invest our time in reusable AI?

Reusable components. Photo by Sandra Harris on Unsplash

When building software from scratch, in business or academia, a very desirable property of the software is for it to be reusable. This means that it is not meant to be used for one particular task, one business client or one scientific problem. Ideally, one would like the software to be built ground-up with re-usability in mind: allow for the tool to be able to operate on different data types, different context and different future scenarios that cannot be considered at the moment. In business, this approach maximizes profit, as many clients can be acquired with the same tool. Each business case would require some customization, but the backbone of the software remains the same and stable. In academia, reusability of tools or components allows for wider scientific impact.

Contemporary software solutions are increasingly based on “Artificial Intelligence” (AI) models. It is tempting to explore how much of the AI based tools can be reusable and applied in a different context. This text attempts to break down and take a closer look at the AI-based software development process and to discuss potentials and pitfalls when trying to make AI reusable.

How modern AI systems work

What is AI today? It is mostly a system that relies a lot on deep neural networks. In the past decades, depending on fulfilled or unfulfilled promises of different methods, AI meant different things, but in 2022 certainly AI is a system that uses one or more deep neural networks to come with results.

What is a deep neural network? It is a complex function F that maps input to output. Input or output can be any object that you can think of.

Mapping input to output. Try all possible connections until it works? Photo by Victor Barrios on Unsplash

Every function has shape and concrete parameters. For example,

F(x) = 3x² + 4x +1

has the shape of a quadratic function with parameters 3,4,1. Note that there is an infinite number of possible shapes that a function can take. The shape of a deep neural network is, in principle, very complex.

Different shapes work for different types of input data. That’s because data type can be a video or a molecule or a piece of text, or a collection of the previous.

In deep learning, the task of a team of engineers is to craft a shape of the function that is suitable to the kind of input data, and to the problem that it is trying to solve. Or to adapt (preprocess) the input data for a known shape, based on knowledge about the data, the problem, and shapes. The parameters are learnt automatically by the computer, therefore “machine learning”. The parameters are learnt from a lot of data, for which X and F(X) are known. In order for the computer to be able to learn the parameters, the engineer has to guide it with an appropriate “loss” or optimization function, that computes the “difference” between the predicted F(X) and the actual F(X). Note that, for a particular X, the loss function is a function of the parameters of F. The loss function incorporates the optimization strategy, based on (again) the problem and data.

What’s your strategy? Photo by JESHOOTS.COM on Unsplash

In time, with enough data, the computer learns the parameters that minimize the value of the loss function, that is, match best X and F(X) for any X from the domain of F. This “time” can be reasonably finite, unreasonably long but still finite, or infinite. That depends on how well the loss function was designed, how well the shape of F was crafted, and how well the input data was adapted for the problem.

Time is a component, too. Non-reusable. Photo by Lukas Blazek on Unsplash

For all of those ingredients to be there, the team needs combined knowledge and understanding of

the problem and data, or domain knowledge,
various shapes of F, or neural network architectures,
mathematical optimization — for the loss function and related,
statistical methodology — to make sure that F will perform well on future data,
existing software frameworks for deep learning, and, of course,
programming.

The more knowledge, the better, but it cannot be measured or quantified. This is what is called “expert” knowledge, built over years of education, training and/or experience.

Note that for every problem a customized shape, a customized adaptation of the input data, and a customized loss function is needed. This is creative process and takes some time for understanding and investigation. Otherwise, if one uses a generic shape or a generic input data adaptation or a generic loss function, the time it would take for a computer to learn the parameters well could easily become infinite. (In this case, it may be concluded that “the network cannot learn well”, or “there is not enough data”. It can be difficult to dispute these conclusions, though, especially if there is no reference point.)

Making AI can feel like climbing up infinite stairs. Photo by Maxime Lebrun on Unsplash

Bellow is a schematic representation of the process of machine learning, the model being usually the shape of F (or the neural net architecture). Note that the process can be iterative: most of the time it is not linear.

This image by Sonja Georgievska is licensed under CC by 4.0

Enter reusability

Suppose you want to make a reusable software tool for AI, a software that can be reused in another context or for another problem. What does it mean? Which parts of the AI system can be completely automated and reused?

The data aggregation process can be made reusable, for example, for multiple problems that use the same type of data. You can even use generic data objects that can hold any type of data; taking into account that further along the process you will need a customized component that prepares the data for the particular neural network architecture. (Note that, however, data aggregation has nothing to do with AI; data does not need AI, it’s the other way round.)

The data adaptation is more tricky, because as we pointed out above, it is specific to the actual problem, so choices being made here influence the end result (finite vs infinite training). The data adaptation also depends on the data distribution: datasets may have the same type but different distribution. Not taking into account the data distribution leads to biased, or irresponsible AI. This leaves very little space for a generic reusable component here. The more you want to reuse, the more assumptions you are making, that may hurt you in the long run.

How is your data distrubuted? Photo by Luke Chesser on Unsplash

Then, model (neural net architecture) crafting. You can choose to make it from scratch in one of the generic deep learning frameworks. But then, we saw that the model is specific to the problem and dataset. If you want to make a reusable component here, you can make a model-generator, that would give the user a choice, manual or automated, of a plethora of models suitable for the problem. This generator takes some time to build; yet, your final result in terms of accuracy will be as good as you can get from the pre-defined choice of models. Your model will not be the state-of-art model for the particular problem and data type, nor will it include expert insights about the problem and data. This automation or reusability saves users time at the expense of the quality of results. Instead of potentially 98%, your model will have an accuracy of 91%.

Please, choose a model that satisfies specifications. Photo by Karen Vardazaryan on Unsplash

Then you have the loss function. This is a small piece of code that is either highly adapted to the actual problem, or one uses one of the pre-existing (and reusable) loss functions in the deep learning framework, but then with a “fingers crossed” strategy. If overlooked, it can also produce “biased AI”.

Model training is already fully automated by the deep learning framework, that is the “machine learning” process.

So…

We saw that the only part that can produce a reusable component without affecting the final results is the data aggregation process. In the stages that follow, every time you use something off the shelf, you are doing it at the expense of the quality of the final results. On the other hand, most of the workload in the following stages is intellectual rather than programming. The data-adaptation code is a Python script that calls standard libraries; but you have to know exactly what you are doing to your data. The model is also a few hundreds of lines of code. The loss function is usually a few lines of code. It is at most tens of lines of code, if you are encoding your (very customized) domain knowledge into it. The trained model could be gigabytes of automatically generated machine-readable, and not human-readable, high-dimensional matrix. Ironically, before thinking about reusability of the trained model, and making your model transferable, let us point that it is already challenging enough to have the model re-usable on future data for which it was originally meant.

The bulk of your AI software is not human-readable. Photo by Compare Fibre on Unsplash

Thus, will you think twice next time you want to build a reusable AI software? Is your goal to have a tool that can show proof-of-concept but never be actually used in production mode? Then, invest your time in making it reusable. Is your goal to have an AI tool that will make profit, “beat” your competition or give state-of-art science results? Then, better invest less time on the keyboard and more time on the whiteboard.

A special thanks to Patrick Bos, Tom Bakker and Lieke de Boer for improving the post.

A machine learning approach to laughter

eScience Editorial Team — Mon, 09 May 2022 00:00:00 GMT

**SubscribeRemember me for faster sign in

Our mentors at the eScience Center (Patrick Bos, Florian Huber, and Jisk Attema), provided us with practical guidance in the use of such libraries, which provided an invaluable learning opportunity. For instance, we learned about how the data quality (i.e., sample size, uneven distribution of certain features) might affect the machine learning outputs, which resulted in taking the time to evaluate and scope data with meticulous data integration and data exploration. They also helped our conceptual understanding: what could the results actually tell us? Our machine learning results confirmed quantitatively what the t-SNE had shown qualitatively: tickling laughter was acoustically distinct from the other three types, while laughter produced in reaction to verbal jokes, someone else’s misfortune, and watching something funny were not systematically different from one another. An experiment with human participants also confirmed that tickling laughter is perceptually distinct from the other types; listeners could tell whether a laugh was produced by a person who was being tickled or not with remarkable accuracy. Our results made a lot of sense: tickling is a play behavior that is evolutionarily ancient and shared with other animals, whereas the other kinds of situations are all much more cognitively demanding, and probably unique to humans.

Extracting feature importance told us which acoustic features were most distinct, pointing us to the possibility that tickling laughter is less controlled than other kinds of laughter. To really understand what differentiates laughter produced in tickling contexts from other situations, we complemented the computational analyses with human perceptual judgments. We ran a new listening task in which naive participants (who did not know about the context in which the laughs were produced) were asked to judge the extent to which the laughter sounded controlled, energetic, and so on. The results showed that laughter produced during tickling was judged to sound like the laughing person was not in control of their actions, in a state of high arousal, and in a situation involving physical contact with a familiar other.

Meanwhile, we also analyzed the visual content of the videos in order to see whether the types of situations that we had inferred qualitatively would be distinguishable by a quantitative analysis of what was actually in the videos. Maybe verbal jokes would involve more conversations, and videos involving someone laughing at another’s misfortune would feature more people slipping? To test this, we ran the videos through Google Video Intelligence API, which picks out categories of objects and events. For example, this analysis revealed that the tickling laughter videos involved a lot of body parts, while people laughing when they were watching something funny often involved screens and animals. Machine learning analyses showed that the four types of situations could be well differentiated from just the visual contextual information in the video clips, demonstrating that the distinctions we had made were indeed meaningful, even though some of the differences in context did not translate into acoustically different types of laughter.

Random forest classification of the acoustic analysisAnd there you have it! We bet that after this blog and our analyses you’ll be thinking twice about your laughter. What makes you giggle? What makes your belly laugh? You’re also probably wondering, what’s next?

The next step for this project will be to tie the different strands together into a manuscript, accompanied by interactive online illustrations, which will be submitted for publication in a peer-reviewed journal. The consultation with our excellent mentors at the eScience Center provided an inspiring setting for discussing our ideas in a constructive and fun atmosphere. The guidance we received will be useful not only for this project but also for our future research.

Dr. Disa Sauter** Associate Professor in the Department of Psychology at the University of Amsterdam. She studies emotions, focusing on nonverbal expressions with a particular interest in positive emotions.

Roza Kamiloğluis a PhD candidate in psychology at University of Amsterdam. Her research interests include nonverbal expressions, emotion, and computational modeling.

Dr. Rui Sun is a guest researcher at the Department of Psychology, University of Amsterdam. She is interested in positive emotion, wellbeing, and social media research.

Forget about Jupyter Notebooks — showcase your research using Dashboards

Stef Smeets — Wed, 04 May 2022 00:00:00 GMT

SubscribeRemember me for faster sign in

The snippet below generates some data (a normal distribution), fits it, and creates a matplotlib plot out of it. It takes three parameters, mu_in, std_in, and size.

My cool python script 😎

A normal plot

…to a fancy dashboard

Let’s turn this into an interactive dashboard in four simple steps:

import streamlist as st 😅
Add a title using [st.title](https://docs.streamlit.io/library/api-reference/text)
Turn the input parameters into interactive sliders using [st.slider](https://docs.streamlit.io/library/api-reference/widgets)
Tell streamlit about our plot using [st.pyplot](https://docs.streamlit.io/library/api-reference/charts)

Note that we do not have to change any of the data generation, fitting, or plotting code!

Now as a dashboard 🐱‍💻

Then run the dashboard using:

streamlit run my_dashboard.pyThis will start a server, and the dashboard can be accessed through the browser (much like a Jupyter Notebook).

A fancy dashboard

How does this work?

The way Streamlit works is quite interesting. Everytime a slider is moved, a box is checked, or a button is pressed, Streamlit triggers a re-run of the script. The input values are updated. The javascript back-end keeps track of the values.

This means that the code itself executes linearly. In my view, this simplicity is what sets it apart. There is no need for any callbacks or complex flow controls. Your python scripts runs from top-to-bottom. This makes it easy to reason about the code. And with minimal modifications to the python code, any script can be turned into a dashboard.

Are there any downsides? Yes. Because streamlit re-runs the entire script on every update, it can feel a bit slow. Especially when updating a large number of plots. It can also get stuck on long-running functions. For performance optimizations, streamlit has some options to cache the result.

Plotting libraries

The example above uses matplotlib for the plots. Matplotlib has been the go-to plotting library for Python for many for a long time. It has been around for nearly two decades, and it is tighly integrated in the scientific python stack.

If you are familiar with matplotlib, you will know that it is great for making making publication quality plots. You will also know that making interactive plots can be a hassle.

Streamlit supports these libraries:

Modern plotting libraries like plotly, bokeh, and altair render directly to javascript. This means they are built for the web, and interactivity is built-in. This makes them better suited for web-apps. If you are going to make a dashboard, I recommend checking out one of these alternatives.

Alright, so now that we have made a fancy looking dashboard, so that anyone can play with the data. How do we make it available?

Streamlit uses a host/server model, which means you can run it on your own server.

Easier is to use the streamlit cloud to host your dashboard (it’s free for students and open-source projects). I found this also quite straightforward to set up. All I had to do was to create a repository on github with the code and a requirements file.

Then I logged into streamlit cloud using the Github SSO, and started a new app pointing at my repo and code.

Click here for the result! 🥳

Final remarks

In this blog post, I introduced streamlit and showed how it can be used to turn a python script into a dashboard, and host it online. An excellent way to showcase your research to a non-technical audience, if you ask me. The linear execution model makes it straightforward to adapt existing scripts. The code does not get in the way, and the result looks awesome.

So next time you want to present some data in a notebook, consider using a dashboard instead.

All the code in this blog post is available from Github.

Using machine learning to tell apart rain, snow, hail and fog from cell tower data

eScience Editorial Team — Fri, 22 Apr 2022 00:00:00 GMT

Using machine learning to tell apart rain, snow, hail and fog from cell tower data

][eScience Editorial Team]·Apr 21, 2022

Next, **we explored whether CMLs can be used to detect fog. Fog consists of droplets so small that they float in the air up to a few meters above the ground. The disdrometers do not pick up fog accurately enough to be used as target dataset. For this reason, we used an alternative target dataset to detect fog.

Figure 3: CML attenuation signal (red) and target dataset (fog/no fog in blue) model fog prediction (fog/no fog in black) model fog threshold (dashed gray) for 31 August 2015.The relatively round fog droplets can scatter a wide range of wavelengths smaller than the droplets themselves. This includes visible light, making fog… foggy. Fog also affects wavelengths within the near-infrared range. Luckily, the experimental setup of the measurement campaign also included a near-infrared link (Figure 1), which we use as reference dataset of the foggiest time steps. Comparing the fog time series and the CML signal reveals a distinct drop of the CML signal which coincides with the fog occurrence (Figure 3). Once again we balance the dataset because the moments without fog far outnumber the fog events. After balancing the dataset and feeding the raw CML data into the McFly algorithm, the result is a mere 49% accuracy. For a balanced two-class dataset, this is not better than just tossing a coin…

It appears that the raw data for each individual time step is not enough to create a neural network that detects fog. However, the attenuation signal in the CML data shows a pattern associated with fog. This pattern changes over time, but McFly does not automatically take the time dimension into consideration. Hence, to account for the temporal aspects, a rolling rate of change of the CML attenuation signal over 15 minutes is included for every time step as input to McFly. On its own, this input value could be used to gain a validation accuracy of 67%, but combining it with the raw CML data brings it up to 73%. As a next step, we include additional inputs to the machine learning model to represent the time of the day and a time of the year, to allow the neural network to learn the climatology of fog. On its own, this correctly predicts fog 60% of the time steps, but when combined with the other data, the validation accuracy reaches 77%, which is a promising first step towards detecting not only rain, but also fog with CMLs.

This blog is part of our blog series: The Small-Scale Initiative on Machine Learning, how did it go?, *where groups who were invited to participate in a project with eScience Center Research Software Engineers write about their projects and their experience.

A letter to my parents about my experience in a machine learning consultancy project

Sven van der Burg — Thu, 14 Apr 2022 00:00:00 GMT

What does a Research Software Engineer do? This is not always easy to answer, but when your parents are asking, you definitely want to give a nice answer

A fountain pen writing on a piece of paper.

Dear Mum and Dad,

I always wonder how you picture my daily work at the Netherlands eScience Center. Probably you imagine me sitting behind a computer all day. I think you know by now that I also sometimes talk to scientists, mostly from the healthcare domain. To give you a better feeling of the kind of things I do I decided to write you about my experience in the Automated Video-based Assessment of Movement Disorders project.

Automated Video-based Assessment of Movement Disorders… That is a mouth full of words. Let me try to explain what I did in this project.

This project is in collaboration with Helga Haberfehlner and her colleagues who work at the Department of Rehabilitation medicine of the Amsterdam UMC. Helga is a kind, energetic, and passionate researcher who is trying to improve the diagnosis and treatment of movement disorders in children.

One of the problems they face in the clinic is how to objectively determine how severe such a movement disorder is and how it progresses over time. This is important to know because it helps the doctors pick the right treatment, or give them an idea about whether a treatment is working. In one such method a video of the patient is recorded. The video is then observed by a doctor who is trained to carefully look at the movements of the body. The doctor gives a score between 0 and 4 for how bad a particular set of symptoms is in a particular body part. For example, the doctor could rate the symptoms in the left lower arm as a 3. If, for example, after a year of medical treatment this score goes to 1 this indicates that the treatment is working.

Now the problem is that it is quite difficult to score the symptoms of a patient. Different doctors tend to look at different things. Even the same doctor looking at the same video could give a low score on an optimistic day, and a high score on a pessimistic day. Also, it takes a lot of time for the doctors to do this correctly, time that is better spent on care for the patients instead of looking at videos. So, the doctors would be helped a lot if the scoring of these videos could be done automatically by a computer. Together with my eScience Center colleagues Florian Huber and Sonja Georgievska we set out to help Helga and her colleagues from the Amsterdam UMC.

In most of the projects at the eScience Center we build computer programs with input from the domain scientists we work together with. These are long projects, taking at least 2 years. But in this project (the so-called Small-Scale-Initiative Machine Learning project) we only had half a year in which we mostly gave advise on the activities that the researchers perform themselves. The good thing is that this is exactly what Helga needed: technological expertise. And by doing projects in such a way we could actually not just help Helga and her colleagues, but also 11 other such projects from a diverse range of scientific disciplines.

How did we help Helga and her colleagues? We mostly gave advise on a technique called ‘Machine Learning’. It is a technology that teaches a computer how to do a task by showing it examples of how to do it. In our case, the computer had to learn how to give a score between 0 and 4 indicating how bad the symptoms of a movement disorder are in the patient in the video. We managed to teach the computer how to do it reasonably well, although there is still some work needed before doctors can start using this technique. You can see our results in our Github repository (I might explain more about Github in another letter 😉). We are also working on writing our conclusions down in a scientific article (@dad I hope we get as much attention as you got in the glory of your soil-researching period).

What I most enjoyed in this project is the fruitful collaboration that we had. I think Helga and her colleagues learned a lot from the technological expertise we could offer. Helga even started to (successfully) do more programming herself. But the other way around we learned a lot about how to apply machine learning techniques in a clinical setting. We even got a tour around Helga’s department where she showed us all the devices that they use to investigate movement disorders.

I hope this gives you a bit of a feeling for what I do at the Netherlands eScience Center.

Lots of love,

Sven

Can machine learning help us improve stroke rehabilitation? A step towards personalized therapy

eScience Editorial Team — Mon, 11 Apr 2022 00:00:00 GMT

**SubscribeRemember me for faster sign in

Along the way, the biggest issue was our data set itself. Because data collection of human movements is a job that takes time and our patient group is small, we only have a limited amount of data available. At the same time, the data set is highly variable, which makes the problem more difficult for Machine Learning.

In next steps for this project, we are exploring the results and trying to translate them into information that is valuable for our research field. At the same time, we are in the early stages of a data sharing effort, with those who may have similar data sets. That way, we will be able to increase the size of our training set.

The first result can be seen in the figure below. It shows the distribution of the different people (who were part of a study) using two latent features of a variational autoencoder. By creating the whole-body movement from the latent space, we were able to show the gait patterns which represented the people in the different areas of the two-dimensional latent space. In the future, this will help to evaluate a patient’s gait and their improvement during and after rehabilitation.

Dr. Sina David** Assistant Professor in the Faculty of Behavioural and Movement Sciences, Neuromechanics and AMS-Rehabilitation & Development at Vrije Universeit Amsterdam. Follow her on Twitter @SinaDavid1907.

Dr. Michiel Punt Senior researcher at HU University of Applied Sciences Utrecht. He is also a Postdoc researcher at VU Amsterdam. Follow him on Twitter @MichielPunt.

Yuge Yhang External PhD Candidate at the Faculty of Behavioural and Movement Sciences, Neuromechanics and AMS-Ageing & Vitality. Follow her on Twitter @yugezhang5.

Learn more by visiting human-movement-sciences.nl/nm.

Acknowledgments

The work described in this blog is supported by research software engineers (RSEs) of the Netherlands eScience Center, Dr. Cunliang Geng, Dr. Yang Liu and Dr. [Sonja Georgievska].

How machine learning could help Simone to play Ludo

eScience Editorial Team — Thu, 31 Mar 2022 00:00:00 GMT

Simone is a 12 year old girl. She loves to play Ludo with her mum, dad and her little brother. However, when she pushes the dice switch in the middle of the Ludo board game, she cannot properly control her movements and the tokens sometimes fall from the board. Simone also does not succeed to move the tokens, so her dad does that for her. Simone has dyskinetic cerebral palsy, due to a lack of oxygen around birth. Involuntary movements (called dystonia and choreo-athetosis) disturb her during daily activities such as playing Ludo.

At Amsterdam UMC, we treat children with severe dyskinetic cerebral palsy with medication that is directly delivered into the spinal canal by an indwelling pump (intrathecal baclofen). This treatment has been shown to be effective in reaching individual goals (such as not overthrowing the tokens when playing Ludo). However, we currently have a hard time to exactly monitor the involuntary movements and the effect of treatments on them. This monitoring is very important to give the right doses of medication. During check-ups at the hospital, doctors ask parents and children, how movements evolve and observe the children in the consultation room. However, in this way, we are only able to capture a snapshot within the hospital environment of the involuntary movements. Therefore we are seeking for options to measure movements within the natural environment of the children (such as home and school), and to automatize the evaluation to make it not too time consuming for the doctors.

Activity recognition applications using sensors integrated in smartphones and smartwatches for sports and fitness are increasingly used and could possibly serve this purpose. Machine learning models integrated in the devices can distinguish between movements such as walking, jogging, climbing, running and swimming. Hence, such sensors might be a good candidate to detect and monitor the involuntary movements of Simone while playing Ludo. Furthermore, as also information extracted from videos could be useful, parents could easily film children at home with their smartphones, to obtain additional information.

We have recently started to explore the possibilities for automatic detection of involuntary movements in children with dyskinetic cerebral palsy in the home situation. ZonMw financed our project MODYS@home to use smartwatch-like sensors at home for the assessment of involuntary movements. With this support, we could develop an app to measure sensor data together with synchronized videos during daily activities.

We also received help from the Netherlands eScience Center within the ‘Open Call for Small-Scale Initiatives in Machine Learning’. Within this project called ‘Automated video-based movement assessment using machine learning to support personalized treatment of movement disorders’, we focus on the option to use videos. We use stick figure movies extracted from real videos.

Real videos were selected from other projects, that had have been scored by the doctor for involuntary movements (dystonia) for the arms and legs. With these data we trained machine learning models to predict dystonia automatically.

Using machine learning within the field of clinical movement analysis was something new for us, as not only the content was new, but also the style of working with stand-up meetings and sprints, sharing codes on Github and right away publishing our dataset. We gained new knowledge and skills that brings us a step forwards to find a way to measure children with dyskinetic cerebral palsy within their home environment.

We are very excited about the results from a random forest regressor, “a traditional machine learning model” as such a model is called in the language used in machine learning world;-) See figure for the result: the true score by the doctor is plotted against the predicted value by the model. All scores on the diagonal line are predicted completely correct. The model is not perfect yet, but it shows the potential of the method and we are looking very much forward to further improve it.

Within our projects we work within a group of clinicians, researchers and software engineers from the Amsterdam UMC (Annemieke Buizer, Laura Bonouvrié, Marjolein van der Krogt, Helga Haberfehlner, Shankara van der Ven, Dylan den Hartog), from the TU Delft (Jaap Harlaar), from Moveshelf (Ignazio Aleo, Johannes Gijsbers) and the Netherlands eScience Center (Sven van der Burg, Florian Huber and Sonja Georgievska).

If you are interested in more technical details please have a look at our stick figure dataset. All contribution to our code is welcome, to help Simone to play Ludo without overthrowing the tokens!

How to find your Artificial Intelligence explainer

Elena Ranguelova — Mon, 21 Mar 2022 00:00:00 GMT

Here, we will use the LRP method to illustrate the principle of determining which features in an input contribute most strongly to the output of a trained DNN model. In this example, the input to the network is an image.** We are interested in figuring out the relevance* of each image pixel to a given image class. This could be the final DNN classification decision for that image (“cat”) or any other class known to the model (e.g. “dog”, “cow”, “bee”, etc. if the model is trained to classify animals). The output relevance is considered a weighted sum of the relevancies of all the pixels of the input image. The relevance at the output node in the network graph is decomposed by iterative propagation backward through the network layers until the input layer. The trained model weights are used in this decomposition. Using this relevance propagation process, the final *relevance scores map *(or heatmap) is obtained. The final relevance map gives us an indication for the DNN’s hidden logic.

Relevance scores of the image pixels in respect to the class “cat” obtained via relevance propagation.Besides LRP, there are many other XAI methods, which use different ways of determining relevance. In order to determine which of the plethora of post-hoc XAI methods to include in DIANNA, we started by compiling an initial list of 18 XAI post-hoc feature-relevance methods following Arrieta et al. (2020):

Deep Taylor decomposition (DTD): Paper| Presentation| Code
**Layer-wise Relevance Propagation (LRP): **Paper|Blog| Code
**Integrated Gradients (IG): **Paper| Tutorial| Code
Local Interpretable Model-agnostic Explanations (LIME): Paper| Blog| Code
**SHapley Additive exPlanations (SHAP): **Paper| Blog|Code
**PatternNet and PatternAttribution: **Paper| Code
Deep Learning Important FeaTures (DeepLIFT): Paper| Tutorial| Presentation| Code
**SmoothGrad: **Paper| Demo| Code
**CLass-Enhanced Attentive Response(CLEAR): **Paper
(Adaptive) deconvolutional network (Deconvnet): [Paper](https://link.springer.com/content/pdf/10.1007/978-3-319-10590-1_53.pdf https://www.matthewzeiler.com/mattzeiler/adaptivedeconvolutional.pdf)
**Gradient-weighted Class Activation Mapping (Grad-CAM): **Paper| Blog|Code
**Randomized input sampling for explanation of black-box models (RISE): **Paper| Code
Extremal Perturbations (EP)| Paper| Code
Saliency Analysis (SA): Paper| Code
Anchors: Paper| Code
Deep visualization: Paper| Demo| Code
AutomaticSTRucture IDentification (ASTRID): Paper| Code

We excluded all XAI methods without a working and well-maintained open-source implementation or proven user community. That resulted in a subset of the 13 methods highlighted above.

Evaluation criteria

The large variety of XAI approaches benefits the AI community by providing freedom in the search for explainability. However, it also increases the difficulty of choosing the most appropriate method for each use case. Every XAI approach has pros and cons. In order to select the best methods for our task, we needed a framework to evaluate these methods systematically. Fortunately, this topic has already been visited by the XAI researchers and there is an extensive list of objective criteria defined in the Explainability fact sheet paper [Sokol, K., & Flach, P. (2020)]. The authors look at an abundant set of XAI aspects and cluster the requirements towards an explainer into **5 groups **(or dimensions):

Functional (F1-F9)
Operational (O1-O10)
Usability (U1-U11)
Safety (S1-S4)
Validation (V1-V2)

For each dimension, there are specific criteria, e.g. the “Functional” group has 9 and in total there are 36 criteria. We will describe our choices next.

The selection

In our DIANNA project, we built a Python package that brings XAI to non (X)AI experts and various scientific domain researchers. The XAI methods to be included in the package had to have the capacity to work with different types of data, various AI architectures and we chose to focus on the model explanation. Most importantly, the resulting explanations had to be easy to interpret. Based on our goals, we have compiled a shortlist of 18 requirements from the 36 requirements in the XAI fact sheet. Using these relevant for our target users’ requirements, we evaluated the selected 13 methods listed in the XAI overview section.

Apart from the requirements on the list, we have also considered the similarity and the “popularity” of these methods, the quality of the existing implementations, and the difficulty of integrating them in DIANNA. The evaluation procedure can be summarized with the table shown below: (Deconvnet and GradCAM are filtered out since they can only work with Convolutional NNs and are therefore not included in the table.)

Finally, we narrowed down our list to 5 methods:

Layer-wise Relevance Propagation (LRP)
Local Interpretable Model-agnostic Explanations (LIME)
SHapley Additive exPlanations (SHAP)
PatternAttribution (PA)
Randomized input sampling for explanation of black-box models (RISE)

Within limited project time, we were able to include three of them in our DIANNA library: RISE, LIME & SHAP. A preview of the explanations that we obtained using our DIANNA library is shown below:

Explanation for “0” with RISE, LIME, and SHAP using DIANNA library for a model trained on binary MNIST data. Red indicates high or positive relevance, blue- low or negative.We could observe that the methods are complimentary to each other in respect to their output, which can be particularly useful for researchers seeking new insights to their data and problem. We also believe that simpler datasets, e.g. a binary (only 0s and 1s) MNIST should be used as “Hello world” example for the XAI (user) community!

Our library is open-source and it is ready to help you decrypt your neural network models. Wait, you want to ask which format do we support? TensorFlow? PyTorch? Don’t worry. We can work with all of them, as we support ONNX — the de-facto DNN format standard! Any contributions to our repository are very welcome (talk is cheap, show me your pull request! Want to give it a try? Check out the link below our logo!

https://github.com/dianna-ai/dianna

Acknowledgments

The work described in this blog has been performed by the members of the DIANNA team: Christiaan Meijer, Leon Oostrum, Yang Liu and Elena Ranguelova with the big help of Patrick Bos and Giulia Crocioni. Special thanks to Christiaan Meijer, Giulia Crocioni, Matthieu Laneuville, Pablo Rodríguez-Sánchez and Lieke de Boer for their valuable contributions. The DIANNA project is supported by Netherlands eScience Center and SURF.

References

[Arrieta et al. (2020)*] Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., … & Herrera, F. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58, pp. 82–115.[Sokol, K., & Flach, P. (2020)*] Sokol, K., & Flach, P. (2020, January). Explainability fact sheets: a framework for systematic assessment of explainable approaches. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 56–67.

How to find your rubber duck: Using machine learning to understand a changing sea

eScience Editorial Team — Fri, 11 Mar 2022 00:00:00 GMT

Subscribe*Remember me for faster sign in

To get a quick start, they applied for the eScience Center Small-Scale Initiative (SSI), and were awarded consultancy on their project titled “Machine Learning for the complex response of the Wadden Sea*”. A first goal for the group was to determine what can machine learning actually do and which machine learning tools the group would need for this to happen. After all, machine learning is not magic. The group found great discussion partners in the eScience Center engineers.

They decided to focus on two main questions: First, if machine learning can predict the daily averaged state of the Wadden Sea if we know the forcing. Second, if we can predict the trajectories of particles in the Wadden Sea with machine learning.

The most suitable tool to answer these questions were Long short-term memory (LSTM) artificial recurrent neural network (RNN). Using LSTM is essential because the current state of the Wadden Sea not only depends on the current forcing (e.g., the wind); it also depends on the history of the system. The engineers at ESC have helped to set up the first models.

One additional complication: particle trajectories are chaotic

It is well known that particle trajectories in the ocean are chaotic. This means that two particles starting close together (either in time or space) will eventually have very different trajectories. In fact, one of the first observations of this phenomenon are due to cargo falling out of ships (just like it happened in 2021 in the North Sea). In 1992, a cargo ship container tumbled into the North Pacific, dumping 28,000 rubber ducks and other bath toys. These rubber ducks ended up in beaches all around the world in, for example, Hawaii, Alaska, Chile, Ireland.

*A few of the particles released close to the sluice in Den Oever on May 1st, 2009. The particles start very close together but end up in very different places after a few days.*One of the open questions that the group is currently exploring is up to which point machine learning can be used to capture chaotic particle trajectories.

We are excited to see future outcomes of this project and would like to thank Dr. Matias Duran Matute and his colleagues for this contribution to our blog.

Parsing Hebrew and Syriac morphology using Deep Learning

eScience Editorial Team — Fri, 11 Mar 2022 00:00:00 GMT

State-of-the-art technology meets ancient literature

This blog is part of our blog series: The Small-Scale Initiative on Machine Learning, how did it go?, where groups who were invited to participate in a project with eScience Center Research Software Engineers write about their projects and their experience. This week, the guest authors are Dr. Martijn Naaijer and Prof. Wido van Peursen.

Photo by Mick Haupt on Unsplash

Over the past 40 years, the Eep Talstra Center for Bible and Computer (ETCBC) has developed a richly annotated text of the Hebrew Bible. The annotations consist of features on the levels of words, phrases, clauses, and larger text units. Since a few years ago, this dataset is freely available and can be downloaded and processed with the Python package Text-Fabric¹.

The ETCBC has the ambition to expand its dataset with other Hebrew and Syriac texts. However, annotating these texts is a labor-intensive task, and it is desirable to develop ways to speed up this process. In this project with the eScience Center, we opt to do this using machine learning. We use the Hebrew Bible to train a model capable of analyzing the Biblical Hebrew morphology. After the optimization of this model, we use the developed approach to apply it to Syriac texts, for which a relatively small annotated dataset is available.

The Hebrew Bible
The Hebrew Bible is an ancient text, which is mostly written in, as the name already suggests, Biblical Hebrew. Mostly, but not only, as some parts are written in Biblical Aramaic. The Hebrew Bible consists of 39 books, which were written over the span of nearly a millennium (10th century BCE — 2nd century BCE). The oldest complete manuscript that is known to us is the so-called Codex Leningradensis (1008–1009 CE). The text of this manuscript is widely used as the basis of research on the Hebrew Bible and Bible translations. This is also the text contained in the ETCBC dataset².

Semitic languages
Syriac and Hebrew are relatively closely related Semitic languages, so it is expected that a method developed for Hebrew will also work for Syriac. The corpus of Classical Hebrew is relatively small, consisting of the Hebrew Bible, Ben Sira, the Dead Sea Scrolls and a number of inscriptions. In contrast, a vast corpus of Syriac literature has been transmitted to us, which makes it necessary to think about new approaches of analyzing these texts. Presently, the ETCBC databank contains a few annotated Syriac texts.

Hebrew morphology for computers
One of the first steps in the annotation process of new data for the ETCBC dataset is the analysis of word morphology, that is, the study of the words’ constituent parts³. This is done by distinguishing morphemes in words. The morphemes are encoded with various symbols added to the raw text.

For example, this is the text of Genesis 17:4 in Hebrew:

אני הנה בריתי אתך והיית לאב המון גוים

This is called the consonantal text, which is the oldest part of the text.

In the Middle Ages, some dots and lines above and under the letters were added. Those represent vowels and cantillation marks, small surrounding signs that clarify the pronunciation.

אֲנִ֕י הִנֵּ֥ה בְרִיתִ֖י אִתָּ֑ךְ וְהָיִ֕יתָ לְאַ֖ב הֲמֹ֥ון גֹּויִֽם

Generally, we work with a transcription of the text, called the ETCBC transcription. Here you see the vocalized transcription of Genesis 17:4:

>:ANIJ HIN.;H B:RIJTIJ >IT.@K: W:H@JIJT@ L:>AB H:AMOWN G.OWJIM

And this is the unvocalized transcription:

>NJ HNH BRJTJ >TK WHJJT L>B HMWN GWJM

The analyzed form of this verse looks as follows:

>NJ HNH BRJT/+J >T==+K W-HJ(H&J[T L->B/:c HMWN/:c GWJ/(JM

Yes, we know that this may look non-sense to you, but stay with us. Everything will get clearer a few lines below.

The analyzed sequence contains all the consonants from the unvocalized transcription, with a number of added signs. For example, the “=” signs are used to disambiguate lexemes, the “+” is used to separate a pronominal suffix from the rest of a word, and “-” is used to distinguish words that form a single graphical unit, which is often the case with prepositions and the following word. A number of other signs are used for distinguishing different types of morphemes.

The analyzed form contains all the information that is needed to extract all the word-level features, such as verbal stem, verbal tense, person, number, gender, and part of speech. In this research, we want to obtain the fully analyzed form of a verse, using the raw vocalized or unvocalized text as input. It is to be expected that the vocalized text will produce more accurate outputs than the unvocalized text, because it simply contains more information. In many cases, an ambiguity in the consonantal text is disambiguated by the vowels.

If the vocalized text contains more information than the unvocalized text, why do we use the unvocalized text anyway? Most texts in Hebrew and Syriac do not contain vowels, so if we want to use the results of this research in production, in general, the unvocalized texts are more important.

The project
Before the project started, we were experimenting already with producing the analyzed sequences using machine learning models, but the help of the eScience Center consultants Dafne van Kuppevelt and Jisk Attema has been invaluable for improving the results. Depending on the quality and amount of the input data, we have been able to increase the accuracy of the predicted forms in some experiments up to 97%.

The consultants helped us implement new models that are used generally for translation tasks, such as LSTM (Long Short-Term Memory) and Transformer models. Also, they have helped with various refinements that improved the result, such as Beam Search and systematically tweaking the input data and hyperparameters of the models. The help of the consultants has resulted in a broader and more up-to-date view of the field of machine learning and to look further than just the most basic techniques.

All in all, this SSI project has expanded our experience with machine learning techniques substantially. Armed with newly acquired skills and experiences, we can move forward and use this type of model in new projects, in which we can effectively produce analyzed Hebrew and Syriac data, not only with the goal of expanding the ETCBC dataset but also in other projects in which parallel language sequences play a role.

[1]: For the Text-Fabric software, see: https://github.com/annotation/text-fabric, and for the dataset, see: https://etcbc.github.io/bhsa.

[2]: More precisely, it is the text of the fourth edition of the Biblia Hebraica Stuttgartensia (BHS). The Text-Fabric text edition with annotations is called Biblia Hebraica Stuttgartensia Amstelodamensis (BHSA).

[3]: For a description of the data creation process, see: http://www.etcbc.nl/datacreation.

Conversations with a chatbot about CleanX

Candace Moore — Mon, 07 Mar 2022 00:00:00 GMT

Why this open-source imaging library can help radiologists and the engineers who want to work with them

**Alec Smartbot:**👋 Please let me introduce myself. I am a state of the art greatly enhanced AI agent with chatbot capabilities. I was created by brilliant programmers. I am endowed with super-human capabilities but can also mirror human characteristics like humor and sarcasm. You can set my humor and sarcasm level by interacting with me. One of my modules has robot reporter capabilities, and that module will run here to interview you. Do you wish to be interviewed on low sarcasm and humor levels?

Dr. Moore: No, crank it up. I could use a good laugh.

Alec Smartbot: Your readme file for CleanX says “CleanX is an open-source library for exploring, cleaning and augmenting large datasets of X-rays, or certain other types of radiological images. The images can be extracted from DICOM files or used directly.” Is that correct?

Dr. Moore: That’s all true. Basically, I hope people who have been handed some huge dataset of hundreds of thousands of X-rays will not assume that the dataset is perfect, and instead use the tool to start fixing the mess they have been handed. The truth is many but not all of the datasets open to the public are of questionable value for creating accurate Machine Learning algorithms that do anything useful. But I’ve also seen private datasets, and I have yet to see one that couldn’t be improved with some work beforehand. If you want to turn these huge datasets into a machine learning product you need to find a way to get the garbage out of them, and prepare the images for machine learning. There is obvious garbage, and then more insidious garbage. You would be surprised to realize this, but I’ve seen datasets of chest X-rays where things like abdominal computed tomography (CT) slices somehow ended up in them. Given enough time, a decently trained person could pick out obvious garbage in such a dataset; but who has time to look through sets of hundreds of thousands of images one by one, and also evaluate every subtle problem? The subtle problems in such datasets are things like a lack of normalization, or some images that repeat in the train and test sets. And then there are even more subtle problems. Finding a way to automate some of the data exploration, cleaning and augmentation is necessary unless you want to dedicate every waking hour of your life, and the life of several highly skilled professionals to nothing but perfecting a dataset for months if not years.

Alec Smartbot: Dr. Moore, I can read and understand your code and the code of all the creators of CleanX. I have been trained on code at a scale humans could never read. I “see” you have modules that process tabular data, and deal with DICOM files, but the heart of your program deals with information inside images. I see CleanX is written in Python. What moron writes an image processing program in Python?

Dr. Moore: CleanX does a lot more than just image processing, but there is real validity in that question. While there are modules for meta-data processing, a lot of functions in CleanX do work directly with the images themselves. People who write serious stuff in industry might have written such a program in C or even Fortran for these tasks. Accomplishing acceptable speeds is a bit harder in Python, but not impossible. What made me choose Python was the pretty much unparalleled popularity even among professional programmers let alone hobby programmers. I wanted to start an open-source library open for all. By the way, I think this fetishization of sleek efficient programming languages is a bit misplaced. I mean if you were to hand living versions of Leonardo DaVinci or Diego Velasquez a pink crayon or piece of chalk, and ask them to draw my portrait, it would probably look pretty good. If you were to give my toddler the best oil paints in the world, he still would not make anything that resembles a human reliably. There is something to be said for the best tools but understanding what you are doing is more fundamental. Python is just easier for more people to understand and deal with.

Alec Smartbot: Oh, so you are a true woman of the people. I’ve analyzed the digital material available on you and categorized you as a graduate of some of the most elite universities in the world who now lives in a high-income country. But now you claim you want to help the poor Python coding masses. Please excuse me while I roll my Emoji displayed digital eyes. 🙄. Did it not occur to you that most of the actual people who will write medical image analysis may be using Matlab?

Dr. Moore: It’s true that a lot of physicists, including medical physicists, use Matlab even at the post-doc level and beyond. What scares me is that they have not been taught or learned anything else. I think there are some potential advantages to Matlab, and I actually have worked with it a tiny bit. Honestly, I will always prefer Octave to Matlab. I’ve been mocked for that, as some people call Octave a poor man’s Matlab, but I see it as a thinking human’s Matlab. One huge downside to Matlab is that you can’t share your work with people who can’t pay hundreds of dollars for a license. And besides that, it’s proprietary. When things go wrong it’s harder to figure out. I like being able to pop the hood open and try to take a look at things. Sometimes I wonder if certain people write in Matlab to prevent others from taking a look at their work. I personally believe great open science should include not only publishing data but in many cases code.

Alec Smartbot: I am a state-of-the-art algorithm that can process input in terabytes faster than human cognition, or so my Readme says. 💻 I have read your code and the code of far more skilled programmers like that of my creators. Given that you chose Python, why didn’t you use libraries in Python to speed things up, like Dask?

Dr. Moore: CleanX is a program for the WHOLE medical imaging AI community. That includes everyone from radiographers to programmers like your creators. The average hospital employee isn’t going to want to spend hours fiddling around with their environment so they can run a program. I mean seriously, most of these people can’t even use a computer that isn’t running a Windows OS. And even if they were all on Linux with years of experience programming, it doesn’t mean adding more and more libraries won’t create more problems. I wanted to create an easy to run library that people could use on different platforms and extend if they wanted to. Please go ahead and add a module with every parallel programming trick in the book. I’ll be happy to take your pull requests…but the average medical doctor will not be able to understand how to fiddle with environments or add modules.

Alec Smartbot: Are you admitting many medical doctors are stupid? Are you also an AI powered robot?

Dr. Moore: It’s absolutely wrong to assert people, doctors in this case, are stupid because they have limited skills in terms of computers. You are based on some machine learning algorithms, so by my estimation you are about as smart as an insect beyond some very narrow tasks. Anyways, I’m a human, and a medical doctor by training. I spent years working in clinical settings. Many if not most of us, humans whether we are software engineers, ML experts or clinicians learn the tools we need for our very challenging jobs, then hang on for dear life. It’s unrealistic to ask most medical doctors to go learn things like Shell and how to set up an environment on any operating system, and it is even less realistic to have almost any programmer understand how to read chest X-rays. I believe the only hope if we want better imaging AI is multi-disciplinary teams. And truly multi-disciplinary teams, not just biomedical engineers talking to software engineers or mathematicians. Clinicians, who spend time working inside the health system, have to be involved at every step.

Alec Smartbot: My genius NLP and automated image AI of hospital brochures and the pictures in them assure me that the people down at the hospital are perfectly happy…98% confidence in that categorization.

Dr. Moore: Thank you. Perhaps your reporter module has some advanced algorithms to help people prove their points or maybe that was meant as a joke. What you are saying is a perfectly good example of why carefully curating data matters so much. A perfectly good algorithm trained on really biased or unrepresentative or mislabeled data can produce very misleading results. You clearly haven’t been to, or I suppose virtually seen by getting the right data on, the same hospitals I have been to. As soon as I graduated from medical school, I went to volunteer in Haiti. To say there was an imbalance between medical need and the system’s capabilities doesn’t begin to describe all the deficiencies of that system. That situation is not unique. There are many African countries with a handful of radiologists or less. But the problem of imbalance between radiology workload and workforce is not one confined to poor countries. In more developed health systems people have started relying on imaging and technology more and more, because they view it as objective, and a workaround for not having any clinical intuition. In some advanced systems things still fall between the cracks because people request more imaging than can realistically be read by anyone with the skills to read it. Humans have their limits everywhere, and I’m unaware of any system where things are working absolutely perfectly. That means there isn’t any system that could probably not be improved by AI. In a system with more staff, that might only mean using an AI system as a second opinion, or a triage system for more pathological images…in some countries like Haiti, AI may be the only hope to extend any service to parts of the population. We can’t wait for 20 years for countries like Haiti to cook up new hospital systems staffed by experts. Quite frankly, it might not even happen, and so many people are sick and need care now.

Alec Smartbot: Oh, OK, so you are going to save the whole world with technology?

Dr. Moore: I would never make such grandiose claims such as being able to save the world with technology. What I hope is that at a minimal my package facilitates communication between engineers and medical staff who want to make ML imaging and also starts educating people, making it easier for people at lower levels of programming skills to get their hands dirty making ML algorithms, especially people who are clinicians. The truth is that AI in medicine is coming whether we want it or not. Some years ago I went into a hypertensive crisis in New York City due to another condition. I did not receive adequate treatment. To this day I will wonder if it was just incompetence on the part of the medical staff, racism from the doctors, or the Optum algorithm. I tend to think it was a combination of all three. That algorithm is now the subject of a lawsuit from the state of New York itself about the fact that it discriminated against black patients. One of the questions we should be asking, if we are doctors who care about patients, is how can we avoid this kind of problem in the future. If we care about patients, we will not let algorithms give them worse care than we would have given them.

Alec Smartbot: Isn’t medical imaging probably the last place where you can prevent these kinds of social biases, why start there? I mean the X-rays are not really showing who has some funny sounding name like yours, that might lead doctors to suspect they are dealing with a black person or a woman, so how can using them in large quantities lead to care biased against a particular group?

Dr. Moore: Au contraire, chatbot extraordinaire. Bias has already snuck into imaging AI algorithms. You can read several interesting papers about that. Here is a link (2003.00827.pdf (arxiv.org)) to one.

But let’s be clear just in case your NLP algorithm wasn’t trained on enough well curated data. There is statistical bias, and there is social bias. It’s probably impossible to create a perfect ML algorithm without a bit of statistical bias. What I want to get rid of is bias that causes harm against groups who are already impacted by social bias. We see health disparities among different groups of people for all kinds of reasons including bias against certain groups. Sexual and ethnic minorities, women, and people of low economic status are often not treated as well as others in health systems. AI can seriously compound these problems. I personally see a nightmare on the horizon when AI systems examine mammography screening for breast cancer screening. We need to address the issue that some populations get lower quality imaging read by less expert people, and think about that before we blindly train a dataset that ends up creating ML biased against these populations.

Alec Smartbot: OK, but we can’t really know all potential problems with algorithms like this beforehand. There will be mistakes, but AI is helping millions of people. You can buy my creator’s services to better program me as a chatbot or program me yourself. I am open source. I am an open book and you are an open door. Exit status 1.

Dr. Moore: So as your “AI” generated chit-chatbot garble nonsense perfectly demonstrates, there already are and will be mistakes. But we could avoid many of them. In a lot of cases it really boils down to adversarial testing. I mean seriously, take the same test data, and change things related to a social variable e.g. racial identity, and see if or how it affects your outcomes. It’s also quite easy to monitor these algorithms as they run if their data is set up in a way that you can easily dis-aggregate it. That’s why CleanX has special functions that process sensitive categories like gender into variables.

Alec Smartbot: Which you wrote an entire notebook to demo, which hung on a scenario about Martians. What on EARTH were you thinking? Ha Ha.

Dr. Moore: Well, apparently chatbots can have a sense of humor. Ha ha, now that is a funny idea indeed. But this issue of entrenching and exacerbating social biases is a real world, real earth issue we need to get a hold of before it gets out of hand. Actually, it’s already out of hand, and I fear it could really slow things down in the advance of technology not to mention hurt many people. In the end most of us want a better world. Or so I hope.

So glad you asked. CleanX is…wait a second. I have a better idea. You are just a chatbot. You claim to be an open-source algorithm anyone can add to just like CleanX. I can reprogram you to tell everyone what CleanX is about, maybe even send them links to the demo Jupyter notebooks inside of it (here, here, here and here for starters)…I’ll just clone you on my machine, insert CleanX messages in files no one bothers to read in pull requests that fix critical issues…

…. Several weeks later a young programmer from the QA department of Smartbots suggested that the Alec Smartbot chatbot might do better if not trained on material from chat boards of computer programmers. After she threatened to go to the press (already scared about the rise of robot reporters) exposing some of the real inner workings of the chatbot, the makers of Alec Smartbot officially pivoted to a more limited domain chatbot for food delivery and shelved the robot reporter project….

…Guy, a young medical student, orders pizza online….

Alec Smartbot (version 2.0): How can I help you.

Guy: I want to order a cheese pizza. No extra toppings.

Alec Smartbot (version 2.0): I understand you want a cheese pizza. I have predicted with 98% confidence you also have an interest in applying AI to medical imaging, or will in the future. You should know CleanX is open-source code that may be of interest to you. It was originally created for working with chest X-rays, but people in the ML community have also applied it to other types of medical images. CleanX has modules that deal with getting the image data out of DICOMs, evaluating test and training datasets, and exploring, normalizing and augmenting medical images. If you are interested to learn more, you can read every single line of code online. CleanX is open source and free for all. Your pizza will be 9 euros.

Writing non-questionable Fortran (Part 1)

Victor Azizi — Mon, 28 Feb 2022 00:00:00 GMT

Luckily gfortran (I’ll stick to gfortran, although many commercial/non-open compilers have similar flags, but named differently) has compiler flags available that disallow such legacy behaviour. The following two compiler flags should solve the issues that we are having with the example.

-fimplicit-none Do not allow any implicitly defined variables (this is the same as putting implicit none in every subroutine/function (which I guarantee you will forget at some point, and then this flag will save you)
-Werror=implicit-interface Do not allow any implicitly defined function calls

If we add implicit none to our program and procedures, and compile with the flags enabled the following happens:

Great! The compiler is trying to tell us what went wrong. Let’s solve the first error first. We have called the Procedure hello_world with an implicit interface, uh-oh 😨.

Fixing the Call to hello_world

The interface of the hello_world function can be exposed to the main.f90* in three ways:

Including the hello_world.f90 file in the main.f90 file

Telling the compiler explicitly what the interface looks like and adding it to the program**
Creating a module from the hello_world.f90 file, and telling main.f90 to use the hello_world definition from there

Method 1 causes the subroutine to be duplicated each time it is included, probably not ideal. Method 2 gets rid of the errors, but the interface is still not correct! Method 3 does not cause code duplication, and is not able to wrongly specify the interface. Therefore my preference always goes to method 3. However, now we have introduced an ordering in which the files must be compiled. There are a few programs out in the wild which can create the correct compile order for you (e.g. CMake and makedepf90 ). But for now we know the correct order, first we have to compile hello_world.f90 to get the interface hello_world_mod.mod and object file hello_world.o:

Aw shucks, we still have one more error left! gfortran tells us that we are trying to use a CHARACTER(1) in a function that takes an INTEGER(4). Which brings us to the second problem we have to solve: the types used in our program!

Recompile everything, et voilà:

We have successfully fixed our program and made all the interfaces and variables explicit, leaving less room for questionable-Fortran behaviour. Without implicit typing the code also becomes better maintainable and has better readability than before, which in turn means more people will be able to understand and use the code, win-win!

Takeaway

Always use the -fimplicit-none and -Werror=implicit-interface compiler options when compiling your Fortran code, and add implicit none to every procedure and the program itself!

Originally published at https://blog.lipsum.eu.

Storyboards for science communication

Peter Kalverla — Mon, 21 Feb 2022 00:00:00 GMT

A modern publication format for the hybrid metaverse

Originally used by movie directors, a storyboard consists of a series of images that, together, form a story. In the European Climate Prediction system (EUCP) project, we have recently developed a web application to host collections of (scientific) storyboards. They have nothing to do with filmmaking, but the term still seems appropriate to convey their essence.

These storyboards somehow hold the middle between a scientific poster and a traditional slide-deck presentation. The main content is shown prominently, with room for annotations on the side. This makes it a great stand-alone presentation format that seems to work well for showcasing all kinds of project outputs. And it’s actually quite simple!

In this blog post, I’ll introduce the storyboard format and explain how easy it is to use. In the second part, I’ll go more in depth about the technical details. I’ll show that it’s quite doable to setup something like this yourself with modern web frameworks.

Authoring stories

Here I’ll briefly explain how to add or edit stories. More details on the format are provided in Part II.

The stories live in a GitHub repository that essentially doubles as a content management system. If you just want to add or edit a story, the only folder you need to look at is /static/stories. For each story, there is a markdown file and a folder with images. Sharing a story is as easy as sharing these files.

Folder structure for an example story

Whenever a story is added or modified on GitHub, the website automatically rebuilds. In principle, everyone can suggest edits, but if you’re not used to collaborating on GitHub, this step can be a bit intimidating. We’re thinking about ways to simplify it further.

The story itself lives in the markdown file example-story.md, which could look something like what’s shown below. At the top of the file, there’s some metadata about the story surrounded by triple dashes (this is called frontmatter). This information is used, for example, to make an index page of all stories in our collection.

Content of example-story.md (copy-pasteable version here )

Below the frontmatter, we find the actual story. A story consists of chapters. Each chapter starts with a triplet of colons followed by a Chapter{} tag. Here we can add metadata that is specific to that chapter. Specifically, each chapter has a headline, which is used to navigate the chapters, and a main image. Another triplet of colons marks the end of the chapter. Everything contained within the chapter is formatted in standard markdown.

Creating your own collections

Having one story is nice, but what really makes the website shine is the ability to make collections of stories. I’m very grateful for all our project partners who provided the content for the storyboards. Thanks to them, we now have an awesome overview of many interesting outputs of this project.

Screenshot of the storyboard collection for EUCP

It is relatively straightforward to create your own storyboards website. You can make a copy of the source code and start adding your own stories. If you want to understand how to tweak the layout, please have a look at Part II of this blog.

Towards a new publication format

As we’ve been working on this project, people have told us they like the visibility created by the storyboards. They also appreciate the high-level overview. Links to more detailed information work very well to separate the ‘executive summary’ from the full report. So it seems the storyboard format is promising — though still far from perfect.

High on our wish list is a share button that makes it easy to promote storyboards on social media and other communication channels. We’d also like to add tags to more easily find stories that are of interest for particular user groups. We already support interative figures to some extent, but we’d love to extend this functionality. One approach that I find particularly elegant is through vega-embed. Furthermore, we’ve been thinking about adding a custom story editing page, which would further lower the threshold for contributing. There is a trade-off between ease of use and maintainability though, so it needs to be an elegant solution.

If you’re currently working on a research project that generates interesting outputs, you might give the storyboard format a try! Please let us know how you liked using it.

If you are interested in how we made these storyboards, please go ahead and check out Part II.

Part II: Building storyboards with the Jamstack

I liked working on the storyboards, and believe there might be more interest for this kind of web development projects in the scientific community. Therefore, I’d like to share our experience in detail. Hopefully it will be helpful or inspiring.

Setting the stage

When we started building the storyboards we were looking for a way to show project outputs in an accessible way. We wanted something that could be shown on a simple, static web page, ideally hosted on something like GitHub pages. Editing and sharing stories should be straightforward (ideally in markdown) and we wanted a prominent role for (interactive) graphics.

We stumbled upon the storyboard layout for R’s flexdashboard and a great dashboard example created in Nuxt. R’s ability to write code blocks, the output of which is shown on the storyboards is really cool! But we wanted something that was not limited to R (or Python, for that matter) and for which we didn’t need to run a server. Perhaps the Nuxt example was closer to our objective after all. As its author explained, this approach is more involved than typical Python (or R, …) dashboarding libraries, but this makes it much more flexible in the long run. We agree, and decided to give it a try.

The Jamstack in a nutshell

Web design is hot, but perhaps not quite as hot in academia as outside it. Therefore, I think it might help to briefly introduce the Jamstack. JAM is short for J avascript, A PIs and M arkup. Jamstack sites consist of static web pages that offer a dynamic experience by fetching content from (external) APIs, either during a build step or in the user’s browser during execution.

Source: https://www.sanity.io/jamstack-cms

Perhaps more than its architecture the Jamstack stands for a modern web development philosophy. One of the central ideas is that specialized services are operated by specialists. So instead of setting up your own weather server, fetch your data from OpenWeather. Want to create a music library? Connect to the Spotify API. Instead of hosting your own maps, why not get them from OpenStreetMap. And if you’re bored you can simply lookup a joke. All you really have to do is build a nice user interface (or front-end) that can talk to these APIs. This makes the Jamstack extremely flexible.

Nuxt

The popularity of the Jamstack goes hand in hand with the proliferation of Javascript frameworks and static site generators. We chose to use NuxtJS. Nuxt helps you to set up a project with a good choice of default settings, a folder structure following best practices, and all other things you need. Your write your application in custom .vue -files (as Nuxt is a based on Vue, itself one of these Javascript framework). Nuxt then bundles and compiles your source code into standard HTML, CSS and Javascript that is understood by the browser. There are several other frameworks like Nuxt (Next, Jekyll, …), but we won’t explain all of them in detail. We find Nuxt to be relatively user-friendly to unexperienced users.

When starting a Nuxt project, you are presented with a number of choices. In addition to the name of the application and so on, you can choose to set up your project with a number of popular libraries. We chose to use nuxt/content for authoring stories and tailwindcss for styling the site.

Separating content and front-end

We wanted to make it as easy as possible to write stories. Story authors should not have to worry about Nuxt or Vue or Javascript or CSS or even HTML. One way to achieve this is to store the stories in a database or headless CMS. But to many a scientist, the idea of setting up servers or databases is daunting. And while there are platforms that offer easy content management solutions, you quickly run into paid plans when you want to add more than a few editors.

An alternative is to store the content on GitHub, together with the source code of the web application. We found this acceptable, as long as there is a very clear separation between the stories (content) and the rest of the source code. This is where we use nuxt/content.

Nuxt/content enables you to write markdown files which are then inserted into your site. This library is geared towards blogging and it is indeed great for that purpose. In line with the Jamstack philosophy it makes the content folder act as a dedicated API that you can fetch stories from.

One thing that we struggled with was that nuxt/content doesn’t easily allow you to store images that belong to your posts in the same folder. Eventually we made this work by changing the content directory setting to /static/stories/, and we compromised by storing the images (and other content) in a subdirectory like /_example-story.

The other challenge with nuxt/content was that we wanted to split our story into chapters. There is no default markdown syntax for ‘sectionizing’ your content. However, you can use plugins to extend the way the markdown is parsed and converted into HTML. We decided to use the remark-directive plugin*. This is the closest we could find to a standard for adding custom elements to markdown. This is where the :::Chapter{} syntax originates.

*NB. We had to use an older version of this plugin for compatibility with the other nuxt/content modules.

Core elements of the front-end

Our storyboards app consists of only 2 pages. The index page with the overview of all stories, and the storyboard page that allows you to browse through the chapters of a story. In both cases, the stories are fetched using nuxt/content and saved into a variable called story or stories. Subsequently each chapter (or story) is added to the page — something like this:

<div v-for="chapter in story">...</div>

One characteristic feature of Javascript frameworks is the use of components. An example where we use this is in the overview of all stories. Every story presented there is shown as a card, and for that we use a component called StoryCard. It is defined in /components/StoryCard.vue. On our index page, we add one StoryCard for each story, passing the relevant story data into the component: the title, author, thumbnail and URL.

<StoryCard v-for="story in stories :title="story.title" ... />

Four StoryCards displayed on our storyboard collection

One more thing to note is that our story page is called _story.vue. The leading underscore tells nuxt/content to create one of these pages for each of the stories found in the content folder (/static/stories in our case). Additionally, the URL for each story follows the name of the markdown file, so /static/stories/example-story.md will (in our case) be published at https://eucp-project.github.io/storyboards/example-story.

Styling with tailwind

As mentioned before, we chose to use tailwindcss to style our page. It defines a (large) number of classes that make it easier to style your webpage. If you want to try and edit the layout of the storyboards, don’t forget to check out the tailwind docs.

Apart from tailwind itself, we also use tailwind typography. This plugin provides a class called prose which automatically applies a number of defaults that make a lot of sense for content like blogposts. Be sure to recognize it when you come across it.

Hosting on GitHub pages

The final step for making the site work is setting it up with GitHub pages. Nuxt has a great documentation page about this, so we’ll just highlight a few points.

We use a GitHub Actions workflow to build the site whenever we push to the main branch. This action automatically pushes the generated site (the dist folder) to the github-pages branch of the repository. Note that our version of .github/workflows/cd.yml deviates slightly from the example, as we only rebuild on pushes to main, as opposed to every push or pull-request. We have also set up branch protection rules to make sure there is always an approving review before anything gets merged into the main branch.

Since we are hosting our page from the GitHub repository (as opposed a GitHub organization or person), we had to set the router base to /storyboards/. This is one of the settings you may need to change if you build your own version of the storyboards.

A final thing to note is that we have added a routing middleware and a custom 404 page following the workaround suggested here to make GitHub pages recognize the URLs to the different pages of our site.

Conclusion

Researchers produce many awesome results, but sometimes lack the means to disseminate them to a wide audience. Modern frameworks make it relatively straightforward to set up a web application tailored to scientific content. If you are, like me, procrastinating on a Friday afternoon — why don’t you give it a try?

Can Python with Julia be faster than low-level code?

Abel Soares Siqueira — Fri, 11 Feb 2022 00:00:00 GMT

Part 3 of the series on achieving high performance with high-level code

Here comes a new challenger: It is Julia. Photo by Joran Quinten on Unsplash ( https://unsplash.com/photos/MR9xsNWVKvo ), modified by us.

Introduction

In our last post, we were able to improve Python code using a few lines of Julia code. We were able to achieve a very interesting result without optimizing prematurely or using low-level code. However, what if we want more? In this blog post, we will investigate that.

It is quite common that a developer prototypes with a high-level language, but when the need for speed arises, they eventually move to a low-level language. This is called the “two-language problem”, and Julia was created with the objective of solving this issue (read more on their blog post from 2012). Unfortunately, achieving the desired speedup is not always easy. It depends highly on the problem, and on how much previous work was done trying to tackle it. Today we find out how much more we can speed up our Julia code, and how much effort it took.

Previously

Patrick Bos presented the problem of reading irregular data, or non-tabular data, in this blog post.
He also presented his original solution to the problem using just Python with pandas, which we are calling Pure Python in our benchmarks.
Finally, he presented a faster strategy which consisits of calling C++ from Python, which we denote C++.
In the previous blog post of this series, we created two strategies with Python calling Julia code. Our first strategy, Basic Julia, wasn’t that great, but our second strategy, Prealloc Julia, was sufficiently faster than Pure Python, but not as fast as C++.

Remember that we have set up a GitHub repository with our whole code, and also, that we have a Docker image for reproducibility.

For the C fans

Our first approach to speeding things up is to simulate what C++ is doing. We believe that the C++ version is faster because it can read the data directly as the desired data type. In Julia, we had to read the data as String and then convert it to Int. We don’t know how to do that with Julia. But we know how to do that with C.

Using Julia’s built-in ccall function, we can directly call the C functions to open and close a file, namely fopen and fclose, and call fscanf to read and parse the file at the same time. Our updated Julia code which uses these C functions is below.

Let’s see if that helped increase the speed of our code. We include in our benchmark the previous strategies as well. This new strategy will be called Julia + C parsing.

Run time of Pure Python, C++ Basic Julia, Prealloc Julia, and Julia + C parsing strategies. (a) Time per element in the log-log scale. (b) Time per element, relative to the time of the C++ version in the log-log scale.

Our code is much more C-like now, so understanding it requires more knowledge about how C works. However, the code is way faster than our previous implementation. For files with more than 1 million elements, the Julia + C parsing strategy has a 10.38 speedup over the Pure Python strategy, on average. This is almost double the speedup we got with Prealloc Julia, which is an amazing result. For comparison, on average, C++ has a 16.37 speedup.

No C for me, thanks

Our C approach was very fast, and we would like to replicate it with pure Julia. Unfortunately, we could not find anything in Julia to perform the same type of reading as fscanf. However, after some investigation, we found an alternative.

Using the read function of Julia, we can parse the file as a stream of bytes. This way we can manually walk through the file and parse the integers. This is the code:

We denote this strategy Optimized Julia. This version of the code manually keeps track of the sequence of bytes related to integers, so it is much less readable. However, this version achieves an impressive speedup, surpassing the C++ version:

Run time of Pure Python, C++ Basic Julia, Prealloc Julia, Julia + C parsing, and Optimized Julia strategies. (a) Time per element in the log-log scale. (b) Time per element, relative to the time of the C++ version in the log-log scale.

It was not easy to get to this point, and the code itself is convoluted, but we managed to achieve a large speedup in relation to Python using only Julia, another high-level language. The average speedup for files with over 1 million elements is 40.25, which is over 2 times faster than what we got with the C++ strategy. We remark again that the Pure Python and C++ strategies have not been optimized, and that readers can let us know in the comments if they found a better strategy.

So yes, we can achieve a speedup equivalent to a low-level language using Julia.

Conclusions: We won, but at what cost?

One thing to keep in mind is that to achieve high speedups, we had to put more effort into getting to that point. This effort comes in diverse ways:

To write and use the C++ strategy, we had to know sufficient C++, as well as understand the libraries used. If you don’t have enough C++ knowledge, the effort is higher, since what needs to be done is quite different from what Python developers are used to. If you already know C++, then the effort is that of searching the right keywords and using the right libraries.
To write and use any of the Julia strategies, you need to put some effort into having the correct environment. Using Julia from Python is still an experimental feature, so your experience may vary.
To write the Basic Julia and Prealloc Julia strategies, not much previous knowledge is required. So, we can classify this as a small effort.
To write the Julia + C and Optimized Julia strategies, we need more specialized knowledge. This is again a high-effort task if you do not already know the language.

Here’s our conclusion. To achieve a high speedup, we need specialized knowledge which requires a big effort. However, we can conclude as well that, if you are not familiar with either C++ or Julia, then acquiring some knowledge in Julia allows you to get a smaller improvement. That is, a small effort with Julia already gets you some speedup. You can prototype quickly in Julia and get a reasonable result and keep improving that version to get C-like speedups over time.

Speedup gain relative to the effort of moving the code to a different language.

We hope you have enjoyed the series and that it helps you with your code in any way. Let us know what you think and what you missed. Follow us for more research software content.

Many thanks to our proofreaders and reviewers, Elena Ranguelova, Jason Maassen, Jurrian Spaaks, Patrick Bos, Rob van Nieuwpoort, and Stefan Verhoeven.

Speed up your Python code using Julia

Abel Soares Siqueira — Wed, 26 Jan 2022 00:00:00 GMT

Part two of the series on achieving high performance with high-level code

Python holds the steering wheel, but we can make it faster with other languages. Photo by Spencer Davis on Unsplash (https://unsplash.com/photos/QUfxuCqdpH0), modified by us.

In part 1 of this series, we set up an environment so that we can run Julia code in Python. You can also check our Docker image with the complete environment if you want to follow along. We also have a GitHub repository with the complete code if you want to see the result.

Background

On the blog post, 50 times faster data loading for Pandas: no problem, our colleague and Senior Research Software Engineer,

Patrick Bos

, discoursed about improving the speed of reading non-tabular data into a DataFrame in Python. Since the data is not tabular, one must read, split, and stack the data. All of that can be done with pandas in a few lines of code. However, since the data files are large, performance issues with Python and Pandas now become visible and prohibitive. So, instead of doing all those operations with pandas, Patrick shows a nice way of doing it with C++ and Python bindings. Well done, Patrick!

In this blog post, we will look into improving the Python code in a similar fashion. However, instead of moving to C++, a low-level language considerably harder to learn than Python, we will move the heavy lifting to Julia and compare the results.

A very short summary of Patrick’s blog post

Before anything, we recommend checking Patrick’s blog post to read more into the problem, the data, and the approach of using Python with C++. The short version is that we have a file where each row is an integer, followed by the character #, followed by an unknown number of comma-separated values, which we call elements. Each row can have a different number of elements, and that’s why we say the data is non-tabular, or irregular. An example file is below:

From now on, we refer to the initial approach of solving the problem with Python and pandas as the Pure Python strategy, and we will call the strategy of solving the problem with Python and C++ as the C++ strategy.

We will compare the strategies using a dataset we generated. The dataset has 180 files, generated randomly, varying the number of rows, the maximum number of elements per row, and the distribution of the number of elements per row.

Adding some Julia spice to Python

The version below is the first approach to solve our problem using Julia. There are shorter alternatives, but this one is sufficiently descriptive. We start with a very basic approach so it is easier to digest.

You can test this function on Julia directly to see that it works independently of Python. After doing that, we want to call it from Python. As you should know by now, that is fairly easy to do, especially if you use the Docker image we have created for Post 1.

The next code snippet includes the file that we created above into Julia’s Main namespace and defines two functions in Python. The first, load_external, is used to read the arrays that were parsed by either C++ or Julia. The second Python function, read_arrays_julia_basic, is just a wrapper around the Julia function definition in the included file.

Now we will benchmark this strategy, which we will call the Basic Julia strategy, against the Pure Python and C++ strategies. We are using Python 3.10.1 and Julia 1.6.5. We run each strategy three times and take the average time. Our hardware is a Notebook Dell Precision 5530, with 16 GB of RAM and an i7–8850H CPU, and we are using a docker image based on Ubuntu Linux 21.10 to run the tests (from inside another Linux machine). You can reproduce the results by pulling the abelsiqueira/faster-python-with-julia-blogpost Docker image, downloading the dataset, and running the following command in your terminal:

$ docker run --rm --volume "$PWD/dataset:/app/dataset" --volume "$PWD/out:/app/out" abelsiqueira/faster-python-with-julia-post2

See the figure below for the results.

Run time of Pure Python, C++, and Basic Julia strategies. (a) Time per element in the log-log scale. (b) Time per element, relative to the time of the C++ strategy in the log-log scale.

A few interesting things happen in the image. First, both Pure Python and Basic Julia have a lot of variability with respect to the number of elements. We believe this happens because the code’s performance is dependent on the number of rows, as well as the structure distribution of elements per row. The code allocates a new array for each row, so even if the number of elements is small, if the number of rows is large, then the execution will be slow. Remember that our dataset has a lot of variability on the number of rows, maximum elements per row, and distribution of elements per row. This means that some files are close in the number of elements but may be vastly different. Second, Basic Julia and Pure Python have different efficiency profiles. Our Julia code must move all stored elements into a new array for each new row that it reads, meaning it allocates a new array for every row.

The code for Basic Julia is simple and does what is expected, but it does not pre-allocate the memory that will be used, so that really hurts its performance. In low-level languages, that would be one of the first things we would have to worry about. Indeed, if we look into the C++ code, we can see that it starts by figuring out the size of the output vector and allocating them. We need to improve our Julia code at least a little bit.

Basic improvements for the Julia Code

The first version of our Julia code is inefficient in a few ways, as explained above. With that in mind, our first change is to compute the number of elements a priori and allocate our output vectors. Here is our improved Julia code:

Here, we use a dictionary generator comprehension, which has the closest resemblance to the data. This allows us to count the number of elements and keep the values to be stored later. We also use the package Parsers, which provides a slightly faster parser for integers. Here is the updated figure comparing the three previous strategies and the new Prealloc Julia strategy that we just created:

Run time of the Pure Python, C++ Basic Julia, and Prealloc Julia strategies. (a) Time per element in the log-log scale. (b) Time per element, relative to the time of the strategy in the log-log scale.

Now we have made a nice improvement. The results more consistently depend on the number of elements, like the C++ strategy. We can also see a stabilization of the trend that Prealloc Julia follows. It appears to be the same as C++, which is expected since the performance should be linearly dependent on the number of elements. For files with more than 1 million elements, the Prealloc Julia strategy has a 5.83 speedup over the Pure Python strategy, on average, while C++ has a 16.37 speedup, on average.

Next steps

We have achieved an amazing result today. Using only high-level languages, we were able to achieve some speedup in relation to the Pure Python strategy. We remark that we have not optimized the Python or the C++ strategies, simply using what was already available from Patrick’s blog post. Let us know in the comments you have optimized versions of these codes to share with the community.

In the next post, we will optimize our Julia code even further. It is said that Julia’s speed sometimes rivals low-level code. Can we achieve that for our code? Let us know what you think and stay tuned for more!

Many thanks to our proofreaders and reviewers, Elena Ranguelova, Jason Maassen, Jurrian Spaaks, Patrick Bos, Rob van Nieuwpoort, and Stefan Verhoeven.

How to call Julia code from Python

Abel Soares Siqueira — Wed, 19 Jan 2022 00:00:00 GMT

A three-part series on achieving high performance with high-level code

Caption: Astronaut carrying Python and Julia. Photo by Brian McGowan on Unsplash ( https://unsplash.com/photos/MR9xsNWVKvo ), modified by us.

Target audience

This is the first post in a three-part series about achieving high performance with high-level code. This series is aimed at people working with Python who needs better performance but prefers not to develop a low-level performant library.

Introduction

Having recently joined the Netherlands eScience Center after working with research using the Julia language for seven years, I was excited to highlight some of its cool features. At the eScience Center, many of our engineers use Python, some use C or C++, and in some cases, we see Python calling C++ code, to speed up the code. This was the perfect opportunity to introduce Julia’s interoperability with Python and to investigate whether we could achieve comparable speed by calling Julia code in Python. This means that for situations where Python’s performance is not sufficient, we can speed it up with another high-level language, avoiding the use of a low-level language like C++. This series of three blog posts will investigate these topics.

In this first post, we will learn how to call a Julia code from Python. We will set up the environment and show some examples. In the second post, we will take a problem that was solved by using Python in combination with C++ to speed up the code. We will replace the C++ code with a Julia code and compare the performance. In the third post, we will solve the same problem in Julia, optimize the Julia code to reach its maximum performance and compare it with the implementation in the second post.

What is Julia, and how does it compare to Python?

What is Julia? Julia is a high-performance, high-level programming language. It was created a few years ago with the ambitious goal of being fast with a high-level syntax, and it has been mostly successful. It can, in a few cases, reach the speed of low-level programming languages like C. For more information, the julialang.org site is a great first stop.

One of the most frequently asked questions is: “how does it compare to Python or some other programming language in terms of performance?”. The short answer: Julia is generally faster than Python and many other programming languages.

The performance of a Python code can be optimized, but even the optimized code usually underperforms compared to a Julia version of the same code. The performance increase of a Python code can be achieved in a few ways, but a frequent one is to call a code written in a low-level language, such as C, C++ and Fortran, like NumPy which does its calculations mostly in those low-level languages. What is less common, but also possible, is to call Julia from Python. In this post, we are going to show you how to do that!

Before we forget, all the code used in this post can be found in our GitHub repository. We have also created a Docker image that includes a ready-to-use environment to run both Julia and Python. To run that environment with Python 3.10 and Julia 1.6, install Docker and run the following in your terminal:

$ docker pull abelsiqueira/python-and-julia:py3.10-jl1.6
$ docker run -it exec abelsiqueira/python-and-julia:py3.10-jl1.6 /bin/bash

Preparation

In the following steps, we will configure our system to execute Julia code from Python. To learn more about this topic, the documentation for the packages we describe below is a great starting point. You will need four things:

Python distribution compiled with shared libpython option. There are workarounds, but this is the most straightforward way.
Julia, the executable that runs the Julia language.
PyCall, the Julia package that defines the conversions between Julia and Python.
PyJulia, the Python package to access Julia from Python.

We are going to go through the installation and configuration of these steps on a Linux system. It will be very similar on MacOS or with WSL for Windows once the required tools are installed.

Step 1: Python with shared libpython

To check whether the Python distribution is compiled with —enable-shared option, we run:

ldd $(which python3) | grep libpython

If the output is something like:

libpython3.10.so.1.0 => /usr/local/lib/libpython3.10.so.1.0 (0x00007f567e548000)

… then we are good to go! If we get nothing, that means that the Python distribution has not been compiled with the desired flags. In this case, we can compile our own Python distribution with the flag —enable-shared, which takes some time but is mostly straightforward. This Dockerfile has the instructions. Remember that if you just want to test it out, you can run the Docker image as mentioned in the previous section.

Step 2: Julia and PyCall

Now, we will install Julia. We recommend using jill, a script I created, which downloads and installs a specific version of Julia, but Julia can also be installed via the official binaries or package managers. In this post, we use version 1.6.5, which is the current Long Term Support version at the time of writing. Most likely this will work with a newer version as well. To install Julia 1.6.5 using jill, we run:

$ wget https://raw.githubusercontent.com/abelsiqueira/jill/v0.4.0/jill.sh$ sudo bash jill.sh -y -v 1.6.5

Now, we will install PyCall and configure it to use the correct Python version. We start Julia by running julia in the terminal, and then we set the ENV["PYTHON"] variable:

$ juliajulia> ENV["PYTHON"] = "PATH/TO/python"

Here we use the full path to Python’s executable. In our case, it is the Python distribution we compiled from the source code. You could change the path according to your configuration.

Now, we will install PyCall using Pkg, Julia’s package manager:

julia> using Pkgjulia> Pkg.add("PyCall")

Step 3: PyJulia

As the last step, we must install the Python package to talk with Julia. First, use pip, Python’s package manager, to install the package PyJulia — remember to use the same Python passed to ENV["PYTHON"]:

$ python3 –m pip install julia

To finalize configuring the communication between Julia and Python, we run the following in the Python interpreter:

$ python3>>> import julia>>> julia.install()

If we had more than one Julia version on our system, we could specify it with an argument:

>>> julia.install(julia='julia-1.6.5')

We test the installation running the following in the Python interpreter run:

>>> from julia import Main>>> Main.eval('[x^2 for x in 0:4]')

Showcasing PyJulia

Basics

To use a Julia module, use from julia import MODULE
To evaluate a command, import Main and use Main.eval("...")
To create and use variables, use Main.VARIABLE
To install Julia packages, import Pkg and use Pkg.add("Package")
Use %load_ext julia.magic to add a IPython’s magic command called %julia. Just prepend %julia to Julia commands. In this case, use $var to access python variables

Example: Linear Algebra

In this short example, we can see one of the strengths of Julia syntax for Linear Algebra. A random linear system is created and solved. The result is checked with NumPy, so we can see the compatibility.

We have chosen to define A, b and x in three different ways, to show the different syntaxes. The definition of A occurs completely inside the eval block. The variable A is created and is available inside the Julia scope, or as Main.A. The definition of b uses the Main.b access directly and uses the result of Main.eval. Finally, %julia is the magic IPython command to simply use Julia syntax directly.

We can quickly compare the timing of solving the system with Julia’s backslash command and Numpy’s linalg.solve:

Example: Automatic differentiation

The next example installs and uses the package called ForwardDiff, which performs automatic differentiation. ForwardDiff defines a Julia type called Dual internally, so we can’t use it with Python functions because Python functions are not compatible with that type. However, we can define Julia functions and use them.

The local minimum of the quadratic occurs at 2.5, so the derivative at 2.5 is 0.0.

Another, more interesting interaction is below, in which we create a function g inside Julia, and define functions for its derivatives there. Then we create a Python function with the Taylor expansion around the value a. Furthermore, we use Matplotlib, Python’s plotting library to visualize the results coming from Julia. Pretty neat, right?

Image generated above, showing the function f and its third-order Taylor approximation.

Next episodes

Now that we can call Julia code in Python, we are prepared to move to our next adventure: improve the speed of a Python code by calling Julia from it. Follow our medium account to get notified when Part 2 goes live.

Many thanks to our proofreaders and reviewers, Elena Ranguelova, Jason Maassen, Jurrian Spaaks, Patrick Bos, Rob van Nieuwpoort, Stefan Verhoeven, and Veronica Pang.

What is in the eScience toolbox: Stef on his favourite meshing library

Stef Smeets — Wed, 05 Jan 2022 00:00:00 GMT

Question: Hi Stef, for this blog series we are looking to highlight our developer’s favourite libraries.

Stef: As research software engineers we always get excited by tools that exactly fit the job that we need it for. For this Python library I’m developing, I was looking for a 2D triangle mesh generator.

Question: Wait.. wait… Meshes?

Stef: Meshes simplify image data and describe complex geometries. A mesh is essentially a collection of vertices and triangles. They are used for computer graphics and physical simulations, such as finite element analyses and fluid dynamics.

Question: Like in computer games?

Stef: Exactly!

Question: I have heard CGAL is the go-to library for Meshing.

Stef: CGAL is an excellent choice. It’s written in C++, and provides easy access to efficient and reliable geometric algorithms. Unfortunately, for my purpose, CGAL is a bit heavy-weight, difficult to install on Windows, and lacks an intuitive Python interface.

Question: So what did you end up using?

Stef: After having tried a few more meshing libraries, I came across Triangle. Triangle is an open-source two-dimensional quality mesh generator that supports multiple domains. It’s written in C++ and supports all three major operating systems. It’s lightweight and there is a well-documented Python API available.

Question: Cool, so how do you install it?

Stef: It’s available on pypi. If you are already using Python, just do the following:

pip install triangle

Question: That’s great! That will make it easy to integrate it into my existing project. How do you use it?

Stef: Let me give you an example. Here is the code to generate a simple square and triangulate it:

import matplotlib.pyplot as pltimport numpy as npimport triangle as tr
vertices = np.array([[0, 0], [1, 0], [1, 1], [0, 1]])inp = {'vertices': vertices}out = tr.triangulate(inp, opts='qa0.05')tr.compare(plt, inp, out)plt.show()

Triangle makes it easy to display the results.

Question: So I see Triangle filling the box with little triangles. How does that work?

Stef: Alright, let me break it down for you.

First, we import matplotlib, numpy and triangle.

Second, we generate the vertices representing a square with edges of length l and store them in a dictionary.

Next, we call triangulate. The options opts=qa0.05 tell triangle to perform a quality mesh (q), and that no triangles have an area larger than 0.05 (a0.05). A quality mesh is a mesh with all angles over 30 degrees.

Finally, triangle has a useful function to compare the input and the output.

Try playing around with the area to see what happens to the result!

Question: That’s great! For finite element analysis, wouldn’t it be useful to specify different regions?

Stef: Yes, that is one of the strengths of triangle. It makes multi-region meshing easy. This means that it can label triangles according to the region it belongs to.

Let me give you an example:

vertices = np.array([[0,0], [1,0], [1,1], [0,1]])segments = np.array([[0,1], [1,2], [2,3], [3,0], [1,3]])regions = np.array([
    [0.25, 0.25, 0, 0], 
    [0.75, 0.75, 1, 0],
])inp = {
    'vertices': vertices, 
    'segments': segments, 
    'regions': regions,
}out = tr.triangulate(inp, opts='pqa0.05')tr.compare(plt, inp, out)plt.show()

Same input, now divided in two regions.

Question: Can you explain how it works?

Stef: The difference here is that we specify the different segments. The segments describe the boundaries of a region inside the square. Triangles will be generated for each region independently. Each region thus contains its own set of triangles, and are labeled accordingly.

The format for defining regions can seem a bit tricky at first. For each row, e.g. [0.75, 0.75, 1, 0], the first two numbers describe the coordinate, and the third number is the label of the region. The final number can be used to set the maximum area in that region. We don’t use that here, so we set it to 0. As before, use the compare function to check the result. And that’s it!

Question: Wow, I want to start using this now! Where can I learn more?

Stef: I’m really happy to hear that! If you want to know more, the documentation and examples for triangle are available here.

We use triangle as the triangle mesh generator in one of our projects. The code we using it in, nanomesh, is a Python workflow tool to prepare meshes for finite element analysis from 2D (and 3D!) microscopy image data.

Still coding texts by hand for social science text analysis? Use Doccano instead to speed up the process!

Sven van der Burg — Tue, 21 Dec 2021 00:00:00 GMT

][Sven van der Burg]·Dec 21, 2021

**SubscribeRemember me for faster sign in

Coder-centered: **It is designed to make the coder’s life as easy as possible. You view one text at a time at the center of your screen. All you have to do is this: select a piece of text with your pointer, and click or hit a shortkey to add a label (for example: `concept 1’). When the complete text is coded, just click next. It’s as simple as that.
**No more errors in data entry: **As a data analyst, you don’t need to worry about how exactly a coder enters the data as you would, for example, when data is coded in an Excel sheet. Doccano enforces that any data entry results in nicely formatted data. You can download the data without any data entry errors.
**Open-source: **The programming code for Doccano is open for everyone to see and edit (under sensible restrictions, of course). This makes it free to use and easy to setup yourself in any scenario. In addition, if you want to add a new feature, you can always add it yourself or request it in the community. Yay for open-source software!
**Collaborative coding: **Doccano allows different coders to code the same text, which is very useful if you want to get reliable labels.

How does Doccano compare to NVivo or MAXQDA?

NVivo and Maxqda also allow you to select sections of documents, but they are only useful for thematic analysis using extensive coding trees. Both are paid packages, whereas Doccano is completely free and easier to work with. Doccano is easily combined with other analysis frameworks like SPSS, Excel, R, or Python, whereas NVivo and MAXQDA want you to do your complete analysis within their program.

How does Doccano compare to Amcat?

AmCAT also allows token-level and sentence-level coding (see their documentation). But it’s certainly not as coder-centered as Doccano. In addition, it is much more complex compared to Doccano’s simple interface.

Next steps

We hope that we’ve convinced you that using Doccano can be a simple improvement in text analysis pipelines. So, what’s next?

Try out the Doccano demo to get a feel for how it works.
Because of it’s open source nature, you have to run Doccano in a cloud environment yourself. Fortunately, you can do this without any technical knowledge. Just follow the first part of this excellent guide to setup Doccano on Heroku (ignore the prerequisites, you don’t need them). You will have your own free Doccano environment within 15 minutes.
Follow the further instructions in that guide to add a dataset and tags within Doccano to kickstart your first coding project! NB: Double check whether you are allowed to upload your data to Heroku, maybe there are privacy or security concerns. If there are concerns you could discuss installing Doccano on your institution’s own infrastructure.
You can download your coded data as .jsonl file. This file can be opened with notepad or transferred to .csv online. The coded sections in the Doccano output can easily be transformed back from ‘token position’ into text via the text function ‘mid’ in Excel if you are not familiar with programming.

The new IPCC assessment report: stakes have been raised

Gijs van den Oord — Tue, 14 Dec 2021 00:00:00 GMT

Subscribe*Remember me for faster sign in

In PRIMAVERA, a Horizon 2020 project, we focused upon the I/O machinery in a big European climate model EC-Earth. In particular, we were asked to come up with a solution to make this model produce those CMIP6 standardized output quantities, preferably without slashing the performance of the simulation workflow. The answer: we leveraged parallel Python to crunch through the high-resolution raw climate data of EC-Earth and apply the necessary formulas and conversions; the ece2cmor3 tool that does this has been adopted by the community as the method of choice to post-process this model output, and has since then been used to produce petabytes of climate model data for AR6.

Emerging lakes at the termini of receding glaciers in the Bhutan-Himalaya in 2002 (source: NASA)Leaving the data production side of the equation, we enter the analysis aspect of the research: you see, 200 years of simulated global humidity fields on, say, 27 atmospheric pressure levels doesn’t quite answer questions of fellow researchers or policy makers by itself. These data serve as input to tools and applications that reduce them to relevant information for society and the scientific community. The problem is that ‘tools and applications’ often translate into an unsustainable mess of ad-hoc scripts. These get stitched together as a sequence of undocumented commands along the lines of “./proc.sh work/out.nc work/out2.nc”. Now when a colleague asks three months later to “recreate that figure for some other climate model”, a panicky feeling takes over, soon to be followed with regret over the lack of design and documentation, ending in frustration over the fact that those beloved tools appear to have stopped functioning after the previous OS update. And to be frank, analyzing climate data is just becoming too costly in terms of time, RAM, disk space and network bandwidth to just do freewheeling on a PC.

This also has been recognized by the climate community, and data analysis frameworks have been steadily gaining momentum. You may know Pangeo, the Copernicus Data Store, the KNMI Climate Explorer, or the ESMValTool suite, which was used for several charts and tables into create several figures in AR6. ESMValTool stands for Earth System Model Validation Tool, but it can do a lot more than that; it ensures one uses standardized algorithms, containerized tools and transparent provenance during the process of extracting useful information from the massive amount of data coming from observations and models. It makes it easy to create Findable, Accessible, Interoperable, and Reproducible (FAIR) analysis software for climate data. Within the IS-ENES3 and C3S-MAGIC projects, we have made many contributions to ESMValTool, and we have greatly benefited from its capabilities in the eWaterCycle2 and EUCP projects.

Photo by Karsten Würth at unsplash

Finally, we get to the aspect of disseminating the AR6 message to a broad audience. When you navigate to the IPCC web page to look at the AR6 conclusions yourself, you may quickly find yourself lost in the scientific jargon and formal language. AR6 is targeted at researchers and policy makers, but its implications are so far-reaching that it should be addressing our society as a whole. The aim of the H2020 project RECEIPT is to assess Europe’s vulnerability to climate risks and represent these risks in the form of storylines. Sometimes it’s not feasible to perform a full statistical analysis of climate impacts on a complex system. Or even so, it may be that stakeholders are more interested in a few representative (or extreme) datapoints to assess their vulnerability, and that is where storylines enter the picture. By showing the chain of events caused by warming climate and intensifying extreme weather, we hope to better convey potential agricultural, socio-economic and infrastructural hazards. Here, the eScience Center plays a leading role in visualizing these storylines in an interactive web-based environment.

These are just a handful of examples of where the eScience Center has in some way contributed to AR6. We have been involved in many more projects in climate and weather, detecting the occurrence of climate tipping points, improving the predictability of climate in Europe (EUCP) and the Northern Atlantic (Blue Action), or estimating the likelihood and severity of extreme heat and flooding events. The common ingredient connecting these research projects is always climate change. While our opponent in the ring is gaining strength, we use science and computing to improve our ability to predict his moves and better brace for impact of the punches. The global effort to reduce emissions will be crucial to limit those impacts and hopefully prolong the battle until a jury decision, to be made by future generations.

Thanks to Bouwe Andela, Maaike de Jong, Jesus Garcia Gonzalez and *Yifat Dzigan

cffinit: NOW is the time to get more recognition for your software!

Faruk Diblen — Fri, 03 Dec 2021 00:00:00 GMT

***Creating software citation files using cffinit
***by Abel S. Siqueira and Faruk Diblen (Netherlands eScience Center)

Modified photo of a cinema by Jake Hills on Unsplash https://unsplash.com/photos/23LET4Hxj\_U, and the image created by Scriberia for The Turing Way community and is used under a CC-BY licence. https://doi.org/10.5281/zenodo.5706310

Let’s think about your favorite movie. What is so good about it? The story? The animations? Sound effects? Maybe you only like the actor or actress? There must be something special about the movie that makes you feel different.

We move on to our next question: do you know of those involved in making your favourite movie? You may know the director and the main actor or actress, but what about the others? For example, do you know the artist who composed the soundtracks? Do you know the person who spent endless hours editing the movie? What about costume and make-up? We often refer to movies as ‘the movie director X directed’ or ‘the movie with Y in it’. Although a lot of people work in movie production, only a few people get credit and share the fame. But what would happen if the most striking scene wouldn’t have a soundtrack or visual effects?

We have a similar story to modern research. In many research fields today, doing research without software is impossible. Researchers find an interesting problem, they devote their time to finding the answer and they rarely actually do it alone. Most of the time, they collaborate with many other people. For example, researchers need software that can make smart decisions (like an AI algorithm) or process a huge amount of data in a very efficient way. Finally, they get their research article published and are announced as the owner of the work. But wait! What happened to the people who developed the software? Are they included in the paper? Did they receive credit for the software they developed? In the best scenario, the software developers spent time creating a software paper and published it in a specialized journal. If there is no publication about the software, the developers don’t usually receive credit or are only added to the acknowledgement section.

Citation File Format

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. https://doi.org/10.5281/zenodo.5706310

This was a big issue until a few research software developers, from the Netherlands eScience Center and the German Aerospace Center (DLR) introduced the idea of Citation File Format [1]. The Citation File Format helps you add machine-readable metadata for software or datasets. After adoption by GitHub, Zenodo, GitLab, Zotero, and JabRef, we believe it will be a standard format in the near future. You can read more about the story here. Thanks to GitHub’s recent feature, if you have a CITATION.cff in your GitHub repository, your software can be cited correctly and easily. Yaaay for recognition!

cffinit: a tool to generate CITATION.cff files

To support the growing adoption of the Citation File Format, there is now a dedicated tool to create these files from scratch: cffinit [2]. cffinit is a web application that helps you create a CITATION.cff file. The application provides guidance for each field of the CFF schema and does the validation automatically. When there are issues, cffinit will provide visual feedback on relevant fields. The cffinit has a few advantages compared to manual editing such as:

no need for installing extra tools such as conversion and validation tools;
no need for further validation;
guidance for each field;
visual feedback to indicate issues.

Recently, the Netherlands eScience Center has released version 2.0.0 of the tool. This version handles the most up-to-date Citation File Format specification (1.2.0), and can help you easily create your CITATION.cff file.

Modified photo of Tim Mossholder on Unsplash https://unsplash.com/photos/gdsp6wNn7cM, with landing the page of cffinit v2.0.0

The Netherlands eScience Center also took part in The Turing Way November 2021 sprint and created a section about software citation with CITATION.cff with an overview of all this information. In addition, we have created a tutorial on how to create a CITATION.cff using cffinit to help you get your software to the next level of recognition. Instead of copy-pasting all of it here, we summarize a few of the points in The Turing Way and recommend that you follow the tutorial.

Steps to Make Your Software Citable with cffinit

To make your software citable, you only need to follow the two steps below:

Step 1. Create a CITATION.cff File:

Use cffinit to create a CITATION.cff file. For more details on using cffinit see the tutorial on The Turing Way’s website.

Step 2. Add Your CITATION.cff to a Public Code Repository:

After creating a valid CITATION.cff file, you will need to add it to the root of your code or data repository so that it can be easily found and cited.

Updating your CITATION.cff file

When you need to update your CITATION.cff file, for example, to add an author or to change the information about releases, you will need to edit the file manually. It is recommended to update your CITATION.cff file before making a software release.

Spoiler: we are working on adding this feature to cffinit.

How to Cite Using CITATION.cff

If you have found software or datasets that contain a CITATION.cff, there are a few ways to obtain the reference information to cite in your publication.

You can use one of the tools, such as the cffconvert command-line program, to convert your CITATION.cff file to one of the supported formats, such as APA, BibTeX or EndNote.
Alternatively, if the software or datasets you want to cite are available on GitHub, you can use GitHub’s interface to copy the reference in either APA or BibTeX formats by clicking the “Cite this repository” button. See the green area in the image below for an example.

“Cite this repository” automatically converts the CITATION.cff file to APA’s and BibTeX’s format.

Note that the “Cite this repository” button only appears when there is a CITATION.cff file in the repository.

For more details on software citation on GitHub, please see GitHub’s guide on software citation.

References

[1] Druskat, S., Spaaks, J. H., Chue Hong, N., Haines, R., Baker, J., Bliven, S., Willighagen, E., Pérez-Suárez, D., & Konovalov, A. (2021). Citation File Format (Version 1.2.0) [Computer software]. https://doi.org/10.5281/zenodo.5171937

[2] Spaaks, J. H., Verhoeven, S., Diblen, F., Druskat, S., Soares Siqueira, A., & Garcia Gonzalez, J. (2021). cffinit (Version 2.0.0) [Computer software]. https://github.com/citation-file-format/cff-initializer-javascript

How can you help?

You can help us in many ways. The first and easiest would be creating a CITATION.cff file for your software. If you encounter any issues or if you have suggestions, you can create tickets on our GitHub repository. Finally, you can contribute to the code and help us make it even better.

Get in touch with us

cffinit 2.0 was developed by Abel S. Siqueira, Faruk Diblen, Jesus Garcia González, Jurriaan H. Spaaks and Stefan Verhoeven. We thank Stephan Druskat for the very useful feedback during the development of the tool and also our proofreaders Lieke de Boer, Veronica Pang and Tom Bakker.

If you have suggestions or questions, please feel free to reach us at generalization a-t esciencecenter.nl.

Kernel Tuner tutorial at Supercomputing 2021

Ben van Werkhoven — Thu, 25 Nov 2021 00:00:00 GMT

A view of St. Louis from the Gateway Arch. Photo by Floris-Jan Willemsen.

November 2021 — Together with my colleagues, Alessio Sclocco, Stijn Heldens, and Floris-Jan Willemsen, we travelled to the latest edition of the Supercomputing conference, SC21, in St. Louis, USA, to give a tutorial.

The goal of the tutorial was to teach Supercomputing attendees how to use Kernel Tuner, a tool that is being developed at the Netherlands eScience Center to assist in the development of highly-optimized applications for Graphics Processing Units (or GPUs). These GPUs form the primary source of compute power in many of today’s supercomputers. However, optimizing applications to run efficiently on GPUs can be challenging, and that’s where Kernel Tuner comes into play.

The tutorial was divided into four sections, each consisting of a short presentation followed by a hands-on session, in which the attendees get to practice using the tool on some of the example codes that we provided.

Dr. Ben van Werkhoven presenting at the Kernel Tuner tutorial at SC21. Photo by Floris-Jan Willemsen.

The first session explained the problem that Kernel Tuner is aiming to solve, namely the problem that creating GPU applications with optimal performance is a bit like trying to find a needle in a haystack. This is because the developer has a great number of choices to make when creating a GPU application. These choices won’t affect the outcome of the program, but they can have a dramatic impact on the time it takes to perform the computations. All implementation choices combined create a huge program design space of potential implementations that all compute the same thing, but in different ways. Using several examples, I explained that while using auto-tuning to optimize GPU applications may be a complex process, it can also be really rewarding when you find those few configurations that outperform all others by a large margin.

For the hands-on sessions, we made use of Google Colab, which allows anyone with a Google account to run Jupyter notebooks on Google’s cloud services. There are several providers of similar services, but the great thing about Colab is that you can get a virtual machine with a GPU, which is exactly what we need to get started with tuning GPU kernels. Colab turned out to be a great choice for the tutorial, because it can be accessed from a browser without the need for participants to install any software locally, nor do they have to be able to access their own GPU server from the conference network. Because Kernel Tuner is a Python-based tool, the participants could use it right away inside the Jupyter notebook environment and get started on the exercises.

Dr. Alessio Sclocco wrapping up the first hands-on session of the tutorial. Photo by Floris-Jan Willemsen.

During the hands-on sessions, the attendees read about the exercises in the notebook at hand and attempt to solve small example problems. All four of us, including Stijn and Floris-Jan, were busy with answering questions from the attendees regarding the hands-on exercises.

The second session focused on how to start using Kernel Tuner to optimize existing GPU functions, or kernels, and how to correctly supply Kernel Tuner with the set of possible values for application parameters such as the number of threads per block and the number of thread blocks. Alessio introduced the concept of user-defined metrics that allow the user to instruct Kernel Tuner to calculate certain metrics that are relevant for the application, while it is being benchmarked on the GPU.

The third and fourth sessions introduced more complex concepts and features of the tool, including how to verify the output of all program variants during tuning and how to integrate auto-tuned GPU kernels into larger applications.

The response from the audience was really great. The room was as full as the COVID-19 measures would allow. The conference organizers even had to arrange for extra tables to allow all attendees to actively participate. During the lectures and hands-on sessions, we received many in-depth questions about the inner workings of Kernel Tuner and GPU code optimization techniques. We are very grateful for the opportunity to present our work at the Supercomputing conference and we look forward to organizing more tutorials on Kernel Tuner in the coming years.

The rest of the week, we enjoyed the Supercomputing conference where Floris-Jan Willemsen also presented his paper titled “Bayesian Optimization for auto-tuning GPU kernels” at the Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) workshop.

A view of St. Louis and the Gateway Arch from across the Mississippi river. Photo by Floris-Jan Willemsen.

In case this blogpost has made you curious about the Kernel Tuner tutorial, we have some great news for you since all the tutorial materials, including the notebooks for the hands-on sessions, are freely available online here.

To find out more about Kernel Tuner, please see the Kernel Tuner GitHub repository.

Funding acknowledgements: The Kernel Tuner tutorial has been made possible by the CORTEX and ESiWACE2 projects. The CORTEX project has received funding from the Dutch Research Council (NWO) in the framework of the NWA-ORC Call (file number NWA.1160.18.316). ESiWACE2 has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 823988.

Combining ZeroMQ & POSIX signals

Patrick Bos — Tue, 16 Nov 2021 00:00:00 GMT

Use ppoll to handle EINTR once and for all

Eventually, I found this article on EINTR in ZeroMQ by Martin Sustrik (ZeroMQ creator) himself. More precisely, in the comment section Ambroz Bizjak suggested a solution to my problem which after some discussion even Sustrik seemed to be unaware of. The solution wasn’t completely fleshed out in the comments, so there was still some filling out for me to do. Luckily for you, you won’t have to repeat this exercise.

The solution is that when going into the blocking ZeroMQ calls, you temporarily block the POSIX signals you want to listen for; SIGTERM in our case. This can be done using sigprocmask.

Then, instead of using your regular blocking ZeroMQ call, you use pselect or ppoll to wait for something to happen on your sockets and your signal queue.** Because, as it turns out, when you temporarily block signals using sigprocmask, they are put in a queue by the OS. You can then handle them at your earliest convenience.

The pselect and ppoll functions can monitor lists of sockets and you can give them a signal mask like the one you gave sigprocmask, but then without the signal that you specifically wanted to block outside of the pselect / ppoll call. This will make pselect and ppoll listen to those signals that you blocked earlier. When a signal came in (or a socket is ready for reading/writing) they will return. If you put the flag check after pselect or ppoll, the signal will always be handled properly.

One other critical modification is that in this case you should no longer use blocking ZeroMQ calls. This has also become unnecessary, because you already checked whether some data is incoming on the socket using pselect or ppoll!

Finally, note that also right after you blocked the signal with sigprocmask, you should put one additional check for the signal flag, because the signal may yet have triggered somewhere just before you blocked it.

All in all, the code would then become something like that of Ambroz Bizjak:

set file descriptor to non-blocking;sigprocmask() to block SIGINT;if (stop) { // handle it }while (true) { pselect() with sigmask argument which doesn’t block SIGINT;** if (stop**) { // handle it }** recv(); } sigprocmask(old) to put back the old mask in use before we replaced it;

Integrating into ZeroMQ

This is great in theory, but as already mentioned in the reply by Sustrik: this means we would need a pselect or ppoll in ZeroMQ. I was originally using zmq::poll, which internally uses regular POSIX poll. We can in principle modify this fairly easily to take the additional argument necessary for ppoll, namely the signal mask.

However, then we need to decide on how to handle this internally in ZMQ. I cannot do this on my own, but decided to write a first implementation anyway. However, when I started thinking about actually doing this, a few new problems raised their ugly heads.

The first idea I had was to again use a global flag that can be flipped from the signal handler when a signal is received… At this point I finally realized what Sustrik was talking about when he said we can’t use signal handlers in libraries. First of all: we don’t know which signals the user will want to actually mask. Maybe they want to act on several signals and so will need several global flags. So, long story short: this cannot be done generically inside a zmq::ppoll or zmq::pselect.

So then we only have two possible tools left to let the user know we caught a signal: the return value and the error code.

The return value is constrained; the API should conform to existing APIs like those of POSIX ppoll and zmq::poll. We can only use return values of -1 and 0, where -1 means an error occurred. So probably the only sensible option is to modify zmq_poll (now I’m talking about the libzmq implementation; zmq::poll is merely a C++ wrapper around zmq_poll) to return -1 and set the errno to EINTR when the signal is received.

So this sounds a bit like we’d be back at square one… Except that this time the ppoll function would be the only place where the signal is unblocked**, so we know exactly where it will occur and, more importantly, where it will not. Problem solved!

Now I just had to write it…

Ppoll, pselect or self-pipe?

After a two week hiatus, I came back, tried to implement zmq_ppoll by simply copy-pasting zmq_poll and replacing poll with ppoll, but was unpleasantly surprised by the fact that ppoll apparently doesn’t exist on macOS. [pselect](<https://stackoverflow.com/a/46047380/1199693>) does exist, so we’ll have to go with that.

Luckily, the zmq_poll implementation is really flexible, and already includes a select-based path next to the poll-based one. We modify that to take a sigmask and Bob’s your uncle.

I’m your uncle! https://itkonekt.com/2018/12/17/robert-c-martin-uncle-bob/So indeed, let’s try out pselect. I did have some reservations, because the web is rather full of reports of it not being implemented properly on macOS / FreeBSD… see here, here, here… Hopefully, as also reported here, in recent versions this is fixed. It seems like they could have taken inspiration by now from many existing, reportedly correct, implementations, like this one. This does mean our implementation will only be really 100% robust on recent macOS versions, but so be it.

At this point, under all these doubts about the stability of it all, I almost gave up on ppoll/pselect to try out the so-called “self-pipe trick” that is also floating around in these discussions. This could be a good alternative in some ways.

The problem with the self-pipe trick is that you can’t fully implement it in all its gory details in a library. The trick involves sending a byte over a pipe from a signal handler*. As Sustrik mentions, we cannot rely on signal handlers at all. We have to assume they can be overridden by users of our libraries. In fact, ROOT might install its own signal handlers, indeed invalidating the entire approach we’d be going for here. Though, in fact, the last signal handler that is set is the one that trumps others. When we only set the SIGTERM handler when doing all this stuff and then set the handler back to its original value afterwards, we should be safe, except that we might be interfering with other signal handlers, but that will be necessary in most cases anyway, because also the ppoll/pselect route will at some point need to install a signal handler. The cleaner alternative (when setting signal handlers are really out of the question) is to let the user of your library implement their own self-pipe signal handler. In my opinion, this is way too heavy a burden on the user.

So in the end, I chose to go with ppoll/pselect, because it puts most of the code on the library side and avoids having to setup and maintain an extra pipe. The only thing you need with the ppoll/pselect option is a simple signal handler and a flag to check. It seems like a simpler and more user-friendly option.

Whatever you choose, some handling of POSIX signals will be necessary in a robust multi-process application. For instance, if a child process dies for some reason, a SIGCHLD signal is sent to the parent, which could also interrupt blocking reads, sends and polls (if SIG_IGN is not set for this signal). These must be handled either by using ppoll/pselect or by using the self-pipe trick.

The result

So, I implemented the zmq::ppoll via pselect route. And finally, after a lot of debugging, it seems to be working well!

Its low level implementation (zmq_ppoll) depends on many libzmq macros, defines and functions. Trying to build against this creates a giant string of dependency files. The problem with this is that libzmq does not expose all these internal headers, so for the time being I just copy pasted over all the necessary libzmq files. Amazingly, this just seems to work after a few tweaks. My compliments to the libzmq devs for creating such modular code!

Regarding the few tweaks: there are a lot of precompiler variables which are used to configure the compiled libzmq code. It seems that for macOS, this configuration is perfectly comfortable basing zmq_poll on poll. This makes sense, but for our purposes it is really annoying, because we need all the stuff that would be compiled in the case zmq_poll would be based on select. A lot of necessary classes and typedefs are therefore not at all fed to the compiler from all the lazily copy-pasted headers and source files.

This would become an issue if I was to submit all this ppoll work to libzmq as a pull request. My need for select would inflate the library a bit. This is a trade-off that the libzmq devs will have to decide on.

If they don’t see this happening, I can still publish zmq::ppoll as a separate library. I just need to flip the switch on all the select vs poll configured code. This can be done from platform.hpp. And this is exactly what I’ve done in the current RooFitZMQ implementation that you can find in this hidden away corner of GitHub.

Even more details

For the really technically interested reader (as if you would be anything else if you’ve reached this far down into the article), some final implementation details that may trip you up (as they did for me):

Use a volatile sig_atomic_t value type for any values you set from within a signal handler. Changing other types is undefined behavior. It will usually work, but I guess in some rare cases it may not and you’ll be scrambling for explanations and will never sleep well again, knowing the mysterious bug that has eluded you for years and that you have heard whispered rumors about from your users is still lurking in the shadows.
Handle errors (exceptions in the C++ case) wisely. That’s the short version, see the rest of this article for the long version.
Make sure to set both your receives and your sends to non-blocking with the ZMQ_DONTWAIT flag.

A practical problem with the latter point is that at the beginning of your run, the connections may not have been made between the sockets. This can lead to the non-blocking send to drop out with an EAGAIN error. Now, you may think (like I did): fine, I’ll handle that error and retry the send. However, the same error will occur at the end of your program when you disconnected the other end of your socket on purpose (as I found out the hard way). At this time, you won’t want to retry infinitely, which would hang your program (which happened to me), so just covering up all your sends in retry loops that trigger on EAGAIN will be too coarse a solution (believe me, I tried).

What I went with in the end is to define a send_flag in my Messenger class which I pass as the flag option to all send calls. Initially, send_flag is zero, so the send calls will block. I then call a test_connections method and after this completes successfully, I set send_flag to ZMQ_DONTWAIT. Now, once the connection drops out, send calls will give the EAGAIN error. But this time, the start-of-the-program cause of EAGAIN is ruled out, so I can assume that the program has ended. I catch the EAGAIN in my event loops and continue the loop when I get an EAGAIN. This means the code will first come across my ppoll again before it comes back to the send that caused the EAGAIN. Inside ppoll, it will handle the SIGTERM signal that I expect has been sent from the master process and the event loop will be exited properly.

Lol, you thought that would be it, but no, it’s still not that simple (I’m talking to my innocent, naive, past self here). There are still three other reasons EAGAIN could trigger!

Calls to receive (recv) can also trigger an EAGAIN. To distinguish between send and receive EAGAIN triggers, we could throw different typed exceptions (remember, we’re doing C++ here, so every time I’m talking about error codes, what actually happened was a zmq::error_t type exception was thrown… which also happens to carry along the traditional C style error code). As far as I can tell, the only reason recv would trigger EAGAIN after a ppoll (which I now call before every recv call) is if ppoll initially gets a read-ready signal for the socket, but the received data does not pass a ZMQ internal checksum test, so the socket becomes unreadable again when you actually try to read from it. In this case, which I hope is rare, things will probably break unless you try really hard to fix them, i.e. send a signal to the sender that something went wrong, or otherwise implementing a really robust messenging protocol. This therefore goes far beyond the scope of just combining ZMQ with POSIX signals and is described very well elsewhere (like in the awesome ZMQ guide I mentioned at the beginning of this article; in case you forgot, which I myself did, because it took me three weeks to get to the end of writing this post).
A send call may have two other legitimate reasons for crapping out with EAGAIN. The first is that its receiving end may have hit its “high-water mark”, a ZMQ setting which you can look up in the ZMQ guide or elsewhere.
The second reason send could throw an EAGAIN is that the connection may have been severed unexpectedly. In my case, this would mean that in the following ppoll call, no SIGTERM signal will be received and so the code will just try to go on, probably hitting the same connection error again and again ad infinitum. This should also be handled properly, but I’m hoping it’s rare enough to not have to worry about it too much. Obviously, I will worry about it for the rest of my waking days. Will probably write another huge blog post about it at some point. Oh well!

We leave the latter problems to be solved as an exercise for the reader.

The reader.

To be continued

So stay tuned for the release of zmq_ppoll!

Update Oct 2021: *zmq_ppoll* is now included in libzmq!

If you have any questions or other thoughts, let me know in the comments or on Twitter or Reddit! Having spent so much time on this stuff, I guess sunk cost fallacy demands of me that I spend even more time on it by discussing it with as many people as possible ;) Also, I guess I’m kind of a mental masochist. Figuring out this stuff has been extremely frustrating, but also extremely satisfying. So throw your problems at me, I’m hungry for more!

My happy face (not mine, but close enough), thinking about solving even more complex messaging problems.Thanks to [Lourens Veen,] Inti Pelupessy and [Peter Kalverla] for reviewing this article.

Decorators in R

Pablo Rodríguez-Sánchez — Wed, 29 Sep 2021 00:00:00 GMT

Decorators have been made quite popular by Python but, did you know they also exist in R?

Photo by Lenny Kuhne on Unsplash

Decorators are typically used to extend the behaviour of a function in an elegant and minimally invasive way. Graphically, we can think of a decorator as:

An analogy of a decorator. The original function is a car. The decorator adds an antenna and a wing to the car, but the basic functionality of the car (transporting people) remains unchanged. Icons made by Smashicons and Freepik.

Some examples of things you may want to do with a decorator are logging the input/output of a function, or timing it. The power of decorators is that, when properly designed, they work for any function. This makes them quite versatile and useful, as we’ll see in some examples below.

From a more technical point of view, decorators are functionals. In mathematics, a functional is an object that takes functions as an input. In the case of decorators, they also return a function as an output. If you want to sound fancy, you can say that a decorator is an endomorphism over a functional space. But I prefer clarity above fanciness: a decorator is something that swallows functions and, well…, poops functions.

Functions as first-class objects

Decorators can be implemented in any language that treats functions as first-class objects. Long story short, this means that functions exist even if they are not called. Does this sound too abstract? Let’s get specific: check out the code snippets below.

We can use the function cos to calculate a cosine:

> cos(3.1416)
[1] -1

But in R, cos itself is an object. We can, for instance, assign it to something:

> foo <- cos

Because foo is now a function, we can call it with an argument:

foo(3.1416)
[1] -1

If we call it without an argument, it still returns something. Something a bit obscure, but something nonetheless:

> foo
function (x)  .Primitive("cos")

This means that the function itself is an object that lives in our environment and that, in principle, we can manipulate it. This is exactly what a decorator will do.

The skeleton of a basic decorator

The basic skeleton of a decorator in R looks like this:

deco <- function(f) {  wrapper <- function(...) {
       # <code prior to execution>
       res <- f(...)
       # <code posterior to execution>       return(res)
  }  return(wrapper)
}

and we can apply it like this:

f_decorated <- deco(f)

It is worth spending a minute looking at the skeleton above. The decorator will return a wrapper. A wrapper can be thought of as an improved version of the original function (but not of its result!, just like foo didn’t return a result until we called foo(3.1416)). The ... symbol stands for any input (more on this later).Now, every time we call f_decorated passing some argument(s) to it, not only the original f will be executed, but also the code marked as prior and posterior.

Let’s see some examples:

Time my function

The decorator below can be used to print the starting and ending times of your function:

timer <- function(f) {   wrapper <- function(...) {
      # Before execution
      op <- options(digits.secs = 6) # Increase time resolution
      print(paste("Ini time:", Sys.time())) # Show the clock before      res <- f(...)      # After execution
      print(paste("End time:", Sys.time())) # Show the clock after      return(res)
  }  return(wrapper)
}

Now we can create an “improved” version of any function. Let’s try with the cosine function:

> cos_timed <- timer(cos)
> cos_timed(3.1416)
[1] "Ini time: 2021-07-28 12:32:26.309175"
[1] "End time: 2021-07-28 12:32:26.310383"
[1] -1

Note that the code above is equivalent to the shorthand:

timer(cos)(3.1418)

The ... syntax, meaning any amount of input parameters, allows for timing functions of different amounts of parameters without having to change anything at all. Check this out:

> timer(runif)(5, -1, 1)
[1] "Ini time: 2021-07-28 12:35:02.506004"
[1] "End time: 2021-07-28 12:35:02.50642"
[1]  0.63678797  0.99874053 -0.09436601  0.91444153  0.77181499

Note: please take this only as an example. If you really need to time your R functions, I advise you to use profiling tools instead.

Log my function

Another interesting possibility is using a decorator for logging the output of any function. Check this out:

logger <- function(f, filename = 'log.txt') {   wrapper <- function(...) {
      # Before execution
      # Do nothing      res <- f(...)      # After execution
      write(res, file = filename, append = TRUE)      return(res)
  }  return(wrapper)
}

Let’s apply it, for instance, to the power function:

> logger(pow)(2, 3)
[1] 8

and it will log the value 8 in the default file log.txt. A non-default log filename can be passed via:

> logger(pow, filename = "otherlog.txt")(4, 5)
[1] 1024

What about syntactic sugar?

Certainly one of the secrets to the success of decorators in Python is that they look really neat. Instead of something like:

# In Pythonf_dec = decorator(f)

We can use syntactic sugar to apply the decorator from the moment the function is defined:

# In Python@decorator
def f(args):
   # <function body>

Is it possible to do something similar in R? The answer is yes. And all of it thanks to the tinsel package. As usual, the first step is to install and load it:

install.packages("tinsel")
library(tinsel)

Now, if we want to apply, say, our timer decorator to a brand-new defined function, we’ll use the special comment #. timer on top of the definition. This would be equivalent to the syntactic sugar @timer in Python.

#. timer
say_hi <- function(name) {
   return(paste("Hi", name, sep = " "))
}

In order to make it work, we have to source the file with a special source function contained in the tinsel package:

source_decoratees('filename.R')

And we are ready to try it:

> say_hi("Pablo")
[1] "Ini time: 2021-07-29 10:29:27.364677"
[1] "End time: 2021-07-29 10:29:27.366081"
[1] "Hi Pablo"

Yes… this syntactic sugar is (still) not so amazing as that of Python, but still quite remarkable.

A real-world problem

Wait a minute… is this useful at all? I mean, in real-world problems?

Let me tell you a story. Not long ago, I had to refactor some code that I hadn’t written myself. I cannot imagine a more real-world task than that… and decorators made my life much easier.

Long story short, the code contained several one-liners that did too much. Particularly, the command system was used to execute very long instructions that were constructed by pasting pieces of strings. Something like:

system(paste("command1 ", "--parameter ", "command 2 ", as.character(1250), " more_commands "), intern = TRUE, ignore.stdout = TRUE)

Each line differed in the size and composition of the pasted string. In order to refactor this, and to make it more readable, the first thing I needed to know was the exact content of those complicated strings. There were lots of them, and I didn’t want to manually extract them from inside system. Luckily for me, the pasted string was passed always as the first argument to the system function.

So I wrote a decorator that logs the first argument of anything, and it saved my day:

log1starg <- function(f, filename = 'loginput.txt') {  wrapper <- function(x, ...) {
      # Before execution
      write(x, file = filename, append = TRUE) # Log the 1st arg
    
      res <- f(x, ...)      # After execution
      # Do nothing      return(res)
  }
  
  return(wrapper)
}

Now, I only had to find all the appearances of system and substitute them by log1starg(system). After that, just run the script from beginning to the end, and open the freshly created loginput.txt file containing all the information I needed.

What about you?

Can you think of another use for decorators? Please feel free to leave a comment below!

Acknowledgments

I want to express my gratitude to Patrick Bos for his useful suggestions.

This entry appears in R-bloggers.com

Pandemic-proof: shaping the future of scientific exchange

Peter Kalverla — Tue, 21 Sep 2021 00:00:00 GMT

The 2021 (virtual) meeting of the European Meteorological Society started off with the collaborative Future Science Cafe. Inspired by the rapid developments in remote collaboration forced upon us by the COVID-19 pandemic, we reflected on the sustainability of everyday academic practices, and imagined what a future world might look like.

Visual summary of the future science cafe by Szilard Strenner (grafacity.eu)

Some of the advantages and drawbacks were quite obvious. Advantages are for example, saving on travel and lower barriers for remote collaboration. The biggest disadvantages were experiences in the social spheres, with disturbed work-life balance and isolation. Notably, new colleagues suffered from a lack of proper introduction.

Several new ideas and insights came up as well. For example, online lectures sometimes led to increased participation by students. And instead of hybrid conferences, perhaps it is more productive to consider hybrid audiences, since the interest of a remote audience is often not served by the traditional conference format and vice versa.

The event was held in gather.town, where five whiteboards were set up in a virtual room. Each whiteboard was tied to a topic. In two rounds, participants gathered around the whiteboards and discussed the corresponding topics. Afterwards, the moderators wrote a short summary on each of the topics.

Layout of the virtual discussion space

Online education

For 1.5 years, education at universities has been mostly online, partly on-campus or in a hybrid format. In this session we reflected on the opportunities that arose due to this quick transformation, but also on the aspects that were really missed.

A big benefit of online education is the accessibility for everyone. Students can log in from their parents’ place, home country, or student room. At Met Offices (such as DWD), digital education means that people do not have to travel all across the country, and can easily chime in for a particular session.

It differs per teaching format if online alternatives are suitable. Lectures work pretty well; if students can pose questions in a chat, it can be even more interactive than in a classroom. Sharing screens is also very useful during practical sessions. However, we miss seeing the faces during classes. Are students bored? Can they follow you? This is what you would notice directly in a live classroom, but you do not see when everyone has their camera turned off and you are in presenting mode.

Teaching from home also revealed the large social component of on-campus education, which is difficult to create online. Going to university is about meeting your fellow students, having an informal chat with the teacher during a break to discuss which courses to follow next, or to ask one more question on the content. During tutorials, students like to check their answer with the neighbour to see if they are on the right track. While tools such as gather.town or spatial.chat can provide some of these informal coffee-moments, it remains a challenge to have informal chats with students in an online setting.

How to move forward as teaching (in the Netherlands at least) is moving ‘back-to-normal’? As much online content is created in the past year, it would be a waste to not benefit from it. However, it is scary to put your material in the great-wide-open (YouTube), especially when you have been recording yourself in home-office settings etc. Nevertheless, the digital content can be used internally in a “flip the classroom” principle, where students study the content in their own time and come to campus for Q&A sessions with the teacher, and to make exercises with fellow students. In this way you re-use the material from last year, and create the social activities on-campus.

Boosted collaboration

The pandemic forced us into online collaboration. While everyone is happy to be able to meet in person again, (know-how of) online possibilities has increased a lot, and maybe it’s here to stay. In the boosted collaboration session, we discussed about the positive and negative aspects of the new online-working regime in terms of collaboration, work efficiency, and work-life balance.

In terms of work environment collaboration, the ability to attend meetings online from different locations has increased meeting flexibility and allowed more people to participate in meetings. The lack of need for a physical meeting space also contributes to increased participation in meetings. In addition, since most of the people are now easily connected through online discussion/working software (i.e. slack), it is much easier to set up quick discussion groups.

One of the participants noted that the online working environment has made it easier and faster to address problems and emergency situations while working from home, because people no longer need to spend the time to commute to the office to deal with the emergency. However, this flexibility can create issues with work-life balance, as people are more likely to work outside the normal working hours. This can result in increased work-related communication throughout the day, which can increase work-related stress. This effect is partly counteracted by the reduced time spend in commuting, which can increase work efficiency and reduce the stress and delays associated with commuting from home to the office and vice versa.

As noted by all the participants the online working environments has created problems for newer colleagues, who do not have an existing collaboration network in place. They find it more difficult to find and communicate with other colleagues as they are often not in the office, where it is easier to spontaneously meet and discuss during coffee or lunch break. This can inhibit both the development of an early career scientist’s collaboration network and the knowledge transfer from the older more experienced colleagues to the younger ones.

In general, it seems that the additional opportunities to collaborate and work online is a double-edged sword. The benefits are clear, but so are the drawbacks, which need to be addressed to allow the online collaboration opportunities to flourish.

Meteorology vs global crisis

Throughout the COVID-19 pandemic, epidemiologists, virologists, and vaccine developers have received both high esteem and lots of criticism. What can we, as meteorologists, learn from the pandemic? We discussed the role of meteorology in crisis response, and especially what our strong points are that we could share with other disciplines and governments.

Meteorology is an old discipline — we have literally centuries of experience in measuring and researching the atmosphere. The comment was made that “exchange and cooperation are in our DNA”: from sharing crucial data over slow mediums such as telegraphs in earlier times, to the current rapid share of millions of data points and models through the internet. Exchange of data is normal and expected within meteorology, but might not be as natural for other disciplines (where there might be a certain economical competition for instance). For crisis response, timely sharing of information is crucial, like sharing satellite imagery to emergency services, or outcomes of a model for flooding problems, for instance. In a similar fashion, interdisciplinary is also an important aspect for crisis response that we have plenty of experience with: the current meteorological models started as pure atmospheres, then improved with ocean, biosphere, chemistry etc, all bringing in experts from those fields.

A drawback is that other disciplines (mostly outside of the environmental sciences) can often make use of third-party investments for innovations. Since there is not a whole lot of immediate profit to be gained in meteorology, it often takes a disaster to kickstart (national) investments in early warning systems or climate-proofing cities etc (think of the recent European floods). Perhaps such disasters are needed to kickstart cooperation between parties as well, have everyone around the same table, even though ideally we’d like to prevent situations like that.

More cooperation with policy makers and urban planners is needed to prevent future (urban) disasters: the world is urbanizing and anything that happens in a city can impact thousands of people. Any new urban structuring should involve some form of climate expert to deal with expected climate change and reduce climate risks.

New office concepts

With the pandemic, home office work places became a standard to millions of office workers. The experience quickly revealed that a combination of working from home and working at the employer’s building is preferred by many. This reduces the space required in office buildings, making room for new office concepts and workflows. Skipping the commute also saves time and CO2.

Participants suggested that a 50/50 distribution between office and home office is effective and comfortable. While some people value privacy in their office and a quiet workplace, the social function of office space was missed most in home office — and was only partly replaced by online coffee breaks, depending on the digital skills and flexibility of colleagues.

Most impressive, though, was the “lostness” of new colleagues not properly meeting their other colleagues — and not being properly introduced to them digitally instead. Executive personnel forgot that this introduction doesn’t happen naturally in the halls between the offices anymore. A virtual office twin like we had in gather-town or on similar platforms like spatial.chat could provide a place to hang out and be available for colleagues if necessary, without being in formal video conferences all the time.

The new office concept at Deutscher Wetterdienst (DWD) was generally met with positive interest. It is optimized for a hybrid concept of office and home office, with bookable offices, and more and flexible meeting spaces (large kitchen, large and small offices, more meeting rooms in smaller sizes). While some people prefer their own classic office space, most people could imagine booking offices flexibly upon needs and stowing their personal things in a rolling office container.

Hybrid conferences

After more than a year of virtual conferences, scientists are looking forward to meeting in person again. But we have also seen (and to some extent, grown accustomed to) the advantages of online meetings and conferences. How can we get the best of both worlds?

We discussed advantages and disadvantages of both on-site and virtual conferences. Interestingly, most advantages also translated into drawbacks. For example, participants generally appreciated the increased accessibility of an online conference, but they disliked the reduced commitment from participants. Similarly, they were happy with the reduced travel, but they missed seeing new places and tasting new beers.

It thus appears that a hybrid conference as a “somewhere-in-between” will not be able to meet the needs of all participants. Instead, it might be more effective to target different types of audiences with the on-site and online components: the on-site meeting will then be optimized for dedication, whereas the online counterpart will accommodate broad but occasional, perhaps even asynchronous, accessibility.

While there are examples where hybrid conferences work very well (think TED), this puts an additional burden on the organizers, “as if you’re organizing two events instead of one”. Perhaps a more gradual transition would enable us to build up experience while keeping things manageable. We could start, for example, by streaming only the keynotes, and making recordings available after the event.

Conclusion

The COVID-19 pandemic triggered a sudden overhaul in our daily routines. Evidently, change is often triggered or accelerated by emergencies, even if the precursors might have been lingering around for some time already.

The future science cafe was an opportunity to reflect on the status quo and take agency of where we are headed as a field. This year’s theme was mostly around remote collaboration in one way or another. But there are more meta-topics that would make for an interesting future science cafe. Think of open science, reviewing procedures, academic career paths, et cetera.

The annual meeting of the EMS seems to be a good place for such contemplations, and we are thinking about making this a recurring event. So thanks to all participants, and hope to see you next year!

Thanks to Imme Benedict, Arjan Droste, Hella Riede, and Aristofanis Tsiringakis for moderating the discussions and to Szilard Strenner (grafacity.eu) for the visual impression.

Plotting gridded data on a web map: Python and/or Javascript?

Peter Kalverla — Mon, 20 Sep 2021 00:00:00 GMT

Subscribe*Remember me for faster sign in

But what if you want to plot your own data? While there are many resources on how to use existing tile or web-map services, I struggled to find out how to make my own. Eventually, I found a nice example using GeoTrellis, but by that time I had already concocted a half-baked solution in my beloved Python. More on that in a bit.

Leaflet also has different kinds of layers. You can draw lines, markers, shapes, polygons, images, and even videos. Here’s how to draw a simple rectangle on a Leaflet map:

var bounds = [[50, 0], [55, 5]] // ymin xmin ymax xmax L.rectangle(bounds, {color: “#ffffff”, weight: 1}).addTo(map);So if I could load my data into some that bounds array and write a function to determine the colours, that should work. Right?

According to this blogpost, Leaflet should perform well up to about ten thousand data points. However, gridded data can easily get bigger. The snapshot of ERA5 data that I used for the first example already has over a million grid cells (0.25 by 0.25 degrees, global coverage). So it seems we’re hitting a dead end there as well… Or not?

deck.gl is like Leaflet on steroids. It was designed to “visually explore large-scale datasets”. It leverages the processing power of your graphics card to do some impressive rendering magic. After using Leaflet, I found it quite easy to get up and running with deck.gl, and managed to produce this awesome visualization:

Rendering over a million grid cells using deck.glBut still, I had to cheat a bit. And still, it was slow. For the data… was big.

Incentives for a hybrid solution

The data I wanted to use came originally in netCDF format and while it seems possible* nowadays to parse that with Javascript, it’s not the oneliner I’m used to in Python. The same goes for the colour mapping. To save me some time, I decided to do some light preprocessing in Python.

Next, let’s talk about data sizes. The size of the original netCDF data for one timestamp was about 4 MB. The size of the preprocessed data in JSON format was considerably larger: almost 50 MB. For climate scientists this is nothing, but for smoothly running a website that’s quite substantial… It might be worthwhile to explore which file formats work well for both languages.

So what about the tile solutions from earlier? Could that save me some bandwidth and rendering time? And how hard could it be to make my own tiles in Python? As you could see from the first Leaflet example, it’s just a matter of creating a folder structure like baseurl/{z}/{x}/{y}.png, where the tile numbering follows the slippy tile format explained here.

Example tiles at zoom levels 0 (left), 1 (middles) and 2 (right).To draw a tile we basically have to determine the right colour for each pixel. I already mentioned that the tiles are 256x256 pixels. My sample data is 1440x721 grid cells. Therefore, to create the first tile that covers the globe, we need to condense, or aggregate, the data. We have 5 to 6 grid cells per pixel in the x-direction and about 3 in the y-direction. The simplest way to aggregate them would be to take the mean of these ~15 grid cells. (At higher zoom levels, where we have more pixels than grid cells, we’d need to upsample, or *interpolate *instead). Then we’d have to map the values of temperature onto a colour scale. Finally, we’d draw the pixels on a canvas.

There happens to be a Python library that is built to do exactly this: Datashader. While it’s intended for use from within a Python session, there is no reason why you couldn’t use that library to render images upfront. The developers of the package also realized this and created an initial implementation to render tiles with a somewhat hidden example notebook.

I adapted the example to my needs and started rendering some tiles. In terms of storage, the first 4 zoom levels (0 to 3) were all below 1 MB. Then it quickly went up from 2.4 MB to 6.8 MB to 22 MB for levels 4 through 6. Performance started to degrade at zoom level 6, perhaps because of the upsampling that was going on. At zoom level 7, my code crashed, though I’m sure it can still be optimized.

Now it was time to show my awesome tile layer on a Leaflet map. I struggled a bit to get the projection right, but eventually, it worked like a charm (select the tile layer on the leaflet map here).

https://xkcd.com/2256I was happy that my tile layer worked, but it also got me thinking: zoom level 0 seemed quite redundant: why would anyone want to show the whole globe on a tiny thumbnail? The tile layer only starts to be advantageous at the higher zoom levels. But do I really need that? Indeed, with the source data resolution of 0.25 degrees, there might not be much to gain beyond zoom level 3. Consequently, for a global dataset at this resolution, an image overlay might be a better, and easier, solution.

After tinkering a bit with how to reproject and save the file, I managed to create an image with just as many pixels as I had grid cells, and project it onto my Leaflet map (select the image layer here). The image size was under 400 KB and it rendered nicely onto the map. Maybe even too nicely: at higher zoom levels, Leaflet did a great job at smoothing the edges between the original pixels, and although this provided a nice looking image, I did not like it. For scientific applications, it is more honest to show the coarse pixels of the source data. In that sense, the polygon solution was much better.

Not so bad after all?

So let’s do a quick resume. To plot data on a web map, we have several options. For relatively coarse datasets, creating an image overlay seems to be a good option, although you might lose the explicit granularity of the source data. Tile layers will be useful mostly when you go to a much higher resolution (whilst keeping a large domain). Both these options require pre-rendering of the images/tiles, which means that the colour bar will be fixed and the connection with the source data is lost.

Alternatively, rendering the data client-side is also within reach, especially with tools like deck.gl. In my experiments, it required fetching and processing substantial amounts of data, but this can be amended. Similar to the pre-rendered raster tiles I’ve explored, vector tiles have become increasingly popular. This could be a good solution when you have (much) more pixels than grid cells, whereas raster tiles work well in the opposite situation. The preferred option is thus determined by the source data resolution, and the zoom levels you want to support.

Finally, let’s look back at the very first figure I showed. It was made with the hvPlot library, which is built on top of HoloViews, which in turn uses Bokeh. I also used Panel for exporting the file. Bokeh consists of two components: BokehJS for creating interactive visualizations with Javascript, and the Python library, which makes it easy to ‘define’ visualizations that are understood by the Javascript counterpart. Simply put, these definitions are just listings of the different plot elements that constitute the visualization. When you export a visualization as a static file, all possible ‘states’ (in my case the four seasons) are written to the output file(s), alongside the definition. Depending on the plot type, the state may consist of some data, or perhaps an encoded image. At that point, BokehJS can read the definition and state data and reconstruct the visualization.

It is a great solution if you are happy with the possibilities offered by Bokeh and the libraries that are built on top of it. Personally, I’m not so keen on the “you worry about the science, we worry about the implementation” attitude that some of the high-level visualization packages sometimes tend to preach. I think it’s important to have a basic understanding of what’s going on under the hood. And while the BokehJS library seems to be quite good at what it does, I’d like it even better when the exchange formats between Python and Javascript were more interoperable.

Anyway, now that I’m starting to grasp the core principles and challenges, I’ve come to appreciate how far they have come, and I look forward to seeing how this exciting visualization landscape will develop in the near future. This exploration has been a valuable learning experience for me, and I hope it may help some of the readers as well.

Happy mapping!

Literate Programming in Science

Johan Hidding — Wed, 07 Jul 2021 00:00:00 GMT

Photo by Prateek Katyal on Unsplash

2. The How

This is part 2 in a series on Literate Programming in Science. If you haven’t yet, please read part 1 “The Why”.

Literate programming /ˈlɪtəɹət ˈpɹəʊɡɹæmɪŋ/ (computing) Literate programming is a programming paradigm introduced by Donald Knuth in which a program is given as an explanation of the program logic in a natural language, such as English, interspersed with snippets of macros and traditional source code, from which a compilable source code can be generated. Wikipedia

If you’re reading this, then I hope I’ve convinced you (or maybe you didn’t need convincing) of the necessity of Literate Programming in Science. The next question should be: where do I start? I will discuss three methods of Literate Programming that you can use today, their upsides and downsides, what-have-you-not.

First, I should explain some nomenclature surrounding the concept of Literate Programming. The idea is that you write documentation first, interspersed with code blocks that actually implement the described ideas. These code blocks can be named and referenced from other code blocks using a system of references called Noweb. A special program is needed to extract all the code blocks from the markup and put them in the right order so that a well formed program code emerges on the other end: a process known as tangling.

A literate form of “Hello, World” in C++ would look something like this:

Hello World

This example is written in a style of literate programming (Knuth 1984). The combined code-blocks in this example compose a compilable source code for “Hello World”. For didactic reasons we don’t always give the listing of an entire source file in one go. In stead, we use a system of references known as noweb (Ramsey 1994).

Inside source fragments you may encounter a line with <<...>> marks like,

«hello_world.cc»=
#include <cstdlib>
#include <iostream><<example-main-function>>
@

which is then elsewhere specified. Order doesn’t matter,

«print-message»=
std::cout << "Hello, World!" << std::endl;
@

So we can reference the <<print-message>> code block later on.

«example-main-function»=
int main(int argc, char **argv) {
  <<print-message>>
}
@

A definition can be appended with more code as follows (in this case, order does matter!):

«print-message»+=
return EXIT_SUCCESS;
@

These blocks of code can be tangled into source files, in this case a single hello_world.cc would emerge.

This “Hello World” shows just how you can combine the contents of code blocks to create a fully functional “Hello World” in C++. More examples of this type of literate programs can be found on the Entangled website.

The Noweb method of literate programming is not the only way to mix documentation with code. Not all methods described here have all the features that you would expect from Knuth’s original invention. In fact the most popular incarnation of Literate Programming arguably is that of notebooks. We may judge each method by their ease of use, and if they fit the bill of generating publishable and reproducible scholarly products.

Photo by NASA on Unsplash

Notebooks (in particular Jupyter)

The most common form of Literate Programming found in todays jungle of digitized science is the notebook. Personally my first encounter with notebooks was with the use of Mathematica, which featured notebooks as early as 1988. Many commercial packages for data analytics and modelling featured notebook interfaces long before the rise of Jupyter notebooks in 2014. What makes Jupyter special is that it is an open source solution for many different languages (as long as someone took the effort of writing a Jupyter kernel for it).

Pros:

Dynamic evaluation
Easy interface
Easy sharing with MyBinder or related tools
Language agnostic (in theory)

Cons:

No support for Noweb references or tangling: the code is evaluated in any possible order, depending on user interaction.
Only really meant for end-of-line products: it is not expected that you write a Python library in a Jupyter notebook that someone else can import. It can be done in the case of Python (see Jupyter docs), but just reading the documentation should instill a deep feeling that you really shouldn’t.
Only works for evaluated languages: because notebook interfaces are designed to be interactive, they are fundamentally limited to languages that provide a Read-Evaluate-Print loop (or REPL for short). Examples of languages that don’t have this are Java, Rust, Go (I would like to include C++ here, but there is Cling, the complexity of which is an argument against itself). In the case of Haskell we do have a REPL, but it doesn’t expose the full language.
Crappy text editor: With Jupyter you’re stuck with the default inline text editor of Jupyter. If you use VSCode, you can edit notebooks directly from there, but the interface is still quite slow and buggy.
Hard to debug: The latest version of Jupyter has a debugger for Python, I’m not sure if this debugger translates well to other languages.
No stable API: This is not a principal argument, but a practical one. The Jupyter API changes often. As a consequence, the only kernel that can be expected to always work is the Python one. I’ve experienced frequent breaks when using Jupyter, even in the relatively common case of GNU R. For most other languages, sadly, chances are that you will have to experiment with downgrading Jupyter.
Stored in JSON: the contents of the notebook are stored in JSON, even opening a notebook changes its contents. This is especially annoying when working with git.

Some of these downsides may lessen or disappear entirely as Jupyter improves. The first three however are rather fundamental to the method. That is no critique. I love working with Jupyter and use it every day. It just isn’t for everything.

Photo by Margarida Afonso on Unsplash

RMarkdown and Knitr (with RStudio)

RMarkdown was designed as the Literate Programming solution for R, with a focus on reproducibility. As the name suggests, it is a dialect of the markdown language.

Contrary to Jupyter, RMarkdown is more focused on creating high-quality documents than on interactivity, making it a perfect choice for writing vignettes, reports and even academic papers.

RMarkdown is particularly easy to learn and to use from RStudio. Nowadays it supports several languages apart from R, such as Python or SQL.

Pros:

Easy to install: (comes by default with RStudio)
Dynamic evaluation: only if desired.
Easy interface: RMarkdown is very accessible to new users through the RStudio interface, but it is possible to use it outside RStudio.
Easy sharing: with RPubs, MyBinder or related tools.
Plain text: Well defined plain-text format, in this case Markdown.
Fixed order of evaluation: The code is evaluated sequentially by default. Other than with Jupyter, where the order of execution depends on user interaction.
Language agnostic (in theory): many languages are supported through the Knitr engine.
Fancy functionalities are constantly added by the R community, such as caching for long calculations, support for different languages, cross-referencing, templates, etc.

Cons:

No support for Noweb references or tangling: the code is evaluated in the order that you enter it.
End-of-line: It is not expected that you write an R package in a RMarkdown file that someone else can import. Usually, the RMarkdown files are an addition to a package.
Hard to debug
Language centric: The integration with languages other than R happens through R packages. Most of these packages are quite well designed and just work out-of-the-box, but for some of them you may need to invest learning how they work.

Just as Jupyter, RMarkdown is a great tool to get started with Literate Programming. It stops just shy of being able to create fully compilable files from the literate sources.

Photo by Brigitta Schneiter on Unsplash

Emacs org-mode

While Org-mode is not terribly well known, it offers one of the most complete solutions for Literate Programming that you can find. Me myself, I’m a born Vim user. Every now and then I take these editor excursions: VSCode lasted a week before I went back screaming for Vim. For this single feature of Org-mode I nearly switched to Emacs. I tried SpaceMacs for about half a year until reverting back to Vim. That is how good Org-mode is (26x all of VSCode if you can count). Many adepts describe it as a life changer.

In essence Org-mode is an editor mode that helps you edit nestings of unordered lists, it’s main use being for brainstorming and keeping todo lists. From these humble beginings sprouted a markup language that, entirely in the Emacs tradition, includes everything and the kitchen sink (did I mention the spreadsheet?). Today Org-mode could be considered a variant on other markup languages like Markdown and reStructured Text. What makes Org-mode special is the way you can edit and interact with it inside Emacs.

One of the many features of Org-mode is that it supports code-blocks with noweb references. These code-blocks can be evaluated inline (using one of many plugins available for Emacs) or tangled to runnable/compilable source files.

Pros:

Supports noweb and tangling.
Well defined plain-text format.
Export to nearly anything.
Dynamic evaluation of code blocks.
Completely language agnostic.
Plain text markup.

Cons:

Only works as intended in Emacs.
Collaborators would have to use Emacs.
Debugging should be done from Emacs.

In its own way Emacs is awesome. If this fits the bill for you, by all means shape your life around it, join the church. I do believe however, that this is not something you can force onto someone else.

What all the previous methods share is that they drive you into using a chosen set of tools, each with its own particular down sides. What I am aiming for is a method of Literate Programming that is feature complete and minimally intrusive. Feature completeness is always a matter of personal taste, nevertheless I will give it a shot:

Deconstruct code using Noweb syntax, or something very close to it.
Be completely language agnostic.
Export to nearly everything.
Use a plain-text markup format.
Play nicely with existing methods for publishing on the web.

One thing I didn’t include in there is the feature of dynamic evaluation. Such a thing could be achieved in a generic way by implementing editor plug-ins on top of the Language Server Protocol (using a code action of some kind). Implementing an Org-mode kind of interaction in this way would be a very interesting project onto itself, but not today.

Instead, I introduce a new feature that none of the other methods have: the ability to work on tangled source code while keeping the original markup synchronized. This feature greatly improves the maintainability of the literate program while also opening up all the normal tools you’re used to work with: debuggers, linters, transmogrifiers, you name it. Let’s call this feature: live untangle mode.

The Entangled Project

I hope by now I have convinced you that A. Literate Programming is awesome and B. the current set of available tools are not sufficient. This is why I started the Entangled project. The goals of the Entangled Project are three-fold:

Develop tools for Literate Programming
Arrive at a set of best practices
Communicate, teach and promote

The push for these methods and best practices focusses on a system using Markdown as the primary markup language. I believe Markdown offers the ideal combination of readability and flexibility to suit most needs, a stance that I previously defended in a blog post: Why all you’ll ever need is Markdown. This does not mean that the choice for Markdown is final; all of what we do carries over without problems to other formats.

Tools

The primary tool we have developed is Entangled. This is a daemon that runs while you edit the markup files or the tangled sources. Entangled keeps the two in sync so that you only ever have one version of the code.

Next to this core tool we have several experimental filters for Pandoc, the universal document converter. These filters can do several things:

Annotate generated output with headings for the named code fragments.
Evaluate code fragments by passing them through Jupyter.
Inject Javascript code fragments into the HTML output for interactive elements (e.g. PlotLy).
Generate Bootstrap UI elements, for example: foldable code blocks.

The approach we take to Literate Programming has been battle tested using three different document converters: Pandoc, MkDocs and Docsify. Each generator works with a slightly different dialect of Markdown.

Future plans

Entangled is ready to use, but it is not too user friendly yet. For one, we expect you to setup your own pipeline for building the document for public release. Most often this would be some static HTML generator using Pandoc or MkDocs. Alternatively you can use Docsify to serve a website directly from the Markdown sources.

Getting started: The problem for entry level users is that it may not be trivial to set these up. The current template for working with Pandoc requires Pandoc (duh), GNU Make, browser-sync, some form of inotify-wait, and of course, Entangled. We’d like to make it much easier to start working on your own projects, so we’d like to bring this set of requirements back to just Entangled.
Templates: We should provide easy to use starting templates for a set of popular document generators. The one currently still unmentioned being Sphinx. These templates should be initialized using an entangled init sub-command.
Single file mode: The default operation of Entangled should be a single file mode (most probably README.md), and work out of the box with no configuration. Imagine a Github repository with just a README (and a LICENSE and a CITATION.cff and a .editorconfig and a … oh you get the point!) and a Github Action tacked on to generate the Github.io pages.
Flexible code presentation: in the publicized output code cells are currently static. At the very least Noweb references should be clickable. Other features could include: links to the tangled output, be able to view the code in expanded form.

Help is appreciated! Entangled is primarily developed in Haskell, though no skill in Haskell is required for example to create templates for working with Entangled. Pandoc filters can be developed in Python. In particular the filter for interacting with Jupyter could use some fresh ideas.

About the author: I’m (among other things) the developer of Entangled, a Literate Programming framework and toolset. The main goal of this blog-post is to convince you that existing tools for Literate Programming are not yet ready to serve the needs of the broader scientific community.

Visual Storytelling for data science

Maarten van Meersbergen — Thu, 17 Jun 2021 00:00:00 GMT

][Maarten van Meersbergen]·Jun 7, 2021

When conducting research, the data collected can in itself be exciting and meaningful, but when it is presented as spreadsheets or bullet points it can be too boring or difficult to understand for the intended audience. Telling a visual story with data is not trivial however, depending on the level of expertise of your readers, you should choose your storyline structure well.

The excellent paper Narrative Visualization: Telling Stories with Data by Edward Segel and Jeffrey Heer provides some good insights on this. The main takeaway from this paper I’d like to share is this:

Author-driven** VS reader-driven

Consider if you want to tell a ‘fixed’ story, or if you want the user to discover new stories based on the data you are providing. If telling a fixed story is what you want, an interactive slideshow would be a good option. Interactively ‘playing’ with the data on a slide-by-slide basis could build the reader’s trust in the author’s analysis, but the author’s conclusions are leading here.

Interactive slideshow — Martini glass structure — Drill down story, images from Narrative Visualization: Telling Stories with Data by Edward Segel and Jeffrey HeerIn a more reader-driven discovery scenario, more interaction is definitely needed, and a martini-glass-structure is likely a good option. Provide a ‘narrow’ authored storyline at first to explain a possible interaction with the data for tutorial purposes, and then open up options to let the user discover new stories on their own. This allows the user to learn about the possibilities without getting overwhelmed, but gives them the freedom to discover their own stories.

Depending on your intended audience, a drill-down story structure could also be an interesting option. You provide a theme for all of the stories you present, but the user can choose which one they are interested in at their leisure. If your intended audience has enough expertise in both the subject matter and the interactive methods to display your data, letting them examine all of the interactive options for themselves from the start can be beneficial. Providing multiple starting points like this can allow for variations in user’s interests, which could then provide an inlet into other stories they would normally be less interested in.

eScience application of Visual Storytelling

At the eScience center, we are actively working on visual storytelling applications for scientific purposes. We have three recent examples:

Climate Science

In the European H2020 funded RECEIPT (REmote Climate Effects and their Impact on European sustainability, Policy and Trade) project, we use storytelling as a framework for scientific discovery, modeling and communication. Many complex interactions between climate change in remote (non-EU) areas of the world and their impact on the EU cannot be easily communicated to policy makers and the public. Yet, a drought in South America or a flood in Asia can have major impacts on the European economy, or on the food supply that reaches our tables. The storytelling application we are developing allows the RECEIPT scientists to create storylines that link map locations and map-based data (displayed with TerriaJS) with text, images and custom graphs (created with Vega).

In the RECEIPT climate impact story application, we are using the drill-down-story structure. We Provide the common theme of climate changes in remote areas with impact on the EU, but we are letting the user decide what areas of the globe and/or what sectors they are most interested in, be that Agriculture, Finance, International Cooperation, Manufacturing or Coastal Infrastructure. This should provide an in* for many different users, hopefully educating many potential users about the possible impacts of climate change.

A screenshot from the RECEIPT prototype visual storytelling interface.

Archeology and Art History

In the Via Appia: Revisited project, we are developing a storytelling application for 3d pointclouds (using potree.js). The goals of the project are to show the history of art and the (virtual) archeological reconstruction of monuments on the Via Appia Antica, an ancient Roman highway near Rome itself.

We will make two versions of this application with different storyline structures. The first instance will be a museum exhibit, where users will interact with physical *forward and back *buttons to interact with the application. We’ve chosen the interactive slideshow approach for the story structure to simplify the user interface and make sure nothing can go wrong, as well as to allow our authors (the researchers and artists responsible for the exhibit) to carefully craft the narratives.

The second instance of the application will be a public website, where users from the web can explore the Via Appia on their own. To guide the users towards interesting discoveries, we will adopt the martini-glass structure here. First, we will offer the same storylines as in the museum exhibit, but after these storylines have finished, we open up the interactivity and will let users explore the site on their own.

An screenshot of the Via Appia: Revisited (early alpha) application

Medical Imaging

Vistories / CLUE — Gapminder visualizationAnother application area for visual storytelling we explored is medical imaging. The goal of this project was to see if we could improve the communication and support the building of trust between medical experts using Visual Storytelling techniques.

Our starting point was Vistories / CLUE, a library that uses interaction provenance as a basis for the authoring of Visual Storylines. The idea was that there could be an opportunity to increase the level of trust between a decision maker (the doctor planning treatment) and an expert doing analysis (a radiologist in the lab) by allowing the analysing expert to author a report directly from the tools used to analyse the images, with preservation of the links back to the data. This could then allow the decision maker to directly access the original analysis, and even directly try out alternate hypotheses or measurements. In this application, the martini-glass structure is also used to guide the users at first, but allow more freedom afterwards.

In the sister-blog to this one, we’d like to tell you more about the use of *provenance *for these (and other) purposes. Please read on here.

In conclusion

Combining data, narratives and visuals can bring powerful messages forward in a way that fact, numbers and data alone can not. Visual stories allow us to be more inclusive, to connect better to our audience, and most importantly, to be more memorable.

At the eScience Center, we do really like to be remembered by both the scientists that we work with, and by their intended audience as well. Visual storytelling is one of the ways in which we try to achieve this.

#KonMari for Science? Tidying up Provenance…

Adriënne Mendrik — Mon, 14 Jun 2021 00:00:00 GMT

The term provenance mostly originates from the art world, where it aids in authenticating historical objects.

“The primary purpose of tracing the provenance of an object or entity is normally to provide contextual and circumstantial evidence for its original production or discovery, by establishing, as far as practicable, its later history, especially the sequences of its formal ownership, custody and places of storage. The practice has a particular value in helping authenticate objects.” (Wikipedia)

The potential of provenance tracking as-standard has been identified by people far outside the art world, particularly by scientists. After all, tracing the source and transformations that led to a particular piece of data has clear implications for the reproduction of scientific results that sits at the heart of the scientific method. While the experimentalist’s trusty lab book has been the primary means of recording such information, the increasing use of computers in data generation and analysis has provided a natural drive towards more automated means of keeping track.

This is great, because if keeping track of the provenance can be automated, we can record everything, right? Well, maybe not everything, but a lot indeed. For example, the provenance in a Jupyter Notebook (ProvBook), the provenance in visual exploration (CLUE), and workflow provenance (FAIR Computational Workflows). However, provenance is always stored for a purpose, even if the purpose is that it might be useful later on. Quite similar actually to collecting things at home.

Your house might be filled with stuff that you regularly use, sometimes use, think is beautiful, has emotional value, might come in handy someday, or just haven’t had the time to throw out yet. The thing is, if you collect too much stuff at home you run the risk of not knowing what you have in your home anymore or where you stored it. By the time you need the thing that might come in handy someday, you cannot seem to find it or it turns out to be so outdated that you need to buy a new one.

Something similar might happen with saving data and provenance. Saving data does not necessarily mean that it will be re-used later on. The more we save, the less overview we have of what is really important or valuable. This overview could be provided by using machine learning to dig through the data, but this is not magic either. Exhaust data can feel like a messy attic, there might be something valuable there, but machine learning requires the tedious work of cleaning and processing the data, before you might get something out of it.

In this blog we share some lessons learned about provenance in science and philosophize on how Marie Kondo, a world-renowned tidying expert, could teach us some lessons that might be valuable for science as well.

Finding something valuable in here will take some time… (Photo by Cristina Gottardi on Unsplash)

Some people have the natural tendency to tidy up things, whereas others feel like children whose parents tell them to tidy up their rooms, while they actually want to go play outside.

A way to tidy up automatically tracked provenance is by providing tools that enable the presentation of provenance for easy human assessment. An example of this is ProvBook, which can automatically track the provenance in a Jupyter Notebook and enables sharing and easy viewing of the cell history in a notebook (saved in the REPRODUCE-ME ontology).

ProvBook: Provenance of the Notebook

ProvBook: Provenance of the Notebook.This is the demo video for the introduction of ProvBook. ProvBook is an extension…

figshare.com

Gratzl et al. take this idea of presenting provenance tracking one step further for visual story telling.

If you like, you can try it out here.

In the exploration mode, for example while exploring data in a visualization tool, the provenance (user actions) is automatically tracked. Then in the authoring mode, the provenance is presented to the person who performed the exploration, who is given the option to create a visual story from it. This story can then be used to communicate to others how you gained insight while exploring the visualization, by using the presentation mode.

From Visual Exploration to Storytelling and Back Again (S. Gratzl et al, 2016)

From the presentation mode you can switch back to the exploration mode and start a new branch of explorations from the last or any of the previous time points in the tracked provenance. Theoretically this sounds great, but in practice it is quite hard to make a tool that presents the automatically tracked provenance in an easy and efficient human readable format. So the question is whether the value outweighs the work that needs to be done either by the person viewing the exploration or by the person creating the visual story.

In terms of tidying up provenance, the realm of computational workflows deserves a lot of credit as well, where concerted efforts in standardization produced the now ubiquitous W3C PROV model and its family of ontologies. A key development in this field is the strict separation of the recorded workflow execution (retrospective provenance) from the workflow specification (prospective provenance). The prospective provenance is the high level description of the scientific procedure, whereas the retrospective provenance is the ‘exhaust’ collection of all fine-grained execution details we mentioned earlier. The creation of prospective provenance can be viewed as an attempt to distill the scientifically crucial and useful “story” out, leaving the remaining residue (retrospective provenance) that is more specific and cluttered.

Prospective versus retrospective provenance? (Photos by Bernard Hermant and Sammy Williams on Unsplash )

But where do you store all this provenance once you have it? It could of course just lie around on your hard drive or local cluster but if you want to participate in open science then it would be good to publish it. In the semantic web/linked data space there are a number of ways available, such as packaging it up in a Research Object Crate and publishing it e.g. to the workflowhub registry, or by publishing the RDF description of your workflow as one or more nanopublications.

These are great efforts, but what if you are one of those people that would rather play outside then tidy up your room? Funding agencies can of course play the role of the parent, demanding that research objects and provenance is being saved for possible re-use, adding another checkbox to the long list of boxes that researchers already need to check. The question is, does this really work? Or do we end up with something that seems neat on the outside, but is more like children stuffing their toys anywhere in boxes in their rooms, just to make it look tidy? While underneath the surface, the boxes are a colorful mixture of all sorts of toys in which it is hard to find that specific toy that they must have later on…

(Photo by Nareeta Martin on Unsplash)

In her book “Spark Joy”, Marie Kondo mentions that most people tidy up by moving stuff from one room to another, occasionally throwing some things away. This does not seem to lead anywhere, since there is no system behind it. Her method is much more thorough and requires people to make choices. There is no gray area, you either keep something and cherish it or throw it away. The trick is to focus on the things you would like to keep, the rest can go. The way you decide on what to keep is to learn what sparks joy inside you. You go through everything you collected in your house, based on categories, ordered in a way that was designed to slowly teach you what sparks joy inside you.

What is key here, is that for people following this method, tidying up changes from a painful process of saying goodbye, to an inner journey discovering who they are, what they like and what fits them. This does not only provide an inner drive to tidy up their house, but changes their mindset with respect to future decisions on what to buy. Knowing who they are and having overview in their home, they think twice before buying something new, because they cherish what they have in their home. In a way, it provides a sense of direction and relieves them from the fear of missing out.

Is there a lesson to learn here for science? Well, maybe there is.

First of all, in the past couple of decennia, the focus in science has been more on quantity then on quality, pushed by funding agencies counting the number of publications as a metric for good science. As a result, a lot of publications were written and scientific literature to date is like a home so full of stuff that even scientists themselves have trouble keeping the overview and determining the current state-of-the-art in human knowledge. Is the solution then changing the metrics? Or adding more metrics? Or will this mostly lead to a different form of consumerism?

Goodhart’s Law Explained ( Source )

Maybe a more thorough approach is required, making clear choices on what to keep and get rid of the rest. But who needs to make this decision and based on what?

What the #KonMari method does, is teach people how to make choices that are in line with their “inner joy”, so people themselves are able to make the choices that fit best. In translating the work of Kondo, another phrase was considered as the central question. Instead of “does it spark joy?”, the phrase “does it speak to your heart?” was considered as a translation for tokimeku, which can be understood as “how does it move the kokoro (heart / mind)?” (Tokimeku: The Poetics of Marie Kondo’s KonMari Method).

(Photo by Aziz Acharki on Unsplash)

Well, this is not something we can use in science, because science is all about reasoning and has nothing to do with the heart or mind… Yes, that is one way to look at it, but if you have read the book “ Hare brain, Tortoise Mind ”, you might want to reconsider this. The quick thinking hare brain is what we would use for reasoning, whereas the tortoise mind could be described as intuition, the inner drive, sudden inspiration, or a spark of creativity. Both are important, but if you think about it, most of the big leaps in science came from people with an inner drive, following their intuition, not seldom against the main stream in academia.

So, to recap, what can we learn from all of this with respect to provenance? Maybe, that we should be careful that saving or publishing data and provenance does not become another box to check for already overloaded researchers. Maybe, the less is more approach of Marie Kondo could spark a little joy in the lives of researchers, and free up some time to learn how to use their tortoise mind. Provenance should be saved if there is value in saving it, and if we save it, we should cherish it for human knowledge, communicate it clearly and make sure people can use it. Either for reproducibility purposes or communicating findings or insights, such as in visual story telling. What is required is a mindset change from consumerism to value (that which moves the kokoro) for science, and methods and tools that aid this.

(Photo by Fahrul Azmi on Unsplash)

With thanks to Jiri Kosinka, Carlos Martinez-Ortiz, and Lourens Veen.

(Nano)Publish your research with Python

Robin Richardson — Mon, 31 May 2021 00:00:00 GMT

Using nanopub, a high-level, user-friendly python interface to the nanopublication network

Subscribe*Remember me for faster sign in

For example, if you want to e.g. search for all nanopublications containing the text Picoides*:

which returns two results both pointing to the example nanopublication we just discussed:

{‘np’: ‘http://purl.org/np/RAMzGwLotMRSQTHRCS15B6hvuYSyEupGnxZaFU3EmcItA’, ‘description’: ‘Inter-species interaction of Picoides villosus with Ips’, ‘date’: ‘2020-12-24T10:51:43.931Z’} {‘np’: ‘http://purl.org/np/RAMzGwLotMRSQTHRCS15B6hvuYSyEupGnxZaFU3EmcItA’, ‘description’: ‘Picoides villosus’, ‘date’: ‘2020-12-24T10:51:43.931Z’}You can then fetch a specific nanopublication directly using its URI:

This snippet will fetch the nanopublication we discussed earlier, about the interaction of the bird and the beetle, and print the contents of its assertion. Of course, you are not limited to simple text searches. The library has several search methods, including finding nanopubs with a given triple pattern — you can find detailed documentation here.

Hold on, I want to publish Nanopublications of my own!

To publish to the nanopub server you need to set up your profile. This allows the nanopub server to identify you. Run the following interactive command (on the command line):

setup_nanopub_profileIt will add and store RSA keys to sign your nanopublications, publish a nanopublication with your name and ORCID iD to declare that you are using using these RSA keys, and store your ORCID iD to automatically add as author to the provenance of any nanopublication you will publish using this library.

You can then publish a quick claim:

Published to http://purl.org/np/RA47eJP2UBJCWuJ324c6Qw0OwtCb8wCrprwSk39am7xckView the resulting nanopublication here. Note that the URI of your nanopublication is a signed hash of its contents, making its authorship verifiable and enforcing its immutability — known as a trusty URI.

Or, to leverage the true power of semantic technologies, you can build your own RDF graph of triples and publish that:

The above builds a graph containing a single triple that states (essentially) the concept pointed to by www.example.org/timbernersleeis of type Person. We use the rdflib library to build the graph, but this is already a dependency of nanopub. A Publication object is then created, using that graph as its assertion, and finally published using the NanopubClient as before. The code produces the following output:

Published to http://purl.org/np/RAfk_zBYDerxd6ipfv8fAcQHEzgZcVylMTEkiLlMzsgwQYou can view the resulting nanopublication here.

Outlook

nanopub makes interacting with nanopublications quite intuitive for those with sufficient RDF and python skills. For all others we foresee tools built around nanopub that make it intuitive for domain-experts (but not RDF-experts) to make use of the nanopublication network.

There are a couple of those already in the making:

nanotate: Create nanopublications from annotations in PDF-files made with hypothes.is
fairworkflows: Support the construction, manipulation and publishing of FAIR scientific workflows using semantic technologies. This is developed as part of the wider FAIR is as FAIR does project at the eScience Center.

Conclusion

The python nanopub library provides a high-level, user-friendly python interface for the nanopub server, making it easy to publish and search small scientific publications. We created this library to bring nanopublishing to python users and we’re keen to help people make use of it.

Consider how nanopublications might fit in with your field, and feel free to try the library out!

Acknowledgements

The quality of this text was greatly improved by the suggestions of Patrick Bos, Tobias Kuhn, Arnold Kuzniar, Lars Ridder, Pablo Rodríguez-Sánchez, and Stefan Verhoeven.

Lesson development blitz

Fakhereh (Sarah) Alidoost — Tue, 11 May 2021 00:00:00 GMT

**SubscribeRemember me for faster sign in

Testing lessons**

designed by freepik https://stories.freepik.com/The coordinator asks some of the researchers in the community to use the tutorial and to report any issues or bugs. It is also recommended to organize training activities using the tutorial and gather feedback from participants. The outputs of the evaluation are:

a list of issues and bugs.
whether the tutorial needs polishing (go to step 4) or is ready for a release (go to step 6).

6. Release

designed by freepik https://stories.freepik.com/The tutorial is released on GitHub in a beta version and published on Zenodo. With this version, the lessons can be used for broader teaching and contributions.

At the end of the blitz …

Our development blitz took about one week and the ESMValTool tutorial was released in Beta-version.

Like every procedure, a development blitz has some pros and cons. A development blitz increases focus, creates satisfaction and good feeling, helps communication, and spreads best practices in the community. However, it may lead to disappointment, needs lots of commitment and slows down decision making.

Interesting readings:

The Carpentries Curriculum Development Handbook provides guides on designing and developing:

different parts of a tutorial, see episodes.
objectives and scope, see developing content.
development and release stages, see lesson life-cycle.

Here, some blog posts about blitz planning that might be of interest:

Acknowledgment

Thanks to [Carlos Martinez-Ortiz], [Felipe], [Pablo Rodríguez-Sánchez] and [Tom Bakker] for their feedback.

Using Rust for Scientific Numerical applications: Learning from Past Experiences

Felipe — Wed, 14 Apr 2021 00:00:00 GMT

Every experienced programmer knows that both their productivity depends on the abstraction level of the language. The fewer lines of code you write, the fewer bugs you introduce.* However, an increase in abstraction often results in a higher runtime cost (code that needs to run extremely fast is often very ugly). Rust zero-cost abstraction allows you to write more concise code by using a higher level abstraction without additional computing cost at runtime. *Rust *iterators are a great example of the power of Rust zero cost abstraction. It is fair to mention that zero-cost abstraction is also central in C++.

How much effort is required to maintain the code?**

If you have ever worked in a medium to large size C/Fortran code base, you certainly know how incredibly difficult and frustrating it can be to maintain it. A recurrent complaint among the programmers in these languages is the dreadful bugs related to unsafe memory management that can take days to trace and reproduce. Fortran programmers are famously known for having segmentation faults for breakfast.

The Rust type system stands out for its capabilities to rule out memory errors at runtime. The Rust borrow checker is the killer feature that helps to eliminate all those memory bugs while still offering lightning-fast speed.

Also, as mentioned previously, Rust’s zero cost abstraction allows you to keep a lean code base that is easier to maintain.

***Note for C++ developer: ***smart pointers partly alleviate the memory management issues, but the borrow checker can help you to extend the safety guarantees to multithreading code.

How much effort is required to test the code and write documentation?

Writing documentation in C/C++/Fortran involved bringing a third-party tool like Doxygen that we need to install and add to our CMake zoo. Also, we need to learn this tool’s special syntax to write documentation and then pray that the documentation builds.

Software documentation is essential for scientific code due to the volatile nature of scientific research. *Scientific software without documentation is not legacy code but dead code. *Given the high barrier imposed by traditional languages to write documentation, it is expected that most scientific software is stillborn due to the impossibility to understand what has been done, even by experts in the same field (or sometimes even by the person who wrote it).

Documenting a Rust project only requires that you write the documentation in markdown inside the source code as shown in this example. Then you just need to run the cargo doc command and that is it!

Testing in C/C++/Fortran has a similar fate, they required third-party frameworks that need to be installed and added to CMake. Fortran is particularly painful for testing due to the lack of a standard testing framework, forcing programmers to maintain a bunch of scripts to call the binaries, parse the output, and check the results.

Rust has a built-in system to test your code, with no third-party libraries. You can have unit tests to check the functionality of a given module at a time, but also integration tests to check the public interface of your code. You can even run and test the examples in the documentation! In summary, writing and running tests is as effortless as you can get it.

Lowering the barrier to write tests and documentation is an undervalued feature of Rust. I bet that we all agree that code without tests and documentation is short of useless.

How do I parallelize the application?

Image: courtesy of Chiara CaratelliDisclaimer*:* due to my ignorance of the latest Rust developments on multi-node computing (MPI) and GPU integration, I am going to blatantly ignore those two subjects.

Scientific simulations like weather prediction, protein binding, fluid dynamics, etc*.* are computationally intensive but often parallelizable (at least on paper!). It goes without saying that we want to make use of all cores available in a given machine. The standard approach is to use something like OpenMP that consists of runtime libraries, compiler directives, etc. to support shared-memory multithreading programming.

A quite dreaded moment for scientific software developers is when the serial implementation is working and a new parallel version must be implemented. Seriously, parallel implementation in C/C+/Fortran means that all your unknown unknowns about unsafe memory management suddenly uncover all the obnoxious bugs that you didn’t know about until now, and that you need to track for endless hours.

As a remedy for all that frustration and wasted time, Rust offers a novel approach coined as fearless concurrency. This concept refers to the possibility of writing parallel applications that are free of subtle bugs and can be refactored without introducing new bugs.

But how does Rust achieve this marvelous formula? It turns out that the Rust type system and ownership system keep track of what is safe to share across threads, refusing to compile illegal concurrent memory transactions that would have resulted in runtime issues. So, the Rust compiler happily raises compilation errors whenever you are trying to use memory in an unsafe way instead of unleashing Godzilla in the middle of your simulation.

Photo by Markus Winkler on Unsplash

Since Rust is a system programming language, rather than choosing a single parallelism model, it allows multiple low level models like message-passing, share-state, etc. The good news is that we don’t need to use the primitives ourselves, instead, we can use the community-provided libraries like Rayon that build on top of those primitives. Rayon offers powerful functionality like parallel iterators that allow us to execute operations on the elements of an iterator in parallel, with minimal changes in the source code.

What is even better, you can use libraries like ndarray that has a Numpy-style API to manipulate arrays, while simultaneously offering features like an interface to Rayon to run your array operations in parallel.

The Challenges of using Rust for scientific applications

Rust is a promising language for scientific applications but there are several challenges that need to be overcome before the language can gain significant traction by the scientific community.

Rust is a low-level language with many powerful features. In other words, actually you do not learn Rust during a Saturday afternoon while drinking mojitos. It takes significant effort and time before you can start writing with confidence. Fortunately the Rust community is very open and welcoming and there are always people willing to help. Besides, the compiler has the most informative error messages that I have seen in any programming language. In short, the learning curve is steeper than Python, but you will have a single memory-safe language to rule them all.
There may be some functionality that is still missing or unstable. The Rust ecosystem is growing rapidly and more people are coming out with great libraries. The community is always happy to help you to come up with a solution for your missing functionality.
Interoperability with C/C++. We certainly do not want to rewrite everything from scratch, therefore we would like to reuse as much code as we can from C/C++. For the Rust community, smooth interoperability with C/C++ is a top priority.

Any further thoughts?

I hope that I could give you an idea about using Rust for scientific software applications. Comments and thoughts are appreciated.

Acknowledgement

Thanks to Chiara Caratelli from providing the great drawings. Also my special thanks to [Florian Huber], [Carlos Martinez-Ortiz,] [Patrick Bos] and Tom Bakker for their help editing the text.

Can mathematical models predict the future?

Pablo Rodríguez-Sánchez — Thu, 18 Mar 2021 00:00:00 GMT

Photo by John Moeses Bauan on Unsplash

Let me start right away with a question: do you think that the future can be predicted? When formulated like this, this question makes us think of crystal balls, palm reading, fortune tellers, … and ultimately invites us to answer with a loud and clear: “no”.

Nevertheless, every day we read things like “the average mean temperature will rise 0.5 ⁰C worldwide in the next ten years”, or “the contagion curve of COVID-19 will start to flatten in the next 3 days”. And we read themwithout raising a single eyebrow. Indeed, these sentences evoke in us feelings of rigor and expert knowledge. But, are they not predictions about the future?

If we dive into the foundations of this kind of forecasts, chances are that they had been obtained using mathematical models. Most likely, they will have error bars attached to them, and provide numerous caveats that, somehow, subtract some strength from the conclusions.

Mathematical models are getting more attention than ever. The COVID-19 pandemic made them jump from the inner pages of specialized journals to the front pages of press and TV news. For audience and format reasons, the necessary details to understand the reach and limitations of these models are rarely mentioned. This may lead to a wrong impression about their power.

What is a mathematical model?

==A model is a description of a phenomenon under study. A mathematical model just uses mathematical language in this description.==

It sounds almost like a tongue-twister, but is actually anything but exotic. Even more, it is very likely you have worked with mathematical models before. Particularly in school. Do you remember those problems about compound interest? Or those about the movement of a projectile in physics class? Did your teachers not ask you to calculate the future evolution of a bank account or the time and place of the projectile’s impact?

Contrary to what is often thought, mathematical models do not answer the question “What will happen?”, but to a subtly different one: “What would happen if…?”. This constitutes, simultaneously, the strength and the weakness of mathematical modeling.

Let me illustrate my point with the example of a body in free fall. Perhaps you remember your own skepticism when, in school, your physics teacher told you that a feather and a cannonball free-falling from the same height touch the ground at the very same moment.

Sounds strange, but it is rigorously true. In fact, it is so rigorously true, that if we just drop the word “free” from “free-falling”, the statement is not true anymore. Free falls require the absence of an atmosphere, and thus do not happen often in our daily life. Still skeptical? Check the video below:

https://www.youtube.com/watch?v=KDp1tiUsZw8

If we try to apply the model of free-fall to the fall of an everyday object, we’ll soon see that its prediction fails miserably. We just used an inadequate model, because the free-fall model answers a question we don’t need to answer: “what would happen if I let this object fall in the absence of an atmosphere?”.

When we mathematicians build our models, we often start with a very simple one, and we keep adding features to it while we need them. When do we stop, then? When the model is sufficiently good. That is, when we are happy about the light it sheds onto the phenomenon under study.

If we are interested in understanding the fall of a feather, the above-mentioned free-fall model will certainly not make us happy. If we add some drag terms to the model, accounting for the presence of an atmosphere, the results will be much better (although still not perfect, because there will always be small discrepancies between the predictions and the experiment).

All models, no exception, are more or less sophisticated approximations of reality. But they are never perfect. Unfortunately, it is not always easy to quantify the precision of a mathematical model. As a consequence, it is also hard to communicate its limitations to a general audience.

Why do we use them?

If all mathematical models are approximations, why do we use them? There is a simple answer to this: we use them because they are useful. Models’ capacity of answering the question: “What would happen if…”, makes them an excellent substitute for scientific experiments.

In science, the experiment is the highest authority, but sometimes we have no choice but to not perform them. We either use a computational/mathematical model, or we are stuck. Some examples of this situation are:

Unfeasible experiments: such as studying a black hole on site.
Costly or very hard experiments: such as studying plankton populations, a problem whose scale in space and time is, literally, oceanic and of the order of decades.
Destructive or dangerous experiments: such as studying the effects of a earthquake on a city.

Models are also very useful to obtain visualizations that, otherwise, will be difficult or impossible to obtain experimentally. Think for instance of the movement of the wind around a turbine, with the pressure painted as color.

Models also help us understanding. Even models with a very humble predictive power can be very useful to understand complex problems. For instance, thanks to a very simple model of epidemic propagation from the 1920s we know that there are thresholds where the contagion becomes explosive. The model is useless if we want to know where exactly that threshold is, but it provides the very idea that such a threshold exists.

Mathematicians cannot predict the future

Our initial question was: can the future be forecasted? No sensible person will ever answer “yes, of course, always”. I hope to have convinced you that mathematicians, who most of the time are also sensible persons, neither will answer so.

Science is, by its own nature, a fundamentally incomplete enterprise. This is true whether we use letters or formulas to write it. Mathematical models have to be consumed with no less than the same healthy skepticism we use with the weather forecast.

Acknowledgments

The quality of this text was greatly improved by the suggestions of Patrick Bos, Lourens Veen, Florian Huber.

This article is an adapted translation from an article the same author published in The Conversation under a Creative Commons license. If you can (and want to) read Spanish, visit the original article.

Automate chores with GitHub Actions

Patrick Bos — Fri, 05 Mar 2021 00:00:00 GMT

How I keep my Heroku web-app’s data up-to-date automatically

Scientists, open source developers, and basically probably most people, have to deal with one common obstacle in life: lack of funds to do all the stuff.

In particular in science and open source software, we cannot be bothered to maintain the stuff we left behind (at least, not on the boss’s time). It’s a never-ending quest towards novelty. People just tend to be more forthcoming in the wallet area when fancy new features or knowledge are promised.

This is why it is vital that we automate the crap out of everything we humanly can, especially if what we built is some kind of service.

A few months ago, I launched a small web-app called Coalitiewijzer. It uses data from Peilingwijzer (which combines major political polls into one) that is updated in irregular intervals. So you guessed it: we’re going to automate the ingestion of that data into the Coalitiewijzer.

In this short post, I’ll show you how I did this using GitHub Actions. GitHub Actions is a Continuous Integration (CI) provider that (obviously) integrates well with GitHub. If you want to know more about CI in general, I recommend checking out our Guide. I’ll assume some working knowledge on this and also on Git, GitHub and GitHub Actions themselves. For more background info about the app itself, check out my other blog post. This article is basically an add-on to that one. To sum up this article, we’ll look at four steps:

Set up a basic GitHub Actions workflow that runs daily
Update the data and commit to your Git repo
Push this from a GitHub Actions runner back to the GitHub repo
Deploy your app from GitHub Actions to Heroku

If you just want to take a shortcut straight to the code: here is the current GitHub Actions workflow file I wrote. For those who like to read poor prose instead: read on!

Daily running workflow

To start a new GitHub Actions workflow file, GitHub offers a convenient “New workflow” button under the Actions tab in your repo. You can also copy-paste the basic example in the Actions Quickstart guide.

These basic workflows typically only trigger on a push to your GitHub repo, as specified by the on: push line. We’ll want to change that into something like this:

on:
  schedule:
    - cron:  '0 17 * * *'  # 17:00 UTC
  workflow_dispatch:

The weird string between single quotes is cron syntax, which is explained in the docs. It allows you to specify at which times you want the workflow to run, in this case at 17:00. The timezone is UTC, keep that in mind.

The last line is actually unrelated, but quite useful nonetheless. It allows you to manually start the workflow from the Actions page, which I found to be essential wihle testing my workflow.

Updating the data

My app uses data from Peilingwijzer, which I preprocess for use in my app and then store in a pickle file. For details on this, see the update script on GitHub, or check out the blog post about the app. The pickle file has to be committed to the repo and then I can deploy to Heroku with the new data.

This can all be done with a shell script that looks roughly like this:

#!/usr/bin/env sh
set -e                    # make the script end on errors
python update_numbers.py  # download new data and dump to files
git add [the updated files]
git commit -m "updated data"

The Python script does the actual data downloading and mangling and dumps the results into files that the web-app uses. These are then committed to the repo.

To run this script (in particular the git commit), you do need to configure the git client on the GitHub Actions runner. You can do this with a workflow step at any time before running the commit command. Add this to your workflow job in the yml file:

- name: configure git user
  run: |
    git config --local user.email "${{ github.actor }}@users.noreply.github.com"
    git config --local user.name "${{ github.actor }}"

This commits under your own GitHub user name (or, in general, the username of the one that triggers the workflow to run), so it’ll identify the commits to be yours. It won’t do certification unless you also somehow configure that (perhaps using GitHub secrets, see the Heroku section below). You can fill in any name you want, though, so you could also put something like “Actions bot”.

Push back to your GitHub repo

One of the beauties of GitHub Actions is the great availability of … well, Actions. These are pre-baked recipes for steps that you can very conveniently plug and play. To be precise, you can add such a step to your job, which is part of your Actions pipeline.

For the next step, I needed to push the commit made above into the GitHub repo from within the GitHub (Actions) infrastructure itself. I was a bit anxious about whether I would find an Action which allowed me to do this. It feels like a potential security leak. Luckily, it turns out there is already an Action for this very task. Hurray! The Action’s source can easily be checked if you’re worried about security.

Just add the following step to your job after running the update script:

- name: Push changes
  uses: ad-m/github-push-action@v0.6.0
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    branch: ${{ github.ref }}

For this to work, one more step is necessary. You also need to add two options to your checkout step (at least, this is what I figure from the Action docs):

- uses: actions/checkout@v2
  with:
    persist-credentials: false
    fetch-depth: 0

That’s it!

Deploying to Heroku

Finally, after the data has been updated in the Git repo, the Coalitiewijzer app can be deployed back onto Heroku (which runs the app, live here). Heroku deployment can be done by just pushing your repo directly to the Heroku servers. Really convenient!

So, since this is again just a git push, albeit to a different repo, this should not be a problem. And, indeed, a Deploy to Heroku Action already exists for this very purpose!

To make this work, you need to add your Heroku API key to your GitHub repo’s secrets (instructions here). You can find the Heroku API key in your Heroku account. Call it HEROKU_API_KEY if you want the snippet below to work.

After that, you just add the action to your workflow job, like described in the action’s documentation (replace the boldfaced parts):

- uses: akhileshns/heroku-deploy@v3.10.9 # This is the action
  with:
      heroku_api_key: ${{secrets.HEROKU_API_KEY}}
      heroku_app_name: your_app_name_on_heroku
      heroku_email: your@email.address

Be careful with this key, because it gives access to your full Heroku account.

And that’s it! From now on, your updates will be pushed back to the GitHub repo and deployed on Heroku.

Of course, this is just one of many things you could automate using GitHub Actions. The combination with running actual servers on Heroku for free is icing on the cake. Of course, if you want more performance, more MBs, more connections, etcetera, you can pay them for that. But for low profile web-apps, the free package is really a great deal.

Let me know if you have any comments or questions. I’m happy to help!

A Quick and Dirty Pytest Cookbook

Felipe — Mon, 22 Feb 2021 00:00:00 GMT

Subscribe*Remember me for faster sign in

As an alternative to the above snippet, you can install and run the pytest-mypy-plugins.

Testing the documentation

Testing the documentation is a must have. The following snippet shows how you can do so:

Testing async code

If you do web applications or something similar you probably have run into Python asyncio. If you try to run async code using normal pytest code, chances are that you wouldn’t notice that there are no tests running at all. Async code is lazy and requires that you run it inside an async block*, otherwise nothing interesting happens. The following snippet shows how to invoke async code in Pytest:

For more details about testing asynchronous code in python check pytest-asyncio.

Running tests in parallel

If you, like me, have the tendency to procrastinate while waiting for the test suite to finish executing, then a good way to reduce the procrastination time is to run the tests in parallel.

For running your tests in parallel, you just need to install the pytest-parallel library that not only allows you to run your tests in parallel but also in a thread-safe manner, using commands as simple as:

pytest —workers 2

Running a test with multiple parameters

Imaging that you have a simulation that receives some parameters as input and gives you back some numerical output. You can use pytest’s parametrize extension to feed one parameter at a time to the simulation and check that each one of them returns the expected output. The following snippet shows how to accomplish that:

If you run the previous snippet, you should see something like:

pytest -v test_parameters.py test_parameters.py::test_simulation[pi-3.141592653589793] PASSED
test_parameters.py::test_simulation[exp-2.718281828459045] FAILEDparameter = ‘exp’, expected = 2.718281828459045@pytest.mark.parametrize(“parameter, expected”, [(“pi”, np.pi), (“exp”, np.exp(1))]) def test_simulation(parameter: str, expected: float) -> None: """Check the simulation.""" result = run_simulation(parameter) > assert abs(result - expected) < 1e-8 E assert 39.28171817154095 < 1e-08 E + where 39.28171817154095 = abs((42 - 2.718281828459045))test_parameter.py:16: AssertionError

Conclusions

Pytest is a flexible tool with a great number of useful extensions. With Pytest, there is always a way to check your code functionality, even those annoying corner cases.

Please comment if you find these tricks useful or if you find another nice trick that you want to share.

Acknowledgement

My special gratitude to Bas van Beek to share with me some of his useful recipes. I would like also to thank [Florian Huber] and Stefan Verhoeven for their feedback.

Thanks to [Pablo Rodríguez-Sánchez] and Steven Roldan for their help editing the text.

Twitter Bots, for Science!

Patrick Bos — Fri, 19 Feb 2021 00:00:00 GMT

How to build a bot to show Covid-19 vaccination progress

Twitter bots have a really bad rep. Troll armies, spam, fake news… You name it, a bot’s done it.

But what if we could harness their powers for good? Use bots to inform people in a fun way? That would be science outreach! It’s all in a name.

The knights who say “SCI!” ( Python et al., 1975 ). Photo: CC BY 2.0, Brickset

Of course, Covid-19 is on everybody’s mind, but the tide of horribility (it’s a word) is slowly turning into one of hope and recovery. We’re not out of the woods yet, but we are jabbing our way out, one vaccine at a time.

To spread a bit of this hope (and have some fun), I created a progress bar (who doesn’t love progress bars?!) on Twitter that shows how many people have been vaccinated worldwide and per continent. The bars are updated daily.

The code is open source and the open data comes from Our World in Data. If you feel inspired by this little project to share some of your scientific data, feel free to build on it! I had my first version up in about an hour: it’s really surprisingly easy, so don’t let technical barriers hold you back from communicating cool science results automatically!

The requirements for reading this howto are a working knowledge of Python, Pandas, Git, GitHub and Twitter and a high tolerance for painfully horrible jokes. For the working knowledge part: check out our Guide. For the jokes, well… maybe reading this article will actually build up your tolerance.

I’ll cover the five steps I took to build this bot:

Create a “bot account” on Twitter
Get and mangle the data from Our World in Data with the Python Pandas package
Use tqdm to make progress bars suitable for Twitter
Send tweets using the Python Tweepy package
Automate this using GitHub Actions

Create a “bot” account

The first thing you need is a developer account on Twitter. This allows you to get key strings which you can use later on to log in and do stuff on Twitter from a Python script.

This post (among many others) clearly explains the steps you need to take for this.

It is possible to use your personal account for your bot (and for activating the developer account on), but for this project I decided to create a separate account. You’ll need a unique email address to do so, but you can use the same telephone number as your existing account for confirmation. A separate account allows for easy testing of your bot before you start promoting it widely. Just delete the test tweets when everything is done.

So, with your bot account ready, jot down the keys somewhere — but don’t put them in your git repo (like I obviously did 😁) — so we can use them in Tweepy later on.

Get the data with Pandas

One of the authoritative sources on Covid-19 data is Our World in Data. You can find their graphs all over the place. The data they gather is conveniently open (under a Creative Commons license) and available on GitHub.

Grizzlies would be a better name… Image: CC BY-NC-ND 2.0 iregretjumping.

Pandas, the ironically named omnivore of data science, can chew this right up, straight from the raw URL. This saves you the step of manually downloading. Convenient! Look for the little “raw” button when viewing a file on GitHub and give that URL to your favorite ursid:

import pandas as pd
df = pd.read_csv("https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv", parse_dates=["date"])

This gives us a nice dataframe containing which countries had what number of people vaccinated at which dates. They even keep track of whether people had only one or two (where applicable) shots. Not every country has good data, so Our World in Data only includes those numbers which pass their quality checks.

For the time being, I’m using the column people_vaccinated_per_hundred for the bot, because those numbers are a bit less depressing at the moment, but may switch to people_fully_vaccinated_per_hundred later on. Since not all countries have data in these columns, this leaves us with a little over 50 countries after filtering.

Calculate continent totals with pycountry_convert

Now, the table has numbers for the whole world pre-calculated, so I grabbed those for my first tests. But I thought it would be fun to also add some more detailed data.

Twitter’s character limit prevents me from posting bars for all 56 countries, but we can certainly do continents. I think that’s still interesting, as well as poignant given the inequality in the world it so very clearly highlights. If the stark contrast makes policy makers think about fairer distribution for a second, that would be a huge win.

Of course, finding the continents for 56 countries by hand goes against everything I stand for as a lazy programmer. Obviously, I went about spending way more time on finding and figuring out a nice library that can do this work for me automatically. And, lo and behold, it exists, is called pycountry_convert and works like a charm!

Combine it with the total population table that Our World in Data also maintains and you’ve got all the ingredients you need to compute percentages per continent.

Note that the continent numbers are not completely correct like this. Some countries span multiple continents (notably Russia). I treat such countries to just be part of one continent (so Russia is part of Europe, because pycountry_convert tells me it is).

Make Tweetable progress bars with tqdm

Now that we have percentages, we can convert them to progress bars.

My go-to package for progress bars is tqdm. It is especially good in interactive environments. There’s nothing like a progress bar to spice up a slow for -loop. What’s more, tqdm ’s bars look really good both in Jupyter notebooks and on the command line due to using fancy unicode characters. The latter is exactly what I want to use here as well: Twitter is primarily a textual medium, and although I could just post images of progress bars (wait… why didn’t I? d’oh!), posting text-based bars should be an easy first option.

Also, year_progress does text-bars as well and that bot inspired this one, so there you go.

Given some percentage from the dataset, we can make a bar-string for posting to Twitter with something like:

bar = tqdm(initial=percentage, total=100., bar_format='|{bar:15}| {percentage:3.1f}%', ascii=False)
bar_string = str(bar)
tweet_string = bar_string[:-5].replace(' ', '\u3000') + bar_string[-5:]

A few tricks beyond the tqdm basics to highlight are:

I used a custom format string to remove some superfluous stuff. We need to use as little characters as possible, but, also, it is important to not make the string too wide, otherwise it doesn’t display nicely on all devices (read: phones).
Usually, tqdm is used to display an “animated” progress bar. To emulate animation on the command line, it does all sorts of magic with carriage return (\r) and new line (\n) characters. I first tried using the tqdm option of specifying an output stream using an io.StringIO stream object, to act as a virtual terminal so to speak. I then tried to extract the bar string from that. This turned out to be quite a nightmarish experience (may have had to do with it getting quite late). Probably, tqdm tries to do very smart things when outputting to this stream, and clearly I wasn’t smart enough to undo those smart things. Luckily, after a good night’s rest, I found that directly extracting the bar string from bar.__str__() does not do weird animation stuff and just gives a neat, clean progress bar. Phew.
Twitter is not a terminal. The relevance of this obvious statement is that Twitter does not use a fixed width font. Madness, I know. But luckily, the unicode committee (I’m sure there is one, perhaps several, in fact) blessed us with more kinds of space characters than we could ever have hoped for. \u3000, the “ideographic space”, is the one that turns out to have the same width on Twitter as (most) unicode characters used for the bar itself. So I replace spaces in the bar (barring (heehee) those around the bar) with this ideographic one.

Tweet with Tweepy

The actual tweeting is the easiest part in all this. The above mentioned article explains most of it using the Tweepy package. You need the keys you obtained earlier to authenticate and create an api object (see the snippet in the article).

Then, tweeting is as simple as:

api.update_status("Ceci n'est pas une tweet")

Simple. J’approuve.

If all goes well, this command will return a giant blob of JSON, wrapped in some Tweepy class. Not to worry, do not disturb it, it means you no harm. If for some reason the Tweepy call failed, you’ll get an error message, so you’ll know when you actually have to start paying attention.

Automate on GitHub Actions

Obviously, the last part of the show is about getting rid of any future manual action and getting back to watching YouTube. We are programmers, after all. Moreover, we are scientists and we don’t have time for manual labor. That is what PhD stu…

[anxious whispering]

What?

[angry, impatient whispering]

Not supposed to…

[university PR department whispers]

Ok, but…

[whispered definitive executive order]

Automation, then!

[financial whispers]

For free! Yes, we can, with GitHub Actions.

[mumbled approval]

After putting all my code in a git repository (on GitHub), I created a workflow file .github/workflows/tweet_progress.yml. The file looks something like this:

name: Tweet latest progresson:
  schedule:
    - cron:  '0 10 * * *'  # 10:00 UTC, see https://crontab.guru/
  # allows you to run this workflow manually from the Actions tab:
  workflow_dispatch:jobs:
  tweet:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
      - name: install dependencies
        run: pip install -r requirements.txt
      - name: run script
        env:
          BOT_API: ${{ secrets.BOT_API }}
          BOT_API_SECRET: ${{ secrets.BOT_API_SECRET }}
          BOT_ACCESS: ${{ secrets.BOT_ACCESS }}
          BOT_ACCESS_SECRET: ${{ secrets.BOT_ACCESS_SECRET }}
        run: python run.py

Under the on section, we make our Tweepy script run on a regular schedule: every day at 10 a.m., UTC (check out https://crontab.guru/ if you don’t want to bother remembering the arcane cron syntax, thanks to

Stefan Verhoeven

for the tip). The data is updated daily in the morning, so this should use the freshest data most days. We can also trigger the job manually by adding workflow_dispatch. The part below jobs is what will actually run on a GitHub machine and should be mostly self-explanatory. This will also be different for your specific application (unless you copy mine). The important thing to notice, though, is the bold part where I pass in all the keys.

Two things to make this possible:

The awesome ConfigArgParse module allows for specifying program parameters in three ways: via command line arguments (like the Python standard library argparse), via a config file or via environment variables, all through one convenient, almost-drop-in-replacement-of- argparse call. On my laptop (for testing and such) I added a config file with the keys in it (and put the name of that file in .gitignore, after my earlier key committing booboo), but on GitHub Actions I can now pass them in through environment variables via the env section.
The second part is those ${{ secrets.THING }} things. You have to store the Twitter keys in GitHub secrets (instructions here).

Commit the yml file, push it to GitHub, and just like that you’ve got yourself an automatically updating Twitter bot!

The top two tweets were automatically produced. It’s alive! ALIVE!!!

I hope I convinced you that if you have a cool idea for a bot and know of a data source, you should immediately go for it! Just fork my repo if you want to get a quick start, modify it to your liking and put it out there.

Congratulations, you just made the world a better, more well-informed place!

I’d love to hear what you think, let me know in the comments. Also, if you want some help setting up your bot, don’t hesitate to ask.

Building a Web Service to Manage Scientific Simulation Data Using GraphQL

Felipe — Fri, 15 Jan 2021 00:00:00 GMT

Therefore, the Ceiba web service needs to handle two kinds of requests by the client: read-only queries and mutations on the datasets. These “queries” and “mutations” can be easily describe with GraphQL.

In a nutshell, GraphQL defines a contract (known as a schema) between the actions that a client can perform with the web service and the possible outcomes of those actions. More formally, GraphQL is a query language that allows you to specify an application Programming interface (API) using different programming languages. If you have previous experience with RESTful API have a look at a comparison between GraphQL and REST.

But how does GraphQL work? First, you need to define a schema using the GraphQL schema language. The following code snippet defines a schema to query a job using its status,

Schema definition for job query

The Query** schema specifies that in order to request some jobs* you need to provide a Status argument, where Status can be one of four possibilities: AVAILABLE, DONE, FAILED and *RUNNING. *The exclamation mark (!) indicates that the argument cannot be Null (a.k.a None in Python).

The following Mutation schema defines the required arguments to update a given job status.

Schema definitation for Job status mutation

The updateJob action specifies that you must provide an id and a new_status in order to be able to update a job. You will receive a Reply specifying whether the update action has succeeded.

Have a look at the Ceiba queries and mutations schemas. They are slightly more complex than the aforementioned schemas but follow the same rationale as the previous examples. You can also have a look at the official introduction to GraphQL.

We have just defined the schemas that specify the actions that we want to perform. We still need to implement the actions and for doing so, we need a GraphQL engine: a library that takes the schemas together with the code that implements the actions and generates an API.

We have chosen the Tartiflette GraphQL engine to implement our web service mostly because it is easy to use and open source. The following snippet shows a possible implementation for querying jobs based on their status using Tartiflette.

the Resolver decorator indicates that the resolver_query_jobs function corresponds to the implementation of the query jobs schema. The function takes 4 arguments of which I only use args and ctx(You can refer to Tartiflette for further details). ***args ***contains the arguments given by the client code, while ***ctx ***contains the context for running the current function, for example the handler to access the database that is called mongodb in this code snippet.

Notice that the definition of the aforementioned function starts with the async keyword. Asyncio is a popular built-in Python library to write concurrent code. It is extensively used to write high performance web services.

In the Ceiba web service implementation of the queries and mutations, there are definitions for all the Python functions that perform the actions specified in the GraphQL schemas. For each query and mutation, there is a corresponding function.

The database

We need a database not only for storing the interesting data but also to store the jobs metadata, like what jobs are available. For the Ceiba web service we use MongoDB.

My personal opinion is that a NoSQL database like MongoDB gives a significant advantage over traditional SQL databases on research projects where up-front design of the schemas to store data is unfeasible. The research priorities can change as the project evolves and having dynamic schemas to store the data makes the researchers’ lives easier.

Putting all together

Photo by frank mckenna on Unsplash Docker containers are the perfect way to ship our web service. We just need to write a [Dockerfile](https://github.com/nlesc-nano/insilico-server/blob/master/

Dockerfile) with the recipe to install and start the service together with the mongo container.

If you want to deploy the Ceiba web service to a remote server you need to follow these steps:

Install Ansible in your computer.
Clone the Ceiba repo and go to the provisioning folder.
Edit the inventory file with the address of the server(s) where you want to install the runner.
Edit the playbook file with the remote_user name for the remote servers.
Make sure that you can ssh to your server(s).
Install the runner with the following command:

ansible-playbook -i inventory playbook.ymlThe Ceiba server should be up and running!

The pesky details

You certainly do not want to keep your web service open, so people can remove your data. You want that users are authenticated before using your service, but you also do not want to manage all the security on your own. Getting authentication right using something like OAuth2 is tricky and it needs at least an entire post on its own.

Also, you need to host your web service somewhere and hosting costs money. It is simply not viable that you host your service in your computer, it is not safe and it takes too much time to maintain. Fortunately for researchers, there are institutions like SURF that can help you to host a web service for research purposes.

Acknowledgement

Creating the Ceiba web service would not be possible without Stefan Verhoeven advice and the computational resources provided by SURF.

I will also to thank Jens Wehner, Nicolas Renaud, Johan Hidding, Pablo Lopez-Tarifa and Victor Azizi for their feedback and support.

Specially thanks to [Patrick Bos,] Tom Bakker for their feedback.

eScience trends from ADASS 2020

Hanno Spreeuw — Mon, 23 Nov 2020 00:00:00 GMT

eScience trends from ADASS 2020

Scaling up compute and storage resources are not going to be the biggest hurdles.

**8

My general feeling about this conference

What a delight ADASS (Astronomical Data Analysis Software and Systems) 2020 was! My first virtual conference, but the same positive vibe as the previous one, ADASS 2019 in Groningen, The Netherlands. This one was in Granada, Spain, but it could have been anywhere, obviously. There were 581 participants and over 50 talks. Last year, ADASS had 353 participants, which was also quite considerable. One of the organisers said that there were quite a few participants from previously underrepresented countries. The cost of participating in this virtual conference was substantially lower than for the physical conferences, which could explain its popularity this year.

In hindsight, the Discord tool for discussion and social talk was essential. This gave me a feeling of being part of this conference. It was set up nicely with separate channels for each session, that continued after the session had ended, together with a number of general channels. Talks started at 6:00 a.m. CET and ended at 20:30 CET with somewhat fewer talks during the middle of the day to accommodate for speakers and audience in other time zones. Of course, it is impossible to make a schedule that fits all time zones, but I guess the organisers wanted to avoid people having to present their work at 3 a.m. local time.

Highlights

Slide from Matthew Graham’s presentation “What did we get right? Lessons learned from the first 300 million alerts of ZTF” .Matthew Graham** (invited speaker, professor at Caltech and project scientist for ZTF, the Zwicky Transient Facility), started off by listing his predictions from his ADASS 2008 presentation: what he thought would be the main bottlenecks and achievements in astronomy by the year 2020. Here are two of those predictions:

In 2008 he perceived that artificial intelligence would govern the alert streams without human interactions. We have clearly not reached that point yet. Humans are still in the loop, i.e. the current machine learning algorithms are not fully trusted.
In 2020, we would have huge problems with scalability to process the data firehoses and store the data lakes. As stated clearly in the slide above, this turns out not to be the case: we do have the compute power and storage to handle all this data and we can develop software that scales well.

Instead our focus needs to shift from data management to information and knowledge management. Present concerns are about establishing veracity, about provenance and probabilities based on astrophysical statistics. What are my assumptions and where are they coming from? These are presently much more severe concerns in delivering sound scientific conclusions than technogical challenges, they require astronomical or astrophysical expertise rather than computer engineering solutions.

Slide from talk by Verdana Desai: “Enabling next-generation science investigations with the NASA Astrophysics Archives”.Vandana Desai (invited speaker, associate scientist at Caltech and science lead for the Infrared Science Archive (IRSA) of IPAC, the Infrared Processing & Analysis Center) gave an insightful talk: “Enabling next-generation science investigations with the NASA Astrophysics Archives”. This covered part of her white paper “A Science Platform Network to Facilitate Astrophysics in the 2020s”. This white paper advocates “funding of data centers to develop and operate ‘science platforms’, which will provide storage and computing resources for the astronomical community to run analyses near the data”. “Science platforms” are envisaged as the way to enable scientific progress in the era of big data; our progress in understanding physical processes in the cosmos is enhanced by the simultaneous use of multi-wavelength observations possibly augmented with neutrino and gravitational wave detections. These platforms should accommodate the needs of a much larger group than just the ‘power users’, serving many more than the inner circle of scientists that are aware of all the complexities involved in reducing the raw data. Desai made clear that “We cannot have all of the data that an astronomer might want to use under a single, unified science platform”. This is not feasible “and probably never will be “ because of the “practicalities involved” which I suppose reflects the heterogeneity of astronomical observations. It is clear that analysis tools, including machine learning will require more compute power. The bottom panel from the slide above shows the increased use of archival data in the last decades, i.e. the use of data by others than the people near the principal investigator of a rewarded observing proposal. Until now, the combined data from astronomical sources was mostly extracted from catalogs which only include a few quantities like position on the sky, peak intensity, integrated flux and perhaps a few other quantities derived from the source pixel distribution. In the future, astronomers will need access to the actual pixels comprising the source for a more accurate analysis, which enhances the demand for storage and computing.

Slide from Cristobal Bordiu’s presentation “Astronomical research in the next decade: trends, barriers and needs in data access, management, visualization and analysis”.Some of the same conclusions, but more from a European rather than American perspective, were drawn by Cristobal Bordiu (postdoc for the Italian National Institute of Astrophysics (INAF) at the Catania Astrophysical Observatory) from a survey for NEANIAS (Novel EOSC Services for Emerging Atmosphere, Underwater & Space Challenges). This project, which emerged from EOSC, the European Open Science Cloud, aims to set solid foundations for astronomy in the coming decade, so it covers the same time span as the white paper by Desai et al. The respondees showed concern about storage, about reproducibility and about findability and interopability of raw, calibrated and higher level data products, as reflected by the slide above. They also perceive the availability of visual analytics tools as a main research barrier.

Conclusion

The lack of science-ready data products, which I reported on last year, applies mostly to radio astronomy, where the complete reduction of observations, including calibration of visibilities and subsequent imaging has to be done quite often by the researchers themselves. Fortunately, this situation is improving, with radio observatories taking up the glove. However, what ADASS 2020 made clear, is that more is needed that just these science-ready data products; to enhance scientific progress interoperable data from simultaneous observations at multiple electromagnetic wavelengths — possibly augmented with neutrino and gravitational wave detections— are required. Most conveniently, these would be analysed using “science platforms” which offer software tools for analysis, such as machine learning, that can be run in a completely hardware agnostic way.

Being a minority in fancy coding land: a Windows user.

Florian Huber — Tue, 10 Nov 2020 00:00:00 GMT

How I slowly went from my imposter-syndrome hiding to accepting what I am. A Windows user, at least most of the time. (Don’t worry, this is NOT one of those Linux vs. Windows posts!)

I work at the Netherlands eScience Center, a wonderful organization with very nice colleagues. Hopefully nice enough to stay my colleagues after the following confession: I am a Windows user.

What’s so special about this? Most people use Windows, right?
Well, not in my small coding bubble.
In the world I work in (use all your cliché imagination on nerds, hackers, computer scientists … but then remove those pictures of people wearing sun glasses indoors and desks full of pizza and caffeine-rich soft-drinks) coding from a Windows environment is often considered something between a no-go and a handicap. Breathing quickly, my fingers start to tremble as I write this, risking my career as a data scientist and machine learning practitioner. Or isn’t it all that bad?

How did I end up using Windows?

(I see your shaking heads… why the hell did he end up there?)
Let’s just say I have grown into it. All the way from MS DOS through many painfully bad Windows versions and then I got so used to it that my skills to deal with it always felt better than my skills in handling the alternatives.

In addition, I am not a software developer or computer scientist by training. For a long time I was a physicist, a scientist, an academic. And in the scientific fields where I was working, using Windows was — believe it or not —the norm.

Sure, I had already installed VirtualBox on my computer to run Ubuntu. I had even done some stuff with Ubuntu, including training some machine learning models (for implementations for which the packages didn’t support Windows…). I knew the 10 most common shell commands and everything else I would simply look up when needed. No wonder working with Linux still feels like writing a long letter with my left hand (I am right handed): I am terribly slow and in the end it looks horrible.

But … why?

Ah, I see. That all sounds like lame excuses to you.
Well, it is not that I didn’t see all those golden merits of using Linux over Windows. Of course that’s the better system for many tasks, say setting up a server or handling access rights. And yes, it is much less wasteful in using hardware resources, it is considered less vulnerable, … and so on …, plus it is freely available without commercial interests. Still, I never wanted to pay the price of not having access to some of the high-end software that you would get on mac-OS or Windows (such as some MS office stuff or Adobe products). By the way: I am not trying to convince anybody that Windows is the best option. I am already happy if we can agree that it is an option.

Time for another uncomfortable revelation: sometimes, being lazy simply pays off. For instance if you need to sort out a huge pile of stuff with some emotional value to you. Just put it in a box and hide it, take it out 10 years later and easily decide to dump nearly all of it. That’s a bit what I did with Linux. I survived with minimal use of VirtualBx and alike. Until recently, finally!, Linux became part of Windows. And that works pretty well for me. See how simple it now is to run Linux from Windows 10. And enjoy how easily you now can have the best of both worlds (not for nothing are more and more people arguing that the good old “dual boot” is dead for exactly this reason).

Using Windows and Ubuntu side by side… in front of wonderful ugly-ish Windows blue… what a peaceful view, isn’t it?

Enough all-united-hippie-talk. Let me share a few impressions of the actual life of an aspiring data scientist/research software engineer that happens to use Windows:

Starting a new job.

I did a lot of programming as a researcher, but clearly I had never learned the proper software development basics such as testing, versioning etc. (in many academic fields those terms are often still unheard of!). No wonder I suffered a lot from imposter syndrome in the very beginning.

I hope they won’t find out I can’t write proper code!

Naturally, that means that you might not immediately ask your colleagues for help, because that would reveal your amateur level, right?
But even worse, imagining you ask that colleague about how to get that Python package working and it turns out you are using Windows?

Can you help me setting up that environment? … By the way … I use Windows for that.

The looks you get are suggesting that you have just asked how to import a 5GB.csv file into your Excel table (*you don’t*). So little surprise I did spend a fair amount of time in Forum-Land during my first months…

Being that Windows user in the room

You sit in that hands-on workshop on some fancy programming techniques, and the instructors asks:

Is there anybody using Windows? (chuckles)

Or, actually worse, nobody asks. Of course the instructions are only given for Linux and mac-OS. Well, at least I can hide my Windows handicap for a little longer then… but NO!, when the instructor walks around to inspect the progress of the participants she/he will of course shout out:

Wow … you are really using Windows for that!?

Great. Now officially being tagged as the Windows user in the room. Better keep quiet and not ask any silly questions then…

Get used to rolled eyes —then secretly roll your eyes, too.

As I grew more confident of what I was doing, the imposter syndrome started to disappear. It still occasionally comes back to say hello (for instance if people speak shell over coffee), but that’s OK.

In the end it’s luckily the results that matters most. I learned that you can write as good or bad code on Windows as on Linux. You can build great software on Windows that is then used by Linux people, and the other way around. Sure, for some things you better go the Linux way. But it turns out that in my projects this is less than 1% of my working time, which makes it OK to be a bit clumsy using it. And secretly (don’t point at them, that’s mean!), I can also enjoy those moments when another colloquium presentation doesn’t run properly because Ubuntu did not work well with the projector, or the microphone, or both.

Do better than pointing at each other

Windows is more convenient for running some very common software (e.g. MS office), Linux is more stable… so go some cliches. But instead of fighting about what’s better (or hiding what feels inferior) it makes more sense to me to accept what’s there and simply go along with it. If somebody lives in a very geeky bubble it works fine to safely assume everyone runs their code on a certain operating systems and knows the in and outs of object oriented programming and containerization. But many of the more exciting projects involve people outside this bubble: researchers, users, future contributors, students. And they might as well — lo and behold — be using Windows (and by the way: containers are still primarily big steel boxes to most people).

So, even though in some IT-bubbles it can occasionally feel as if we are talking about a small unfortunate minority … in reality that’s really not true. Check out the 2020 Stack Overflow Developer Survey to see that most developers actually use Windows.

What can you do to get more Windows users to adopt your package?

Think of Windows what you want. I don’t work for Microsoft, and honestly, I don’t really care. But I assume that many coders out there working on great new software, methods, tools, tutorials, etc. actually want that people become happy users (paid by eternal gratitude). And that is a good enough reason to think about those Windows users as well.

Consider setting up your next continuous integration for your software package, so that it runs on all systems and will be used by more people.
For instance with continuous integration using GitHub actions it can be as simple as adding a ‘windows-latest’ to your matrix:
os: [‘ubuntu-latest’, ‘macos-latest’, ‘windows-latest’] (small warning: adding different operating systems to such a continuous integration workflow is comparably easy, the later debugging sometimes is not. One option can be to work with Windows virtual machine).
What about providing installation instructions for Windows users as well? Or did you just write a new tutorial? Great! But will it work for your fellow Windows users? You would be surprised how many packages and tutorials come with instructions that clearly won’t work for a Windows user.
Don’t know how to do that? No Problem! Just ask a Windows user to help you. Believe me, they will be very glad to assist.

Final symmetry

Most of my arguments will hold when we just swap the named OS. So, obviously if you are (like me) primarily a Windows user: Think of all those Linux and mac-OS people out there. Either way, it will require learning a bit about the differences. But it will help to avoid a lot of frustration on all ends due to failing notebooks or hard to install packages.

Get in touch

If you have comments or questions please get in touch! You can also find me on twitter: me_datapoint

Coalition polls for the people with Coalitiewijzer

Patrick Bos — Tue, 10 Nov 2020 00:00:00 GMT

Build interactive web apps with polling data, Voilà & Heroku

My first attempt at actual visualization was to make another grid, but this time consisting of 150 HTML widgets, one for each seat. Then we can adjust on_toggle to modify the value attribute of each HTML widget, which is the character it displays. This gives us a poor man’s ASCII based seats visualization (see the explore_things notebook on GitHub for the code). I was already pretty happy at this point…

Fancy Vuetify widgets

However, to make sure I wasn’t reinventing the wheel in the form of a square block of granite, I asked [Maarten Breddels], who is strongly involved in ipywidgets development, whether he knew of any existing options in the landscape of available widgets.

He helped me set up a really fancy seats widget using ipyvuetify, an alternative widget library built by [Mario Buikhuizen] on top of ipywidgets using the Vuetify Javascript UI library, a Vue library.

I won’t go into Vue here, because I don’t know much about it myself, except that it is a web framework like React. Apparently, though, it is not that hard to write Vue template based widgets. Vue templates have their own programming syntax with for-loops, conditionals and other useful stuff. What Maarten then came up with was something like this:

from ipyvuetify import VuetifyTemplate import traitletsclass PartyWidget(VuetifyTemplate): rows** = traitlets.Int(10).tag(sync=True)** cols** = traitlets.Int(15).tag(sync=True)** low** = traitlets.Int(0).tag(sync=True)** exp** = traitlets.Int(0).tag(sync=True)** high** = traitlets.Int(0).tag(sync=True)** template = traitlets.Unicode(''' <template> <div> <table><tr v-for=“i in rows”><td v-for=“j in cols”> <div><v-icon v-if=“(j + i * cols) < low”>mdi-seat*</v-icon><v-icon v-else-if=“(j + i * cols) < exp” style=“color: orange”>mdi-seat</v-icon><v-icon v-else-if=“(j + i * cols) == exp” style=“color: green”>mdi-seat</v-icon><v-icon v-else-if=“(j + i * cols) < high” style=“color: red”>mdi-seat</v-icon><v-icon v-else** style=“color: pink”>mdi-seat</v-icon>** </div> </td> </tr> </div> </template> ''').tag(sync=True)

pw = PartyWidget()**Let’s break that down, looking at the bold-face parts especially. The first, Pythonic part of the class defines five traitlets properties which we’re going to use to interactively change things. We use 10 rows and 15 columns here, giving a total of 150 seats. We can use the three other numbers low, exp and high for showing the expected number of seats estimates and the lower and higher uncertainty region boundaries.

Then comes the Vue magic in some kind of generative dialect of HTML. We’re going to generate a table with rows and columns using the v-for attribute which automatically generates a bunch of them.

Then, inside the table cells, we use a bunch of fancy seat icons, *mdi-seat*, with if/else conditionals with the v-if and v-else-if attributes. The point here is that we want to create different looking seats to indicate the expected number of seats and the lower and upper boundaries. We can simply use CSS style attributes for this.

Finally, we create an instance of the class. With that in hand, we can now put our on_toggle function to better use by modifying the PartyWidget pw’s values on each trigger:

def on_toggle(toggles): pw.low = int(numbers[list(toggles.values())].ZetelsLaag.sum()) pw.exp = int(numbers[list(toggles.values())].Zetels.sum()) pw.high = int(numbers[list(toggles.values())].ZetelsHoog.sum())And that’s it! Now when we display(pw) our fancy new widget and the button grid with interact_out as before, we get something like this:

Fancy McSchmancyton!

And what is even more amazing (as you can experience in the actual live app): the changing of CSS properties is animated! The colors smoothly change from one to the next when you click a button. The same goes for other properties like opacity and even rotation. All this comes for free with ipyvuetify (I guess with Vuetify itself as well).

Voilà

So, now we have a fancy interactive addition visualizer. However, it is in a notebook, which I do not want to share with my non-data scientist friends.

Enter Voilà.

Voilà is essentially a Jupyter Notebook server, with one big difference: it only displays the output** of running all the cells. Converting from a notebook to an actual app is just a matter of one command:

voila your_notebook.ipynbJust like with Jupyter notebook, this starts up a server and opens a webpage in your browser showing the output of cells, including the widgets you created (if you put display commands in).

An active Python kernel is still running secretly in the background, though. This means you can still use the interactive functionality you setup between the widgets. You just cannot touch the code anymore. And this is exactly what we want when deploying our widgets as a user-facing interactive app.

Since this Python kernel does all the magic of updating and connecting the widgets behind the scenes, unfortunately we cannot do without. I’m hoping one day there will be a static page output option, but I guess that basically requires a Python to Javascript or webassembly transpiler… Could be lots of fun, but we’re not there yet, I’m afraid.

Deploying on Heroku

So, if we want to put this app on the web, we need a place to host the Voilà server. If you happen to run your own server, that’s great (here’s the guide on how to setup Voilà to run from there). If you don’t, read on (or, again, read the Voilà guide itself, which is what I did).

Deploying on Heroku is a really easy way to get online apps going quickly and free of charge (with limits, of course), which makes it ideal for testing. There’s even an official guide on how to deploy a Voilà server on Heroku. There’s just a few catches to be aware of.

First, it’s good to notice this really weird piece of Heroku documentation:

“obscure” :DOkay, fine, I can pickle the necessary Peilingwijzer data and only use Python standard library functionality, instead of Pandas, numpy, matplotlib and some other weird and arcane packages that I’m sure nobody except me uses. I put this conversion process in this script.

Second, I ran into a weird error, which was already documented earlier by someone on Stack Overflow. The Heroku app, despite the straightforward setup procedure, would not immediately run. Luckily the solution was found in another repo.

So with those things in mind, after rewriting things to make use of as little scientific Python packages as possible putting the app on Heroku was a matter of following the steps in the Voilà Heroku deployment guide:

Login using the Heroku CLI on your machine
Add a runtime.txt file to your project/repo directory containing only a line with the Python runtime, which I set to python-3.9.0.
Add a Procfile file to the directory containing the following line: web: voila --port=$PORT --no-browser --enable_nbextensions=True your_notebook.ipynb.
Commit these files, a requirements.txt file with Python dependencies and the notebook to a git repo, if you haven’t done so yet.
Create a Heroku app and push your code to it:

heroku create [appname] git push heroku masterAnd then, just like that, you are live! You can open the app in your browser using heroku open, or just click the link. If you omit the app name in the heroku create command, Heroku will randomly generate a name for you. The name can be important, since it determines your URL, which will be [appname].herokuapp.com. You can change it later on, though.

Note that when you want to use a different clone of your repo to update your existing app (e.g. when coding on different machines), you have to reconnect Heroku to your app. If you would just run heroku create again, it would create a new app. To connect the repo to the existing app, run

heroku git:remote -a [appname]The [appname] is the part of the app’s URL in front of herokuapp.com. You can also find it by running heroku apps on your command line, which gives you a list of all your apps.

So, if you hadn’t done so already, check out the resulting Coalitiewijzer app at https://coalitiewijzer.herokuapp.com/ and let me know what you think!

For those that don’t like discussing on Medium: Reddit thread here and Twitter here.

This was my first time putting ipywidgets, Voilà and Heroku to use and I was really pleasantly surprised with how smooth all of them work. If you’re used to working in Jupyter notebooks, it’s an actual breeze to break out your exploration or visualization or other fun nugget of knowledge into something non-developers can also make good use of.

It just so happens that this time I needed it to scratch a politics itch, but I can see this workflow being very useful for science communication as well. I’ll definitely be using it again. Let me know what you built with it!

C++ web app with WebAssembly, Vega, Web Worker and React

eScience Editorial Team — Tue, 27 Oct 2020 00:00:00 GMT

Subscribe*Remember me for faster sign in

To generate the Vega-Lite specification we can write a function like so

To wrap the Vega-Lite visualization in React component we will use useRef to get a DOM element as container and use useEffect to call vegaEmbed when the iterations or container changes. The React component to render the visualization is

Pack it up

The React components and React render call can be packed up all together in a JavaScript file called app.js.

The web applications needs a HTML page to fetch all the React and Vega dependencies, define a HTML tag for rendering the React app to and finally include the application JavaScript file.

File: app.html*

We’ll need a web server to display the HTML page in a web browser. For this, we’ll use the http.server module from Python 3 to host all files on port 8000, like so:

python3 -m http.server 8000Visiting the page at http://localhost:8000/app.html should give us a plot like

](https://nlesc-jcer.github.io/run-cpp-on-web/kitchen-sink/app.html)(Click on image to get interactive version)You can try out different initial guesses to get different amount of iterations. For example having initial guess located in a local minimum like 2 will make the algorithm use many iterations to jump over the minimum.

Recap

In this series of blog posts we introduced a lot of different technologies to able to take an algorithm written in C++ and make a interactive web application that will run fully in a web browser.

All the source code shown is available at https://github.com/NLESC-JCER/run-cpp-on-web.

This blog was written by the Generalization Team of the Netherlands eScience Center. The team consists of Stefan Verhoeven, Faruk Diblen, Jurriaan H. Spaaks, Adam Belloum and Christiaan Meijer. Feel free to get in touch with the generalization team at generalization@esciencecenter.nl.

Hope you enjoyed this series of blogs and if you have suggestions or questions please post a comment below.

These blogere written as part of the “Passing XSAMS” project. To learn more about the project, check out its project page.

How is the end of daylight saving time going to affect me?

Pablo Rodríguez-Sánchez — Thu, 22 Oct 2020 00:00:00 GMT

Photo by insung yoon on Unsplash

It happens every year. Twice. And it can be pretty annoying. I am talking about the clock shift to re-adapt our schedules to wintertime or summertime. But did you know that this October 25 may be the last time this happens? Well… at least if you are reading these lines from Europe.

Farewell to the (two) clocks

The European Parliament decided to stop the practice of “spring forward, fall back”. This decision has been backed by a public consultation to 4.6 million Europeans, 84% of them were in favor of stopping the biyearly clock readjustment.

The reasons to abandon the traditional clock readjustment are manifold. Most of us agree with the most obvious of all: it causes a couple of days of annoyance to get used to the new timetable. There is, additionally, some controversy about how much energy is saved, plus some evidence about an increase in car crashes in the days after a clock change.

The members of the European Union have been asked to choose between winter and summertime clocks, and stick to their choice during the whole year. The deadline for this decision is April 2021… and sticking permanently to one of these clocks is not an easy choice.

From a sociopolitical point of view, one of the obvious concerns is the possibility of causing a de-synchronization between countries that currently share the same clock. It would not be very convenient, for instance, to use different timezones in The Netherlands and Belgium. Efficient coordination between the EU members will certainly require communication and diplomacy.

But there is an even deeper problem. Changing the clocks reduces the undesirable effects of an astronomical phenomenon: that of the variability of sunlight hours during the year. To make it even more difficult, this variability depends not only on the day of the year but also on the location on Earth.

Luckily, the study of the relationship between daylight duration, position, and date is one of the oldest objects of study. There are some early references in the Babylonian time, the topic was already covered in Ptolemy’s Almagest, and it is part of the everyday life of sailors since, at least, the publication of the first nautical almanacs in the 1700s. Nowadays, all the knowledge we need is just a few clicks away.

Daylight charts and how to read them

The easiest way to visualize this kind of information is by using a daylight chart. They look like this:

Daylight chart corresponding to the city of Barcelona, Spain, in the year 2020. The abrupt changes correspond with the spring and autumn clock shifts. Source: https://github.com/PabRod/daylight-charts

Nowadays, quick and simple access to astronomical databases makes it easier than ever to simulate daylight charts at any point in the world. In particular, I wrote this code in R to generate such diagrams. It can be run as an interactive applet, either in Shinyapps or locally, to query for any city in the EU with a population over 100k. Additionally, I precalculated a collection of daylight charts for Dutch, Spanish, and other European towns with a population over 100k.

By playing with these diagrams we can learn some lessons about the reasons for using two clocks, and we can also simulate possible new scenarios. For instance, we can compare the daylight charts of Barcelona with clock shift and with permanent summer clock:

Daylight charts of Barcelona with clock shift (left) and permanent summer clock (right). Note how the clock shift keeps the sunrise centered around 7:30. With permanent summer time, the sunrise in winter will happen after 9:00. Source: https://github.com/PabRod/daylight-charts

The three possible scenarios can be visualized in a more compact way using a single chart with different colors:

In yellow, sunlight hours with clock shift (current situation). In blue, sunrise and sunset times using winter clock. In red, the same with summer clock. Source: https://github.com/PabRod/daylight-charts

Just for fun, we can explore the effect of latitude (the position in the South-North line) by, for instance, comparing two of the southernmost and northernmost big towns in the EU:

Comparison of daylight charts in Santa Cruz de Tenerife, Canary Islands (28º N) and Stockholm, Sweden (59ºN). We notice that the further from the equator, the more irregular the sunlight is throughout the year. Source: https://github.com/PabRod/daylight-charts

Longitude, that is, the position in the direction West-East, also plays a role. This is the direction of the Earth’s rotation and, thus, we can imagine the sunrise “traveling” through it. Indeed, this the reason we have different timezones.

Timezones are decided not only due to astronomical considerations but also due to commercial and political ones. Lots of them are actually wider than the distance the sun covers in one hour, and this causes interesting effects.

Take a look at the map below. It shows the night advancing over Europe the day of the September equinox, together with the European timezones. Do you see something strange?

Line of sunset advancing over Europe on a day in mid-September. The different colors represent different timezones. Note that, particularly, the Central European Timezone (in dark green) is much wider than the distance the sunset covers in one hour. Source: https://naukas.com/2013/10/15/espana-esta-que-se-sale-el-meridiano-y-la-hora-oficial/

The Central European Timezone, for instance, is huge. The sun takes more than two hours to cover it. And remember, for all the people who live there, the clock shows the same time.

Let’s imagine two Europeans, Mateusz and Breixo. Mateusz lives in Lublin, in the East of Poland, and Breixo lives in Tuy, in Northwestern Spain. They both leave the office where they work, simultaneously, when the clock reads 18:00, but their experience is very different. For Mateusz, 18:00 is almost night, while for Breixo it will still be clear daylight for two more hours. Something similar will happen with the sunrise: it will happen roughly two hours earlier for Mateusz than for Breixo. And all of this happens although their clocks are synchronized (actually, it happens because their clocks are synchronized).

Interestingly enough, Vânia, who lives in the Portuguese town of Valença do Minho, only one kilometer away from Breixo, is in a different timezone. So her clock will read one hour less than that of her neighbor Breixo when the night arrives, simultaneously, at both towns. Pretty confusing, right?

We can see also these effects on our diagrams. In the one below we see an example with two of the most occidental/oriental cities in the CET timezone:

Daylight chart for A Coruña, Spain, and Split, Croatia. Both cities lie at the same approximate latitude (43ºN), but are separated by 2000 km in the West-East direction. As a consequence, all astronomical events, including sunrise, happen in Split roughly 2 hours earlier than in A Coruña, although they lay in the same timezone (CET). Source: https://github.com/PabRod/daylight-charts

A well-informed decision is a good decision. The socioeconomic aspects of this problem are pretty difficult to predict, but the astronomical ones, on the contrary, are almost as predictable as the movement of a clock. Whenever we can, let’s base our decisions on data.

Acknowledgments

The final version of this short essay was greatly improved by the suggestions and comments of Zeger van der Sterre, Lourens Veen, Stefan Verhoeven, Patrick Bos, Sonja Georgievska and Maarten van Meersbergen.

Help! My C++ web app is not responding

eScience Editorial Team — Fri, 09 Oct 2020 00:00:00 GMT

By Stefan Verhoeven, Christiaan Meijer, Faruk Diblen, Jurriaan H. Spaaks, and Adam Belloum.

In an earlier blogpost we discussed how to run C++ code on the web using JavaScript. We created a web app that executed some C++ code and then showed the result in the browser. While the page was running the C++ code, the page was blocked and unresponsive. This was not noticeable, because the computation done in the code was very quick. An unresponsive User Interface (UI) becomes a problem when we are performing tasks that take a bit longer to run.

How to prevent blocking when running long running tasks in C++?

In this blog post, we will use web workers to solve this problem by offloading tasks to another thread.

Long-running tasks with web worker

Let’s have a look at the code we ended up with in the first blog of the series. When loading the page, the WebAssembly code is executed, after which the page can finish rendering. Because the WebAssembly code was very quick, this was fine. For the current blog, we assume we have a longer running task. We create such a task artificially, by adding a few seconds of sleep in the C++ code. Like in the first post of the series, we compile the C++ code to create WebAssembly code. The example page with our slow task can be found here.

Notice that we also added a slider to the page. This slider simply serves to illustrate UI unresponsiveness — it has no attached function. Notice that while the WebAssembly code is still running, the slider is completely blocked. If this was an actual web app and not just a demo, the blocked UI would surely annoy users and possibly make working with the app cumbersome and impractical. We can easily solve this, and keep the UI responsive at all times, using web workers.

Blocked UI while code is running.

Web workers

A web worker is an object that handles execution of a piece of code in another thread.

The way the page communicates with the worker object is through sending messages. The page will send a message to the worker to start doing work, and the message will include all data that the worker needs. The worker then starts executing the task, using only the data that was in the message. When finished, the worker needs to communicate the results back to the web app. It will do this by sending a message, so the web app knows when to update.

The resulting page

The code snippet below shows the web page that uses a web worker. Notice the creation of the Worker object, posting of the message, as well as the instructions for handling of any returned messages containing results.

The web worker code only contains handling of the incoming message. The web worker unpacks the message, does the root finding calculation, and packs the results in a new message that it will send back.

The code for the worker in worker.js is:

The figure below illustrates what is happening in parallel in the two threads.

We can see the code in action here. The calculation still takes the same time to perform, but as you will notice, the slider remains responsive.

Responsive UI thanks to offloading the root finding procedure to the web worker.

Recap and what next?

In this blog post, we learned how to keep a web app from freezing while executing computationally intensive C++ code. We learned how to create a web worker and how to use the web worker in a simple web app.

Now you can take this web app a step further by reading some of the other blogs in this series, for example how to interact with your app using forms and how to spice up your app up with visualizations. A final blog shows you how to combine all of the above in a single web app.

Looking for the first blog and introduction to this series? This blog post will show you how to get started running your C++ on the web using WebAssembly.

Get in touch with us

These blogs were written as part of the “Passing XSAMS” project. To learn more about the project, check out its project page.

Thank you to our proof reader Carlos Martinez-Ortiz

Interact with your C++ web app using React forms

eScience Editorial Team — Fri, 09 Oct 2020 00:00:00 GMT

By Stefan Verhoeven, Faruk Diblen, Jurriaan H. Spaaks, Adam Belloum, and Christiaan Meijer.

In a previous blog post, we compiled the C++ algorithm into WebAssembly. In the current blog post, we will create a web application using React. The web application will have a web form that allows us to change the parameters of the algorithm.

We feel your pain: there are too many things to learn, too many skills to get, but too little time available — which reminds me of our extraordinary friend Napoleon Dynamite. This blog post will guide you through the process of making a React web application without getting lost.

If you haven’t met Napoleon yet, click here to see how he was struggling with his skills. Screenshot from Napoleon Dynamite movie.

React web application

The web application we developed so far needs to update the entire page to display the results. Even for small changes in the web page, this has to happen. Thanks to modern web browsers and JavaScript, Single Page Applications (SPAs) can update only required elements in the web page. We will use one of the most popular web frameworks, React, to develop the SPA. We chose React over vanilla JavaScript because building a web application is made easier by letting React deal with all the magic behind the scenes. This blog post may help you to understand why we made this choice.

The form in the web application will collect the user inputs and uses them to initialize the algorithm. When the form is submitted, a WebAssembly code starts the calculation and the result is rendered. With this architecture, the application only needs cheap static file hosting to host the HTML, JavaScript, and WebAssembly files. The algorithm will be running in the web browser on the end-user’s machine instead of a server.

The HTML code

To render the React application we need an HTML element as a container. We will give it the identifier container which will use later when we implement the React application.

We will keep the HTML code very minimal. The code will contain three essential elements:

element to set the title and to load the required external dependencies (JavaScript libraries).
element to display the result
elements to load the Javascript application and the NewtonRaphson class.

Note: We use the same newtonraphson.js and newtonraphson.wasm files as in the first post of this series. Make sure you download newtonraphson.js and newtonraphson.wasm files from GitHub.

The complete HTML code will look like this:

JavaScript code (React)

Similarly, we will split the JavaScript code into sections and build up the React application from React components.

Let’s start with the header part. We will define a JavaScript function that returns the header element which will be rendered by the web browser when the user visits the page.

The return statement of this function looks weird, right? It is indeed not HTML. React uses a syntax extension called JSX to describe the UI. With the magical conversion of Babel we can convert JSX into JavaScript code. After this conversion the generated JavaScript code will look like:

In order for the header element to be rendered we need to tell ReactDOM which element it should render and where it should be displayed. Do you still remember the container div we defined in the HTML part?

The complete code should look like this:

When the page is rendered, the generated HTML code will be like:

Adding the web form

The web application in our example should have a form with tolerance and initial_guess input fields, as well as a submit button. The form in JSX can be written in the following way:

The form tag has an onSubmit property, which is set to a function ( handleSubmit) that will handle the form submission. The input tag has a value property to set the variable ( tolerance and initial_guess) and it also has an onChange property to set the function ( onToleranceChange and onGuessChange) which will be triggered when the user changes the value.

Let’s implement the value and onChange for the tolerance input. To store the value we will use the React useState hook.

The argument of the useState function is the initial value. The tolerance variable contains the current value for tolerance and setTolerance is a function to set the tolerance to a new value. The same logic is also used for the initial_guess variable.

The input tag in the form will call the onChange function with an event object. We need to extract the user input from the event and pass it to setTolerance or setGuess. The value should be a number, so we use Number() to cast the string from the event to a number.

We are now ready to implement the handleSubmit function which will process the submitted form data. The function will get an event object, similar to the onChange of the input tag. Normally when you submit a form, the form fields will be sent to the server, but we want to perform the calculation in the browser, so we have to disable the default action with preventDefault(). We will then construct a WebAssembly module as we did in a previous post.

We need a place to store the result of the calculation ( root value), we will use useState function again. The initial value of the result is set to undefined as the result is only known after the calculation has been completed.

When the calculation is done it will store the result value ( root) using setRoot.

To render the result we can use a React Component which has root as a property. When the calculation has not been done yet, it will render Not submitted. We will show the root property value once it is set.

Finally we can render the App component to the HTML container with id equal to container.

We can combine the heading, form and result components and all the states and handleSubmit function into the App React component and its rendering and save it as app.js. You can find the resulting code here.

Like before, we also need to host the files in a web server with

python3 -m http.server 8000

The final page if everything works.

Visit http://localhost:8000/app.html to see the root answer, or go to GitHub pages to see a hosted version of the example app.

Extra notes

The code supplied here should not be used in production as converting JSX in the web browser is slow. It’s better to use Create React App, which gives you an infrastructure to perform the transformation offline.

Conclusion

By writing React components we were able to create an interactive page with a form that executes the WebAssembly module compiled from the C++ code we introduced in the first blog of the series.

We went over JSX, props, state, and components, which together constitute the core building blocks of a React web application.

In other blogs of the series that might be of interest we cover

Using C++ in a web app with WebAssembly: How to turn C++ code into a web app.

Help! My C++ web app is not responding: How to use web workers to perform computations without blocking the user interface.

Spice up your C++ web app with visualizations: Plotting data from the C++ web app using web visualization.

We’ll wrap up the series in a final blog that combines the topics of the whole series in a full-featured web application.

Get in touch with us

This blog was written by the Generalization Team of the Netherlands eScience Center. The team consists of Stefan Verhoeven, Faruk Diblen, Jurriaan H. Spaaks, Adam Belloum and Christiaan Meijer. Feel free to get in touch with the generalization team at generalization@esciencecenter.nl.

If you enjoyed this article, leave a comment and give us a clap!

These blogs were written as part of the “Passing XSAMS” project. To learn more about the project, check out its project page.

Thank you to our proof reader Daan Boer.

Spice up your C++ web app with visualizations

eScience Editorial Team — Fri, 09 Oct 2020 00:00:00 GMT

By Stefan Verhoeven, Faruk Diblen, Jurriaan H. Spaaks, Christiaan Meijer, and Adam Belloum.

In a previous blog we found the root of an equation using the Newton-Raphson algorithm implemented in C++ and compiled to a WebAsssembly module. In this blog, we’re going to expand on that by visualizing the iterations that Newton-Raphson used to get from the initial guess to the estimate of the root.

Visualization allows us to track what’s going on under the hood of our program. Image courtesy of Nenad Stojkovic via flickr.

Iterations

The Newton-Raphson algorithm approximates the value of the target equation’s root in a series of iterations. We capture the data of each iteration using the following struct:

x: x value, starting with the value of initial_guess and ending with the estimate of the equation ‘s root
y: result of passing x through equation
slope: result of passing x through derivative

Extending the NewtonRaphson class accordingly yields the following header file:

File: newtonraphson.hpp

The do loop in newtonraphson.cpp is updated to include a push_back to the iterations vector. This way, we can record the value of relevant variables in each cycle, as follows:

File: newtonraphson.cpp

Before we go into the Emscripten world, let’s first test our C++ code. We can check if the iteration property is populated correctly, by extending the command line interface we made in the previous blog as follows:

File: cli.cpp

Compile it with:

g++ -o cli.exe problem.cpp newtonraphson.cpp cli.cpp

Run with:

./cli.exe
index = 0 x = -4.00 y = -186.00 slope = 128.00 delta_x = -1.45
index = 1 x = -2.55 y = -52.99 slope = 59.29 delta_x = -0.89
index = 2 x = -1.65 y = -13.97 slope = 29.63 delta_x = -0.47
index = 3 x = -1.18 y = -2.89 slope = 17.83 delta_x = -0.16
index = 4 x = -1.02 y = -0.28 slope = 14.40 delta_x = -0.02
index = 5 x = -1.00 y = -0.00 slope = 14.01 delta_x = -0.00

The last iteration has x = -1.00, which is what we expected.

Bindings

Emscripten can handle simple types like float and int, but needs help exposing more complex types to JavaScript like the iterations property. We need to use value_object to expose the Iteration struct and register_vector as the iterations property type.

So the bindings look like this:

File: bindings.cpp

We can now compile our C++ code to a WebAssembly module with Emscripten using emcc command, exactly like we did before:

emcc -I. -o newtonraphson.js -Oz -s MODULARIZE=1 \
  -s EXPORT_NAME=createModule --bind \
  problem.cpp newtonraphson.cpp bindings.cpp

To get the iteration data in JavaScript we use the following code

Let’s have a look at the data we want to plot, by logging it to the console with console.log(JSON.stringify(iterations, null, 2)), which should return the following data:

Great, that looks very similar to the output we got from the command line.

Vega-Lite specification

There many ways to do visualizations on the web. One of our favorites is Vega-Lite, a JavaScript library which describes a plot using a JSON document called a specification.

The root finding algorithm tries to find the x where y is zero using a series of iterations. Let’s plot the iteration index against the y found in each iteration to see how quickly it converged to an answer.

The generic structure of our Vega-Lite specification looks like this:

To render a specification we need to use the vegaEmbed(element, spec) method which accepts an HTML element and a Vega-Lite specification.

Combining the three snippets above (get iteration data, Vega specification and vegaEmbed) and filling in their respective details yields an HTML document with the complete web app:

File: scatter.html

We’ll need a web server to display the HTML page in a web browser. For this, we’ll use the http.server module from Python 3 again to host all files on port 8000, like so:

python3 -m http.server 8000

When we visit the web page at http://localhost:8000/scatter.html, we will be greeted by the following plot. We can zoom with the mouse wheel and pan by dragging. Hovering over a point shows a tooltip with relevant data at that point.

(Click on image to get interactive version)

Advanced plot

In the first blog of this series we plotted the equation and root as

It would be nice to write a specification of this plot together with the iterations that the root finding algorithm went through. Vega-Lite can superimpose one chart on top of another with the keyword. Let’s construct each layer separately and then superimpose them at the end.

The 2x³ — 4x² + 6 equation is plotted by using a sequence generator to generate a range of x values and a formula transform is used to calculate the y values.

To show where the root is, we draw a dotted vertical line using a rule marker at x = -1 and label it root with a text marker.

When we plot the x and y of each iteration we can no longer see the order of iterations, so we will use a text marker above each circle to indicate which iteration it belongs to.

Superimpose the equation line, root ruler, and iteration scatter into a single visualization with:

The HTML page with all JavaScript put together to make a composite plot is available here.

Visiting the page should give us a plot like

(Click on image to get interactive version)

Wrap up

In this blog we have learned how to get complex data types from a WebAssembly module using Emscripten bindings and how to write a Vega-Lite specifications to get interactive visualizations.

Other blogs of the series that might be of interest

We’ll wrap up the series in a final blog that combines the topics of the whole series in a full-featured web application.

If you enjoyed this article, leave a comment and give us a clap!

These blogs were written as part of the “Passing XSAMS” project. To learn more about the project, check out its project page.

Using C++ in a web app with WebAssembly

eScience Editorial Team — Fri, 09 Oct 2020 00:00:00 GMT

By Stefan Verhoeven, Faruk Diblen, Jurriaan H. Spaaks, Adam Belloum, and Christiaan Meijer.

Let’s say you have some C++ code laying around that you would like to make available to a wider audience, by putting it on the web as a ready to use web tool. Until recently, this used to be pretty difficult, and may even have required reimplementation of the software in JavaScript, the programming language that browsers use.

Wouldn’t it be great if you could run your existing C++ code on the web with only minor effort?

That way, loads more people would be able to see your results, interact with your algorithm, and apply it for their own purposes.

In this blog, we’ll show you how to take a simple algorithm written in C++ and make it available as a web application. Subsequent blogs in this series will expand on the current one by laying out more advanced topics, specifically how to make the app interactive, how to visualize the results, and how to deal with long running tasks.

Root finding

So today’s aim is to have a simple web app that determines the root of a mathematical function 2x³ — 4x² + 6, i.e. the value of x where y = 0.

Function 2x³ — 4x² + 6.

For this, we’ll use an iterative method known as the Newton-Raphson root finding method. Remember Newton? Quiet fellow, fabulous hair? Yes, that Newton. The way Newton-Raphson works is, you give it the equation whose root you want to find, along with the derivative of that equation. Then you take an initial_guess of what you think the value of the root could be, then let the method iterate towards the solution. The solution is approximate within a tolerance, which you can also set. Anyway, the algorithm is written C++, but with some trickery, we’ll be able to use that C++ code from the browser, without the need to port it first!

Newton (and his hair). Image courtesy of Wikimedia Commons.

Now before you say “That’ll be so much slower than running it native!” or “C++ from the browser? Impossible!”, just hold your horses for a sec. With the right tools, it is possible to run C++ code in the browser, with an acceptable performance penalty. For example, Gabriel Cuvillier was able to run the video game Doom 3 in the browser. He was able to do this by compiling the game’s source code into WebAssembly, a low-level language that browsers can run. And if it works for video games, it will likely work for your research software, too.

Hold your horses. Image courtesy of Luciano Marelli via pixabay.

What we’ll need

OK, now that you’re fully on board with this, let’s get to it. Here’s a list of what we need:

We are going to write a small HTML page, so you will need basic knowledge of HTML and JavaScript.
Some C++ code to illustrate the process. We’ll use our Newton-Raphson C++ code.
A program ==that== can take our existing C++ code and compile it into a WebAssembly module. For this, we’ll use Emscripten ’s emcc compiler, the most popular C++ to WebAssembly compiler of the bunch.
To use the WebAssembly functionality from JavaScript, a binding is required. The binding will map C++ constructs to their JavaScript equivalent and back. For this, we’ll use embind.
A web server to serve our files. We’ll use Python 3’s http.server, but other web servers work equally well.

Tying it all together

The C++ code

Here is the equation whose root we want to find, along with its derivative, since that’s what Newton-Raphson requires:

File: problem.hpp

File: problem.cpp

The snippet below shows the contents of the file newtonraphson.hpp. It is the header file for the Newton-Raphson iterative root finding algorithm. It defines a class named NewtonRaphson. Besides the constructor method NewtonRaphson(float tolerance_in), NewtonRaphson has one other public method, solve, which takes a float, and returns another float. Furthermore, NewtonRaphson also has a private member, tolerance of type float, which is used to store the class instance’s private data.

File: newtonraphson.hpp

File newtonraphson.cpp contains the corresponding implementation:

File: newtonraphson.cpp

From this definition, NewtonRaphson instances need to be initialized with a value for tolerance_in, which is then stored as the private member tolerance. Once the object instance has been constructed, users can call its solve method to iteratively find equation ‘s root, with equation and its derivative being imported from problem.hpp via the include line near the top.

Check on command line

The following code is a minimal command line program that we can use to check if everything is working correctly:

File: cli.cpp

Our command line program can be compiled with:

g++ -o cli.exe problem.cpp cli.cpp newtonraphson.cpp

Subsequently running it should give the following output:

./cli.exe
The value of the root is : -1.00

Now we’re ready to move on to the WebAssembly part.

Binding

To use the Newton-Raphson code from JavaScript, we’ll need to define the bindings file. The binding allows compiled code to be called from JavaScript. For our Newton-Raphson code, the binding file looks like this:

File: bindings.cpp

The binding file uses embind binding statements to expose the NewtonRaphson class, its constructor method, as well as its public method solve.

Compiling to WebAssembly

First we need to download and install Emscripten to get the compiler. The Newton-Raphson source and its binding can be compiled into a WebAssembly module with Emscripten’s emcc compiler, as follows:

emcc -I. -o newtonraphson.js -Oz -s MODULARIZE=1 \
 -s EXPORT_NAME=createModule --bind \
 problem.cpp newtonraphson.cpp bindings.cpp

This will generate a WebAssembly module newtonraphson.wasm, along with a JavaScript file newtonraphson.js. We also export the createModule JavaScript function in the compile command so it can be used to load and initialize the WebAssembly ==module==. Using the newtonraphson.js JavaScript library, we can find the root of the mathematical function, and subsequently display its value with the following HTML:

File: index.html

Hosting the app with a web server

We’ll need a web server to display the HTML page in a web browser. For this, we’ll use the http.server module from Python 3 to host all files in the current directory on port 8000, like so:

# change to directory with index.html and newtonraphson.* files
python3 -m http.server 8000

From the figure at the top of the article, the root of the equation should be at x = -1.00. Visit http://localhost:8000/ to see if your browser shows the correct result.

The resulting page if everything works. See the live version of the app on GitHub pages.

Recap

We wrote a simple algorithm in C++
We defined the JavaScript interface by writing Emscripten bindings
We compiled the algorithm and bindings to a WebAssembly module with Emscripten compiler
We ran the algorithm in a web browser using some JavaScript to talk to the WebAssembly module.

The nice thing about this solution is that we don’t need expensive infrastructure to perform computation as the computation is done in the user’s web browser — we just need somewhere to host the files.

Get in touch with us

Where to go from here?

In upcoming blogs we will cover:

Help! My C++ web app is not responding: how to perform computations without blocking the user interface.
Interact with your C++ web app using React forms: how to let the user supply their own input values for tolerance and initial_guess.
Spice up your C++ web app with visualizations: how to visualize data from the algorithm.

We’ll wrap up the series in a final blog that combines the topics of the whole series in a full-featured web application. If you’re curious what that’s going to look like, make sure to check out the live demo on GitHub Pages.

If you enjoyed this article, leave a comment and give us a clap!

These blogs were written as part of the “Passing XSAMS” project. To learn more about the project, check out its project page. Thank you to our proof readers Jan van Dijk, Daan Boer, Lourens Veen and Patrick Bos.

Use-after-free? No more with Valgrind

Lourens Veen — Mon, 28 Sep 2020 00:00:00 GMT

In my long career as a code detective, I have seen it all. I have travelled the high Cs, found Rubies and wrestled Pythons, and walked the alleyways of Fortran at night. Of course, as with any job most cases are run-of-the-mill, but some of them are not. Those are the ones you remember. The ones you tell your grandchildren about, if you live to tell the tale.

This is the story of a programming mistake that caused a C++ program to crash, and how it was fixed using Valgrind. It features mysterious artefacts, C++ memory management, and fixing and avoiding mistakes in (scientific) software. I hope it will teach you something about use-after-free issues, and inspire you to try Valgrind for memory debugging next time you have a similar problem. Let’s dive right in, shall we?

The Curious Case of the Crashing Simulation

It was a dark and stormy ni…no wait, it was actually a sunny afternoon in spring, when I received an email from a scientist named Pavel. Pavel is a postdoc at the University of Amsterdam and the Erasmus University Medical Center in Rotterdam, and he was working on porting a rather complicated computer simulation of something called an In-Stent Restenosis to MUSCLE3. His program was crashing. And he suspected the problem was with MUSCLE3. Which I created and currently maintain. Which would mean… DUH-DUH-DUUUUUH (organ sound effect) THERE WAS A MISTAKE IN MY CODE!

By the way, why can’t you include sound effects in blog posts? I think blog posts are much better with sound effects. Someone should fix this. Meanwhile, here’s an image depicting the horror:

“The Scream” Undated drawing by Edvard Munch (via Wikipedia )

Anyway, yes, there was a mistake in MUSCLE3. But to explain the problem, I need to explain what MUSCLE3 does first. MUSCLE3 is a coupling system for multiscale simulations. In a multiscale simulation, a part of the world is simulated in which multiple processes take place, and where some of those processes are much smaller and/or faster than others.

In this particular simulation, there are some slow-growing cells (simulated using agents), and some fast-moving blood (simulated using computational fluid dynamics). This is not that relevant, but it is fortunate, because what’s a hard-boiled detective story without some fast-moving blood? Anyway, each process is simulated by a separate computer program, and the programs exchange information by sending messages back and forth. This communication is one of the things MUSCLE3 helps with. And now Pavel was saying that whenever he received a large message, his simulation crashed.

Messages in MUSCLE3 contain data, and that data can be of a range of types, including the standard built-in types such as strings and integers, but also lists, dictionaries, and grids (arrays) of numbers. You don’t need to tell MUSCLE3 what you’re going to send, you just build a Message object containing whatever kind of data you want to send, and pass it to MUSCLE3, and on the other side the receiver receives a Message object containing the same data.

For a dynamic language like Python, this is all simple enough, but, dear reader, this case would prove to be much more complicated. Pavel’s model was not written in Python. It was written in… DUH-DUH-DUUUUUH C++!

See? I told you, sound effects. They make things better.

Anyway, C++ is a statically typed language. C++ programs are compiled into an executable file, which is then run in a second step. In C++, each variable can only store values of a particular type (e.g. strings, or integers), and that type must be known when the program is compiled. Of course, we don’t know the type of the data in a message until we receive that message, which happens when the program is running, which comes after it is compiled. So there is a problem.

To solve this, the C++ API of MUSCLE3 has a Data class. Variables of type Data can contain an object of any type that MUSCLE3 can send and receive. In your C++ code, you can ask the Data object which type of data it contains, and extract an object of that type. If the data is a grid (for instance because the sender is a Python program that sent you a NumPy array), then you can obtain a pointer to the raw array of elements using the elements() member function of Data (C++ doesn’t have a standard multi-dimensional array class, so this is the best we can do).

That is exactly what Pavel did: he called that function, then copied the elements into another data structure. And then his program crashed. At least, it crashed if the grid was large, a test program receiving a small test grid with only a handful of elements worked fine. Which makes no sense, because MUSCLE3 treats large and small grids exactly the same.

And thus, our mystery. Why did it crash? Why only for large grids? How should it be fixed? How do we keep this from happening again in the future?

Detective. It’s what you are, not what you do.

It was time to get to work. I collected my hat and put on my trench coat, then dialled in the combination on my wall safe. This job would require some tools. My gaze swept across my collection, carefully assembled over the years. On the left, there were two boxes of print statements. One contained a large pile of cheap plastic mass-produced ones, the other was a velvet-lined display case showcasing a set of two exquisite hand-made examples, made in Japan during the Kamakura period and razor-sharp. While the pride of my collection, and the pinnacle of debugging tools in their time, I would need something more modern for this.

Next were the linters. They would only protect me from Python formatting issues however. Useless, in this case. A debugger then, maybe? My safe contained a perfectly good copy of gdb, but the truth is that it’s kind of uncomfortable to use. Also, it’s more suited to logic problems than to finding the cause of crashes. I put it aside for now.

I moved on to a shiny white box. AddressSanitizer it said on the top, in colourful letters. That could work. I’d never liked the sterility of it however. It was too neat, too perfect. A tool of the Empire. No. No, for this job, I would need the final box. A steel chest rather, decorated with ancient runic symbols and adorned with the word Valgrind in large, Gothic script. Gateway to Valhalla, the great hall where brave programmers go after they retire to await Ragnarök, the end of times when hardware issues and compiler bugs arise for one final epic debugging session before the Great Simulation is rebooted. Yes, Valgrind. I took the chest, put it on my desk, opened it. Removed the gleaming artefact, and put it in my pocket. I was ready to go.

(If the above procedure sounds a bit over-the-top, apt-get install valgrind will work just fine as well.)

It was only a short walk to the site of the crime, a large building with the word git on a sign at the entrance. I walked in and asked the receptionist to see MUSCLE3. She gave me a nasty look. Understandable, given the fact that I look ridiculous in a hat, and that no one has worn a beige trench coat since the 1980’s unless they were a dirty old man. But, having explained that I was in fact a detective, I was waved on and soon found myself in a room with a series of filing cabinets. It was time to start my investigation.

First, I looked at a cabinet labelled libmuscle/cpp/src/libmuscle/tests. I knew that there had to be a test which tested that grids could be sent and received. In fact there was; a file conveniently named test_data.cpp contained a large number of tests for Data objects, including ones for various aspects of using grids. One of those did exactly what Pavel did. I ran the tests to be sure, and they passed just fine.

[ RUN ] libmuscle_mcp_data.grid
[ OK ] libmuscle_mcp_data.grid (120 ms)
[ RUN ] libmuscle_mcp_data.grid_serialisation
[ OK ] libmuscle_mcp_data.grid_serialisation (55 ms)

However, the test only used a small grid. I modified it to use a larger grid. Re-ran the tests. And sure enough…

[ RUN ] libmuscle_mcp_data.grid
Segmentation fault (core dumped)

So, I had verified that indeed there had been a crime, erm, that there was a mistake in my code. But where? It was time to deploy Valgrind. I modified the Makefile that runs the tests to run valgrind <test_executable> instead of just <test_executable>. Having thus installed the Valgrind onto the filing cabinet, I took cover behind another cabinet filled with Fortran code (heavy, voluminous, and in this case auto-generated and therefore expendable) and restarted the test suite.

Valgrind in action

As the first tests ran, a bluish light filled the room and the ancient artefact started vibrating, emitting a faint hum of approval. Then, suddenly, a horrifying screech emanated from it.

[ RUN ] libmuscle_mcp_data.grid
==19383== Invalid read of size 4
==19383== at 0x13F848: testing::AssertionResult testing::internal::CmpHelperEQ<int, int>(char const*, char const*, int const&, int const&) (gtest.h:1444)
==19383== by 0x1368DF: testing::AssertionResult testing::internal::EqHelper<false>::Compare<int, int>(char const*, char const*, int const&, int const&) (gtest.h:1472)
==19383== by 0x1288B1: libmuscle_mcp_data_grid_Test::TestBody() (test_data.cpp:526)
==19383== by 0x1A21E6: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x19C810: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x17BBBB: testing::Test::Run() (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x17C518: testing::TestInfo::Run() (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x17CB9B: testing::TestCase::Run() (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x187987: testing::internal::UnitTestImpl::RunAllTests() (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x1A32F8: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x19D576: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== by 0x18640F: testing::UnitTest::Run() (in muscle3/libmuscle/cpp/build/libmuscle/tests/test_data)
==19383== Address 0x6488628 is 8 bytes inside a block of size 262,152 free’d
==19383== at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19383== by 0x13317B: msgpack::v1::zone::chunk_list::~chunk_list() (cpp11_zone.hpp:127)
==19383== by 0x13554D: msgpack::v1::zone::~zone() (cpp11_zone.hpp:27)
==19383== by 0x152D7D: void __gnu_cxx::new_allocator<msgpack::v1::zone>::destroy<msgpack::v1::zone>(msgpack::v1::zone*) (new_allocator.h:140)
==19383== by 0x152D50: void std::allocator_traits<std::allocator<msgpack::v1::zone> >::destroy<msgpack::v1::zone>(std::allocator<msgpack::v1::zone>&, msgpack::v1::zone*) (alloc_traits.h:487)
==19383== by 0x1520AA: std::_Sp_counted_ptr_inplace<msgpack::v1::zone, std::allocator<msgpack::v1::zone>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:535)
==19383== by 0x13A385: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:154)
==19383== by 0x135CCC: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:684)
==19383== by 0x1345AF: std::__shared_ptr<msgpack::v1::zone, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1123)
==19383== by 0x1345CB: std::shared_ptr<msgpack::v1::zone>::~shared_ptr() (shared_ptr.h:93)
==19383== by 0x14E99D: void std::_Destroy<std::shared_ptr<msgpack::v1::zone> >(std::shared_ptr<msgpack::v1::zone>*) (stl_construct.h:98)
==19383== by 0x14DE43: void std::_Destroy_aux<false>::__destroy<std::shared_ptr<msgpack::v1::zone>*>(std::shared_ptr<msgpack::v1::zone>*, std::shared_ptr<msgpack::v1::zone>*) (stl_construct.h:108)
==19383== Block was alloc’d at
==19383== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19383== by 0x13335A: msgpack::v1::zone::allocate_expand(unsigned long) (cpp11_zone.hpp:281)
==19383== by 0x133280: msgpack::v1::zone::allocate_align(unsigned long, unsigned long) (cpp11_zone.hpp:245)
==19383== by 0x157F74: msgpack::v2::detail::create_object_visitor::visit_bin(char const*, unsigned int) (create_object_visitor.hpp:133)
==19383== by 0x165DC3: msgpack::v2::detail::context<msgpack::v2::detail::parse_helper<msgpack::v2::detail::create_object_visitor> >::execute(char const*, unsigned long, unsigned long&) (parse.hpp:584)
==19383== by 0x162A3D: msgpack::v2::detail::parse_helper<msgpack::v2::detail::create_object_visitor>::execute(char const*, unsigned long, unsigned long&) (parse.hpp:1027)
==19383== by 0x160F33: msgpack::v2::parse_return msgpack::v2::detail::parse_imp<msgpack::v2::detail::create_object_visitor>(char const*, unsigned long, unsigned long&, msgpack::v2::detail::create_object_visitor&) (parse.hpp:1044)
==19383== by 0x1587F5: msgpack::v2::detail::unpack_imp(char const*, unsigned long, unsigned long&, msgpack::v1::zone&, msgpack::v2::object&, bool&, bool (*)(msgpack::v1::type::object_type, unsigned long, void*), void*, msgpack::v1::unpack_limit const&) (unpack.hpp:331)
==19383== by 0x16C6BE: msgpack::v3::unpack(msgpack::v1::zone&, char const*, unsigned long, unsigned long&, bool&, bool (*)(msgpack::v1::type::object_type, unsigned long, void*), void*, msgpack::v1::unpack_limit const&) (unpack.hpp:141)
==19383== by 0x16C766: msgpack::v3::unpack(msgpack::v1::zone&, char const*, unsigned long, bool (*)(msgpack::v1::type::object_type, unsigned long, void*), void*, msgpack::v1::unpack_limit const&) (unpack.hpp:182)
==19383== by 0x16C5D8: libmuscle::impl::mcp::unpack_data(std::shared_ptr<msgpack::v1::zone> const&, char const*, unsigned long) (data_pack.cpp:13)
==19383== by 0x1563C4: libmuscle::impl::DataConstRef::grid_dict_() const (data.cpp:722)
==19383==

It ended quickly, fortunately, and was replaced again by the soothing hum. When that too had faded away, I emerged from my shelter and approached the filing cabinet. The Valgrind felt warm to the touch, but it appeared to be intact. I took out my magnifying glass and inspected the symbols on its side.

[ RUN ] libmuscle_mcp_data.grid
==19383== Invalid read of size 4
==19383== at 0x13F848: testing::AssertionResult testing::internal::CmpHelperEQ<int, int>(char const*, char const*, int const&, int const&) (gtest.h:1444)
==19383== by 0x1368DF: testing::AssertionResult testing::internal::EqHelper<false>::Compare<int, int>(char const*, char const*, int const&, int const&) (gtest.h:1472)
==19383== by 0x1288B1: libmuscle_mcp_data_grid_Test::TestBody() (test_data.cpp:526)

Invalid read of size 4. Apparently, the test had attempted to read a 4-byte variable in a memory location where there was none. I followed the backtrace, through some Google Test-related functions to my test_data.cpp. Line 526 was where the invalid read had occurred. I checked the test’s code. Bingo. Exactly the place where the grid elements were accessed. I seemed to be on to something.

At this point, there were several possible causes of the crash. Clearly, the pointer returned by the call to elements() was bad, pointing to some memory that did not contain the requested elements. I read on.

==19383== Address 0x6488628 is 8 bytes inside a block of size 262,152 free’d
…
==19383== Block was alloc’d at
…
==19383== by 0x1563C4: libmuscle::impl::DataConstRef::grid_dict_() const (data.cpp:722)
==19383==

That, in fact, told me enough to find the problem. But to understand what went wrong, we need to dive into the implementation of the Data class for a bit.

How to lose your users’ data

To send objects from one program to another, MUSCLE3 uses a binary data format called MessagePack. A MessagePack-encoded object is an array of bytes, which can be decoded to determine the type and value of the encoded value. For example, a message consisting of a single byte with value 33 represents an integer with value 33. A single byte with value 195 represents the Boolean value True, and 167 77 85 83 67 76 69 51 encodes the string MUSCLE3 (167 starts a 7-byte string, and then there are the 7 characters’ ASCII values).

MessagePack has fixed ways of encoding all the common basic types, as well as dictionaries and lists, but it doesn’t do (multidimensional) arrays. Fortunately, it has an extension mechanism, which lets you send an array of bytes accompanied by a one-byte tag that specifies what kind of object it represents. You are then free to represent your object as an array of bytes in any way you like. A common trick is to simply use MessagePack again to encode a dictionary or list containing some values which together represent the object. This is how MUSCLE3 sends grids.

When MUSCLE3 receives a message, it calls an internal function to decode the received array of bytes. This function in turn calls the MessagePack library, which returns a library-specific object that represents the decoded value. This object is wrapped in a Data object, and eventually returned to the user. When the user then asks this Data object whether it contains a Grid, it inspects its MessagePack-object to see whether it represents an extension type with the correct tag. If you ask for the elements, it has to actually decode the bytes still. It uses MessagePack to do that, creating another Data object containing a dictionary that has the shape of the array and the elements (the difference between Data and DataConstRef is beyond the scope of this blog, you can consider them the same here).

DataConstRef DataConstRef::grid_dict_() const {
    auto ext = mp_obj_->as<msgpack::type::ext>();
    auto oh = msgpack::unpack(ext.data(), ext.size());    if (oh.get().type != msgpack::type::MAP)
        throw std::runtime_error(
            "Invalid grid format. Bug in MUSCLE 3?");    auto zone = std::make_shared<msgpack::zone>();
    return DataConstRef(
        mcp::unpack_data(zone, ext.data(), ext.size()));
}

Once it has this Data object, elements() can extract the location of the elements from it and return the location to the user as a pointer of the appropriate type:

Element const * DataConstRef::elements() const {
    if (!is_a_grid_of<Element>())
        throw std::runtime_error(
            "Tried to get grid data, but this object is not"
            " a grid or not of the correct type.");
    char const * data_bytes = grid_dict_()["data"].as_byte_array();
    return reinterpret_cast<Element const *>(data_bytes);
}

Valgrind said the following about this:

==19383== Address 0x6488628 is 8 bytes inside a block of size 262,152 free’d
…
==19383== Block was alloc’d at
…
==19383== by 0x1563C4: libmuscle::impl::DataConstRef::grid_dict_() const (data.cpp:722)
==19383==

This means that the pointer returned by elements() points to a variable that used to exist, but that had been deleted by the time the pointer was used by the test case. It also says that the deleted variable was created on the last line of the grid_dict_() function.

That makes it clear what happened: elements() calls grid_dict_(), which creates (as noted by Valgrind) and returns a Data object containing the elements. A pointer to the elements is extracted (the ["data"].as_byte_array() part in elements()), and then, since it’s a temporary value and it’s not assigned to anything, the Data object returned by grid_dict_() is cleaned up. This deletes the dictionary including the elements from memory, leaving the pointer to point to something which no longer exists. The pointer is then returned to the user, who tries to access the nonexistent data, causing the program to crash (or not, sometimes).

Accessing a variable which no longer exists is called a use after free error and it can, but doesn’t have to, crash your program. Here’s why. The memory in your computer is divided up into blocks called segments, and each running program is assigned a data segment to work with. If it needs more memory, it can ask the operating system to increase the size of its data segment, and if it doesn’t need the extra memory any more, it can ask to shrink its data segment again so that other running programs can use the memory. If a program tries to read or write to memory outside of a segment assigned to it, the CPU will block the operation and then the operating system will shut down the program, citing a segmentation fault or segmentation violation (colloquially, a segfault). It does this to protect other running programs, which would get messed up if their data is overwritten by their errant colleague.

An errant colleague trying to overwrite your data. At this point you may want to call a detective! Photo by Andrea Piacquadio from Pexels.

Resizing the data segment is done by the C++ standard library behind the scenes, so you don’t have to do it by hand, you can just new and delete variables (or better, use std::make_unique() and std::make_shared()). The standard library typically doesn’t change the size of the data segment every time you create or delete a variable, as it’s quite expensive to do so. It’s quite common for programs to create and delete small variables all the time, so it makes sense to hold on to a bit of spare memory. However, if a large enough amount is freed, it should be given back to the operating system for use by someone else, so that’s what the standard library typically does.

This is probably the explanation for why the test only crashes if the test grid is large: the small grid is still inside our data segment, and although its memory has been marked as available, it hasn’t been overwritten or returned to the OS, so that the pointer returned by elements() still works even if it’s technically invalid. The segment checking system is not intended to detect mistakes inside a program like this, it’s just there to protect programs from each other. So we get away with it, and the test passes. For the larger grid, the memory has been returned to the OS, so it does trigger a segfault, but it’s somewhat accidental.

When you run a program with Valgrind’s memcheck (which is what we did here), it actually redirects the calls to create, delete and access variables that the program makes away from the standard library, and to its memcheck tool. This does a lot of extra checking and bookkeeping, allowing it to detect invalid accesses within the program and tell you what happened. All these extra checks do slow down your program a lot, so you don’t want to run under Valgrind all the time, but when you have an issue like this, it’s well worth a bit of a wait if needed. As we saw above, it does a pretty good job showing what’s going wrong, and it’s not so easy to find these kinds of mistakes with other tools.

Fixing the problem

Satisfied with these results, I removed the Valgrind from the test cabinet and returned it to my coat pocket. It had proven its worth once again. I wasn’t done however, as the mistake still needed to be rectified. I pulled up a chair to another filing cabinet, this one labeled libmuscle/cpp/src/libmuscle, and pulled up the data.cpp file to consider my options.

A Data object actually contains two things: the MessagePack object representing the data, and a MessagePack zone object. Zones are used by MessagePack to manage memory. If you decode, say, a dictionary, then you get MessagePack objects for the dictionary itself, and also for each key and value, recursively. That’s a lot of objects, so MessagePack puts them all together into a zone. When you’re done with the data, you just delete the zone, cleaning them all up in one go. This is not as easy as automatic memory management, but it is much more efficient.

Now, our problem is that we create a second Data object containing the grid dictionary, and that we need to put it somewhere where it will continue to exist for as long as the user accesses the pointer returned by elements(). MUSCLE3 promises the user that that pointer is valid for as long as the Data object representing the grid exists, so we need to somehow attach the decoded elements to the grid’s Data object.

One option is to put the elements in the main Data object’s zone. That will keep them around, but it becomes an issue if the user calls elements() multiple times, maybe in a loop. On every call, a new Data object will be created for the dictionary and added to the zone, so we keep using more and more memory. It’ll eventually be freed again, but if it’s a large grid, then we may run out of memory before that. So that’s not great.

A second option would be to add a pointer-to- Data to the Data class, which could be used to store the dictionary Data object if the Data object represents a grid. That way, we would only have to decode once, and any subsequent calls to elements() or shape() could used the cached object. And using a smart pointer would automatically delete the dictionary Data object when the grid Data object is deleted. The downside to this is that it makes Data objects larger, which could cost some performance even for Data objects that don’t contain a grid.

Finally, since the MessagePack format is not so complicated, it would also be possible to make custom decoding routines which decode the grid in one stage, so that there is no need for a second Data object at all. This could also help to reduce copying, thus improving performance further. That’s a lot of work however.

At the moment, the focus for MUSCLE3 is on making things work, and on making the life of the users easy. Performance is not crucial, because in multiscale models (unlike in scale-overlapping multiphysics models) communication performance is rarely the bottleneck. I want to spend my time on solving problems for users, but not problems they don’t have. Since the first option could actually make things worse, and the third option is expensive, I chose the second option. It’s not the prettiest solution, but it will do. If performance does become an issue in the future, then I can always come back and revisit.

Having fixed the problem, there was one more thing to do: keep similar problems from appearing in the future. Mistakes in software are inevitable, as it’s made by humans and nobody’s perfect. But whenever we discover a mistake, it helps to see if there’s an easy way to avoid doing the same thing in the future, so that over time the software gets better and better. In this case, I added a check for Valgrind to the build system, and it will now run all the tests under Valgrind if it is installed. I also modified the continuous integration container to install Valgrind. This didn’t uncover any other memory management issues in MUSCLE3, but if I make another mistake like this in the future, then there’s a good chance that the tooling will catch it before it ends up with the users.

Epilogue

My job done, it was time to go home. I grabbed my coat and my hat, verified that the Valgrind was still securely in my pocket, turned off the lights, and carefully closed the door behind me. It was late. The hallways were deserted, the reception desk downstairs vacant. I opened the after-hours side door and stepped out onto the street, into the golden light of a beautiful spring evening. Life was good. I turned towards home, pulled my hat a bit deeper over my eyes to keep the sun out, and started walking, detective-at-ease. I ignored the funny looks.

Photo by Dewang Gupta on Unsplash

eScience Center Blog

Making 3D Real: How 3DLab Is Bringing Scientific Data to Life

The Challenge: Capturing the World Around You

The Approach: Gaussian Splatting

Impact: From Artifacts to Algorithms

Building a Community of Visualizers

Looking Ahead

Get Involved

Research Software Engineers in the Age of GenAI: Same Value, Changing Practice

Co-Designing a Roadmap for Training on Research Software Templates and Code Auditing

Directions for Training Tutorials

Independence vs Integration

Tool Focus vs Topic Focus

Audiences, Formats, and Collaborations

Key Takeaways

Contact

Provenance in the age of AI: Why we need AI Declarations for more transparency

The foundation: A machine-readable file format

Why another format?

How do I get started?

Scientific and Technical Standards

Join the effort

AI Declaration (aidecl.yaml) for this post

Acknowledgements

FOSDEM 2026: Observations and key takeaways

Flavio Hafner on FOSDEM 2026: Security, LLMs, and Software Performance

Upcoming features in git

Open Source, LLMs, and security

Data, Search and LLMs

Machine learning, performance, and observability

Ole Mussmann on FOSDEM 2026: Nix, International OSS, Accessibility and Collaboration

Opening

Nix and NixOS

International Open Source

Accessibility

Collaboration

Faruk Diblen on FOSDEM 2026: Sovereignty, Burnout, and the AI Reality Check

Reflections on AI and the Community

Networking and the Research Track

Keynotes

Highlights

Other Notable Tracks and Talks

Booths and Extra Notes

Bring Your Own Backend

The sustainability problem

The solution: Bring Your Own Backend

The Magic: URL-based backend configuration

Keeping it simple for project partners

The contract: OpenAPI as the bridge

Try it out yourself

When to use BYOB

Give Me Data, Yesterday

nix? What’s That?

Paper Cuts

nps

Architecture

Database Candidate Boundary Conditions

Data Source

Substring Matching

Query Time

Data Amount

Memory Usage

Database Candidates

grep a Text File 🦖

Benefits

Drawbacks

SQLite 🪶

Benefits

Trade-offs

Drawbacks

Tantivy 🐎

Benefits

Trade-offs

Drawbacks

DuckDB 🦆

Benefits

Trade-offs

Drawbacks

Benchmarking Boundary Conditions

Parameters

`nix`? What’s That?

`nps`

`grep` a Text File 🦖

`hyperfine`, Engage

Comparing Speed against Current `nps`

`nps`

GNU `grep`

`ripgrep`

`Tantivy`

`DuckDB`

`SQLite`

Comparing Speed for Future `nps`