Controversial until it’s not

Open data is a long shot. That’s why it’s so important in a field like particle physics, where contracts are short, but experiments run for decades. Meet Zach, who is one of the drivers behind ATLAS’s open data efforts.

Imagine CERN in the distant future: decades after the LHC has turned off, a new hadron collider is about to launch. Without preserved open data, an entire generation of particle physicists would lack the hands-on experience needed to handle data from the new machine.

One answer to this challenge is that the data collected by the collaborations outlive the collaboration itself. When there is no one around anymore who has been running the collaboration, the most effective way the data stay accessible to anyone is if they are, in fact, available to everyone. Thus, open data can bridge that long gap and get future generations of particle physicists ready for the machine that follows the LHC. To ensure this very long-term vision of keeping the data not only accessible, but also findable, interoperable, and reusable, it needs much effort to get the documentation and metadata right as well as a stable infrastructure provided by CERN’s IT.

In this and the following stories we explore what making data open entails and who is behind these efforts. In this story, we follow Zach Marshall, who co-leads a team of about nine people preparing Open Data for ATLAS, a team that at the time of writing this article has just been honoured with the “2026 ATLAS Outstanding Achievement Award” by the ATLAS collaboration. When asking Zach why he does it, he says he’d always thought it was a good idea to make data open. ATLAS had been releasing open data since 2014, but initially they were focussing on releasing data for education and outreach.

 And if you are wondering whether it is still controversial to work on Open Data: In 2026, the team behind the Open Data efforts at ATLAS consisting of 9 researchers has just been honoured with the 

However, there was some resistance from within the collaboration about releasing open data for research. The use case for education seems much clearer at first glance: Anyone can use authentic data to teach basic or advanced particle physics concepts at undergraduate, or even high-school, level. But if you are a particle physicist and want to work with ATLAS data, it is usually possible to collaborate with ATLAS directly.

When Zach became computing coordinator in 2021, however, he saw the benefits of open data not only in education, but also in research, and pushed for releasing the research-level data too. The first open data for research by ATLAS were eventually released in July 2024. It certainly helped that by that time, ATLAS had endorsed the LHC Open Data Policy, requiring that all research data be published within a certain period after the end of each run. 

A curated approach.

ATLAS and CMS reflect different open data philosophies: CMS releases bulk datasets, most of them Monte Carlo (MC) simulations, for general use, whereas ATLAS publishes a curated set of a few hundred datasets tailored for complete, worked analyses. Stay tuned to read more about CMS’s Open Data efforts in one of the next stories.

No matter which approach is taken, it requires the effort of an open data team. However, it is small and almost entirely voluntary. In ATLAS, nobody is hired to work on open data as their main job, and hardly anyone spends more than a quarter of their time on it, so people come and go as their other commitments allow, which makes long-term planning harder. It also makes the work fast and rewarding, in Zach’s words “as soon as somebody says, this is where I will invest my time, that piece can go very far, very fast”. Much of the effort is carried by students, and the team has done well out of summer-student projects, where a single self-contained notebook makes a natural few-week task.

The people who come along, Zach notes, “usually don’t need motivation”, so the harder part is reaching them in the first place. The team advertises inside the collaboration, in the outreach, education and computing talks at ATLAS Week and around each new release, and every so often someone simply finds them through a web search and is surprised how large the effort is. The most reliable way to win someone over is to show them a concrete use for the data in their own work, such as a course they are about to teach where they can use the open data for their exercises. 

Open data also feeds back into the collaboration.

Despite the large efforts of making everything coherent, of course not everything can be perfect when the data are released. But this doesn’t have to be a disadvantage. Releasing open data is an opportunity for external people to check the collaboration’s work. Just as even collaboration papers which already undergo a sophisticated internal peer review process also have to go through external peer review before being published in a journal, releasing open data can be seen as an external peer review for the data, metadata, and software.

This is exemplified very well in ATLAS’s latest open data release, which was that of the Open Event Generation Data. This sharing of Open Event Generation Data has the potential to significantly reduce the computing resources needed for MC simulations and help in reinterpretation of results and thus the search for new physics in existing data. It was the first time ATLAS has released this type of data after it had been requested by the community of phenomenologists for years. So the open data team had put things out in a preliminary form for some early adopters to test, and this turned out to be very fruitful as the ATLAS team could fix bugs inside their event generators, such as wrong units or inconsistent metadata.

Open Data is an impetus for cleaning up the metadata.

Accurate, rich metadata is essential to make open data useful, but getting technical metadata right across multiple databases is challenging. Different samples often lack human-readable descriptions or contain minor errors like incorrect physical units or inconsistent keywords.

For the first research release of around 300 samples the team could still fix everything by hand, laborious but finite. For the Open Event Generation Data, with several thousand samples, that is no longer possible, so the fixes have to be made at the root: agreeing on clear guidelines, such as consistent rules for how keywords are set, convincing the coordinators responsible for each source to adopt them, and automating the enforcement so that violations get flagged without anyone having to check by hand.

This is another way open data feeds back into the collaboration, because those guidelines then apply to every new sample, not only the ones bound for release, so the internal metadata grows cleaner over time as well. The problem is that the backlog of older samples keeps its inconsistent formats for months or even years, so the full benefit only shows up well down the line.

Once there is coherent documentation and guidelines, it doesn’t only help the open data but also internally, giving easier-to-use documentation and cleaner metadata. “Cleaning it up for the outside world is a good impetus for us to clean it up for ourselves as well.”

Looking back from the future.

Looking back at the journey he took, Zach says, “Open Data has been a great opportunity for me to develop my own skill set, try new things, and produce something genuinely useful for the broader community.” And probably, this is the most important lesson for open data, but also open science in general: the true value of it only becomes apparent in hindsight, as we have already seen during our time travel at the beginning of this story.

There is precedent for this. Some of the clearest cases for open data in HEP come from experiments that have since switched off, where the data live on and keep producing science close to twenty years after the beams have stopped. The H1 experiment at DESY or the BaBar experiment at SLAC are good examples, their data being preserved and continuing to yield results or informing future accelerator projects. For an experiment that has ended, Zach says, the value is obvious, which is a useful reminder that the payoff of opening data often arrives well after the effort of releasing it.

“I hope that we’ll look back on the open data we have released and say, that is a pretty big deal that we managed to do that the first time, and now it’s just accepted. That’s the funny thing about open science, it’s controversial until suddenly it’s not. And then it’s as though it’s always been that way.”

Useful links

The ATLAS open data team has developed plenty of materials on the occasion of the first open data tutorial

The easiest way to get started with ATLAS’s Open Event Generation Data is through this notebook.