Empowering weather and climate forecast
One of the goals of MAELSTROM is to produce production-ready machine learning applications for weather and climate. At the end of February this year, MET Norway launched a deep-learning model developed by MAELSTROM on their public weather app Yr . This model—a U-Net with 1.3 million parameters—predicts 2 meter temperature for the Nordics for up to 58 hours into the future. The model combines information from a high-resolution numerical weather prediction model and temperature measurements from Netatmo’s network of citizen weather stations to create locally accurate predictions.
Thanks to MAELSTROM, the 6 million unique Yr users we have in Norway, Sweden, Denmark, Finland, Estonia, Latvia, and Lithuania can enjoy more accurate temperature forecasts to better plan their day.
Weather forecasting is undergoing a transformation by integrating artificial intelligence (AI) into traditional physics-based models. Mariana Clare, a mathematician from the University of Oxford, specialized in numerical methods and fluid modeling, intertwining mathematics with real-world applications. Her journey led to a PhD in simulating coastal erosion and waves, during which she explored machine learning for weather prediction. Joining the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2022, Mariana focused on enhancing forecast uncertainty information for the Destination Earth initiative. She pioneered the development of the AI-based Integrated Forecasting System (AIFS), which derives weather forecasts from past developments rather than physical laws. By introducing noise into machine-learning models, Mariana aims to create a reliable ensemble forecasting system. Despite its computational efficiency and promising performance, the AIFS requires further development to match the breadth and resolution of traditional models. Mariana envisions a future where physical and machine-learning models complement each other in providing high-resolution, accurate forecasts.
ECMWF will hold a workshop together with ESA's European Space Research Institute:
Machine Learning for Earth System Observation and Prediction
7-10 May 2024 | ESA/ESRIN, Frascati (IT) | Free to attend
The workshop aims to explore the integration of machine learning (ML) and deep learning (DL) techniques with traditional Earth System Observation and Prediction (ESOP) methods. It highlights the significant advancements and potential synergies between ML/DL and conventional tools in satellite observations, weather and climate models, and post-processed model outputs. By leveraging ML/DL, researchers aim to enhance understanding and prediction capabilities of Earth's dynamics, contributing to goals such as the European Commission's Green Deal and Digital Strategy. The workshop spans four days and includes oral and poster presentations, along with parallel working groups aimed at addressing challenges, proposing solutions, and determining future directions in ML/DL for ESOP. The workshop seeks to generate a comprehensive report to advance the fusion of ML/DL with traditional ESOP techniques.
See workshop details | Register here
How are we doing? We're collecting feedback. How useful are our apps and datasets? How easy is it to access them and work with them? Please take the time and complete our survey.
We will host our second workshop and second bootcamp next week at the premises of ECMWF in Reading, UK. We're thrilled to have gotten 242 registrations from 59 (!) countries on 6 continents!
This could become very influential:
Classical numerical models of the atmosphere exhibit biases due to incomplete process descriptions and they are computationally highly demanding. Recent AI-based weather forecasting models reduce the computational costs but lack the versatility of conventional models and do not provide probabilistic predictions.
The AtmoRep project asks if one can train one neural network that represents and describes all atmospheric dynamics. The FZJ team around Martin Schulz recently published a paper describing how this novel, task-independent stochastic computer model of atmospheric dynamics provides superior results for a wide range of applications: nowcasting, temporal interpolation, model correction, and counterfactuals.
Congrats to our MAELSTROM team members Bing Gong, Michael Langguth, Martin Schulz, and their FJZ colleagues!
Read the full paper here.
AtmoRep project page
We're approaching two major events before MAELSTROM turns into the home stretch! The registration for both our second Dissemination Workshop and our second Boot Camp are open. Be sure to reserve your place.
MAELSTROM partner and coordinator ECMWF has four (!) machine learning positions available. If you are a machine learning and/or HPC expert and want to walk the last mile towards operational use of ML in numerical weather predictions and climate services, get excited and check here!
Our second hackathon and our second dissemination workshop, which were both originally planned for spring 2023, will be held in the week of November 6
"We need more women in science." "We support women in science." - that's said easily. But when we took a closer look, we discovered lots of open questions and hidden conflicts. We conducted a range of interviews, checked existing initiatives and sources of information - but still have questions. Take a look and tell us what you think!
Congratulations to Thorsten Höfler, who was honored for his pioneering contributions to large-scale parallel processing systems and supercomputers. We are excited to have such a high-caliber contributor to science among the ranks of our team. Read more here.
Registration is now closed!
43 hand-selected high caliber participants will join us for our first boot camp at Jülich Supercomputing Centre Sept 27 – Sept 30, 2022.
Hands-on work will allow them to learn more about the use of machine learning for weather and climate prediction in general, work with cutting-edge ML applications, test new workflow tools, and use one of Europe's fastest supercomputer!
If you want to try out Germany's fastest supercomputer, hang with ECMWF's and JSC' scientific staff and other high-caliber people from across Europe, or be one of the first to try methods that may usher in a new era of weather & climate prediction:
You may be a candiate for our Boot Camp taking place Sept 27 through Sept 30 at Forschungszentrum Jülich!
Eligible for participation are Master and PhD students. You should have a basic background either in meteorology or machine learning or both. Some limited knowledge of Python and standard ML frameworks (e.g. TensorFlow or PyTorch) is preferred. Ideally, you are also familiar with Jupyter Notebooks and git, though this is not required.
This event will be held in English.
There will be no registration fee, but travel and accommodations must be paid by your home institution (or yourself).
Register here until 15th Aug 2022
We at MAELSTROM are no poster child either, no. Our team is a great bunch of people from all over the world, young and old, with all different kinds of backgrounds – but also mainly male.
HPC is about to change science and technology, and experience tells, that whatever changes science and technology, will sooner or later change our societies, our culture. Our hopes are that it will change the world for the better. But for this aim, women's perspectives, skills and opinions must be a full part of HPC.
A related event worth putting in your diaries is the Women in HPC — ISC’22 Workshop. Y'all probably have the ISC 2022 on your radars anyway, so make sure to allow time on Thu 02, Jun, 2022 9:00 am - 1:00 pm.
60 participants, five MAELSTROM speakers, three speakers from our EuroHPC partner projects, and three speakers from the wider science and technology community, interactive polling and subsequent discussions made our first dissemination workshop a high-profile forum on the present and the future of ML-based weather & climate forecasting in a HPC context. To download the presentations that were shown, and watch the recordings of our talks, visit the ECMWF page.
150 registrations for our first workshop! Yikes.
We want to make this a truly interactive event, and we're honored to welcome speakers of some of our EuroHPC sister projects: TimeX, Red-Sea and Deep-Sea.
Also, we were able to win more high-caliber speakers:
Jussi Leinonen (MeteoSwiss) will be talking about "Time-Consistent Downscaling of Atmospheric Fields with Generative Adversarial Networks";
Ryan Abernathey (Columbia) will introduce "Pangeo: An Open Source Ecosystem for Data-Intensive Science";
Thorsten Kurth (from our partner NVIDIA) will feature "Deep Learning for Earth Sciences in the HPC Context".
Luckily, this event is digital, so the size of our venue is flexible. You can still be part of it:
Visit the ECMWF page to register.
Let's celebrate all the great work of women in science. See, for example, a couple of fantastic female individuals working at ECMWF.
There has been been a 4-hour Crash Course in machine learning for Earth System Science at the "Center for Earth System Observation and Computational Analysis" (CESOC) by MAELSTROM Coordinator Peter Dueben that included both lectures and hands-on exercises.
The company Graphcore has used the MAELSTROM dataset on the emulation of gravity wave drag parametrisation for performance tests with their Intelligence Processing Units (IPUs). See Graphcore's blog here.
What data scientist do can be pretty abstract to non-datanerds. We felt we would like to give everybody some feeling for how important it is to understand and predict weather and climate better, but also how difficult it is. Finally, however, what awesome possibilities a combination of high performance computing and machine learning put in our hands! Have a look and tell us what you think!
Hello machine learning and weather & climate scientists! Learn about the first versions of our machine learning tools and solutions, including architectures and loss functions. Get the results of a survey of customized machine learning solutions here.
We're excited to have reached a big milestone end of August: a series of datasets went public.
Dataset for energy production forecast
Dataset for 2m temperature downscaling
Dataset for ensemble predictions
Datasets for 2m temp. and precipitation short-range forecasts
Dataset to emulate radiation
Feedback and comments will be much appreciated.
Why is project MAELSTROM urgently needed, and how will it impact weather & climate prediction, machine learning and supercomputing? Read in ECMWF's science blog how MAELSTROM will radically change W&C predition with new ML tools, HPC-ready development platforms for data scientists and high-performance compute system designs: ECMWF Science Blog
A paper on physics emulation using machine learning has been accepted for publication in the Journal of Advances in Modelling of the Earth System (JAMES). This work creates a MAELSTROM dataset to emulate a piece of ECMWF’s weather forecast model and will feature in JAMES as part of a new special collection on using machine learning for earth system modelling. The work emulates the parametrisation of non-orographic gravity wave drag (NOGWD), a key physical process in the accurate modelling of stratospheric winds and temperatures. Using neural networks we accurately emulate this physical process even over long simulation periods and by using a high-fidelity reference scheme, we are able to improve prediction quality when compared to the standard scheme. Creating and publishing a dataset will enable other researchers to test machine learning methods and seek to produce even better emulators. More information
At ISC 2021 - one of the largest virtual community get togethers for high performance computing (HPC), machine learning (ML) and high performance data analytics (HPDA) - MAELSTROM member Peter Dueben has chaired a focus session on machine learning for weather forecasting. Here, Jason Hickey from Google Research provided an overview on the latest developments of machine learning applications for now-casting, MAELSTROM member Martin Schultz from Forschungszentrum Juelich talked about whether bigger is always better for deep learning applications in air quality, and Duncan Watson-Parris from the University of Oxford showed us results on the emulation of climate models. More information
The MAELSTROM project has been kicked-off via a virtual meeting. The meeting started with a short talk by our Project Officer Daniel Opalka and continued with presentations and discussions of the different work packages. The public had the opportunity to learn more about the project during a public plenary presentation. The recording of the presentation can be found here.
Great news! In January 2020, we have submitted our application - and now we got notice that the European Commission has signed our grant agreement. We're excited to get the chance to turn the page and write a new chapter of weather and climate forecasting, but equally a new chapter of HPC-based machine learning software environments and hardware configuration.