06 Oct A workflow solution to model and forecast coastal ocean processes and water quality supporting ports and aquaculture industries
The use case was developed by Stichting Deltares with support from EGI.eu, using cloud compute resources from GRNET and CREODIAS. The use case has also been deployed and tested at CESNET. In the near future the use case will be expanded to expanded to a hybrid deployment where parts of the workflow can be run in the cloud and other parts, namely running the Delft3D FM model, will be run on an HPC.
For questions, please contact: bjorn.backeberg@deltares.nl or lorinc.meszaros@deltares.nl
With the European Commission preparing to mobilise resources to implement a digital twin of the Earth, there is a need to easily build and deploy modelling workflows across the heterogenous cloud, high-performance and high-throughput computing infrastructures available in Europe. Within C-SCALE we are developing an interoperable workflow solution that will allow users to easily produce coastal hydrodynamic and water quality simulations and forecasts for any geographic area of interest.
Background
The European Copernicus programme has made significant investments to produce publicly available data that can contribute to operational productivity in different sectors. Unfortunately, many of the potential beneficiaries lack the ability to select, extract and utilize much of the data they require to improve their activities.
Particularly for the ports and aquaculture industries, state of the art high resolution hydrodynamic and water quality data has the potential to improve efficiency and cut operating costs. Additionally, such data facilitates improved understanding of the surrounding environment and facilitates more accurate forecasts of future conditions, supporting:
- Early warning of risk factors such as storms, harmful algal blooms, faecal contamination and other hazards.
- Real time crisis management mitigating impacts of pollution accidents by identifying appropriate responses.
- Information for planning operations which require accurate and reliable meteorological, hydrodynamic and water quality forecasts.
Within this use case we are working towards a workflow solution on the European Open Science Cloud’s compute and data infrastructures that will enable users to produce coastal hydrodynamic and water quality simulations and forecasts for any geographic area of interest using Delft3D Flexible Mesh.
Building on previous developments, docker containers have been created that contain python scripts for downloading, preprocessing and postprocessing data. These make use of the Copernicus Marine Services’ MOTU Client, the Climate Data Store’s API and the python scripts developed for Copernicus COASTSERVproject. The model is run using the Delft3D Flexible Mesh docker container, and the results can easily be visualized in a JupyterHub environment connected to the computing resources.
The workflow solution is available at the C-SCALE HiSea Use Case repository including an example model of the Aegean Sea developed for the HiSea project.
Challenge
Experts developing hydrodynamic and water quality models for hind- and forecasts, typically need to overcome challenges related to
- setting up, calibrating and validating model schematisations
- getting the necessary input data to force the models, especially for model schematisation covering large areas and long simulation periods
- converting the data to formats that the model can read
- compiling and running the model source code on heterogenous computing infrastructures
- postprocessing and visualising unstructured data from flexible mesh models
The workflow solution under development in C-SCALE is addressing challenges 2, 3 and 5.
Cloud and HPC compute services offered by the C-SCALE federation were identified as appropriate solutions for these types of model simulations. Particularly for users that want to run very large, partitioned, model schematisation, HPC compute services are needed. For the implementation on GRNET’s cloud, the download, preprocessing and postprocessing scripts were containerised using docker and the Delft3D Flexible Mesh docker container was used to run the model in the cloud.
Future plans include wrapping the workflow components in a workflow engine such as snakemake or cylc and expanding the workflow to be able to run the model on an HPC using the Delf3D Flexible Mesh Singularity container.
Support from C-SCALE
The HiSea use-case is part of the C-SCALE project and activities started at the beginning of the C-SCALE project. The use case has been implemented in close collaboration between Deltares, EGI.eu and GRNET and has been deployed and tested in CREODIAS and CESNET’s OpenStack clouds. In will in the near future a hybrid workflow will be deployed leveraging GRNET’s cloud and HPC infrastructure. The user and provider teams have had constructive interaction, progressively working towards improved solutions addressing the use case requirements.
C-SCALE services used
Deltares is now using the following C-SCALE services:
- Cloud compute at GRNET, CREODIAS and CESNET
- HPC compute at GRNET
- EGI Check-in to access cloud compute
- SRAM to access HPC
More information and relevant publications
Backeberg, B., Z. Šustr, E. Fernández, G. Donchyts, A. Haag, J. B. R. Oonk, G. Venekamp, B. Schumacher, S. Reimond and C. Chatzikyriakou (2022). An open compute and data federation as an alternative to monolithic infrastructures for big Earth data analytics. Big Earth Data, https://doi.org/10.1080/20964471.2022.2094953
Testimony by researchers at Deltares
“C-SCALE enables us with an easy access to distributed compute and data resources together with an amazing support from IT and data experts of the EOSC providers.”
In this specific case, we used
- C-SCALE Cloud compute to do data downloads, preprocessing, model running and post-processing
- C-SCALE HPC for running models at scale
- C-SCALE AAI (EGI Check-in and SRAM) to enable easy access to compute and data resources.
Sorry, the comment form is closed at this time.