Distributed computing aims to offer tools and mechanisms that enable the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources in a transparent way. The research done in this team is based on the past expertise of the group, and on extending it towards the aspects of distributed computing that can benefit from this expertise. The team at BSC has a strong focus on programming models and resource management and scheduling in distributed computing ...

Provide an integrated, comprehensive, modular modelling and testing framework

Develop multi-scale workflows applicable beyond the identified test-areas enabling improved physical understanding and progress beyond state-of-the-art in the earthquake process.

Develop and implement 6 DTCs covering earthquake-related aspects over long and short time scales

Test the 6 DTC-E at 4 relevant sites: Euro-Med (SD8), Central Apennines and Alto-Tiberina (SD9), Bedretto Lab (SD10) and the Alps (SD11).

Develop and implement 1 DTC for data-informed Probabilistic Tsunami Forecasting (PTF) (DTC-T1)

Test the DTC-T1 through demonstrators at 4 relevant sites: Mediterranean sea coast (SD4), Eastern Sicily (SD5), Chilean cost (SD6), and Eastern Honshu coast in Japan (SD7).

Develop and implement 4 DTCs for volcano-related extremes: volcanic unrest (DTC-V1), forecast of volcanic ash clouds and fallout (DTC-V2), lava flows (DTC-V3), and volcanic gases (DTC-V4).

Test the 4 DTC-V through demonstrators at 3 relevant European sites: Mt. Etna in Italy (SD1), and Grímsvötn and Fagradalsfjall in Iceland (SD2 and SD3 respectively).

Develop and implement 1 DTC for Anthropogenic Geophysical Extreme Forecasting (AGEF) with 4 workflow outcomes: forecasting of long-range responses of georeservoirs (TC-AGEF1), forecasting of late responses of georeservoirs (TC-AGEF2), modelling of the largest magnitude (TC-AGEF3), and induced seismic hazard map estimation (TC-AGEF4).

Test the DTC-A through demonstrators at 2 relevant European sites: Strasbourg geothermal site in France (SD12) and KGHM copper ore mine in Poland (SD13).

Team created to publish applications during COMPSs Tutorials, and share them among participants.

This team is to publish workflows executed from WPs that are not the three main pillars of the project.

It explores the modelling of natural catastrophes – in particular, earthquakes and their associated tsunamis shortly after such an event is recorded.

It develops innovative adaptive workflows for climate and for the study of Tropical Cyclones (TC) in the context of the CMIP6 experiment, including in-situ analytics.

It focuses on the construction of DigitalTwins for the prototyping of complex manufactured objects integrating state-of-the-art adaptive solvers with machine learning and data-mining, contributing to the Industry 4.0 vision.

Project that aims to create the NeuroPlat portal for neurodrug design

Provenance registration is becoming more and more important, as we increase the size and number of experiments performed using computers. In particular, when provenance is recorded in HPC environments, it must be efficient and scalable. In this paper, we propose a provenance registration method for scientific workflows, efficient enough to run in supercomputers (thus, it could run in other ...

Session during the Innovative HPC workflows for industry (https://eflows4hpc.eu/event/innovative-hpc-workflows-for-industry/) that describes how Workflow Provenance is recorded with COMPSs: the background on the tools used, how the recording has been designed, and how to use it and inspect metadata.

Cheat sheet that serves as a quick guide to the many advanced features that PyCOMPSs offers. Updated for COMPSs release version 3.3.2.

COMPSs Matrix Multiplication resourceUsage profiling example.

MN5 MSIZE=20 BSIZE=768 7 Nodes (6 workers) (--num_nodes=7 --worker_in_master_cpus=0).

• Total number of tasks: 20^3 = 8000
• Maximum code parallelism: 20^2 = 400
• Total cores: 112*6 = 672
• Maximum utilisation: 400 / 112 = 3,57 Nodes

Overall stats from "pycompss inspect":

│ └── overall 
│ ├── matmul_tasks 
│ │ └── multiply 
│ │ ├── maxTime = 91,111 ms 
│ │ ├── executions = 8,000 
│ │ ├── avgTime = 84,839 ms 
│ │ └── minTime = 79,278 ms
...

Application that perform the multiplication between matrices. In this experiment, a new profiling visualization is available, showing the resource usage such as CPU, memory, data read and written to disk, and data sent and received over the network.

CWL + RO-Crate Workflow Descriptions

This repository stores computational workflows described using the Common Workflow Language (CWL) and enriched with metadata using Research Object Crate (RO-Crate) conforming to the Workflow Run RO-Crate profile.

Each workflow is contained in its own directory (e.g., WF5201, WF6101, ...). Inside each workflow directory you will typically find at least:

• The CWL workflow definition (with the same name as the directory, e.g., WF5201.cwl). ...

CWL + RO-Crate Workflow Descriptions

This repository stores computational workflows described using the Common Workflow Language (CWL) and enriched with metadata using Research Object Crate (RO-Crate) conforming to the Workflow Run RO-Crate profile.

Each workflow is contained in its own directory (e.g., WF5201, WF6101, ...). Inside each workflow directory you will typically find at least:

• The CWL workflow definition (with the same name as the directory, e.g., WF5201.cwl). ...

Name: Matrix Multiplication Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...

Name: Matrix Multiplication Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...

Lysozyme in water full COMPSs application

Lysozyme in water full COMPSs application, using dataset_small

Lysozyme in water full COMPSs application

Name: SparseLU Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

The Sparse LU application computes an LU matrix factorization on a sparse blocked matrix. The matrix size (number of blocks) and the block size are parameters of the application.

As the algorithm progresses, the area of the matrix that is accessed is smaller; concretely, at each iteration, the 0th row and column of the current matrix are discarded. ...

COMPSs Matrix Multiplication, out-of-core using files. Hypermatrix size used 2x2 blocks (MSIZE=2), block size used 2x2 elements (BSIZE=2)

Name: Matrix multiplication with Files, reproducibility example, without data persistence Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number ...

Name: Matrix multiplication with Files, reproducibility example Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying ...

Lysozyme in water full COMPSs application run at MareNostrum IV, using full dataset with two workers

Lysozyme in water sample COMPSs application

Cluster Comparison COMPSs application

Cholesky factorisation COMPSs application

K-means COMPSs application

Wordcount reduce version COMPSs application

Wordcount merge version COMPSs application

Solutions to the exercises on the Workflow Provenance part, during January 2024 COMPSs Tutorial.