eFlows4HPC project aims at providing workflow software stack and an additional set of services to enable the integration of HPC simulations and modelling with big data analytics and machine learning in scientific and industrial applications. The project is also developing the HPC Workflows as a Service (HPCWaaS) methodology that aims at providing tools to simplify the development, deployment, execution and reuse of workflows. The project demonstrates its advances through three application Pillars ...

Provenance registration is becoming more and more important, as we increase the size and number of experiments performed using computers. In particular, when provenance is recorded in HPC environments, it must be efficient and scalable. In this paper, we propose a provenance registration method for scientific workflows, efficient enough to run in supercomputers (thus, it could run in other ...

Session during the Innovative HPC workflows for industry (https://eflows4hpc.eu/event/innovative-hpc-workflows-for-industry/) that describes how Workflow Provenance is recorded with COMPSs: the background on the tools used, how the recording has been designed, and how to use it and inspect metadata.

Cheat sheet that serves as a quick guide to the many advanced features that PyCOMPSs offers. Updated for COMPSs release version 3.3.2.

COMPSs Matrix Multiplication resourceUsage profiling example.

MN5 MSIZE=20 BSIZE=768 7 Nodes (6 workers) (--num_nodes=7 --worker_in_master_cpus=0).

• Total number of tasks: 20^3 = 8000
• Maximum code parallelism: 20^2 = 400
• Total cores: 112*6 = 672
• Maximum utilisation: 400 / 112 = 3,57 Nodes

Overall stats from "pycompss inspect":

│ └── overall 
│ ├── matmul_tasks 
│ │ └── multiply 
│ │ ├── maxTime = 91,111 ms 
│ │ ├── executions = 8,000 
│ │ ├── avgTime = 84,839 ms 
│ │ └── minTime = 79,278 ms
...

Application that perform the multiplication between matrices. In this experiment, a new profiling visualization is available, showing the resource usage such as CPU, memory, data read and written to disk, and data sent and received over the network.

Lysozyme in water full COMPSs application. Added new WRROC profile: Provenance Run Crate

Application that perform the multiplication between matrices.

Name: Random Forest Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum4 This is an example of Random Forest algorithm from dislib. To show the usage, the code generates a synthetical input matrix. The results are printed by screen. This application used dislib-0.9.0

Name: GridSearchCV Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5

GridSearch of kNN algorithm for the iris.csv dataset (https://gist.githubusercontent.com/netj/8836201/raw/6f9306ad21398ea43cba4f7d537619d0e07d5ae3/iris.csv). This application used dislib-0.9.0

Name: GridSearchCV Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5

GridSearch of kNN algorithm for the iris.csv dataset (https://gist.githubusercontent.com/netj/8836201/raw/6f9306ad21398ea43cba4f7d537619d0e07d5ae3/iris.csv). This application used dislib-0.9.0

Name: Matrix Multiplication Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...

Name: Matrix Multiplication Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...

Lysozyme in water full COMPSs application

Name: KMeans Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5

KMEans for clustering the housing.csv dataset (https://github.com/sonarsushant/California-House-Price-Prediction/blob/master/housing.csv). This application used dislib-0.9.0

Name: TruncatedSVD (Randomized SVD) Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5

TruncatedSVD (Randomized SVD) for computing just 456 singular values out of a (4.5M x 850) size matrix. The input matrix represents a CFD transient simulation of air moving past a cylinder. This application used dislib-0.9.0

Name: SparseLU Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

The Sparse LU application computes an LU matrix factorization on a sparse blocked matrix. The matrix size (number of blocks) and the block size are parameters of the application.

As the algorithm progresses, the area of the matrix that is accessed is smaller; concretely, at each iteration, the 0th row and column of the current matrix are discarded. ...

COMPSs Matrix Multiplication, out-of-core using files. Hypermatrix size used 2x2 blocks (MSIZE=2), block size used 2x2 elements (BSIZE=2)

Name: Matrix multiplication with Files, reproducibility example, without data persistence Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number ...

Name: Matrix multiplication with Files, reproducibility example Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs

Description

Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.

If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying ...

Name: Matmul GPU Case 1 Cache-ON Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: Minotauro-MN4

Matmul running on the GPU leveraging COMPSs GPU Cache for deserialization speedup. Launched using 32 GPUs (16 nodes). Performs C = A @ B Where A: shape (320, 56_900_000) block_size (10, 11_380_000)             B: shape (56_900_000, 10)   block_size (11_380_000, 10)             C: shape (320, 10)                block_size ...

Name: Matmul GPU Case 1 Cache-OFF Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs 3.3 Machine: Minotauro-MN4

Matmul running on the GPU without Cache. Launched using 32 GPUs (16 nodes). Performs C = A @ B Where A: shape (320, 56_900_000) block_size (10, 11_380_000)             B: shape (56_900_000, 10)   block_size (11_380_000, 10)             C: shape (320, 10)                block_size (10, 10) Total dataset size 291 ...

Name: K-Means GPU Cache OFF Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: Minotauro-MN4

K-Means running on GPUs. Launched using 32 GPUs (16 nodes). Parameters used: K=40 and 32 blocks of size (1_000_000, 1200). It creates a block for each GPU. Total dataset shape is (32_000_000, 1200). Version dislib-0.9

Average task execution time: 194 seconds

Name: K-Means GPU Cache ON Contact Person: [email protected] Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: Minotauro-MN4

K-Means running on the GPU leveraging COMPSs GPU Cache for deserialization speedup. Launched using 32 GPUs (16 nodes). Parameters used: K=40 and 32 blocks of size (1_000_000, 1200). It creates a block for each GPU. Total dataset shape is (32_000_000, 1200). Version dislib-0.9

Average task execution time: 16 seconds