Parallel performance

The MultiH2Lib is an interface to one of two parallel runners; a multiprocess and a MPI runner.

The multiprocess runner is typically easier to install as it does not require MPI, but it is limited to the number of CPUs in one PC or HPC node.

The MPI runner on the other hand is able to distribute the H2Lib execution to CPUs across HPC nodes.

H2Lib automatically switches between the two runners depending on the avilablilty of MPI.

Generate timing results

The cell below writes the file docs/check_parallel_performance.py which run 10s of simulations of the DTU 10MW reference turbine. The timings for different number of instances are saved in a netcdf file.

%%writefile check_parallel_performance.py
import numpy as np
import xarray as xr
import time
import socket
from tqdm import tqdm
from h2lib_tests.test_files import tfp

def get_multih2lib_performance(N):
    try:
        with MultiH2Lib(N, suppress_output=True) as mh2:
            t0 = time.time()
            mh2.init(htc_path=tfp + 'DTU_10_MW/htc/DTU_10MW_RWT_no_output.htc', model_path=tfp + 'DTU_10_MW')
            t1 = time.time()
            mh2.run(10)
            t2 = time.time()
    except ChildProcessError:
        return
    return t1 - t0, t2 - t1, t2-t0

if __name__=='__main__':
    from h2lib import MultiH2Lib
    from multiclass_interface import mpi_interface
    if sys.argv[-1]=='mpi':
        mpi_interface.activate_mpi()
        mpi_interface.TERMINATE_ON_CLOSE = False
    else:
        mpi_interface.mpi = False
    name = ['multiprocess','mpi'][mpi_interface.mpi]

    if mpi_interface.main:
        print (f"Collect parallel performance data at {socket.gethostname()} using {name}")

    N_lst = [1,2,4,8,16,32,64]
    res = np.array([get_multih2lib_performance(N) for N in tqdm(N_lst, disable=not mpi_interface.main)])
    if not mpi_interface.mpi or mpi_interface.rank==0:
        ds = xr.Dataset({k:('N', res[:,i]) for i,k in enumerate(['Initialization', 'Simulation','Total'])},
                   coords={'N':N_lst})
        ds.to_netcdf(f'parallel_performance_{name}_{socket.gethostname()}.nc')

Writing check_parallel_performance.py

To run the script created above on the DTU HPC cluster, Sophia, a working python environment with h2lib and h2lib_tests is needed. The multiprocess and mpi runner result can be obtained with the following commands:

MultiProcess runner

srun --partition workq --nodes 1 --exclusive --pty python check_parallel_performance.py

or

srun --partition workq --nodes 1 --exclusive --pty bash
python check_parallel_performance.py

Note, the number of H2Lib instances is not limited by the number of cores, but you may run into a Too-many-open-files error. To solve that, the maximum allowed number of open file descriptors can be increased to N by ulimit -n N.

MPI Runner

srun --partition workq --mpi=pmix_v2 --nodes 2 --ntasks-per-node 32 --ntasks 64 --exclusive --unbuffered python check_parallel_performance.py mpi

or

srun --partition workq --nodes 2 --mpi=pmix_v2 --ntasks=64 --exclusive --pty bash
mpirun -n 64 python check_parallel_performance.py mpi

Note, with this approach the number of H2Lib instances is limited by the number of mpi_workers, i.e. N<=ntasks

Plot the timing results

c_lst = [plt.plot([], label=l)[0].get_color() for l in ['Initialization', '10s simualation', 'Total']]
plt.plot([],'-',color='gray', label='Multiprocess, 1 node (32 CPUs)')
plt.plot([],'--', color='gray', label='MPI, 2 nodes (64 CPUs)')

for s, n in zip(['.-','.--'], ['parallel_performance_multiprocess_sophia.nc', 'parallel_performance_mpi_sophia.nc']):
    ds = xr.load_dataset(n)
    plt.plot(ds.N, ds.Initialization, s, color=c_lst[0])
    plt.plot(ds.N, ds.Simulation, s, color=c_lst[1])
    plt.plot(ds.N, ds.Total, s, color=c_lst[2])
plt.ylabel('Time [s]')
plt.xlabel('H2Lib instances [-]')
plt.legend()
plt.grid()

It is seen that the initialization time is almost linear with the number of instances. This is problably due to input file reading.

The simulation time of the MPI runner is almost constant, while the simulation time of the Multiprocess runner is nearly constant up to 32 instances.

Note that this is the optimal performance example as there is no communication between the instances, the time step loop occurs in the h2lib fortran code and the output file writing is disabled.