Compiling applications in Mahti

General instructions

Whenever possible, use the local disk on the login node for compiling software.
- Compiling on the local disk is much faster and shifts load from the shared file system.
- The local disk is cleaned frequently, so please move your files elsewhere after compiling.

Building MPI applications

C/C++ and Fortran applications can be built with GNU or AMD compiler suites. The GNU compilers are loaded by default. AMD compilers can be loaded using the Modules system with the command:

module load aocc

Different applications function better with different suites, so the selection needs to be done on a case-by-case basis.

The MPI environment in Mahti is OpenMPI, and when building MPI applications all compiler suites can be used with the mpicc (C), mpicxx (C++), or mpif90 (Fortran) wrappers.

The compiler options for different suites are different. The recommended basic optimization flags are listed in the table below. It is recommended to start from the safe level and then move up to intermediate or even aggressive, while making sure the results are correct and the program's performance has improved.

Optimisation level	GNU	AMD (clang)
Safe	-O2 -march=native	-O2 -march=native
Intermediate	-O3 -march=native	-O3 -march=native
Aggressive	-O3 -march=native -ffast-math -funroll-loops	-O3 -march=native -ffast-math -funroll-loops

A detailed list of options for the GNU and AMD compilers can be found on the man pages (man gcc/gfortran) when the corresponding programming environment is loaded, or in the compiler manuals (see the links above).

List all available versions of the compiler suites:

module spider gcc
module spider aocc

Building OpenMP and hybrid applications

Additional compiler and linker flags are needed when building OpenMP or MPI/OpenMP hybrid applications:

Compiler suite	OpenMP flag
GNU and AMD	-fopenmp

Building serial applications

For building serial applications, one needs to use compiler suite specific compiler command:

Compiler suite	C	C++	Fortran
GNU	gcc	g++	gfortran
AMD	clang	clang++	flang

Building GPU applications

CUDA is the recommended programming model for Nvidia GPUs and CSC provides it as an environment module. OpenACC and OpenMP offloading programming models can also be used, but they are not part of the CSC supported software stack.

CUDA

The CUDA compiler (nvcc) takes care of compiling the CUDA code for target GPU device and passing the rest of the code to a non-CUDA compiler (i.e. gcc). For example, to load the CUDA 12.6.1 environment together with the GNU compiler:

module load gcc/10.4.0 cuda/12.6.1

To generate code for a given target device, you can tell the CUDA compiler what compute capability the target device supports. On Mahti, the GPUs (Ampere V100) support compute capability 8.0. Specify this using -gencode arch=compute_80,code=sm_80.

For example, compiling a CUDA kernel (example.cu) on Mahti (for C or C++ codes):

nvcc -gencode arch=compute_80,code=sm_80 example.cu

Compile a CUDA Fortran code named example.cuf

nvfortran -gpu=cc80 example.cuf

OpenACC and OpenMP offloading

Warning

OpenACC support is provided through the NVIDIA nvc and nvc++ compilers. However, it is important to note that the support can be somewhat limited and may lack certain functionalities and they are not integrated to the rest of the module tree.

Warning

If you enable the modules with the following instructions, your environment may not work normally. The module purge command is necessary and loading any other modules together with nvhpc ones may break your environment and is not supported by CSC. For additional information about OpenACC support, the CSC service desk should be contacted.

The compilers can be accessed through the NVIDIA HPC SDK modules which are included in the SDK installation. They can't be accessed directly and you have to enable them by adding the search path manually as follows:

module purge
module use /appl/opt/nvhpc/modulefiles

After adding the modules to the search tree you have to load the desired combination of compilers, MPI and CUDA. The recommended combination is nvhpc-hpcx-cuda, for example:

module load nvhpc-hpcx-cuda12/25.1

OpenACC

To enable OpenACC support, one needs to give -acc flag to the compiler.

To generate code for a given target device, tell the compiler what compute capability the target device supports. On Mahti, the GPUs (Ampere A100) support compute capability 8.0.

For example, to compiling C code that uses OpenACC directives (example.c):

nvc -acc example.c -gpu=cc80

For information about what the compiler actually does with the OpenACC directives, use -Minfo=all.

For Fortran code:

nvfortran -acc example.F90 -gpu=cc80

For C++ code:

nvc++ -acc example.cpp -gpu=cc80

OpenMP Offloading

To enable OpenMP Offloading, the options -mp=gpu is required

For example, compile a C code with OpenMP offloading:

nvc -mp=gpu example.c -gpu=cc80

For Fortran code:

nvfortran -mp=gpu example.F90 -gpu=cc80

For C++ code:

nvc++ -mp=gpu example.cpp -gpu=cc80

The nvc++ compiler supports codes that contain OpenACC, OpenMP Offloading and C++ parallel algorithms in the same code, for such case you can compile with:

nvc++ -stdpar -acc -mp=gpu example.cpp -gpu=cc80

Building software using Spack

Spack is a flexible package manager that can be used to install software on supercomputers and Linux and macOS systems. The basic module tree including compilers, MPI libraries and many of the available software on CSC supercomputers have been installed using Spack.

CSC provides a module spack/v0.17-user on Mahti that can be used by users to build software on top of the available compilers and libraries using Spack. It is also possible to install different customized versions of packages available in the module tree for special use cases. See here for a short tutorial on how to install software on CSC supercomputers using Spack.