Here, we concern ourselves with running SeisSol on the LUMI-G partition; that is, on the GPU partition.

The nodes then consist of:

  • 1× AMD Epyc 7A53 (Zen 3) CPU, configured with 4 NUMA domains

  • 4× AMD Instinct MI250x GPUs, thus 8 GCDs in total

Due to the 8 GCDs, we will launch SeisSol with 8 processes per node. The architecture settings we will need for SeisSol are milan for the CPU architecture (optimizing for Zen 3), and gfx90a for the GPU architecture (targeting the MI250X). As device backend, we use HIP, and for the SYCL implementation, we use AdaptiveCpp.

Installing Modules (without Spack)

Here, we go for a build using amdclang and AdaptiveCpp. We begin by setting up an environment. Firstly, choose a folder you want to install SeisSol to and navigate to it. Run pwd and copy the path there. Run the following script there.

export SEISSOL_BASE=$(pwd)


Next, we load the necessary modules for our SeisSol build. We set the compilers to the cray compiler wrappers (which in our case use amdclang internally).

module load LUMI/23.09 partition/G
module load cpeAMD/23.09
module load rocm/5.6.1
module load amd/5.6.1

module load Boost/1.82.0-cpeAMD-23.09
module load Eigen/3.4.0

module load cray-hdf5-parallel/
module load cray-netcdf-hdf5parallel/
module load cray-python/3.10.10

export CC=cc
export CXX=CC
export FC=ftn

We also require a small hotfix for pkg-config, as required by easi (and subsequently also SeisSol) right now. It is needed to work correctly with the Cray environment (only the folders hdf5-parallel and netcdf-hdf5parallel are included by default; but these do not contain the non-parallel pkgconfigs):

export PKG_CONFIG_PATH=/opt/cray/pe/netcdf/$PKG_CONFIG_PATH

Next, we also start up our Python installation. The virtual environment sets additional paths for e.g. executables to our prefix directory automatically.

python -m venv $SEISSOL_PREFIX
source $SEISSOL_PREFIX/bin/activate
pip install setuptools
pip install numpy
pip install git+
pip install git+
pip install git+

Then, we can start installing the modules. For convenience, we also add Ninja as a build tool here first. (TODO: remove and replace by module load buildtools/23.09)

git clone --branch v1.12.0 --depth 1
mkdir -p ninja/build
cd ninja/build
make -j 10 install
cd ../..

Next, we choose AdaptiveCpp. Note that we need to switch off everything but ROCm for the installation to work smoothly.

git clone --branch v23.10.0 --depth 1
mkdir -p AdaptiveCpp/build
cd AdaptiveCpp/build
ninja install
cd ../..

The rest of the packages can be installed as usual.


tar -xvf parmetis-4.0.3.tar.gz
cd parmetis-4.0.3
sed -i 's/IDXTYPEWIDTH 32/IDXTYPEWIDTH 64/g'  ./metis/include/metis.h
make config cc=mpicc cxx=mpicxx prefix=$SEISSOL_PREFIX
make install
cp build/Linux-x86_64/libmetis/libmetis.a $SEISSOL_PREFIX/lib
cp metis/include/metis.h $SEISSOL_PREFIX/include
cd ..

YAML-CPP can be installed as follows:

tar -xf 0.8.0.tar.gz
mkdir -p yaml-cpp-0.8.0/build
cd yaml-cpp-0.8.0/build
ninja install
cd ../..

For easi, Eigen and libxsmm, the default instructions suffice.


git clone --recursive --depth 1
mkdir -p ASAGI/build
cd ASAGI/build
ninja install
cd ../..

For LUA:

tar -xf lua-5.4.6.tar.gz
cd lua-5.4.6
cd ..

For easi (depending on the former two):

git clone --recursive --depth 1
mkdir -p easi/build
cd easi/build
ninja install
cd ../..

WARNING: libxsmm gives errors at the moment. We recommend to only resort to PSpaMM for the time being (add ``-DGEMM_TOOLS_LIST=PSpaMM`` to your CMake configuration of SeisSol)

For libxsmm (note that we need 1.17 sharp; the latest main will not work as intended with the generator):

git clone --branch 1.17 --depth 1
cd libxsmm
make generator
cp bin/libxsmm_gemm_generator $SEISSOL_PREFIX/bin
cd ..

Compiling SeisSol

Finally, it’s time to clone SeisSol and build it.

However, we need to apply a small hotfix here, since the Cray compiler environment does not work with AdaptiveCpp (it causes problems with finding MPI, the filesystem headers etc.). As a workaround, we compile SeisSol with amdclang directly, and add the necessary flags from the Cray environment as compiler flags (that can be done by CC --cray-print-opts=all, the same with cc and ftn).

In total, we get the following:

git clone --recursive seissol
mkdir -p seissol/build
cd seissol/build
CC=amdclang CXX=amdclang++ CFLAGS=$(cc --cray-print-opts=all) CXXFLAGS=$(CC --cray-print-opts=all) cmake .. -GNinja -DPRECISION=single -DDEVICE_BACKEND=hip -DDEVICE_ARCH=gfx90a -DHOST_ARCH=milan -DORDER=4 -DASAGI=ON -DNUMA_AWARE_PINNING=ON -DCMAKE_INSTALL_PREFIX=$SEISSOL_PREFIX

Optionally, you can install SeisSol to $SEISSOL_PREFIX.

Running Jobs

Attached is a job script which does the pinning for us. The pinning on the LUMI nodes needs some special attention, since 8 out of the 64 cores are reserved for the OS (cf. ).

Also, for now we disable the HSA_XNACK feature, as it is known to cause problems with ROCm 5.6 (cf. ).

#!/usr/bin/env bash
#SBATCH --job-name=seissol   # Job name
#SBATCH --nodes=<NUMBER-OF-NODES>               # Total number of nodes
#SBATCH --account=<your-project>  # Project for billing
#SBATCH --mail-user=<your-mail>
#SBATCH --time=01:00:00       # Run time (d-hh:mm:ss)
#SBATCH --output=seissol-output.log # Name of stdout output file
#SBATCH --error=seissol-error.log  # Name of stderr error file
#SBATCH --partition=standard-g  # Partition (queue) name
#SBATCH --ntasks-per-node=8     # 8 MPI ranks per node
#SBATCH --gpus-per-node=8       # Allocate one gpu per MPI rank
#SBATCH --mail-type=all         # Send email at begin and end of job
#SBATCH --exclusive
#SBATCH --requeue

cat << EOF > select_gpu

exec \$*

chmod +x ./select_gpu


export HSA_XNACK=0

export OMP_PLACES="cores(3)"
export OMP_PROC_BIND=close

export SEISSOL_FREE_CPUS_MASK="52-54,60-62,20-22,28-30,4-6,12-14,36-38,44-46"

srun --cpu-bind=mask_cpu:${CPU_BIND} ./select_gpu ./SeisSol_Release_sgfx90a_hip_6_elastic parameters.par