.. SPDX-FileCopyrightText: 2018 SeisSol Group SPDX-License-Identifier: BSD-3-Clause SPDX-LicenseComments: Full text under /LICENSE and /LICENSES/ SPDX-FileContributor: Author lists in /AUTHORS and /CITATION.cff .. _env_vars: Environment variables ===================== SeisSol can be tuned with several environment variables. Configuration ------------- An executable can hold several configurations (see ``EXTRA_CONFIGS`` in the build parameters). ``SEISSOL_CONFIGURATION`` chooses the one a run computes in, by its name, e.g. ``SEISSOL_CONFIGURATION=elastic-linearck-o4-f32-stroud``, for a parameter file that does not name one with ``Configuration`` in its ``equations`` section; a configuration named there takes precedence. Without either, a run takes the first configuration built. So a parameter file written for an executable of a single configuration runs unchanged in an executable of several. The proxy takes the configuration of ``SEISSOL_CONFIGURATION`` as well, unless ``--config`` gives one. Communication thread -------------------- By default, any SeisSol run with more than one MPI rank will use a communication thread to advance the MPI progress engine. For that, it is required to leave at least one thread vacant in your OpenMP thread placing map, cf. the SuperMUC-NG example below. If you do not want to use a communication thread, you may set ``SEISSOL_COMMTHREAD=0`` to make SeisSol poll the progress from time to time. Load balancing -------------- When running with multiple ranks, SeisSol will estimate the node-level performance, enabling better load balancing. For that, it runs the so-called "Mini SeisSol" benchmark. As its name already hints, it simulates a small test workload on each node; thus estimating the performance of all nodes relative to each other. The number of elements per node assigned during the partitioning will be resized according to these values. As a result, the partitioning of runs may become non-deterministic, and the initialization procedure may take a little longer, especially when running only on a single node with multiple ranks. To disable it, set ``SEISSOL_MINISEISSOL=0``. In a run with several configurations (see :ref:`several_configurations`), a cell costs according to its configuration. With multiple ranks, Mini SeisSol then also times the time, local and neighbor kernels of each configuration of the run on every rank, and takes the median of the ranks for the cost of a cell of each configuration relative to one of the configuration of the run. The partitioning and the LTS clustering weight each cell with it. Without the measurement (on a single rank, or with ``SEISSOL_MINISEISSOL=0``), the relative cost comes from the hardware FLOPs of the kernels, weighted with the size of a real (4 bytes in single precision, 8 in double precision); that estimate sees nothing of the machine. Persistent MPI operations ------------------------- Since SeisSol has a static communication pattern which means, we issue the same MPI transfer requests. Thus, we use persistent MPI communication which may reduce communication latency. To disable persistent communication, set ``SEISSOL_MPI_PERSISTENT=0``. Then, SeisSol will use ``MPI_Isend`` and ``MPI_Irecv`` instead. To explicitly enable the persistent communication, set ``SEISSOL_MPI_PERSISTENT=1``. Right now, it is enabled by default. GPU-Specific Environment Variables ---------------------------------- GPU-specific environment variables are listed on its :ref:`own page `. Output ------ * ``SEISSOL_IO_VERTEXFILTER``: whether the coinciding points of an order 0 output (that is, one written without ``wavefieldvtkorder``, ``surfacevtkorder`` or ``vtkorder``) are merged into one. On by default; the merging happens within a rank, so points on a partition boundary stay duplicated. Also accepted under its previous name ``SEISSOL_VERTEXFILTER``. * ``SEISSOL_IO_META_BLOCK_SIZE``: the size, in bytes, of the blocks the metadata of an HDF5 output file is gathered into. Unset, HDF5 keeps its own small default. A large value, e.g. ``1048576`` as the previous writer had, keeps the many small metadata writes of a file together on a parallel file system, but every file then holds at least one such block in full, used or not -- most of a file of a few receivers. * ``SEISSOL_IO_FLUSH``: whether an output file written at every step (a time series, the data of an Xdmf output) is flushed after each step. On by default, so that the output up to the last step is complete on disk should a run end early. A flush syncs the file on every rank, which a parallel file system may take its time over with thousands of ranks; turned off, the file is complete once the simulation ends. Two further variables are, for historic reasons, named ``XDMFWRITER``; they are also accepted with a ``SEISSOL_IO_`` prefix, which is the name to use: * ``XDMFWRITER_ALIGNMENT``: how far the bulk data of an output is aligned in the file. A bulk write straddling a stripe boundary makes more than one storage target take part in a single write, which serializes them; what the right value is depends on the file system, so there is no default. * ``XDMFWRITER_BLOCK_SIZE``: no longer read. The previous writer gathered the data of an output into blocks of this size before writing them; the MPI-IO hints for collective buffering below (``cb_nodes``, ``cb_buffer_size``) take its place. The MPI-IO hints the output passes through are read the same way, under the names ROMIO understands and with an ``MPIO_`` in front, so ``SEISSOL_IO_MPIO_ROMIO_DS_WRITE`` sets ``romio_ds_write``. Recognized are ``ind_rd_buffer_size``, ``ind_wr_buffer_size``, ``romio_ds_read``, ``romio_ds_write``, ``cb_buffer_size``, ``cb_nodes``, ``romio_cb_read``, ``romio_cb_write``, ``striping_factor`` and ``striping_unit``. Which of them help is again a property of the file system. .. _asynchronous-output: Asynchronous output ~~~~~~~~~~~~~~~~~~~ In addition to the variables in SeisSol, the `ASYNC `__ library provides some tuning variables listed in the `wiki `__. .. _optimal_environment_variables_on_supermuc_ng: Optimal environment variables on SuperMUC-NG -------------------------------------------- On SuperMUC-NG, we recommend using SeisSol with async output in thread mode. Also, we recommend using hyperthreading capabilities (that is, using 96 CPUs instead of 48. 2 threads out of 96 are used as communication threads). Here are some proposed environment variables to be added before invoking SeisSol in your batch file: .. code:: bash export MP_SINGLE_THREAD=no unset KMP_AFFINITY export OMP_NUM_THREADS=94 export OMP_PLACES="cores(47)" export XDMFWRITER_ALIGNMENT=8388608 export XDMFWRITER_BLOCK_SIZE=8388608 export SC_CHECKPOINT_ALIGNMENT=8388608 export SEISSOL_CHECKPOINT_ALIGNMENT=8388608 export SEISSOL_CHECKPOINT_DIRECT=1 export ASYNC_MODE=THREAD export ASYNC_BUFFER_ALIGNMENT=8388608 A complete batch script for SuperMUC-NG can be found in the chapter about :ref:`SuperMUC-NG `. In previous versions of SeisSol, you had to explicitly compile the software with ``-DCOMMTHREAD=ON``. That is not necessary anymore, as any configuration with more than one MPI rank uses the communication thread by default.