ASAGI

ASAGI#

The software package ASAGI can be used to map gridded simulation properties of the domain to the mesh used for a SeisSol simulation. ASAGI reads NetCDF files, which follow the COARDS Convention for netCDF files. This convention in particular states that:

“1-dimensional netCDF variables whose dimension names are identical to their variable names are regarded as “coordinate variables”.”

Here is an example of the structure of an ASAGI file (generated by ncdump -h test_ASAGI.nc), describing two 2D arrays (strike_slip and dip_slip) indexed by variables u and v.

netcdf test_ASAGI {
types:
  compound material {
    float strike_slip ;
    float dip_slip ;
  }; // material
dimensions:
        u = 40 ;
        v = 20 ;
variables:
        float u(u) ;
        float v(v) ;
        material data(v, u) ;
}

Typically, 1D, 2D, and 3D ASAGI files are used in SeisSol setups.

Installing ASAGI#

Make sure that the compiler and Python packages you use are the same as for SeisSol later on. First clone ASAGI with:

git clone git@github.com:TUM-I5/ASAGI
# git clone https://github.com/TUM-I5/ASAGI.git
cd ASAGI
git submodule update --init

Run cmake, and compile with:

mkdir build && cd build
CMAKE_PREFIX_PATH=$NETCDF_BASE
cmake .. -DFORTRAN=OFF -DSHARED_LIB=OFF -DSTATIC_LIB=ON -DCMAKE_INSTALL_PREFIX=$HOME
make -j 48
make install
# (known errors: 1.Numa could not found - turn off Numa by adding -DNONUMA=ON . )

Building SeisSol with ASAGI support#

Enable the option ASAGI=ON in the using cmake, and compile easi with ASAGI=ON as well.

Generating the NetCDF input file#

Using python#

The most straightforward way to generate ASAGI file is to use the netCDF4 module of python. A typical example which generates a 2D ASAGI file can be found here.

Using asagiconv#

Asagiconv (located here) allows querying data, vizualising and exporting to NetCDF data from the 3D Velocity Model for Southern California. For more detail, see ASAGI docu.

Velocity models given as structured grids#

Asagi expects a 1D, 2D, 3D (or higher dimensions) structured grid NetCDF files. Such files can be generated from an ASCII file using the command: ncgen -b asagi_example.txt
Here is a typical example for the ASCII file:
netcdf asagi_example {
types:
  compound material {
    float rho ;
    float mu ;
    float lambda ;
  }; // material
dimensions:
    x = 3 ; // Number of points in x-direction
    y = 2 ; // Number of points in y-direction
    z = 1 ; // Number of points in z-direction
variables:
    float x(x) ;
    float y(y) ;
    float z(z);
    material data(z, y, x) ;
data:
  x = 2, 2.5, 3 ; // Grid points in x-direction (must have the same spacing)
  y = -1, 0 ; // Grid points in y-direction (must have the same spacing)
  z = 0 ; // Grid points in z-direction (must have the same spacing)

  data =
  {1, -1, 10}, // rho,mu,lambda for x0, y0, z0
  {2, -2, 11}, // rho,mu,lambda for x1, y0, z0
  {3, -3, 12}, // rho,mu,lambda for x2, y0, z0
  {4, -4, 13}, // rho,mu,lambda for x0, y1, z0
  {5, -5, 14}, // rho,mu,lambda for x1, y1, z0
  {6, -6, 15} ; // rho,mu,lambda for x2, y1, z0
}

SeisSol parameter file#

A simple example file setting the elastic properties using EASI can be found here.

Such a file would be called adding in the namelist equation:

MaterialFileName = 101_asagi.yaml

In this example, the ASAGI file describes 2D arrays. The AffineMap is therefore needed to define the unit vectors used for indexing the 2D arrays. Note that the variables in the affine map can have different names than x, y or z (actually it should be preferred to avoid confusion). An AffineMap may also be used for 3D arrays, in case the coordinates variables are not aligned with the Cartesian coordinate system.

Performance Tuning#

By default, SeisSol assumes that the whole ASAGI file fits onto a single rank of the simulation. As a very rough estimate, up to 20 GiB-large files should be usable in modern systems.

For larger files, there are several tuning options (via environment variables):

  • SEISSOL_ASAGI_MPI_MODE (default OFF): distribute the ASAGI grid via MPI, thus decreasing the memory load per rank.

    • OFF: not distributed; the file resides on each MPI rank as a whole.

    • COMM_THREAD: distribute; use a communication thread to handle data exchanges. (needs the communication thread within SeisSol to be enabled)

    • WINDOWS: distribute; use unidirectional MPI functionalities for data exchanges. (does not need an extra communication thread)

  • SEISSOL_ASAGI_NUMA_MODE (default OFF): behavior for different NUMA domains on one node.

    • OFF: disables multithreading for ASAGI reading alltogether.

    • ON: enables NUMA-aware data distribution; always fetches the data from the respective NUMA domain

    • CACHE: enables NUMA-aware data distribution; caches data from other NUMA domains

  • SEISSOL_ASAGI_SPARSE (default NO): optimize for a sparse grid, i.e. cache (only relevant with MPI)

  • SEISSOL_ASAGI_BLOCK_SIZE (default 64): the block size for MPI/NUMA distribution

  • SEISSOL_ASAGI_CACHE_SIZE (default 128): the number of blocks cached for MPI/NUMA (for NUMA, if the NUMA mode is set to cache; for MPI only if SEISSOL_ASAGI_SPARSE is enabled)

  • SEISSOL_ASAGI_NUM_THREADS (default 0): the thread count for ASAGI, including the communication thread (if used). Only used if SEISSOL_ASAGI_NUMA_MODE is not OFF. Also, if its value is given as 0, it is set equal to the number of cores visible to ASAGI.

For larger files, it is advantageous to try SEISSOL_ASAGI_MPI_MODE=WINDOWS and SEISSOL_ASAGI_NUMA_MODE=ON first; followed by other options. Depending on the MPI implementation (you will need to test this by yourself), SEISSOL_ASAGI_MPI_MODE=COMM_THREAD might work better. The SEISSOL_ASAGI_NUMA_MODE=CACHE option might give a little more speedup if you only access few grid points per rank.

Further documentation#

See the ASAGI documentation . Note that the variables it refers to are ASAGI-internal variables; for how to use them, see

Known issues#

  • There is a bug when using ASAGI with MPI, if not all ranks request data (e.g. in the case of many ranks and a fault). A workaround is described in SeisSol/SeisSol#46.

  • With the default options, large ASAGI files (i.e. larger than the local memory on a node/rank) will fail during model initialization. In this case, set SEISSOL_ASAGI_MPI_MODE to either WINDOWS or COMM_THREAD.

Contents