1. Build Guide¶
LvArray uses a CMake based build system augmented with BLT. If you’re not familiar with CMake, RAJA has a good introduction. LvArray has a dependency on RAJA and as such requires that the RAJA_DIR
CMake variable defined and points to a RAJA installation.
1.1. CMake Options¶
In addition to the standard CMake and BLT options LvArray supports the following options.
- Adding additional targets
The following variables add additional build targets but do not alter the usage or functionality of LvArray.
ENABLE_TESTS
: defaultON
Build the unit tests which can be run with
make test
. The unit tests take a long time to build with the IMB XL and Intel compilers.
ENABLE_EXAMPLES
: defaultON
Build the examples,
ENABLE_TESTS
must also beON
.
ENABLE_BENCHMARKS
: defaultON
Build the benchmarks,
ENABLE_TESTS
must also beON
.
DISABLE_UNIT_TESTS
: defaultOFF
Use with
ENABLE_TESTS=ON
to disable building the unit tests but still allow the examples and benchmarks.
ENABLE_DOCS
: defaultON
Build the documentation.
- Third party libraries
LvArray has a hard dependency on RAJA along with multiple optional dependencies. Of the following variables only
RAJA_DIR
is mandatory.RAJA_DIR
The path to the RAJA installation.
ENABLE_UMPIRE
: defaultOFF
If Umpire is enabled. Currently no part of LvArray uses Umpire but it is required when using CHAI.
UMPIRE_DIR
The path to the Umpire installation, must be specified when Umpire is enabled.
ENABLE_CHAI
: defaultOFF
If CHAI is enabled, CHAI also requires Umpire. Enabling CHAI allows the usage of the
LvArray::ChaiBuffer
.
CHAI_DIR
The path to the CHAI installation, must be specified when CHAI is enabled.
ENABLE_CALIPER
: defaultOFF
If caliper is enabled. Currently caliper is only used to time the benchmarks.
CALIPER_DIR
The path to the caliper installation, must be specified when caliper is enabled.
ENABLE_ADDR2LINE
: defaultON
If
addr2line
is enabled. This is used inLvArray::system::stackTrace
to attempt to provide file and line locations for the stack frames.
ADDR2LINE_EXEC
: default/usr/bin/addr2line
The path to the
addr2line
executable.
- Debug options
The following options don’t change the usage of LvArray but they are intended to make debugging easier.
LVARRAY_BOUNDS_CHECK
: defaultON
iff in a CMakeDebug
build.Enables bounds checks on container access along with checks for other invalid operations.
ENABLE_TOTALVIEW_OUTPUT
: defaultOFF
Makes it easier to inspect the
LvArray::Array
in TotalView. This functionality is highly dependent on the version of TotalView used.
1.2. Using LvArray Your Application¶
Once LvArray has been installed if your application uses CMake importing LvArray is as simple as defining LVARRAY_DIR
as the path to LvArray install directory and then adding find_package(LVARRAY)
. This will export a lvarray
target that can then be used by target_link_libraries
and the like.
1.3. Host Configs¶
Host config files are a convenient way to group CMake options for a specific configuration together. There are a set of example host configs in the host-configs
directory. Once you’ve created a host config file you can use scripts/config-build.py
to create the build directory and run CMake for you. An example usage would be python ./scripts/config-build.py --hc host-configs/LLNL/quartz-clang@10.0.0.cmake
.
> python scripts/config-build.py --help
usage: config-build.py [-h] [-bp BUILDPATH] [-ip INSTALLPATH]
[-bt {Release,Debug,RelWithDebInfo,MinSizeRel}] [-e]
[-x] [-ecc] -hc HOSTCONFIG
Configure cmake build. Unrecognized arguments are passed on to CMake.
optional arguments:
-h, --help show this help message and exit
-bp BUILDPATH, --buildpath BUILDPATH
specify path for build directory. If not specified,
will create in current directory.
-ip INSTALLPATH, --installpath INSTALLPATH
specify path for installation directory. If not
specified, will create in current directory.
-bt {Release,Debug,RelWithDebInfo,MinSizeRel}, --buildtype {Release,Debug,RelWithDebInfo,MinSizeRel}
build type.
-e, --eclipse create an eclipse project file.
-x, --xcode create an xcode project.
-ecc, --exportcompilercommands
generate a compilation database. Can be used by the
clang tools such as clang-modernize. Will create a
file called 'compile_commands.json' in build
directory.
-hc HOSTCONFIG, --hostconfig HOSTCONFIG
select a specific host-config file to initalize
CMake's cache
1.4. Submodule usage¶
LvArray can also be used as a submodule. In this case the configuration is largely the same except that LvArray expects the parent project to have imported the third party libraries. For example if ENABLE_UMPIRE
is ON
then LvArray will depend on umpire
but it will make no attempt to find these library (UMPIRE_DIR
is unused).
1.5. Spack and Uberenv Builds¶
LvArray has an associated Spack package. For those unfamiliar with Spack the most important thing to understand is the spec syntax. For those interested the LvArray package implementation is here the important part of which is reproduced below.
class Lvarray(CMakePackage, CudaPackage):
"""LvArray portable HPC containers."""
homepage = "https://github.com/GEOSX/lvarray"
git = "https://github.com/GEOSX/LvArray.git"
version('develop', branch='develop', submodules='True')
version('tribol', branch='temp/feature/corbett/tribol', submodules='True')
variant('shared', default=True, description='Build Shared Libs')
variant('umpire', default=False, description='Build Umpire support')
variant('chai', default=False, description='Build Chai support')
variant('caliper', default=False, description='Build Caliper support')
variant('tests', default=True, description='Build tests')
variant('benchmarks', default=False, description='Build benchmarks')
variant('examples', default=False, description='Build examples')
variant('docs', default=False, description='Build docs')
variant('addr2line', default=True,
description='Build support for addr2line.')
depends_on('cmake@3.8:', type='build')
depends_on('cmake@3.9:', when='+cuda', type='build')
depends_on('raja')
depends_on('raja+cuda', when='+cuda')
depends_on('umpire', when='+umpire')
depends_on('umpire+cuda', when='+umpire+cuda')
depends_on('chai+raja', when='+chai')
depends_on('chai+raja+cuda', when='+chai+cuda')
depends_on('caliper', when='+caliper')
depends_on('doxygen@1.8.13:', when='+docs', type='build')
depends_on('py-sphinx@1.6.3:', when='+docs', type='build')
LvArray also has an uberenv
based build which simplifies building LvArray’s dependencies along with optionally LvArray using spack.
> ./scripts/uberenv/uberenv.py --help
Usage: uberenv.py [options]
Options:
-h, --help show this help message and exit
--install Install `package_name`, not just its dependencies.
--prefix=PREFIX destination directory
--spec=SPEC spack compiler spec
--mirror=MIRROR spack mirror directory
--create-mirror Create spack mirror
--upstream=UPSTREAM add an external spack instance as upstream
--spack-config-dir=SPACK_CONFIG_DIR
dir with spack settings files (compilers.yaml,
packages.yaml, etc)
--package-name=PACKAGE_NAME
override the default package name
--package-final-phase=PACKAGE_FINAL_PHASE
override the default phase after which spack should
stop
--package-source-dir=PACKAGE_SOURCE_DIR
override the default source dir spack should use
--project-json=PROJECT_JSON
uberenv project settings json file
-k Ignore SSL Errors
--pull Pull if spack repo already exists
--clean Force uninstall of packages specified in project.json
--run_tests Invoke build tests during spack install
--macos-sdk-env-setup
Set several env vars to select OSX SDK settings.This
was necessary for older versions of macOS but can
cause issues with macOS versions >= 10.13. so it is
disabled by default.
Two simple examples are provided below.
quartz2498 > ./scripts/uberenv/uberenv.py --install --spec="@develop %clang@10.0.1"
This will build RAJA (LvArray’s only hard dependency) and LvArray and install them in ./uberenv_libs/linux-rhel7-ppc64le-clang@10.0.1
. By default libraries are built in the RelWithDebInfo
CMake configuration.
quartz2498 > ./scripts/uberenv/uberenv.py --spec="@develop %gcc@8.3.1 ^raja@0.12.1 build_type=Release"
This will install RAJA in the same location but it will be built in the Release
configuration and instead of building and installing LvArray a host-config will be generated and placed in the current directory. This can be useful for developing or debugging.
Currently uberenv
only works on the LLNL toss_3_x86_64_ib
and blueos_3_ppc64le_ib_p9
systems. Further more only certain compilers are supported. On the TOSS systems clang@10.0.1
, gcc@8.3.1
and intel@19.1.2
are supported. On BlueOS clang-upstream-2019.08.15 (clang@9.0.0)
, clang-ibm-10.0.1-gcc-8.3.1 (clang@10.0.1)
, gcc@8.3.1
and xl-2020.09.17-cuda-11.0.2 (xl@16.1.1)
are supported. Adding support for more compilers is as simple as adding them to the appropriate compilers.yaml
file.
Adding support for a new system is easy too, you just need to create a directory with a compilers.yaml
which specifies the available compilers and a packages.yaml
for system packages and then pass this directory to uberenv with the --spack-config-dir
option.
For reference two more complicated specs are shown below
lassen709 > ./scripts/uberenv/uberenv.py --install --run_tests --spec="@develop+umpire+chai+caliper+cuda %clang@10.0.1 cuda_arch=70 ^cuda@11.0.2 ^raja@0.12.1~examples~exercises cuda_arch=70 ^umpire@4.0.1~examples cuda_arch=70 ^chai@master~benchmarks~examples cuda_arch=70 ^caliper@2.4~adiak~mpi~dyninst~callpath~papi~libpfm~gotcha~sampler~sosflow"
This will use clang@10.0.1
and cuda@11.0.2
to build and install RAJA v0.12.1 without examples or exercises, Umpire v4.0.1 without examples, the master branch of CHAI without benchmarks or examples, and caliper v2.4 without a bunch of options. Finally it will build and install LvArray after running the unit tests and verifying that they pass. Note that each package that depends on cuda gets the cuda_arch=70
variable.
quartz2498 > ./scripts/uberenv/uberenv.py --spec="@tribol+umpire %intel@19.1.2 ^raja@0.12.1 build_type=Release ^umpire@4.0.1 build_type=Release"
This will use intel@19.1.2
to build and install RAJA V0.12.1 in release and Umpire v4.0.1 in release. Finally it will generate a host config that can be used to build LvArray.