Table of contents
Installed oneAPI Base toolkit and oneAPI HPC toolkit on 9/2021 for test purposes. Installed on compute nodes, head nodes, Mark6'es.
Base toolkit: Develop performant, data-centric applications across Intel® CPUs, GPUs, and FPGAs with this foundational toolset. HPC toolkit: Build, analyze, and scale applications across shared- and distributed-memory computing systems.
For additional info from Intel see https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html
Components
The installation is under /opt/intel/oneapi/
The installation provides its own Intel IPP and Intel MPI Library.
It also comes with Intel VTune Profiler, Cluster Checker, Trace Analyzer and Collector, and Intel compilers (CC is icc, CXX is icpc).
Repository
See Intel web pages. For CentOS on our cluster got a by default disabled /etc/yum.repos.d/oneAPI.repo with:
[oneAPI] name=Intel® oneAPI repository baseurl=https://yum.repos.intel.com/oneapi enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
The repo should be enabled temporarily only! Avoids accidentally pulling any Intel-customized RHEL/CentOS packages into the vlbi cluster.
gxr -c "yum-config-manager --enable oneAPI" --group /hardware/nodes/mark6,/hardware/nodes/compute (gxr -c "yum install -y ...) gxr -c "yum-config-manager --disable oneAPI" --group /hardware/nodes/mark6,/hardware/nodes/compute
Intel oneAPI Packages
Packages for devel nodes:
yum install intel-basekit yum install intel-hpckit yum install intel-oneapi-vtune-2021.7.1-492
Runtime packages for nodes:
gxr -c "yum install -y intel-oneapi-clck" --group /hardware/nodes/mark6,/hardware/nodes/compute gxr -c "yum install -y intel-oneapi-runtime-mpi \ intel-oneapi-runtime-ipp \ intel-oneapi-compiler-shared-runtime-2021.3.0.x86_64 \ intel-oneapi-compiler-shared-common-runtime-2021.3.0.noarch \ intel-oneapi-mpi-2021.3.1 \ intel-oneapi-vtune-2021.7.1-492" --group /hardware/nodes/mark6,/hardware/nodes/compute # Dependency bug in runtimes, need to install also /opt/intel/oneapi/lib/intel64/libimf.so via: gxr -c "yum install -y intel-oneapi-runtime-compilers-2021.3.0" \ --group /hardware/nodes/mark6,/hardware/nodes/compute
Environment
Activate with:
source /opt/intel/oneapi/setvars.sh
For DiFX, the usual setup_difx script(s) should have:
#!/bin/bash
## Get oneAPI environment
# Prepare IPPROOT, CMPLR_ROOT, I_MPI_ROOT, ...
if [[ "$I_MPI_ROOT" == "" ]]; then
. /opt/intel/oneapi/setvars.sh
fi
if [[ "$IPPROOT" == "" ]]; then
export IPPROOT=$ONEAPI_ROOT/lib/intel64/
fi
# Intel MPI
export OPENMPIROOT=$I_MPI_ROOT
# Intel Compilers
# Note on icc, vs icpc, vs icx, vs icpx, vs icl:
# https://software.intel.com/content/www/us/en/develop/articles/porting-guide-for-icc-users-to-dpcpp-or-icx.html
export CC=icc
export CXX=icpc
export MPIXCC=${OPENMPIROOT}/bin/mpicxx
export MPICXX=${OPENMPIROOT}/bin/mpicxx
# Add Intel-libs path in case the current host has only the runtimes but not entire
# devel toolkit installed; the automatic setvars.sh does not appear to cover that case
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/oneapi/lib/intel64/
## Rest of DiFX setup
# note: OpenMPI params in DIFX_MPIRUNOPTIONS and the OMPI_MCA_* env vars probably irrelevat for Intel MPI
