UCX Fabric Support¶
!!! note The network provider is an immutable property of a DAOS system. Changing the network provider to UCX requires that the DAOS storage is reformatted.
To enable DAOS UCX support on InfiniBand fabrics, the following steps are needed:
A supported version of MLNX_OFED must be installed before DAOS is installed. This is the same for libfabric and for UCX: DAOS only supports the NVIDIA-provided MLNX_OFED stack, not the inbox drivers. Refer to the DAOS Support Matrix for information about supported MLNX_OFED releases.
mercury-ucxRPM package needs to be manually selected for installation: For the technology preview, the
mercurypackage is provided in two different versions, which are mutually exclusive:
mercuryRPM does support libfabric, but not UCX. This RPM will be installed by default, and must be used in non-InfiniBand environments.
mercury-ucxRPM is also provided, which supports both libfabric and UCX. This RPM must be used in InfiniBand environments when the intention is to use UCX. It may also be used in InfiniBand environments if the intention is to use libfabric. Attempts to install this RPM in non-Infiniband environments will fail, because it has a dependency on UCX packages.
At DAOS installation time, to enable UCX support the new
mercury-ucxRPM package must be explicitly listed in order to prevent the installation of the default
mercurypackage (which does not include the UCX support). For example, using the
yumpackage manager on EL8:
# on DAOS_ADMIN nodes: yum install mercury-ucx daos-admin # on DAOS_SERVER nodes: yum install mercury-ucx daos-server # on DAOS_CLIENT nodes: yum install mercury-ucx daos-client
- To change an existing DAOS installation from libfabric to
UCX, the default
mercuryRPM first needs to be un-installed, and the
mercury-ucxRPM must be installed instead. To prevent the removal of DAOS altogether (it has a package dependency on mercury), the
rpmcommand with the
--nodepsoption should be used:
# on EL8: rpm -e --nodeps mercury yum install mercury-ucx # on Leap15: rpm -e --nodeps mercury zypper install mercury-ucx
- To update from an earlier DAOS version (with libfabric)
to DAOS 2.4 with
UCX, the recommended path is to first perform a standard DAOS
RPM update (which will update the default
mercurypackage). After the update, the
mercuryRPM package can be replaced by
mercury-ucxas described above.
After UCX support has been enabled by installing the
package, the network provider must be changed in the DAOS server's
configuration file (
A sample YML file is available on
The recommended setting for UCX is