High-Speed Fabric Support¶
Control Plane and Data Plane¶
DAOS uses two types of network communication between nodes:
-
Management traffic in the DAOS control plane. The control plane uses gRPC/TCP for communication between servers, from the DAOS admin nodes to all DAOS servers (through the
dmgcommand and the DAOS management API), and from thedaos_agentdaemon on the DAOS client nodes to all DAOS servers (but not to other clients). -
Application I/O in the DAOS data plane. The DAOS data plane uses the DAOS Collective and RPC Transport (CaRT) layer for communication between DAOS clients and DAOS servers. CaRT relies on the Mercury RPC framework, which provides RDMA transfer capabilities if the underlying high-performance fabric supports it.
When planning the deployment of a DAOS system, one question that needs to be decided early on is whether the control plane traffic will use the same physical network as the data plane (for example, an IP-over-IB interface on the InfiniBand NIC in an InfiniBand network), or whether the control plane will use a separate physical network (like an administrative Ethernet network).
-
For example, if there is a reliable 25GbE management network over which all DAOS servers, DAOS clients and DAOS admin nodes can communicate, then it is possible to use that network for the control plane.
-
On the other hand, if the management network of the DAOS servers is isolated from the client nodes' management network (which is a best practice to protect the server-side management interfaces), then the control plane must use TCP over the data plane network to guarantee that the DAOS servers, clients and admin nodes can communicate with each other.
The control plane network is configured in the daos_server.yml,
daos_agent.yml, and
daos_control.yml files by using the IP addresses (or hostnames) of
either the nodes' management network interfaces or an IP-over-IB interface
of their high-speed NICs in the control plane configuration sections.
The settings in these three configuration files must be consistent.
Given an 8-node cluster of nodes n[0001-0008], where:
- nodes n[0001,0004,0008] are defined as management service replicas, and
- the '-ibs1' node name suffix represents an IP-over-IB interface,
the following example would designate a control plane network using the IP-over-IB interface:
# daos_agent.yml
mgmt_svc_replicas:
- n0001-ibs1
- n0004-ibs1
- n0008-ibs1
# daos_agent.yml
access_points:
- n0001-ibs1
- n0004-ibs1
- n0008-ibs1
# daos_control.yml
hostlist:
- n0001-ibs1
- n0002-ibs1
- n0003-ibs1
- n0004-ibs1
- n0005-ibs1
- n0006-ibs1
- n0007-ibs1
- n0008-ibs1
Note
On dual-socket DAOS servers with two InfiniBand NICs, the second InfiniBand
NIC also has an IP-over-IB interface, address, and name (which are used by the
DAOS engines).
By default, the DAOS control plane (daos_server process)
binds to 0.0.0.0, so it listens on all network interfaces.
To limit the control plane to a specific interface, the control_iface:
setting in the daos_server.yml configuration file can be used.
To use the management network interface, replace the above hostnames with the hostnames of the management network interfaces.
The rest of this document assumes that the control plane is using IP-over-IB over the data plane's high-speed NICs, and focuses on data plane configuration.
Please refer to Partition isolation for control plane traffic over InfiniBand for important control plane configuration information regarding InfiniBand networks.
Mercury installation¶
The CaRT layer used by the DAOS data plane is part of the main DAOS RPM packages. CaRT in turn uses the Mercury RPC framework, which is provided as separate RPMs within the DAOS packages directory. For NVIDIA based InfiniBand and RoCE Ethernet fabrics, Mercury's UCX backend is the recommended plugin. For all other fabrics, Mercury's libfabric backend is used.
Info
DAOS version 2.9 uses Mercury version 2.4.1.
Mercury backends are dynamically loaded, and there is no RPM dependency
in the base mercury RPM on a specific backend
(The base mercury RPM will be automatically installed,
because the DAOS RPMs have an RPM dependency on mercury).
Depending on the intended
fabric provider, the corresponding Mercury backend RPM (mercury-libfabric
or mercury-ucx) must be explicitly installed.
Note
mercury-libfabric will be installed by default if mercury-ucx is not installed.
It is also possible to install both Mercury backends on the same node.
Failing to install the Mercury backend that corresponds to the fabric
provider configured in daos_server.yml will result in a runtime error,
and DAOS will not start.
For example, when a libfabric provider is used but the mercury-libfabric
RPM is not installed, an error similar to this will be reported
in the DAOS server log:
na.c:570 na_plugin_scan_path() Could not open plugin (libna_plugin_ofi.so)
The Mercury backend RPMs have dependencies on libfabric and UCX, respectively, and it is recommended to install the fabric's host software stack before installing DAOS to satisfy these dependencies.
Fabric host software stack installation¶
Each high-performance fabric typically provides its own host software stack. To get support from the fabric vendor in case of networking issues, it is vital that a supported version of the vendor-provided host software stack is used. Linux distributions' "inbox" versions are often outdated and vendor support will not be available until the system is updated to a recent version of the vendor-provided host software stack.
This means that for NVIDIA-provided fabrics, a recent version
of the DOCA-OFED stack must be installed, which will provide the UCX
libraries that are required by the mercury-ucx backend RPM.
For all other fabrics, libfabric needs to be installed. The host fabric stacks for HPE Slingshot and Cornelis Networks Omni-Path ship with a version of libfabric that has been validated with those fabrics. In case the fabric host stack does not provide its own libfabric, the DAOS packages directory does include an RPM build of libfabric that can be used, for example when running on generic Ethernet interfaces. The following section contains details on the versions of libfabric that are used with the various fabric providers.
Details about the exact versions of the fabric host software stacks that have been validated with DAOS can be found in the DAOS Support Matrix, which also provides more information and references for the supported high-performance fabrics.
UCX Fabric Support¶
For NVIDIA InfiniBand and RoCE fabrics, DAOS supports UCX as an alternative to the default libfabric network stack. The UCX provider is fully supported since DAOS 2.4.
To enable DAOS UCX support on NVIDIA InfiniBand or RoCE fabrics, the following steps are needed:
-
A supported version of DOCA-OFED must be installed before DAOS is installed on all DAOS servers and clients. DAOS only supports the NVIDIA-provided DOCA-OFED stack for UCX, not Linux inbox drivers or distribution packages. Refer to the DAOS Support Matrix for information about supported DOCA-OFED releases.
-
The
mercury-ucxRPM package must be manually selected for installation. The basemercuryRPM package ships by default with themercury-libfabricpackage unlessmercury-ucxis also installed. Themercury-ucxRPM contains the UCX plugin that is required for enabling UCX support. This RPM must be installed when UCX needs to be used. Attempts to install this RPM on systems without the required UCX packages will fail. -
When installing DAOS, explicitly list the
mercury-ucxRPM package if it was not already installed in the previous step. For example, using thednfpackage manager on EL9: ```bash # on DAOS_ADMIN nodes: dnf install mercury-ucx daos-admin
# on DAOS_SERVER nodes: dnf install mercury-ucx daos-server
# on DAOS_CLIENT nodes: dnf install mercury-ucx daos-client ```
After UCX support has been enabled by installing the mercury-ucx
package, the network provider in the DAOS server's
configuration file (/etc/daos/daos_server.yml) should be changed.
A sample YAML file is available on GitHub.
The recommended setting for UCX is provider: ucx+dc_x (see the following
sections).
Verify the installation and provider configuration before starting the system. On each DAOS server, run:
daos_server network scan
The output should list ucx+dc_x for the configured InfiniBand or RoCE
interface. On each DAOS client, run daos_agent net-scan and verify that
ucx+dc_x is listed for the expected interface. If the provider is missing,
check that the DOCA-OFED stack and mercury-ucx are installed on that node.
NIC firmware update¶
The adapter firmware on the Network Interface Cards (NICs) of the high-speed fabric is a critical component of any parallel storage system, and DAOS is no exception. Backlevel firmware or inconsistent firmware levels across nodes in the fabric can cause operational issues that are hard to debug and resolve. When installing the host software stack for the fabric, adapter firmware levels should always be checked and updated to the correct level. Refer to the respective fabric's documentation for details.
Multiple NICs per host¶
Both DAOS servers and DAOS clients may have more than one high-speed NIC,
with IP addresses in the same network range.
Some sysctl configuration is needed to ensure proper operation in such
scenarios, in particular around ARP resolution. This is discussed in the
Pre-deployment Checklist.
Provider selection and configuration¶
On DAOS servers, the fabric provider is configured in the daos_server.yml
configuration file. Details for each supported fabric are given below.
The daos_server network scan command displays all network interfaces that
are recognized on a DAOS server for the provider that is set in the
daos_server.yml configuration file:
# daos_server network scan
DAOS Server config loaded from /etc/daos/daos_server.yml
---------
localhost
---------
-------------
NUMA Socket 0
-------------
Provider Interfaces
-------- ----------
ucx+dc_x ibs1
-------------
NUMA Socket 1
-------------
Provider Interfaces
-------- ----------
ucx+dc_x ibP1s3
To list all providers that are recognized for each of the server's
network interfaces, add the --ignore-config option. If an expected provider
is missing in the daos_server network scan --ignore-config output,
check if all the host software stack components are installed:
# daos_server network scan --ignore-config
---------
localhost
---------
-------------
NUMA Socket 1
-------------
Provider Interfaces
-------- ----------
ucx+rc ibP1s3
ucx+rc_v ibP1s3
ucx+ud ibP1s3
ucx+dc ibP1s3
ucx+dc_x ibP1s3
ucx+rc_mlx5 ibP1s3
ucx+tcp ibP1s3
ucx+rc_x ibP1s3
ucx+ud_verbs ibP1s3
ucx+ud_x ibP1s3
ucx+all ibP1s3, ibP1s3
ucx+dc_mlx5 ibP1s3
ucx+rc_verbs ibP1s3
ucx+ud_mlx5 ibP1s3
ucx+ud_v ibP1s3
-------------
NUMA Socket 0
-------------
Provider Interfaces
-------- ----------
ucx+ud_mlx5 ibs1
ucx+tcp bond0, ibs1
ucx+rc ibs1
ucx+rc_mlx5 ibs1
ucx+rc_verbs ibs1
ucx+rc_x ibs1
ucx+dc_mlx5 ibs1
ucx+dc_x ibs1
ucx+ud_v ibs1
ucx+ud_verbs ibs1
ucx+all bond0, ibs1, ibs1
ucx+dc ibs1
ucx+rc_v ibs1
ucx+ud_x ibs1
ucx+ud ibs1
DAOS clients do not require explicit fabric provider configuration.
They receive their provider configuration from the DAOS servers through the
daos_agent. This can be checked with daos_agent dump-attachinfo.
DAOS clients do need to have the correct Mercury backend and fabric
host stack software installed to be able to communicate with the servers.
The daos-agent net-scan command displays all fabric providers that are
recognized on the DAOS client for each of its network interfaces.
If an expected provider is missing in the daos_agent net-scan output,
check if all the host software stack components are installed:
# daos_agent net-scan
---------
localhost
---------
-------------
NUMA Socket 0
-------------
Provider Interfaces
-------- ----------
ucx+rc_mlx5 ibs1
ucx+ud_v ibs1
ucx+dc ibs1
ucx+dc_x ibs1
ucx+rc_x ibs1
ucx+ud_verbs ibs1
ucx+ud_x ibs1
ucx+rc ibs1
ucx+rc_v ibs1
ucx+rc_verbs ibs1
ucx+ud ibs1
ucx+ud_mlx5 ibs1
ucx+tcp bond0, ibs1
ucx+all bond0, ibs1, ibs1
ucx+dc_mlx5 ibs1
Unless they are also configured as DAOS servers or DAOS clients, DAOS admin nodes only use the control plane, so fabric provider configuration does not apply to the admin node role.
Generic TCP Fabrics with libfabric TCP¶
DAOS can run over any standard TCP/IP network, using the libfabric
ofi+tcp provider. This provider does not support RDMA transfers,
and instead emulates this capability by using the standard operating
system's TCP stack, which typically implies
higher CPU utilization, higher latency, and lower bandwidth than
the other providers. The TCP provider should only be used on fabrics
where none of the other fabric providers can be used.
The required configuration in the daos_server.yml file is minimal:
provider: ofi+tcp
engines:
-
fabric_iface: eth4
-
fabric_iface: eth5
To achieve good performance over TCP, the operating system's TCP/IP stack
has to be tuned for performance. To benefit from the performance of modern
fabrics such as 400Gbps Ethernet, larger settings than the OS defaults are
often needed for many of the networking settings. In particular,
the TCP buffer sizes in net.ipv4 (and the corresponding
settings in net.core) as well as other performance-related
TCP settings should be reviewed and tuned. The optimized settings need to be
made persistent, for example through a configuration file in /etc/sysctl.d/.
In addition to sysctl tuning, the MTU size of all network interfaces
in the fabric should be set to the largest supported value.
On native Ethernet fabrics, this is typically MTU=9000.
The MTU can be set in network interface configuration files like
/etc/sysconfig/network-scripts/ifcfg-eth0
or an equivalent nmcli configuration.
RoCE on NVIDIA Ethernet with DOCA-OFED and UCX¶
While DAOS runs on any TCP network with the ofi+tcp provider,
DAOS also supports RDMA over Converged Ethernet (RoCE)
which will provide higher performance at lower CPU utilization.
In NVIDIA Ethernet environments, the recommended provider for
RoCE is ucx+dc_x
and a current DOCA-OFED level has to be installed on all hosts.
Note that while the ofi+verbs provider should also support RoCE,
the verbs provider has scalability limitations and is not recommended.
Many NVIDIA network adapters, such as ConnectX-7, are
Virtual Protocol Interface (VPI) adapters. Each port can be set
to either InfiniBand mode (mode 1) or Ethernet mode (mode 2).
By default, all ports are set to InfiniBand. To change a port to
Ethernet, make sure that the mst package is installed (it should
be part of the DOCA-OFED installation), and run the following steps to
determine the device name of the port to be changed, change the link type,
and reboot the node to make the change effective.
For example, with ConnectX-7 adapters (MT4129):
nmcli conn delete ib0
mst start
mst status # note the device type of the adapter to use
DEV="mt4129_pciconf0"
mlxconfig -d /dev/mst/$DEV query | grep LINK_TYPE
echo "y" | mlxconfig -d /dev/mst/$DEV set LINK_TYPE_P1=2
shutdown -r now
DEV="mt4129_pciconf0"
mlxconfig -d /dev/mst/$DEV query | grep LINK_TYPE
The recommendations in the TCP section regarding OS-level tuning of the
TCP stack (sysctl settings, MTU size) are also beneficial for a RoCE setup.
There will likely be other traffic on the high-speed Ethernet interfaces
that will benefit from this tuning, even if the RoCE transport itself does
not need this.
Example of the network-related configuration settings for RoCE
in the daos_server.yml file:
provider: ucx+dc_x
engines:
-
fabric_iface: ens1np0
env_vars:
- UCX_SOCKADDR_TLS_PRIORITY=rdmacm
- UCX_IB_FORK_INIT=n
-
fabric_iface: enP1s3np0
env_vars:
- UCX_SOCKADDR_TLS_PRIORITY=rdmacm
- UCX_IB_FORK_INIT=n
Depending on the size of the fabric, other settings, such as the CaRT timeout,
(set with crt_timeout: in the global section of the daos_server.yml file)
and/or the SWIM timeout settings (set as SWIM_* environment variables
within the env_vars: section of both engines) may also need to be adjusted.
These settings depend on the cluster size and fabric details,
and there is no general recommendation to deviate from the defaults.
NVIDIA InfiniBand with DOCA-OFED and UCX¶
The recommended provider for InfiniBand fabrics is ucx+dc_x,
and a current DOCA-OFED level has to be installed on all hosts.
Example of the network-related configuration settings for InfiniBand
in the daos_server.yml file:
provider: ucx+dc_x
engines:
-
fabric_iface: ibs1
env_vars:
- UCX_SOCKADDR_TLS_PRIORITY=rdmacm
- UCX_IB_FORK_INIT=n
-
fabric_iface: ibP1s3
env_vars:
- UCX_SOCKADDR_TLS_PRIORITY=rdmacm
- UCX_IB_FORK_INIT=n
HPE Slingshot with libfabric CXI¶
The recommended provider for Slingshot fabrics is ofi+cxi,
and a current level of the HPE Slingshot Host Software (SHS) stack
has to be installed on all hosts. The settings below apply to SHS 14.0 and later.
The SHS stack includes its own version of libfabric,
which gets installed into /opt/cray/libfabric/<version>/lib64/.
For DAOS engines and DAOS clients to use the SHS version of libfabric,
the LD_LIBRARY_PATH needs to be set to point to this path.
On the DAOS servers, this is done in the env_vars section of the engines.
The following example of the network-related configuration settings for Slingshot
in daos_server.yml also includes libfabric and Mercury tunables.
While the detailed settings depend on the size of the system,
this is a good starting point for Slingshot environments.
provider: ofi+cxi
fabric_auth_key: "4:64512:65535" # Client VNI range to accept <svc_id:vni_min:vni_max>
engines:
-
fabric_iface: hsn0
fabric_iface_port: 128 # Valid CXI PIDs are in the range [0-510] (see fi_cxi manpage)
env_vars:
- D_MRECV_BUF=16 # Number of preposted multi-recv buffers
- D_MRECV_BUF_COPY=4 # Remaining buffer threshold until start of payload copy
- FI_CXI_RDZV_THRESHOLD=20480 # Needed when CXI RNR is disabled
- FI_CXI_OFLOW_BUF_SIZE=8388608 # Needed when CXI RNR is disabled
- FI_CXI_REQ_BUF_MIN_POSTED=8 # Needed when CXI RNR is disabled
- FI_CXI_REQ_BUF_SIZE=8388608 # Needed when CXI RNR is disabled
- FI_CXI_RX_MATCH_MODE=hybrid # Needed when CXI RNR is disabled
- LD_LIBRARY_PATH=/opt/cray/libfabric/2.3.1/lib64 # Path to the SHS version of libfabric
- FI_CXI_OPTIMIZED_MRS=0 # Needed with mercury <= v2.4.1
- NA_OFI_SKIP_DOMAIN_OPS=1 # Needed with mercury <= v2.4.1
- SWIM_TRAFFIC_CLASS=low_latency # Separate traffic for SWIM messages
-
fabric_iface: hsn1
fabric_iface_port: 256
env_vars:
- [...] # Identical to settings above
The cache monitor for the memory registration (MR) cache for RDMA transfers is one of the key settings for Slingshot. Refer to the "kdreg2" sections in the HPE Slingshot Host Software Installation and Configuration Guide and Host Software Administration Guide.
On the DAOS servers, the MR cache monitor is disabled by default (FI_MR_CACHE_MONITOR=disabled).
On the DAOS clients, the recommendation is to use the kdreg2 cache monitor.
The following tunables should be set in the user environment
on all DAOS client nodes
(for example, through /etc/environment, /etc/profile.env, or a systemd service).
See the DAOS section in the Host Software User Guide:
FI_CXI_RX_MATCH_MODE=hybrid
FI_MR_CACHE_MONITOR=kdreg2
Note that setting FI_CXI_RX_MATCH_MODE=hybrid globally (when RNR is disabled) may affect
performance of all client applications (including MPI applications). In this particular case,
a more fine-grained alternative is to rely on custom modulefiles that can be loaded when DAOS
must be used. It may also be beneficial to adjust the FI_MR_CACHE_MAX_SIZE and
FI_MR_CACHE_MAX_COUNT tunables.
General TCP and Ethernet tuning recommendations for Slingshot can be found
in the TCP performance tuning and Ethernet tuning sections of the
HPE Slingshot Host Software Administration Guide.
The tuning script referenced therein is located in
/opt/slingshot/utils/<version>/bin/slingshot-eth-tuning.
VNI Configuration¶
Note
The following section currently only applies to PBS environments.
To isolate RDMA traffic between DAOS clients, it is highly recommended for security to enable VNIs (Virtual Network Identifiers). A VNI is a protection key used by the Slingshot network to provide isolation between applications. Because VNIs define an isolated PID space for a given NIC, endpoints must use the same VNI in order to communicate. To allow this type of communication between DAOS clients and servers, a VNI range must be properly configured on servers and client nodes must be configured to only select a VNI within that specified range.
This is done in multiple steps:
- A CXI service (see documentation) that accepts a
range of VNIs must first be configured on DAOS server nodes. This is done by
using the cxi_service command. This command has been augmented in SHS 14.0 to specify
VNI ranges in its yaml configuration file:
yaml
vnis:
vni_min: 64512
vni_max: 65535
One can then create a CXI service by running cxi_service create -y <service_yaml_file>, which produces a new CXI service identifier.
- The second step is to configure DAOS servers to use the specified VNI range and CXI service.
This is controlled by the fabric_auth_key parameter in the daos_server.yml file. For example:
yaml
fabric_auth_key: "4:64512:65535" # Client VNI range to accept <svc_id:vni_min:vni_max>
where 4 is the CXI service identifier, and 64512 and 65535 are the minimum and maximum VNIs, respectively.
- The third step is to configure the vnid service (see VNI range documentation
for additional details) that interacts with the PBS scheduler and the PALS daemons running on the
compute nodes. One must first resize the default application pool to accommodate the desired VNI
range (which usually spans the entire VNI range) and create a new pool by running the following
command:
bash
vnidctrl --url <vnid_url> pool update "applications" 1024 64511 "applications"
vnidctrl --url <vnid_url> pool create "daos" 64512 65535 "jobs"
Effectively, this command creates a new pool named "daos" with the specified VNI range, "jobs"
meaning that the VNIs allocated from that pool are per job.
- Finally, one can test the configuration by running the following command within a PBS job:
bash
mpiexec --daos env | grep SLING
SLINGSHOT_VNIS=1116,1115,64529
SLINGSHOT_DEVICES=cxi0
SLINGSHOT_SVC_IDS=5
Note the --daos flag passed to the mpiexec command, which ensures that the DAOS VNI (the third VNI in the SLINGSHOT_VNIS list) is set.
Warning
Through this method, all daos commands and applications that interact with DAOS can only be
run within a PBS job and mpiexec must be used with the --daos flag to ensure proper DAOS VNI
configuration.
Cornelis Omni-Path with libfabric TCP or VERBS¶
For Omni-Path fabrics, the libfabric psm2 and opx providers, which are
available for MPI message-passing applications, cannot be used with DAOS due
to some functional gaps.
The libfabric tcp provider is supported on Omni-Path.
As with all high-speed fabrics, the MTU size is important to achieve good performance
with TCP. The default MTU size on Omni-Path is 2048, but it can be increased to 10240
by editing the configuration file for the Omni-Path fabric manager /etc/opa-fm/opafm.xml
and restarting the fabric manager (systemctl restart opafm). In the default
configuration, there is a single MulticastGroup and its MTU needs to be changed
to 10240. If multiple MulticastGroups exist, change the MTU for all of them:
<MulticastGroup>
<Create>1</Create>
<MTU>10240</MTU>
<Rate>100g</Rate>
<!-- <SL>0</SL> -->
<QKey>0x0</QKey>
<TClass>0x0</TClass>
</MulticastGroup>
After restarting the fabric manager, hosts will automatically pick up the
increased MTU size, for example the ip a show command will
show an MTU size of 10236 (10240 minus a 4-byte header):
# ip a show ibs3d1 | head -1
9: ibs3d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 10236 qdisc mq state UP group default qlen 1000
This MTU change will enable line-rate performance on Omni-Path 100 fabrics. But it will not be possible to saturate the 400Gbps link bandwidth of Omni-Path CN5000 fabrics with TCP, even with the increased MTU size.
Cornelis has implemented a Verbs API over CN5000, and the ofi+verbs
provider is the recommended provider for DAOS on CN5000.
To achieve the best performance, it is recommended to enable the Cornelis
HFI service (previously called Bulk Transfer Service (BTS))
in the Omni-Path SuperNIC driver:
cat /sys/module/hfi1/parameters/use_bulksvc
echo "options hfi1 use_bulksvc=Y" >> /etc/modprobe.d/hfi1.conf
dracut –f
reboot
cat /sys/module/hfi1/parameters/use_bulksvc
Before enabling the HFI service, querying the use_bulksvc driver parameter
will return N. When the service is enabled, it should return Y.
This needs to be done on each host in the CN5000 fabric.
Example of the network-related configuration settings for Omni-Path
CN5000 with the ofi+verbs API in the daos_server.yml file:
provider: ofi+verbs
engines:
-
fabric_iface: ibs3d1
-
fabric_iface: ibs6d1
Depending on the size of the fabric, other settings like the CaRT timeout
(set with crt_timeout: in the global section of the daos_server.yml file)
and/or the SWIM timeout settings (set as SWIM_* environment variables
within the env_vars: section of both engines) may also need to be adjusted.
But these settings depend on the cluster size and fabric details,
and there is no general recommendation to deviate from the defaults.