Skip to content

DAOS in Docker

This section describes how to build and deploy Docker images allowing to simulate a small cluster using DAOS as backend storage. This small cluster is composed of the following three nodes:

  • The daos-server node running a DAOS server daemon managing data storage devices such as SCM or NVMe disks.
  • The daos-admin node allowing to manage the DAOS server thanks to dmgcommand.
  • The daos-client node uses the DAOS server to store data.

At this time only emulated hardware storage is supported by this Docker platform:

  • SCM (i.e., Storage Class Memory) are emulated with standard RAM memory.
  • NVMe disks are emulated with a file device.

Warning

Virtual Docker networks such as bridge are not yet well supported by DAOS. Thus, one physical network interface (i.e., loopback interface is also not well supported) of the host should be chosen for being used by the containers through the Docker host network.

Prerequisites

To build and deploy the Docker images, docker and optionally docker-compose shall be available. The docker host should have access to the Docker Hub and Rocky Linux official repositories.

The host hardware must support virtualized device access, and it must be enabled in the system BIOS. On Intel® systems, this capability is named Intel® Virtualization Technology for Directed I/O (VT-d). Once enabled in BIOS, IOMMU support must also be enabled in the Linux kernel. Exact details depend on the distribution, but the following example should be illustrative:

# Enable IOMMU on CentOS 7
# All commands must be run as root/sudo!

$ sudo vi /etc/default/grub # add the following line:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

# After saving the file, run the following to reconfigure
# the bootloader:
$ sudo grub2-mkconfig --output=/boot/grub2/grub.cfg

# If the command completed with no errors, reboot the system
# in order to make the changes take effect
$ sudo reboot

Finally, HugePages Linux kernel feature shall be enabled on the docker host. At least, 4096 pages of 2048kB should be available. The number of HugePages allocated could be checked with the following command:

$ sysctl vm.nr_hugepages

The default size of a huge page, the number of available HugePages, etc. could be found with the following command:

$ cat /proc/meminfo | grep -e "^Huge"

The platform was tested and validated with the rockylinux/rockylinux:8.4 official docker image. However other RHEL-like distributions should be supported.

Warning

Some distributions are not yet well supported such as rockylinux/rockylinux:8.5: - issue DAOS-10046: management of HugePages with the spdk library.

Configuring HugePages

First, the Linux kernel needs to be built with the CONFIG_HUGETLBFS (present under "File systems") and CONFIG_HUGETLB_PAGE (selected automatically when CONFIG_HUGETLBFS is selected) configuration options.

To avoid memory fragmentation, HugePages could be allocated on the kernel boot command line by specifying the "hugepages=N" parameter, where 'N' = the number of HugePages requested. It is also possible to allocate them at run time, thanks to the sysctl command:

$ sysctl vm.nr_hugepages=8192

It is also possible to use the sysctl command to allocate HugePages at boot time with the following command:

$ cat <<< "vm.nr_hugepages = 8192" > /etc/sysctl.d/50-hugepages.conf
$ sysctl -p

Building Docker Images

Base DAOS Image

The first image to create is the daos-base image which is not intended to be used as it, but as a base image for building the other three daos images. This first image could be built directly from GitHub with the following command:

$ docker build --tag daos-base:rocky8.4 \
    https://github.com/daos-stack/daos.git#release/2.0:utils/docker/vcluster/daos-base/el8

This Docker file accepts the following arguments:

  • RHEL_BASE_IMAGE: Base docker image to use (default "rockylinux/rockylinux")
  • RHEL_BASE_VERSION: Version of the base docker image to use (default "8.4")
  • BUST_CACHE: Manage docker building cache (default undefined). To invalidate the cache, a random value such as the date of the day shall be given.
  • DAOS_AUTH: Enable DAOS authentication when set to "yes" (default "no")
  • DAOS_REPOS: Space-separated list of repos needed to install DAOS (default "https://packages.daos.io/v2.0/EL8/packages/x86_64/")
  • DAOS_GPG_KEYS: Space-separated list of GPG keys associated with DAOS repos (default "https://packages.daos.io/RPM-GPG-KEY")
  • DAOS_REPOS_NOAUTH: Space-separated list of repos to use without GPG authentication (default "")

For example, building a DAOS base image with authentication enabled could be done with the following command:

$ docker build --tag daos-base:rocky8.4 --build-arg DAOS_AUTH=yes \
    https://github.com/daos-stack/daos.git#release/2.0:utils/docker/vcluster/daos-base/el8

It is also possible to build the daos-base image from a local tree with the following command:

$ docker build --tag daos-base:rocky8.4 utils/docker/vcluster/daos-base/el8

DAOS Nodes Images

The three images daos-server, daos-admin and daos-client could be built directly from GitHub or from a local tree in the same way as for the daos-base image. Following command could be used to build directly the three images from GitHub:

$ for image in daos-server daos-admin daos-client ; do \
    docker build --tag "$image:rocky8.4" \
        "https://github.com/daos-stack/daos.git#release/2.0:utils/docker/vcluster/$image/el8"; \
  done

The Docker file of the daos-server image accepts the following arguments:

  • DAOS_BASE_IMAGE: Base docker image to use (default "daos-base")
  • DAOS_BASE_VERSION: Version of the base docker image to use (default "rocky8.4")
  • DAOS_AUTH: Enable DAOS authentication when set to "yes" (default "no")
  • DAOS_HUGEPAGES_NBR: Number of HugePages to allocate for SPDK (default 4096)
  • DAOS_SCM_SIZE: Size in GB of the RAM emulating SCM devices (default 4)
  • DAOS_BDEV_SIZE: Size in GB of the file created to emulate NVMe devices (default 16)
  • DAOS_IFACE_NAME: Fabric network interface used by the DAOS engine (default "eth0")

Note

The IP address of the network interface referenced by the DAOS_IFACE_NAME argument will be required when starting DAOS.

The Docker file of the daos-client and daos-admin images accepts the following arguments:

  • DAOS_BASE_IMAGE: Base docker image to use (default "daos-base")
  • DAOS_BASE_VERSION: Version of the base docker image to use (default "rocky8.4")
  • DAOS_AUTH: Enable DAOS authentication when set to "yes" (default "no")
  • DAOS_ADMIN_USER: Name or uid of the daos administrator user (default "root")
  • DAOS_ADMIN_GROUP: Name or gid of the daos administrator group (default "root")

From a local tree, a more straightforward way to build these images could be done with docker-compose and the following commands:

$ docker-compose --file utils/docker/vcluster/docker-compose.yml -- build

The same arguments are accepted but they must be defined in the Docker Compose environment file utils/docker/vcluster/.env. For example, on system with non standard network interface names such as "ino1" instead of "eth0", the following command could be used to update the Docker Compose environment file.

$ sed -i -e s/eth0/eno1/ utils/docker/vcluster/.env

Warning

For working properly, the DAOS authentication has to be enabled in all the images (i.e., nodes images and base image).

Running the DAOS Containers

Via Docker Commands

Once the images are created, the containers could be directly started with docker with the following commands:

$ export DAOS_IFACE_IP=x.x.x.x
$ docker run --detach --privileged --name=daos-server --hostname=daos-server \
    --add-host "daos-server:$DAOS_IFACE_IP" --add-host "daos-admin:$DAOS_IFACE_IP" \
    --add-host "daos-client:$DAOS_IFACE_IP" --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
    --volume=/dev/hugepages:/dev/hugepages  --tmpfs=/run --network=host \
    daos-server:rocky8.4
$ docker run --detach --privileged --name=daos-agent --hostname=daos-agent \
    --add-host "daos-server:$DAOS_IFACE_IP" --add-host "daos-admin:$DAOS_IFACE_IP" \
    --add-host "daos-client:$DAOS_IFACE_IP" --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
    --tmpfs=/run --network=host daos-agent:rocky8.4
$ docker run --detach --privileged --name=daos-client --hostname=daos-client \
    --add-host "daos-server:$DAOS_IFACE_IP" --add-host "daos-admin:$DAOS_IFACE_IP" \
    --add-host "daos-client:$DAOS_IFACE_IP" --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
    --tmpfs=/run --network=host daos-client:rocky8.4

The value of the DAOS_IFACE_IP shall be replaced with one of the network interfaces which was provided when the images were built.

Once started, the DAOS server waits for the administrator to format the system. This can be done using the following command:

$ docker exec daos-admin dmg -i storage format

Upon successful completion of the format, the storage engine is started, and pools can be created using the daos admin tool. For more advanced configurations and usage refer to the section DAOS Tour.

Via docker-compose

From a local tree, a more straightforward way to start the containers could be done with docker-compose and the following commands:

$ docker-compose --file utils/docker/vcluster/docker-compose.yml -- up --detach

Note

Before starting the containers with docker-compose, the IP address of the network interface, which was provided when the images have been built, shall be defined in the Docker Compose environment file utils/docker/vcluster/.env.

As with the docker command, the system shall be formatted, pools created, etc...

Via Custom Scripts

From a local tree, the bash script utils/docker/vcluster/daos-cm.sh could be used to start the containers and setup a simple DAOS system composed of the following elements:

  • 1 DAOS pool of 10GB (i.e., size of the pool is configurable)
  • 1 DAOS POSIX container mounted on /mnt/daos-posix-fs

This script could also be used to respectively stop and monitor the containers.

More details on the usage of daos-cm.sh command could be found by running the following command:

$ utils/docker/vcluster/daos-cm.sh --help
Back to top