Run Datamover test - DAOS v3.0
Author: DAOS Project
Description: Distributed Asynchronous Object Storage

Skip to content

# DataMover

Note

The example below makes use of a DAOS pool and its POSIX directory setup as part of Run IOR and mdtest. It also makes use of data written to the same pool as part of Run dbench.

## 'daos' utility (single process)

Use the `daos filesystem copy` command to copy data between containers.

1. Create a new container:

    $ daos cont create test_pool test_cont2 --type=POSIX
    Successfully created container cec2bb35-dc65-4397-9672-8ef2607e31ea type POSIX
      Container UUID : cec2bb35-dc65-4397-9672-8ef2607e31ea
      Container Label: test_cont2
      Container Type : POSIX

1. Query the pool before copying data:

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 59 GB, min:2.4 GB, max:2.5 GB, mean:2.5 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 742 GB, min:31 GB, max:33 GB, mean:31 GB

1. Copy data from POSIX directory into the new DAOS container:

    $ daos filesystem copy --src /tmp/daos_dfuse/ --dst daos://test_pool/test_cont2
    Successfully copied to DAOS: daos://test_pool/test_cont2
        Directories: 554
        Files:       33341
        Links:       0

Note

The `daos://test_pool/test_cont2` destination path utilizes the `test_pool` pool and `test_cont2` container labels

1. Query the pool to confirm the data copy (Free space has been reduced from last pool query):

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 59 GB, min:2.4 GB, max:2.5 GB, mean:2.5 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 698 GB, min:29 GB, max:31 GB, mean:29 GB

1. Copy data from the new DAOS container to POSIX directory:

    $ mkdir /tmp/daos_dfuse/daos_container_copy
    $ daos filesystem copy --src daos://test_pool/test_cont2 --dst /tmp/daos_dfuse/daos_container_copy
    Successfully copied to DAOS: /tmp/daos_dfuse/daos_container_copy
        Directories: 554
        Files:       33341
        Links:       0

1. Query the pool to confirm data copy (Free space has been further reduced from last pool query):

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 59 GB, min:2.4 GB, max:2.5 GB, mean:2.5 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 655 GB, min:27 GB, max:29 GB, mean:27 GB

1. Check data inside the POSIX directories:

    $ ls -latr /tmp/daos_dfuse/
    total 20971520
    -rw-r--r-- 1 hendersp ldap 21474836480 Mar 19 17:46 testfile
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 17:55 test-dir.0-0
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 21:50 clients
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:48 daos_container_copy
    $
    $ ls -latr /tmp/daos_dfuse/daos_container_copy
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:37 clients
    -rw-r--r-- 1 hendersp ldap 21474836480 Mar 19 22:46 testfile
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:48 test-dir.0-0

## mpifileutils (multi-process)

Use `mpifileutils` to move data between containers.

1. Build mpifileutils package:

Load mpich module or set it's path in your environment

    $ module load mpich
    $ module list
    Currently Loaded Modulefiles:
     1) mpich/5.0.0b1.lua

or

    $ export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
    $ export PATH=<mpich bin path>:$PATH

Install dependencies

For RHEL:

    $ sudo dnf install -y daos-devel libattr-devel bzip2-devel

For SLES:

    $ sudo zypper install -y daos-devel libattr-devel libbz2-devel

Build other dependencies

    $ installdir=${HOME}/software/mpifileutils
    $ mkdir $installdir

    $ cd /tmp
    $ mkdir deps; cd deps
    $ wget https://github.com/hpc/libcircle/releases/download/v0.3/libcircle-0.3.0.tar.gz
    $ wget https://github.com/llnl/lwgrp/releases/download/v1.0.4/lwgrp-1.0.4.tar.gz
    $ wget https://github.com/llnl/dtcmp/releases/download/v1.1.4/dtcmp-1.1.4.tar.gz
    $ wget https://github.com/libarchive/libarchive/releases/download/v3.5.1/libarchive-3.5.1.tar.gz

    $ tar -zxf libcircle-0.3.0.tar.gz
    $ cd libcircle-0.3.0
    $ ./configure --prefix=$installdir
    $ make install
    $ cd ..

    $ tar -zxf lwgrp-1.0.4.tar.gz
    $ cd lwgrp-1.0.4
    $ ./configure --prefix=$installdir
    $ make install
    $ cd ..

    $ tar -zxf dtcmp-1.1.4.tar.gz
    $ cd dtcmp-1.1.4
    $ export LD_LIBRARY_PATH=$installdir/lib64:$LD_LIBRARY_PATH
    $ ./configure --prefix=$installdir --with-lwgrp=$installdir
    $ make install
    $ cd ..

    $ tar -zxf libarchive-3.5.1.tar.gz
    $ cd libarchive-3.5.1/
    $ ./configure --prefix=$installdir
    $ make install
    $ cd ..

Build mpifileutils

    $ wget https://github.com/hpc/mpifileutils/releases/download/v0.11.1/mpifileutils-v0.11.1.tgz
    $ tar -zxf mpifileutils-v0.11.1.tgz
    $ cd mpifileutils-v0.11.1/
    $ mkdir build; cd build
    $ cmake .. -DWITH_LibArchive_PREFIX=$installdir -DWITH_DTCMP_PREFIX=$installdir -DWITH_LibCircle_PREFIX=$installdir -DCMAKE_INSTALL_PREFIX=$installdir -DWITH_DAOS_PREFIX=/usr/ -DENABLE_DAOS=ON -DMPI_C_COMPILER=${HOME}/software/mpich/bin/mpicc -DMPI_CXX_COMPILER=${HOME}/software/mpich/bin/mpicxx
    $ make -j install
    $ cd

    $ export LD_LIBRARY_PATH=${HOME}/software/mpifileutils/lib64/:$LD_LIBRARY_PATH
    $ export PATH=${HOME}/software/mpifileutils/bin/:$PATH
    $ which dcp
    ~/software/mpifileutils/bin/dcp

1. Create a new container:

    $ daos cont create test_pool test_cont3 --type=POSIX
    Successfully created container 86e9c9a3-8667-4ad4-9a0c-364a318acb6b type POSIX
      Container UUID : 86e9c9a3-8667-4ad4-9a0c-364a318acb6b
      Container Label: test_cont3
      Container Type : POSIX

1. Query the pool before moving data:

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 59 GB, min:2.4 GB, max:2.5 GB, mean:2.5 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 655 GB, min:27 GB, max:29 GB, mean:27 GB

1. Move data from POSIX directory into the new DAOS container:

    $ mpirun -hosts $CLIENT_NODES -np 16 -wdir /tmp/daos_dfuse ${HOME}/software/mpifileutils/bin/dcp --bufsize 64MB --chunksize 128MB /tmp/daos_dfuse daos://test_pool/test_cont3/
    [2026-03-20T18:01:05] Walking /
    [2026-03-20T18:01:09] Walked 66910 items in 2.577 secs (25964.570 items/sec) ...
    [2026-03-20T18:01:09] Walked 66910 items in 2.582 seconds (25914.574 items/sec)
    [2026-03-20T18:01:09] Copying to /
    [2026-03-20T18:01:09] Items: 66910
    [2026-03-20T18:01:09]   Directories: 228
    [2026-03-20T18:01:09]   Files: 66682
    [2026-03-20T18:01:09]   Links: 0
    [2026-03-20T18:01:09] Data: 40.254 GiB (633.000 KiB per file)
    [2026-03-20T18:01:09] Creating 228 directories
    [2026-03-20T18:01:09] Original directory exists, skip the creation: `//' (errno=17 File exists)
    [2026-03-20T18:01:09] Creating 66682 files.
    [2026-03-20T18:01:12] Copying data.
    [2026-03-20T18:01:27] Copied 363.957 MiB (1%) in 15.000 secs (24.264 MiB/s) 1684 secs left ...
    [2026-03-20T18:19:46] Copied 810.250 MiB (2%) in 1114.103 secs (744.721 KiB/s) 55564 secs left ...
    [2026-03-20T18:19:46] Copied 40.254 GiB (100%) in 1114.103 secs (36.999 MiB/s) done
    [2026-03-20T18:19:46] Copy data: 40.254 GiB (43222794240 bytes)
    [2026-03-20T18:19:46] Copy rate: 36.999 MiB/s (43222794240 bytes in 1114.103 seconds)
    [2026-03-20T18:19:46] Syncing data to disk.
    [2026-03-20T18:19:46] Sync completed in 0.001 seconds.
    [2026-03-20T18:19:46] Fixing permissions.
    [2026-03-20T18:19:53] Updated 67790 items in 6.090 seconds (11130.953 items/sec)
    [2026-03-20T18:19:53] Syncing directory updates to disk.
    [2026-03-20T18:19:53] Sync completed in 0.001 seconds.
    [2026-03-20T18:19:53] Started: Mar-20-2026,18:01:03
    [2026-03-20T18:19:53] Completed: Mar-20-2026,18:19:53
    [2026-03-20T18:19:53] Seconds: 1129.781
    [2026-03-20T18:19:53] Items: 67790
    [2026-03-20T18:19:53]   Directories: 1108
    [2026-03-20T18:19:53]   Files: 66682
    [2026-03-20T18:19:53]   Links: 0
    [2026-03-20T18:19:53] Data: 40.254 GiB (43222794240 bytes)
    [2026-03-20T18:19:53] Rate: 36.485 MiB/s (43222794240 bytes in 1129.781 seconds)

1. Query the pool to verify data was copied (free space has been reduced):

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 59 GB, min:2.4 GB, max:2.5 GB, mean:2.4 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 569 GB, min:23 GB, max:27 GB, mean:24 GB

1. Move data from the new DAOS container to POSIX directory:

    $ mkdir /tmp/daos_dfuse/test_cont3_copy
    $ mpirun -hosts $CLIENT_NODES -np 16 -wdir /tmp/daos_dfuse ${HOME}/software/mpifileutils/bin/dcp --bufsize 64MB --chunksize 128MB daos://test_pool/test_cont3/ /tmp/daos_dfuse/test_cont3_copy
    [2026-03-20T18:31:28] Walking /
    [2026-03-20T18:31:31] Walked 67790 items in 2.685 secs (25248.913 items/sec) ...
    [2026-03-20T18:31:31] Walked 67790 items in 2.686 seconds (25241.379 items/sec)
    [2026-03-20T18:31:31] Copying to /test_cont3_copy
    [2026-03-20T18:31:31] Items: 67790
    [2026-03-20T18:31:31]   Directories: 1108
    [2026-03-20T18:31:31]   Files: 66682
    [2026-03-20T18:31:31]   Links: 0
    [2026-03-20T18:31:31] Data: 40.254 GiB (633.000 KiB per file)
    [2026-03-20T18:31:31] Creating 1108 directories
    [2026-03-20T18:31:31] Original directory exists, skip the creation: `/test_cont3_copy/' (errno=17 File exists)
    [2026-03-20T18:31:31] Creating 66682 files.
    [2026-03-20T18:31:37] Copying data.
    [2026-03-20T18:31:52] Copied 305.402 MiB (1%) in 15.012 secs (20.345 MiB/s) 2011 secs left ...
    [2026-03-20T18:49:34] Copied 752.625 MiB (2%) in 1076.404 secs (715.984 KiB/s) 57877 secs left ...
    [2026-03-20T18:49:34] Copied 40.254 GiB (100%) in 1076.404 secs (38.295 MiB/s) done
    [2026-03-20T18:49:34] Copy data: 40.254 GiB (43222794240 bytes)
    [2026-03-20T18:49:34] Copy rate: 38.295 MiB/s (43222794240 bytes in 1076.404 seconds)
    [2026-03-20T18:49:34] Syncing data to disk.
    [2026-03-20T18:49:34] Sync completed in 0.001 seconds.
    [2026-03-20T18:49:34] Fixing permissions.
    [2026-03-20T18:49:40] Updated 67790 items in 6.317 seconds (10731.073 items/sec)
    [2026-03-20T18:49:40] Syncing directory updates to disk.
    [2026-03-20T18:49:40] Sync completed in 0.001 seconds.
    [2026-03-20T18:49:40] Started: Mar-20-2026,18:31:31
    [2026-03-20T18:49:40] Completed: Mar-20-2026,18:49:40
    [2026-03-20T18:49:40] Seconds: 1089.640
    [2026-03-20T18:49:40] Items: 67790
    [2026-03-20T18:49:40]   Directories: 1108
    [2026-03-20T18:49:40]   Files: 66682
    [2026-03-20T18:49:40]   Links: 0
    [2026-03-20T18:49:40] Data: 40.254 GiB (43222794240 bytes)
    [2026-03-20T18:49:40] Rate: 37.829 MiB/s (43222794240 bytes in 1089.640 seconds)

1. Query the pool to very data was copied (free space has been further reduced):

    $ dmg pool query test_pool
    Pool e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa, ntarget=24, disabled=0, leader=1, version=1, state=Ready
    Pool health info:
    - Rebuild idle, 0 objs, 0 recs
    - Data redundancy: normal
    Pool space info:
    - Target count:24
    - Storage tier 0 (SCM):
      Total size: 74 GB
      Free: 58 GB, min:2.4 GB, max:2.5 GB, mean:2.4 GB
    - Storage tier 1 (NVME):
      Total size: 786 GB
      Free: 482 GB, min:19 GB, max:23 GB, mean:20 GB

1. Check data inside the POSIX directories:

    $ ls -latr /tmp/daos_dfuse/
    total 20971520
    -rw-r--r-- 1 hendersp ldap 21474836480 Mar 19 17:46 testfile
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 17:55 test-dir.0-0
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 21:50 clients
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:48 daos_container_copy
    drwxr-xr-x 1 hendersp ldap         104 Mar 20 18:49 test_cont3_copy
    $
    $ ls -latr /tmp/daos_dfuse/test_cont3_copy
    total 20971520
    -rw-r--r-- 1 hendersp ldap 21474836480 Mar 20 18:49 testfile
    drwxr-xr-x 1 hendersp ldap         104 Mar 20 18:49 daos_container_copy
    drwxr-xr-x 1 hendersp ldap         104 Mar 20 18:49 test-dir.0-0
    drwxr-xr-x 1 hendersp ldap         104 Mar 20 18:49 clients
    $
    $ ls -latr /tmp/daos_dfuse/daos_container_copy
    total 20971520
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:37 clients
    -rw-r--r-- 1 hendersp ldap 21474836480 Mar 19 22:46 testfile
    drwxr-xr-x 1 hendersp ldap         104 Mar 19 22:48 test-dir.0-0

Note

For more details on datamover, reference DAOS Support on the mpifileutils website.

## Clean Up

1. Remove the copy created using datamover

    $ rm -rf /tmp/daos_dfuse/daos_container_copy

1. Remove dfuse mountpoint:

    $ clush -B -w $CLIENT_NODES 'fusermount3 -uz /tmp/daos_dfuse'
    $ clush -B -w $CLIENT_NODES rm -rf /tmp/daos_dfuse

1. List containers to be destroyed:

    $ daos pool list-containers test_pool
    Containers in pool test_pool:
      Label
      -----
      test_cont
      test_cont2
      test_cont3

1. Destroy Containers:

    $ daos container destroy test_pool test_cont
    Successfully destroyed container test_cont
    $ daos container destroy test_pool test_cont2
    Successfully destroyed container test_cont2
    $ daos container destroy test_pool test_cont3
    Successfully destroyed container test_cont3
    $

    $ daos pool list-containers test_pool
    No containers.

1. List Pools to be destroyed:

    $ dmg pool list
    Pool          Size   State Used Imbalance Disabled
    ----          ----   ----- ---- --------- --------
    autotest_pool 47 GB  Ready 2%   0%        0/24
    test_pool     786 GB Ready 0%   0%        0/24

1. Destroy Pool:

    $ dmg pool destroy autotest_pool
    Pool-destroy command succeeded
    $ dmg pool destroy test_pool
    Pool-destroy command succeeded
    $

    $ dmg pool list
    No pools in system

1. Stop Agents:

    $ clush -B -w $CLIENT_NODES "sudo systemctl stop daos_agent"

1. Stop Servers:

    $ clush -B -w $SERVER_NODES "sudo systemctl stop daos_server"