Run IOR and mdtest - DAOS v2.8
Author: DAOS Project
Description: Distributed Asynchronous Object Storage

Skip to content

# Run IOR & mdtest

The `ior` benchmarking tool can be used to generate an HPC type I/O load. This is an MPI-parallel tool that requires MPI to start and control the IOR processes on the client nodes. For this purpose, any MPI implementation can be used. IOR also has an MPI-IO backend, and in order to use this MPI-IO backend with DAOS, an MPI stack that includes the DAOS ROMIO backend has to be used to build and run IOR. Refer to MPI-IO Support for details.

In addition to the default POSIX API, IOR also natively supports the DFS backend that directly uses DAOS File System (DFS) I/O calls instead of POSIX I/O calls. Details on the DAOS DFS backend for IOR can be found in the README_DAOS at the IOR github repository.

The `mdtest` tool to benchmark metadata performance is included in the same repository as the IOR tool. Like `IOR`, it is also an MPI-parallel application.

The Performance Tuning section of the Administration Guide contains further information on IOR and mdtest.

## Build ior and mdtest

    $ module load mpich # or any other MPI stack

    $ cd /tmp
    $ git clone https://github.com/hpc/ior.git
    $ cd ior
    $ git checkout 4.0.0
    $ ./bootstrap
    $ mkdir build; cd build
    $ ../configure --with-daos=/usr --prefix=$HOME/software/ior
    $ make
    $ make install

## Run ior

This example uses the default IOR `API=POSIX`, and requires that the DAOS POSIX container is dfuse-mounted at `/tmp/daos_dfuse` on all client nodes. Refer to DFuse (DAOS FUSE) for details on dfuse mounts.

Example dfuse-mounted DAOS POSIX container:

    $ dmg pool create --size=25% -u ${USER}@ test_pool
    Creating DAOS pool with 25% of all storage
    Pool created with 8.64%,91.36% storage tier ratio
    -------------------------------------------------
      UUID                 : e0630b72-68e5-4dbc-b6ec-e1b1c201f8aa
      Service Leader       : 1
      Service Ranks        : [0-1]
      Storage Ranks        : [0-1]
      Total Size           : 860 GB
      Storage tier 0 (SCM) : 74 GB (37 GB / rank)
      Storage tier 1 (NVMe): 786 GB (393 GB / rank)

    $ daos cont create test_pool test_cont --type=POSIX
    Successfully created container 90e3d9a3-2eab-4f08-b912-7c468e5dce56 type POSIX
      Container UUID : 90e3d9a3-2eab-4f08-b912-7c468e5dce56
      Container Label: test_cont
      Container Type : POSIX

    $ clush -B -w $CLIENT_NODES mkdir /tmp/daos_dfuse
    $ clush -B -w $CLIENT_NODES dfuse /tmp/daos_dfuse test_pool test_cont
    $ clush -B -w $CLIENT_NODES "df -h | grep fuse"
    ----------------
    client-[1-2]
    ----------------
    dfuse                                                       802G   16G  787G   2% /tmp/daos_dfuse
    $

The per-task performance over a dfuse mount is limited. To obtain better performance, the POSIX API can be used in conjunction with the IOIL interception library. For best performance IOR can be run with `API=DFS`, passing in the DAOS pool and container information (`ior -a DFS --dfs.pool=$DAOS_POOL --daos.cont=$DAOS_CONT ...`).

    $ module load mpich # or any other MPI stack

    $ mpirun -hosts $CLIENT_NODES -np 10 ${HOME}/software/ior/bin/ior -a POSIX -b 2G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile

    IOR-4.0.0: MPI Coordinated Test of Parallel I/O
    Began               : Thu Mar 19 17:42:44 2026
    Command line        : ${HOME}/software/ior/bin/ior -a POSIX -b 2G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile
    Machine             : Linux brd-233.daos.hpc.amslabs.hpecorp.net
    TestID              : 0
    StartTime           : Thu Mar 19 17:42:44 2026
    Path                : /tmp/daos_dfuse_test/testfile
    FS                  : 801.2 GiB   Used FS: 1.9%   Inodes: -0.0 Mi   Used Inodes: 0.0%
    Participating tasks : 10

    Options:
    api                 : POSIX
    apiVersion          :
    test filename       : /tmp/daos_dfuse_test/testfile
    access              : single-shared-file
    type                : independent
    segments            : 1
    ordering in a file  : sequential
    ordering inter file : no tasks offsets
    nodes               : 2
    tasks               : 10
    clients per node    : 5
    repetitions         : 1
    xfersize            : 1 MiB
    blocksize           : 2 GiB
    aggregate filesize  : 20 GiB
    verbose             : 1

    Results:

    access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
    ------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
    Commencing write performance test: Thu Mar 19 17:42:44 2026
    write     101.04     101.04     0.098970    2097152    1024.00    0.005258   202.69     0.003326   202.70     0
    Verifying contents of the file(s) just written.
    Thu Mar 19 17:46:08 2026

    Commencing read performance test: Thu Mar 19 17:47:54 2026

    read      30912      30992      0.000321    2097152    1024.00    0.004187   0.660818   0.003185   0.662533   0
    Max Write: 101.04 MiB/sec (105.95 MB/sec)
    Max Read:  30911.67 MiB/sec (32413.23 MB/sec)

    Summary of all tests:
    Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
    write         101.04     101.04     101.04       0.00     101.04     101.04     101.04       0.00  202.69653         NA            NA     0     10   5    1   0     0        1         0    0      1 2147483648  1048576   20480.0 POSIX      0
    read        30911.67   30911.67   30911.67       0.00   30911.67   30911.67   30911.67       0.00    0.66253         NA            NA     0     10   5    1   0     0        1         0    0      1 2147483648  1048576   20480.0 POSIX      0
    Finished            : Thu Mar 19 17:47:55 2026

## Run mdtest

Use mdtest to create 30K files using `API=POSIX` (as for IOR, using `-a DFS` will provide much better performance):

    $ mpirun -hosts $CLIENT_NODES -np 10 ${HOME}/software/ior/bin/mdtest -a POSIX -z 0 -F -C -i 1 -n 3334 -e 4096 -d /tmp/daos_dfuse -w 4096
    -- started at 03/19/2026 17:55:06 --

    mdtest-4.0.0 was launched with 10 total task(s) on 2 node(s)
    Command line used: ${HOME}/software/ior/bin/mdtest '-a' 'POSIX' '-z' '0' '-F' '-C' '-i' '1' '-n' '3334' '-e' '4096' '-d' '/tmp/daos_dfuse' '-w' '4096'
    Path                : /tmp/daos_dfuse_test
    FS                  : 36.4 GiB   Used FS: 16.8%   Inodes: 2.3 Mi   Used Inodes: 6.8%
    Nodemap: 1010101010
    10 tasks, 33340 files

    SUMMARY rate: (of 1 iterations)
       Operation                     Max            Min           Mean        Std Dev
       ---------                     ---            ---           ----        -------
       File creation                2568.636       2568.636       2568.636          0.000
       File stat                       0.000          0.000          0.000          0.000
       File read                       0.000          0.000          0.000          0.000
       File removal                    0.000          0.000          0.000          0.000
       Tree creation                 496.015        496.015        496.015          0.000
       Tree removal                    0.000          0.000          0.000          0.000
    -- finished at 03/19/2026 17:55:20 --