Data Mover - DAOS v2.6
Author: DAOS Project
Description: Distributed Asynchronous Object Storage

Skip to content

# Data Mover

The Dataset Mover is a collection of multiple tools that allow users to copy and serialize data across DAOS and POSIX file systems. There is support for data movement across POSIX, DAOS, and HDF5 containers. There is also support for serializing and deserializing a DAOS container, where a representation of the container is stored on a POSIX filesystem in an HDF5 file(s) and can be restored to a new DAOS container.

## Overview of Tools

These tools are implemented within the `daos` command.

- `daos filesystem copy` - Copy between POSIX containers and POSIX filesystems using the `libdfs` library.
- `daos container clone` - Copy any container to a new container using the Object API (`libdaos` library).

These tools have MPI support and are implemented in the external MpiFileutils repository.

- `dcp` - Copy between POSIX containers and POSIX filesystems using the `libdfs` library, or copy between any two DAOS containers using the Object API (`libdaos` library).
- `dsync` - Similar to `dcp`, but attempts to only copy the difference between the source and destination.
- `daos-serialize` - Serialize any DAOS container to an HDF5 file(s).
- `daos-deserialize` - Deserialize any DAOS container that was serialized with `daos-serialize`.

More documentation and uses cases for these tools can be found here.

Build instructions for these tools can be found here.

## DAOS Tools Usage

### `daos filesystem copy`

There are two mandatory command-line options; these are:

Command-line Option | Description
--- | ---
--src=daos://<pool>/<cont> | <path> | the source path
--dst=daos://<pool>/<cont> | <path> | the destination path

#### Examples

Copy a POSIX container to a POSIX filesystem:

    $ daos filesystem copy --src daos://<pool>/<cont> --dst <posix_path>

Copy from a POSIX filesystem to a sub-directory in a POSIX container:

    $ daos filesystem copy --src <posix_path> --dst daos://<pool>/<cont>/<sub_dir>

Copy from a POSIX container by specifying a UNS path:

    $ daos filesystem copy --src <uns_path> --dst <posix_path>

### `daos container clone`

There are two mandatory command-line options; these are:

Command-line Option | Description
--- | ---
--src=daos://<pool>/<cont> | <path> | the source container
--dst=daos://<pool>[/<cont>] | <path> | the destination container

The destination container must not already exist.

#### Examples

Clone a container to a new container with a given label:

    $ daos container clone --src /<pool>/<cont> --dst /<pool>/<new_cont>

Clone a container to a new container with an auto-generated UUID:

    $ daos container clone --src /<pool>/<cont> --dst /<pool>