Detailed Pool Operations: Interactive Rebuild Controls - DAOS v2.8
Author: DAOS Project
Description: Distributed Asynchronous Object Storage

Skip to content

# Detailed Pool Operations: Interactive Rebuild Controls

## Introduction

Although the self-healing policy can control whether a rebuild will occur, occasionally an administrator may need to stop a rebuild already started and restart it later with a `rebuild start` command (or, perform an alternate action/rebuild such as direct reintegration). For this purpose, DAOS provides the following interactive rebuild control command-line interfaces:

- `dmg pool <label> rebuild stop [--force]`
- `dmg pool <label> rebuild start`
- `dmg system rebuild stop [--force]`
- `dmg system rebuild start`

The system-level commands (e.g., `dmg system rebuild stop`) apply to all pools in the DAOS system (i.e., they have the same effect as if multiple `dmg pool rebuild stop` commands are issued, one per pool).

Upon stopping a pool's rebuild, its rebuild state as reported by `dmg pool query` will be an idle state, and an error status=`-2027` (`-DER_OP_CANCELED` DAOS error code).

The effect of a `rebuild stop` command is "one shot", meaning only a pool's currently-running rebuild is stopped and there is no persistent effect on future operations. Subsequent self-healing automatic recovery, pool service leader switch, or administrator command (e.g., system/pool exclude, reintegrate, drain, pool extend) can trigger new rebuild(s). When a rebuild is stopped, whatever progress it had made in reconstructing the data in the pool is not retained — a subsequent `rebuild start` command (or rebuild started any other way) will start from the beginning (i.e., stop/start is not a pause/resume interface).

Some circumstances where rebuild stop/start controls may be helpful are:

- Due to temporary network errors, a pool's rebuild fails and gets retried. If the instability persists for some time, rather than have DAOS automatically retry the rebuild (and fail) repeatedly, the administrator may choose to stop such rebuilds. Later, the administrator can start the rebuilds, or perform alternate recovery actions (e.g., directly reintegrate lost engine(s) excluded but whose rebuilds did not finish).
- A system with many tens of pools are all rebuilding simultaneously (requiring substantial CPU and network resources in the DAOS system). And the administrator deciding in such cases to stagger an overall recovery by manually commanding pools to rebuild in smaller batches.
- A long-running rebuild might be stuck (e.g., due to any unforeseen DAOS bug).

## Rebuild Phases

An important detail of the DAOS rebuild design/implementation is that it proceeds in two phases: the rebuild itself, and a following "reclaim" phase to clean up space that rebuild used while it was operating.

For a successful rebuild, the sequence is:

1. Run `op:Rebuild`
2. After successful rebuild, update the pool map so that target states reflect the completion of the rebuild.
3. Run `op:Reclaim` to clean up

For a failed rebuild, the sequence is:

1. Run `op:Rebuild`
2. Upon failure, determine if the `op:Rebuild` should be retried. Two cases are: 2a. case 1: for network and certain "retryable" errors, rebuild shall be retried. 2b. case 2: for other errors, revert the pool map target states such that (for most rebuild variants - drain, reintegrate, and extend) the rebuild will not be retried. Note: rebuilds initiated due to target exclusion do get retried, though a discussion of this behavior is out of scope here.
3. Run `op:Fail_reclaim` to clean up
   - If `Fail_reclaim` itself failed, retry `Fail_reclaim`
   - If `Fail_reclaim` succeeded, and the original error was retryable (case 1), run `op:Rebuild` again.

A `rebuild stop` command (without `--force`) will immediately stop a rebuild in the `op:Rebuild` phase. During `op:Reclaim`, the command is rejected with a `-DER_BUSY` error (the rebuild has already succeeded). During `op:Fail_reclaim`, the command succeeds (no error returned to the administrator), but the `Fail_reclaim` cleanup is allowed to finish — once it completes, the normal retry of the original `op:Rebuild` is suppressed, and the rebuild transitions to the stopped state. The `--force` option to `rebuild stop` is intended for the pathological case where `Fail_reclaim` itself is repeatedly failing and retrying in a loop; `--force` will terminate the `Fail_reclaim` immediately (provided it has already failed at least once).

The `dmg pool query` command can be used to monitor rebuild state transitions after issuing a `rebuild stop`. See the section Rebuild Stop Command Responses for details on the responses returned in different circumstances.

## Example Usage: Stop a Single Pool Rebuild, then Direct Reintegration

A system has detected the loss of an engine (rank 3) that has 8 storage targets. A corresponding rebuild has launched on pool `p1` after the engine's exclusion from the pool map. The administrator decides to stop this rebuild (perhaps because it is possible to quickly remedy the rank 3 engine issue, restart it, and reintegrate it into the system and pool). This will result in a single rebuild for the reintegration, rather than two rebuilds (for the initial exclusion, and later for the reintegration).

1. Observe the pool `p1` is rebuilding after the fault is detected.

The pool state reflects 8 disabled targets (`disabled=8`) corresponding to the exclusion of engine rank 3. Rebuild is underway (`busy` rebuild state).

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild busy, 1 objs, 524288000 recs
    - Data redundancy: degraded

2. Stop the rebuild and monitor with multiple `pool query` commands until the stop is confirmed (`stopped, state=idle, status=-2027`). Notice some rebuild state changes while waiting.

Using a command at single-pool scope, no output is expected if the request is successfully sent to the storage system.

    $ dmg pool rebuild stop p1

Run `dmg pool query` a few times, though not too frequently. The `--health-only` option is useful to keep the overall query lighter-weight (by interacting with the pool service leader engine, and that engine not interacting with all pool engines while it is handling the query).

Rebuild state output `stopping` along with `state=busy` and `status=-2027` indicates the stop command is being processed, and rebuild has not been entirely stopped yet.

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild stopping (state=busy, status=-2027)

Rebuild state (temporarily) transitioned to `busy` without an error status indicates the `op:Rebuild` is no longer running, and a reclaim phase is now running (in this case, `op:Fail_reclaim`, since a stopped rebuild is processed in the same way as a failed rebuild).

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild busy, 0 objs, 0 recs

Rebuild state `idle` with `status=-2027` indicates the rebuild is now stopped.

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild stopped (state=idle, status=-2027)
    - Data redundancy: degraded

3. Restart engine rank 3, wait for it to join the system, directly reintegrate it back into pool `p1`, and wait for the rebuild to finish successfully.

    $ dmg system start --ranks=3
    # Repeat dmg system query commands until engine rank 3 shown in the joined state
    $ dmg system query
    Rank      State
    ----      -----
    [0-2,4-7] Joined
    3         Stopped

    $ dmg system query
    Rank  State
    ----  -----
    [0-7] Joined

    # Now, reintegrate engine rank 3 into pool p1
    $ dmg pool reintegrate --ranks=3 p1

Run `dmg pool query` a few times, possibly with the `--health-only` option, and not too frequently.

First, it may be seen that the pool map has been updated for the reintegrating engine rank 3 (pool map `version=85` instead of 77, and targets `disabled=0` instead of 8). And it could be that the pool rebuild has not yet started, with state reflecting the same state following the previous (stopped) rebuild (`state=idle, status=-2027`).

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
    Pool health info:
    - Rebuild stopped (state=idle, status=-2027)
    - Data redundancy: normal

Rebuild is now `busy` (performing the reintegration) according to rebuild state.

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
    Pool health info:
    - Rebuild busy, 1 objs, 455081984 recs
    - Data redundancy: normal

Rebuild is still `busy`, though object / record counts have been reset. Also, the pool map version has increased to 93 (previously 85). This indicates the `op:Rebuild` has completed, and the rebuild is cleaning up in `op:Reclaim` phase. The pool map was updated to promote the engine rank 3 targets from `UP` (during reintegration) to `UP_IN` (reintegration complete, ready for client I/O).

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
    Pool health info:
    - Rebuild busy, 0 objs, 0 recs
    - Data redundancy: normal

Rebuild is still `busy` in `op:Reclaim` phase.

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
    Pool health info:
    - Rebuild busy, 1 objs, 0 recs
    - Data redundancy: normal

Rebuild (including reclaim) has finished since state is `done`. Engine rank 3 has been reintegrated into the pool.

    # all done
    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
    Pool health info:
    - Rebuild done, 1 objs, 0 recs
    - Data redundancy: normal

## Example Usage: Stop Multiple Pools Rebuilding

Consider a DAOS system with multiple pools, labeled `p1`, `p2`, etc. A `daos_engine` suffers a fault and the system detection of the engine loss results in pool map updates and all pools rebuilding. The administrator would prefer to stop the exclusion rebuilds, repair/restart the engine, and directly reintegrate the engine into the system and all pools.

1. Observe the engine is excluded from the system, and pools begin rebuilding after the fault is detected.

    $ dmg system query
    Rank      State
    ----      -----
    [0-2,4-7] Joined
    3         Stopped

The stopped engine will eventually also be reported as `Excluded` from the system

    $ dmg system query
    Rank      State
    ----      -----
    [0-2,4-7] Joined
    3         Excluded

Attempting to stop pool rebuilds too soon (before they have actually started) will produce an error. The `dmg system rebuild stop` command reports how many pools had the request successfully issued (in this case, 0 successful pools).

    $ dmg system rebuild stop
    System-rebuild stop request succeeded on 0 pools
    ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist

The following output shows that pool `p2` has excluded targets and rebuild has started: - `State` column shows `TargetsExcluded` - `Disabled` column reports 8 out of 64 targets are disabled, corresponding here to the lost engine rank 3 targets. - `Rebuild State` column shows `busy`

Also, pool `p1` has not excluded targets yet, and has not started rebuild: - `State` column is `Ready` - `Disabled` column reflects 0 targets disabled. - `Rebuild State` column is `done`, reflecting state from a previously-completed rebuild.

    $ dmg system list-pools -v
    Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
    ----- ----                                 -----           -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
    p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready           [0-1,4-6] 105 GB   37 GB    28%           0 B       0 B       0%             0/64     None           done
    p2    dcfe63e0-e10f-464d-b18c-0915e52e048c TargetsExcluded [0-2,6-7] 49 GB    4.6 GB   62%           0 B       0 B       0%             8/64     None           busy

Shortly after this moment, both pools p1 and p2 should show `Rebuild State` = `busy`.

2. Stop all pool rebuilds with one command, and confirm they have stopped.

The successful `system rebuild stop` command will confirm that it succeeded for all (in this case 2) pools.

    $ dmg system rebuild stop
    System-rebuild stop request succeeded on 2 pools

When the rebuilds have been stopped, the `Rebuild State` is expected to be `idle` (seen in both `dmg system list-pools` and `dmg pool query` command output). And the rebuild status will show error status `-2027` (`-DER_OP_CANCELED` DAOS error code)

    $ dmg system list-pools -v
    Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
    ----- ----                                 -----           -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
    p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 TargetsExcluded [0-1,4-6] 105 GB   37 GB    28%           0 B       0 B       0%             8/64     None           idle
    p2    dcfe63e0-e10f-464d-b18c-0915e52e048c TargetsExcluded [0-2,6-7] 49 GB    4.6 GB   62%           0 B       0 B       0%             8/64     None           idle

    $ dmg pool query --health-only p1
    Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=102, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild stopped (state=idle, status=-2027)
    - Data redundancy: degraded

    $ dmg pool query --health-only p2
    Pool dcfe63e0-e10f-464d-b18c-0915e52e048c, ntarget=64, disabled=8, leader=7, version=46, state=TargetsExcluded
    Pool health info:
    - Disabled ranks: 3
    - Rebuild stopped (state=idle, status=-2027)
    - Data redundancy: degraded

3. Restart the affected engine, reintegrate engine and monitor rebuilds until finished.

    $ dmg system start --ranks=3
    # Repeat dmg system query commands until engine rank 3 shown in the joined state
    $ dmg system query
    Rank      State
    ----      -----
    [0-2,4-7] Joined
    3         Stopped

    $ dmg system query
    Rank  State
    ----  -----
    [0-7] Joined

    $ dmg system reintegrate --ranks=3

Use `dmg system list-pools` or `dmg pool query` = (with short delays between commands) to monitor the reintegration rebuilds on the pools.

Both pools have had their target states updated due to the reintegration command, and corresponding reintegration rebuilds started, based on the `dmg system list-pools` command output: - `Disabled` column reports 0 disabled targets (was 8 targets disabled previously) - `Rebuild State` column indicates `busy` for both pools.

    $ dmg system list-pools -v
    Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
    ----- ----                                 ----- -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
    p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready [0-1,4-6] 120 GB   42 GB    28%           0 B       0 B       0%             0/64     None           busy
    p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%           0 B       0 B       0%             0/64     None           busy

Wait for rebuilds to complete (i.e., for `Rebuild State` to report `done` for both pools):

    $ dmg system list-pools -v
    Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
    ----- ----                                 ----- -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
    p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready [0-1,4-6] 120 GB   42 GB    31%           0 B       0 B       0%             0/64     None           done
    p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%           0 B       0 B       0%             0/64     None           done

## Example Usage: Stop and Restart a Rebuild Failing in Fail_reclaim

For some cases in which rebuild fails, `Fail_reclaim` runs successfully, and the process repeats, an ordinary `rebuild stop` command (without `--force` option) is sufficient. However, if the `Fail_reclaim` itself fails, and retries itself repeatedly, eventually the administrator may need to terminate this bad loop with the `--force` option.

    # Administrator has been monitoring pool rebuild state with dmg pool query and noticed occasional
    # rebuild failures, and retries - with busy and failed states seen in a cycle.

    # Attempt to stop the rebuild
    $ dmg pool rebuild stop <pool_label>

    # Continue to monitor with dmg pool query, though the pool rebuild continues to run
    # and the `stopped` state is never seen. Observe more rebuild busy / failed cycles.

    # Forcibly stop the rebuild (since it may be looping in `Fail_reclaim`)
    $ dmg pool rebuild stop --force <pool_label>

## Rebuild Stop Command Responses

The `rebuild stop` command behavior depends on the current rebuild phase. The following subsections describe the responses in each case.

### No Rebuild Currently Running

When no rebuild is currently running, the command will report a "nonexist" error:

    # system-level command applying to two pools, both of which have not started rebuilding yet
    $ dmg system rebuild stop
    System-rebuild stop request succeeded on 0 pools
    ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist

    # single pool scope command applied to one pool that is not currently rebuilding
    $ dmg pool rebuild stop p2
    ERROR: dmg: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist

### Rebuild Finished Successfully, Reclaim in Progress

When the rebuild stage has successfully finished and is in its `op:Reclaim` cleanup stage, `dmg` will report a busy error:

    $ dmg system rebuild stop
    System-rebuild stop request succeeded on 0 pools
    ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_BUSY(-1012): Device or resource busy, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_BUSY(-1012): Device or resource busy

### Rebuild Failed, Fail_reclaim in Progress

When the rebuild has failed and is in its `op:Fail_reclaim` cleanup stage, `rebuild stop` (without `--force`) succeeds with no error. The `Fail_reclaim` is allowed to run to completion, and once it finishes, the normal retry of the original `op:Rebuild` is suppressed. The rebuild then transitions to the stopped state (`state=idle, status=-2027`).

    $ dmg pool rebuild stop <pool_label>
    # No error output — command accepted, stop deferred until Fail_reclaim finishes

    # Monitor with pool query until the rebuild reaches the stopped state
    $ dmg pool query --health-only <pool_label>
    # ... Rebuild busy (Fail_reclaim still running) ...

    $ dmg pool query --health-only <pool_label>
    # ... Rebuild stopped (state=idle, status=-2027) ...

If the `Fail_reclaim` itself is repeatedly failing and retrying (never reaching a successful completion), the `--force` option can be used to terminate it immediately:

    $ dmg pool rebuild stop --force <pool_label>

Note: `--force` only takes effect if the `Fail_reclaim` has already failed at least once. On the first attempt (before any failure), `--force` behaves the same as a non-force stop (deferring until `Fail_reclaim` finishes).

### Another Rebuild Queued for the Same Pool

If another rebuild task is already queued for the same pool (e.g., a reintegration was scheduled after the current rebuild), the `rebuild stop` command will return a no-permission error:

    $ dmg pool rebuild stop <pool_label>
    ERROR: dmg: pool-rebuild stop failed: DER_NO_PERM(-1001): No permission