DAOS Documentation¶

Documentation on DAOS is available in the following places:

The https://daos.io/ website contains general information about the DAOS Foundation, a subproject of the Linux Foundation. This website is maintained by the DAOS Foundation and uses Wordpress for content management.
The DAOS Community Wiki https://wiki.daos.io/ is a redirection to https://daosio.atlassian.net/wiki/spaces/DC, an Atlassian Wiki space.
Documentation for end users and administrators of DAOS is available at https://docs.daos.io/. This website provides versioned documentation for each DAOS release. Its web pages are created from the contents in the daos-stack/daos/docs/ trees of the DAOS release branches in Github. This website also cross-reference the "DAOS Internals" documentation (in the "Developer Zone" section).
The dmg and daos commands have man-pages that are created at DAOS build time through hidden manpage options of these commands. The contents of these man-pages is embedded in the Go code for the commands. The man-pages currently only output concatenated lists of all subcommands and their options. See src/control/cmd/dmg/README.md and src/control/cmd/daos/README.md.
Most DAOS commands and subcommands have a --help option which prints one-line descriptions of the (sub)command and its options. The contents of these help texts is embedded in the Go code (or C code) for the commands.
Developer-facing "DAOS Internals" documentation of the DAOS software is maintained as Markdown files within the daos-stack/daos/src/ source tree of the DAOS Github repository. Make sure to access the release branch (master, release/2.8, etc.) for the version of DAOS you are working with.
DAOS API documentation is generated from the DAOS source files using doxygen. Running doxygen during the website deployment process for https://docs.daos.io/ creates HTML pages in docs/doxygen/html/, which are then made available in the "Developer Zone" section of https://docs.daos.io/.

The following sections describe how the multi-versioned documentation on https://docs.daos.io/ is generated from the documentation Markdown files in the Github daos-stack/daos/docs/ tree (using the mkdocs and mike tools, and the doxygen tool for the API documentation).

Configuration files in Github¶

Mkdocs configuration¶

The MkDocs tool is a static site generator that creates web pages from Markdown files. The mkdocs command is driven by a single configuration file, mkdocs.yml in the toplevel directory of the DAOS project on github. Among other project information, this file defines the menu structure for the navigation bar of the website that is being created. Each DAOS release branch uses its own version of mkdocs.yml, as some information in that file is version specific or uses URLs that are pointing to that specific DAOS version.

Note

Absolute paths or hardcoded version numbers in the mkdocs.yml file should not be changed to relative URLs/paths, as that will break some of the processing by downstream tools. These specifics need to be updated once when a new release branch is created, and again when an existing release branch that was under development reaches General Availability (GA). Examples with detailed steps can be found below.

The mkdocs tool creates a sites subdirectory in the toplevel directory of the DAOS project on github, in which the HTML documents for the static website are created. To prevent this temporary subdirectory to be accidentally committed into github, the .gitignore file contains a site/ line.

Mike configuration¶

The mike plugin is used with Materials for MkDocs to generate versioned documentation for each of the DAOS release branches. Its configuration information is stored in the versions.json file that resides in a new gh-pages branch in a checked out version of the DAOS repository. Note that this file is not committed into the main DAOS-Stack github repo. It only exists in the local gh-pages branch on the system on which the webpages are generated.

The versions.json file should not be edited directly. The mike command should be used to define which versions of the documentation are being created, which aliases should be set up, and which version should be the default version to display. See below for details.

Doxygen configuration¶

The configuration file for the doxygen tool is Doxyfile, located in the toplevel directory of the DAOS project on github.

Installing the mkdocs Software¶

On the machine where the DAOS documentation is to be built, GIT needs to be set up so the daos-stack/daos project can be checked out. In addition, the MkDocs package and plugins need to be installed. Running pip install mike should install the prerequisite packages.

Depending on the operation system and Python environment, prerequisite software and the mkdocs plugins may have to be explicitly installed. For example, on a Windows laptop running Cygwin:

pip3 install alabaster sphinxcontrib-applehelp sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-qthelp

pip3 install --user mkdocs
pip3 install --user mkdocs-material
pip3 install --user mike
export PATH="$HOME/.local/bin:$PATH"
mike --version

Checking out DAOS from github¶

To create the DAOS documentation webpages, it is recommended to work on a clone of the DAOS project on github that is separate from any checked out version that may be used for code development work.

cd ~/dev
git clone git@github.com:daos-stack/daos.git daos-website
cd daos-website
git branch           # should be on master branch
grep site .gitignore # should show "site/"
grep doxy .gitignore # should show "docs/doxygen/"
mike list            # should show nothing on a new setup

Creating the static website contents¶

To create the static website contents, the following process is used. For each DAOS release branch that should get included on the website:

the git release branch is checked out
running doxygen creates a temporary version of the doxygen documentation for this release in the docs/doxygen/html subdirectory.

Note

To prevent these files from being accidentally committed into github, the .gitignore file contains a docs/doxygen/ line.

mike deploy is called with a title, a name for the release, and optional alias name(s) for this release. These will be used in the multi-versioned website's navigation system.
mike first uses mkdocs to create a temporary copy of the static webpages for this release in the site/ subdirectory.
mike will then copy that site/ contents into the temporary gh-pages branch, with a directory name that matches the release name specified on the mike deploy invocation.
Any aliases for that release will be created as symlinks in the gh-pages branch, pointing to the directory with the main release name.

This process is repeated for each release. After all versions have been processed this way, invoking mike default will set the version that is displayed on the website by default. That command creates a toplevel index.html in the gh-pages branch which contains a redirect to the version that was selected as the default.

A complete example for a website where DAOS 2.6 is the current "latest" version, and the master branch is the "2.7" development branch for what will eventually become DAOS 2.8:

mike delete --all

git checkout master
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.7 - master" master v2.7 v2.8  2>&1 | tee deploy-log.master.txt
rm -rf docs/doxygen/html

git checkout release/2.6
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.6 - latest" v2.6 latest  2>&1 | tee deploy-log.v2.6.txt
rm -rf docs/doxygen/html

git checkout release/2.4
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.4 - deprecated" v2.4  2>&1 | tee deploy-log.v2.4.txt
rm -rf docs/doxygen/html

mike set-default latest
mike list

After release/2.8 has been branched, but before it becomes generally available (GA), the website version structure should be updated like this:

mike delete master # will also delete the v2.7 v2.8 aliases

git checkout master
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.9 - master" master v2.9 v3.0 2>&1 | tee deploy-log.master.txt
rm -rf docs/doxygen/html

git checkout release/2.8
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.8 - rc" v2.8 rc  2>&1 | tee deploy-log.v2.8.txt
rm -rf docs/doxygen/html

mike list

And finally, when DAOS version 2.8 is released this will be set as the new latest release:

mike delete rc
mike delete latest

git checkout release/2.8
rm -rf docs/doxygen/html 2>/dev/null
doxygen
mike deploy -t "v2.8 - latest" v2.8 latest 2>&1 | tee deploy-log.v2.8.txt
rm -rf docs/doxygen/html
mike set-default latest

mike list

It is useful to save the logs from the mike deploy invocation. These logs will contain warnings about pages that exist but are not linked anywhere in the navigation structure, and other issues.

The new website can be tested locally by running mike serve on the machine where the website is being created. This will start mike's built-in webserver. The new contents can then be inspected by pointing a browser (running on the same machine) to http://localhost:8000/.

Staging the new website contents to docs.daos.io¶

Because the version aliases are symlinks, and because the website contents consists of thousands of small files, copying it to the actual webserver is best done by creating and transferring a tarfile, which can then be un-tarred on the webserver and will keep the symlinks intact.

As a best practice, a new website version is first deployed in a staging area so it can be validated in the same webserver environment in which the live website is running:

export WEBSERVER="docs.daos.io"
export TARFILE="docs-daos-io.tgz"

git checkout gh-pages

tar czvf $TARFILE versions.json index.html v?.? latest master

ssh $WEBSERVER "rm -r docs.daos.io/staging && mkdir docs.daos.io/staging"
scp $TARFILE $WEBSERVER:docs.daos.io/staging
ssh $WEBSERVER "cd docs.daos.io/staging && tar xz $TARFILE && rm $TARFILE"

rm $TARFILE

git checkout master

At this point, the new website can be inspected by using the https://docs.daos.io/staging/ URL.

Go-live of the new website contents on docs.daos.io¶

To make the website in the staging area the active website, there's a small go-live.sh script on the webserver that will archive the current live version and replace it with the one in the staging area:

$ cat go-live.sh
D=`date "+%Y-%m-%d"`
mv docs.daos.io old-docs.daos.io.$D
mv old-docs.daos.io.$D/staging docs.daos.io
mkdir docs.daos.io/staging

Miscellaneous tips and tricks¶

Beware of relative URLs in links¶

The mkdocs tool creates a directory (with an index.html in it) for each Markdown file that it processes. So a documentation file like docs/user/container.md in the github master branch will become this URL on the website: https://docs.daos.io/master/user/container/index.html.

This additional directory level may break relative URLs in hyperlinks in the Markdown files. On the other hand, adding that additional directory level to those relative URLs will break the links when looking at the (un-processed) Markdown files directly in github.

There is no good fix. In some places absolute paths to website URLs are used to make sure those hyperlinks work in both the raw Markdown and the rendered HTML pages on the website. The drawback of this approach is that those URLs will contain hardcoded release numbers, but those can be quickly updated by a search and replace when a new release branch is created.

Resolving Spell Checker Issues¶

The DAOS CI performs spell checking on PRs. If the spell checker reports an error for a legitimate word (or acronym), the error can be resolved by adding the term to the wordlist that is maintained in the utils/cq/words.dict file.

Always call mike on a release branch, not gh-pages¶

You must be in an actual DAOS release branch, not in the gh-pages branch, to run mike commands. Otherwise the tool will report errors like this:

$ git branch
* gh-pages
  master

$ mike list
error: [Errno 2] No such file or directory: 'mkdocs.yml'; pass --config-file or set --remote/--branch explicitly