Introduction

What is OpenSVC

OpenSVC is an open-source software product and the name of the company developing and supporting it. In production since 2009, OpenSVC consists of:

• Agent: A supervisor, clusterware, container orchestrator, and configuration manager (Apache 2.0 licensed since v3).
• Collector: Optionally, aggregates agent data and manages configuration, infrastructure, and resources (Apache 2.0 licensed).

Why OpenSVC

• Service Mobility: Ensures high availability and scalability.
• Configuration Management: Handles thousands of servers and clusters with minimal dependencies.
• Inventoring: Tracks assets, storage, networks, and services.
• Orchestration: RESTful API, self-service portal, provisioning, fine-grained delegation.
• Auditing: Monitors performance, alerts, and audits infrastructure and services.
• Productivity Boost: Reduces infrastructure maintenance and allows for task delegation.

Requirements

OpenSVC runs on Linux servers.

Contribute

OpenSVC projects, including this documentation, are hosted on Github.

Install Git

sudo apt-get install git
git config --global user.name "First Last Name"
git config --global user.email "first.lastname@domain.com"

Clone the agent and book projects

cd /tmp
git clone https://github.com/opensvc/book.opensvc.com.git
git clone https://github.com/opensvc/opensvc.git

Build the agent

(cd opensvc && make om)

Install mdbook

From https://github.com/rust-lang/mdBook/releases

Make autogenerated documentation

(cd book.opensvc.com && OM=../opensvc/bin/om make)

Contribute Documentations

(cd book.opensvc.com && && mdbook serve --open)

Modify the documentation source files (.md located in the src directory and referenced in src/SUMMARY.md)

Command Line Output Coloring

When producing command line output, it is expected to follow the steps below to preserve colors, so as to provide a better experience for futures readers:

Install ansi2html

From https://github.com/ralphbean/ansi2html

generate raw html code from cli:

$ om node print devs --color=yes | aha -n

Edit the .md document and copy/paste the previous output enclosed between <pre> and </pre>:

<pre>
        <span style="font-weight: bold">centos71.opensvc.com                        </span>  <span style="font-weight: bold">Type  </span>  <span style="font-weight: bold">Size</span>  <span style="font-weight: bold">Pct of Parent</span>  
        `- <span style="color: #aa5500">vda                                      </span>  linear  15g   -              
           |- <span style="color: #aa5500">vda1                                  </span>  linear  500m  3%             
           `- <span style="color: #aa5500">vda2                                  </span>  linear  14g   96%            
              |- <span style="color: #aa5500">centos_centos71-swap               </span>  linear  1g    10%            
              `- <span style="color: #aa5500">centos_centos71-root               </span>  linear  13g   89%            
                 |- <span style="color: #aa5500">loop2                           </span>  linear  50m   0%             
                 |  |- <span style="color: #aa5500">testsvc1-lv1                 </span>  linear  20m   40%            
                 |  `- <span style="color: #aa5500">testsvc1-lv2                 </span>  linear  20m   40%            
                 |- <span style="color: #aa5500">loop1                           </span>  linear  100m  0%             
                 |  |- <span style="color: #aa5500">testsvc3-lv2                 </span>  linear  20m   20%            
                 |  |- <span style="color: #aa5500">testsvc3-lv1-real            </span>  linear  52m   52%            
                 |  |  |- <span style="color: #aa5500">testsvc3-lv1              </span>  linear  52m   100%           
                 |  |  `- <span style="color: #aa5500">testsvc3-osvc_sync_lv1    </span>  linear  52m   100%           
                 |  `- <span style="color: #aa5500">testsvc3-osvc_sync_lv1-cow   </span>  linear  8m    8%             
                 |     `- <span style="color: #aa5500">testsvc3-osvc_sync_lv1    </span>  linear  52m   650%           
                 `- <span style="color: #aa5500">loop0                           </span>  linear  100m  0%             
                    |- <span style="color: #aa5500">testsvc2-lv1                 </span>  linear  52m   52%            
                    `- <span style="color: #aa5500">testsvc2-lv2                 </span>  linear  20m   20%            
    </pre>

The result looks like:

	centos71.opensvc.com                          Type    Size  Pct of Parent  
	`- vda                                        linear  15g   -              
	   |- vda1                                    linear  500m  3%             
	   `- vda2                                    linear  14g   96%            
	      |- centos_centos71-swap                 linear  1g    10%            
	      `- centos_centos71-root                 linear  13g   89%            
	         |- loop2                             linear  50m   0%             
	         |  |- testsvc1-lv1                   linear  20m   40%            
	         |  `- testsvc1-lv2                   linear  20m   40%            
	         |- loop1                             linear  100m  0%             
	         |  |- testsvc3-lv2                   linear  20m   20%            
	         |  |- testsvc3-lv1-real              linear  52m   52%            
	         |  |  |- testsvc3-lv1                linear  52m   100%           
	         |  |  `- testsvc3-osvc_sync_lv1      linear  52m   100%           
	         |  `- testsvc3-osvc_sync_lv1-cow     linear  8m    8%             
	         |     `- testsvc3-osvc_sync_lv1      linear  52m   650%           
	         `- loop0                             linear  100m  0%             
	            |- testsvc2-lv1                   linear  52m   52%            
	            `- testsvc2-lv2                   linear  20m   20%            

Decorating words

Awesome font icons can be inlined in the documentation. To make it easier and to enforce a common set of icon and color, include fragments are available in the src/inc/ directory.

Install

Branch

We feed packages in 3 different branches:

• dev is unstable.

  Every candidate Pull Request causes a new package to be spawned here for OpenSVC QA purpose.

• uat is for testing.

  OpenSVC will push there pre-release packages and packages that contain a candidate fixes for known issues that client are encouraged to validate.

• prod is stable.

  It is the default and recommended branch.

Execute one of the following variable settings in a shell, and the code block corresponding to the operating system.

BRANCH=dev
BRANCH=uat
BRANCH=prod

Debian, Ubuntu

# Set ID (ubuntu, debian) and VERSION_CODENAME (bookworm, bullseye, noble)
source /etc/os-release

BRANCH=${BRANCH:-prod}
DISTRIB=${VERSION_CODENAME:-noble}

# Import opensvc gpg signing keys
# -------------------------------
curl -s -o- https://packages.opensvc.com/gpg.public.key.asc | \
    sudo gpg --dearmor --output /etc/apt/trusted.gpg.d/opensvc-package-pub.gpg --yes

#
# Add the opensvc repository to apt sources
# -----------------------------------------
cat - <<EOF | sudo tee /etc/apt/sources.list.d/opensvc.list 
deb https://packages.opensvc.com/apt/$ID $BRANCH-opensvc-v3-$DISTRIB main
deb-src https://packages.opensvc.com/apt/$ID $BRANCH-opensvc-v3-$DISTRIB main
EOF

#
# Install the opensvc server
# --------------------------
sudo apt update
sudo apt install opensvc-server

#
# Enable the systemd unit and start the server
# --------------------------------------------
sudo systemctl enable --now opensvc-server

Red Hat Enterprise Linux 7

# Set ID (rhel) and VERSION_ID (7.2, ...)
source /etc/os-release

BRANCH=${BRANCH:-prod}
DISTRIB=${ID}${VERSION_ID%.*}

#
# Add the opensvc repository to apt sources
# -----------------------------------------
cat << EOF >/etc/yum.repos.d/opensvc.repo
[opensvc]
name=OpenSVC Packages RHEL \$releasever - \$basearch
baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/
enabled=1
gpgcheck=0
EOF

#
# Install the opensvc server
# --------------------------
sudo yum makecache
sudo yum install opensvc-server

#
# Enable the systemd unit and start the server
# --------------------------------------------
sudo systemctl enable --now opensvc-server

Red Hat Enterprise Linux 8+

# Set ID (rhel) and VERSION_ID (8.10, ...)
source /etc/os-release

BRANCH=${BRANCH:-prod}
DISTRIB=${ID}${VERSION_ID%.*}

#
# Add the opensvc repository to apt sources
# -----------------------------------------
cat << EOF >/etc/yum.repos.d/opensvc.repo
[opensvc]
name=OpenSVC Packages RHEL \$releasever - \$basearch
baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/
enabled=1
gpgcheck=1
gpgkey=https://packages.opensvc.com/gpg.public.key.asc
EOF

#
# Install the opensvc server
# --------------------------
sudo dnf makecache
sudo dnf install opensvc-server

#
# Enable the systemd unit and start the server
# --------------------------------------------
sudo systemctl enable --now opensvc-server

SuSE Linux Enterprise Server

# Set ID (rhel) and VERSION_ID (8.10, ...)
source /etc/os-release

BRANCH=${BRANCH:-prod}
DISTRIB=${ID}${VERSION_ID%.*}

#
# Add the opensvc repository to apt sources
# -----------------------------------------
cat << EOF >/etc/zypp/repos.d/opensvc.repo
[opensvc]
name=OpenSVC Packages SLES \$releasever - \$basearch
baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/
enabled=1
autorefresh=1
gpgcheck=1
gpgkey=https://packages.opensvc.com/gpg.public.key.asc
EOF

#
# Install the opensvc server
# --------------------------
sudo zypper --gpg-auto-import-keys --non-interactive refresh
sudo zypper install opensvc-server

#
# Enable the systemd unit and start the server
# --------------------------------------------
sudo systemctl enable --now opensvc-server

➡️ See Also

• Installed Files

Agent Configuration

Concepts

The agent uses ini configuration files.

Considering a configuration like:

[env]
bar = 1
bar@n2 = 2

• env is a section
• bar is a option
• env.bar is a keyword.
• env.bar=1 is a keyword operation.
• 1 is the env.bar keyword value.
• @n2 is a node scope for the keyword env.bar

Policies

• If a keyword appears in both node.conf and cluster.conf, the value from node.conf takes precedence.

• Sections only accept recognized keywords, with the exception of the [env] and [labels] sections, which are open.

• More specific scoped values override less specific ones.

  With the above section in a svc1 object configuration:

    # on n1:
    $ om svc1 config eval --kw env.bar
    1
  
    # on n2:
    $ om svc1 config eval --kw env.bar
    2
  

Syntax validation

A syntax check is performed before finalizing any modifications made with either the set or edit commands.

om cluster config edit

om cluster config set --kw hb#test.type=unsupported

A direct modification to the configuration file is not validated and may disrupt the cluster. In such cases, you can perform a post-hoc validation using:

# verify the syntax of cluster.conf
om cluster config validate

# verify the syntax of node.conf
om node config validate

# verify the syntax of a svc configuration
om svc1 config validate

Node Configuration

Set the Node Environment

sudo om cluster config update --set node.env=PRD

The node.env setting is used to enforce the following policies:

• Only production services are allowed to start on a production node.
• Only production nodes are allowed push data to a production node.

Supported node.env values:

The setting is stored in /etc/opensvc/cluster.conf.

Set Node Jobs Schedules

The agent executes periodic tasks.

Display the scheduler configuration and states:

$ sudo om node schedule list
NODE      ACTION           LAST_RUN_AT                NEXT_RUN_AT           SCHEDULE      
eggplant  pushasset        2025-01-20T01:31:17+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  
eggplant  checks           2025-01-20T16:40:20+01:00  0001-01-01T00:00:00Z  @10m          
eggplant  compliance_auto  2025-01-20T05:34:49+01:00  0001-01-01T00:00:00Z  02:00-06:00   
eggplant  pushdisks        2025-01-20T02:42:29+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  
eggplant  pushpkg          2025-01-20T00:16:38+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  
eggplant  pushpatch        2025-01-20T01:50:37+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  
eggplant  sysreport        2025-01-20T00:58:22+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  
eggplant  dequeue_actions  2023-08-03T14:05:50+02:00  0001-01-01T00:00:00Z                
eggplant  pushhcs          2025-01-15T18:00:59+01:00  0001-01-01T00:00:00Z  @1d           
eggplant  pushbrocade      0001-01-01T00:00:00Z       0001-01-01T00:00:00Z                

Schedule configuration:

# Set a job schedule
om node config update --set "brocade.schedule=02:00-04:00@120 sat,sun"

# Disable a job schedule
om node config update --set "brocade.schedule=@0"

➡️ See Also

• Agent Scheduler

Register on a Collector

Set a Collector Url

By default, the agent does not communicate with a collector.

To enable communications with a collector, the node.dbopensvc node configuration parameter must be set. The simplest expression is:

om cluster config update --set node.dbopensvc=collector.opensvc.com

Here the protocol and path are omitted. In this case, the https protocol is selected, and the path set to a value matching the standard collector integration.

Advanced Url Formats

The following expressions are also supported:

om cluster config update --set node.dbopensvc=https://collector.opensvc.com
om cluster config update --set node.dbopensvc=https://collector.opensvc.com/feed/default/call/xmlrpc

The compliance framework uses a separate xmlrpc entrypoint. The node.dbcompliance can be set to override the default, which is deduced from the node.dbopensvc value.

om cluster config update --set node.dbcompliance=https://collector.opensvc.com/init/compliance/call/xmlrpc

Register the Node

The collector requires the nodes to provide an authentication token (shared secret) with each request. The token is forged by the collector and stored on the node in /etc/opensvc/node.conf. The token initialization is handled by the command:

om node register --user my.self@my.com [--app MYAPP]

If --app is not specified the collector automatically chooses one the user is responsible of.

A successful register is followed by a node discovery, so the collector has detailled information about the node and can serve contextualized compliance rulesets up front. The discovery is also scheduled daily, and can be manually replayed with:

om node push asset
om node push pkg
om node push patch
om node checks

To disable collector communications, use:

om cluster config update --unset node.dbopensvc
om cluster config update --unset node.dbcompliance

Or if the settings were added to node.conf

om node config update --unset node.dbopensvc
om node config update --unset node.dbcompliance

Extra System Configurations

Linux LVM2

OpenSVC controls volume group activation and desactivation. Old Linux distributions activate all visible volume groups at boot, some even re-activate them upon de-activation events. These mechanisms can be disabled using the following setup. It also provides another protection against unwanted volume group activation from a secondary cluster node.

This setup tells LVM2 commands to activate only the objects tagged with the hostname. Opensvc makes sure the tags are set on start and unset on stop. Opensvc also purges all tags before adding the one it needs to activate a volume group, so opensvc can satisfy a start request on a service uncleanly shut down.

/etc/lvm/lvm.conf

Add the following root-level configuration node:

tags {
    hosttags = 1
    local {}
}

And add the local tag to all local volume groups. For example:

sudo vgchange --addtag local rootvg

Finally you need to rebuild the initrd/initramfs to prevent shared vg activation at boot.

/etc/lvm/lvm_$HOSTNAME.conf

echo activation { volume_list = [\"@local\", \"@$HOSTNAME\"] } >/etc/lvm/lvm_$HOSTNAME.conf

Cluster Configuration

Upon agent installation, the node is considered part of its own 1-node cluster.

In /etc/opensvc/cluster.conf:

• cluster.secret is initialized to a random value.
• cluster.name is initialized to a random value.

Bootstrap a new cluster

If the node joins an existing cluster, skip this section.

Add Heartbeats

If the cluster seed node has no heartbeat setup, a unicast heartbeat with default settings will be automatically added on first join.

This default heartbeat requires every nodename to be resolved to an ip address reachable on 1215/tcp.

If this requirements are not met, you can setup one or more custom heartbeats on the seed node before joins.

For example, a custom heartbeat configuration would be:

    om cluster config update --set hb#1.type=unicast --set hb#1.port=1216

The new heartbeats are visible in the top section of the monitoring command output:

    om mon

➡️ See Also

• Heartbeats

Add Stonith methods

Stonith is optional. Skip to the next section if not concerned.

On a new cluster, the stonith configuration can be applied on the first node. The joining nodes will fetch this configuration from this joined node.

For example, a dummy stonith configuration would be

om cluster config update --set stonith#node2.command=/bin/true

This configuration will execute /bin/true on the node taking over a service which was previously running on the now stalled node2.

Good, isolated fencing packages are freely available. For one, https://github.com/ClusterLabs/fence-agents

Add Arbitrators

Arbitrators are optional. Skip to the next section if not concerned.

The arbitrator configuration can be applied on any node of the cluster.

om cluster config update --set arbitrator#1.name=relay1 \
                      --set arbitrator#1.secret=10231023102310231023102310231023

This configuration will ask for the agent on node relay1 for its vote in a quorum race, if needed to get a majority.

The arbitrator#1.secret value comes from the cluster.secret value on the arbitrator relay1.

➡️ See Also

• Quorum

Join a Cluster

The joining node can choose to join any of the cluster node already joined.

On the joined node node1, generate a join token:

$ sudo om daemon auth --role join

On the joining node node2:

sudo om cluster join --token <token> --node node1

Leave a Cluster

sudo om cluster leave

➡️ See Also

• Listener
• Monitor
• Scheduler
• Quorum
• Heartbeats
• Orchestration
• DNS

Cluster Storage Pools

Services can use volume resources to:

• Abstract the disks and filesystems layout, which are hosting specificities, from the service deployment. A development cluster can for example define pools on a ceph cluster, while a production cluster can define pools on fc arrays.

• Enable service redeployment while retaining the data.

In this case the translation from volumes to disks and filesystems is delegated to the storage pool drivers.

Pools are defined in the node configuration. Each pool is identified by its name (the section suffix). For example, a pool#tank section defines a pool named tank.

The default pool always exist, even if not defined in the node configuration. If not explicitely changed, the default pool driver is directory.

Volumes

• A volume resource drives a volume object, automatically created upon service provisioning if not already existing.

• The volume is hosted in the same namespace than its users.

• If not explicitely set, the volume object name is <consumer name>-vol-<volume resource index>. For example, a svc1 service with a volume#1 resource will create a svc1-vol-1 volume object.

• A volume object can be referenced by multiple services in the same namespace.

• On provision, a service adds itself as a child of the volume objects mapped via volume resources. Due to this parent/child relation, stopping a volume object is delayed until all its consumers are stopped.

• On unprovision, a service removes itself from the children list of the volume objects mapped via volume resources.

• A consumer service instance stop also stops its node-affine volume object instances if the consumer service is the only child of the volume service.

• A consumer service instance start always tries to start its node-affine volume object instances.

Volume Resources Keywords

• Reference

Access Modes

• roo Read Only Once

• rwo Read Write Once (default)

• rox Read Only from multiple instances

• rwx Read Write from multiple instances

Access Mode to Volume Topology

• ..x access modes imply the volume is configured in a flex topology (active on all service nodes).

• ..o access modes imply the volume is configured in a failover topology (active on only one service node).

Volume Resource Parameter Requirements

• ..x + shared=true format=false requires a shared block storage (SAN array, a rados blockdev gateway, …)

• ..x + shared=true format=true requires either a shared block storage (SAN array, a rados blockdev gateway, …) plus a cluster filesystem (gfs2, ocfs, …), or a distributed cluster filesystem (CephFS, GlusterFS, NAS, …)

Pool Selector

A volume resource requires a size and capabilities from the pool, via its size, access, shared and format keywords.

If <i class="fa fa-wrench"></i>pool is not set explicitely to a pool name, the pool selector will return the available pool matching those criteria with the most free space.

Pool Drivers

directory

Capabilities

rox, rwx, roo, rwo

Layout

A volume object from this type of pool contains:

• a fs.directory resource, with path=<pool head>/<volume fqdn>.

Keywords

• Reference

drbd

Capabilities

rox, rwx, shared, blk, roo, rwo

Layout

A volume object from this type of pool contains:

If a vg is defined in the pool configuration,

• a fs resource, with dev=<drbd devpath>
• a drbd resource, layered over a logical volume of the pool vg
• a lv resource

If a zpool is defined in the pool configuration,

• a fs resource, with dev=<drbd devpath>
• a drbd resource, layered over a zvol of the pool zpool
• a zvol resource

If the pool configuration has neither vg nor zpool set,

• a fs resource, with dev=<drbd devpath>
• a drbd resource, layered over a logical volume
• a lv resource
• a vg resource
• a loop resource, with image file hosted in the pool defined path or in <PATHVAR>/pool/<poolname>/

Keywords

• Reference

freenas

Capabilities

roo, rwo, shared, blk, iscsi

Layout

A volume object from this type of pool contains:

• a disk.disk resource named, with name=<volume fqdn>

If the consumer has format=true (default), the volume object also contains:

• a fs. resource, with mnt=/srv/<volume fqdn>

Keywords

• Reference

loop

Capabilities

rox, rwx, roo, rwo, blk

Layout

A volume object from this type of pool contains:

• a disk.loop resource, with file=<pool head>/<volume fqdn>.img

If the consumer has format=true (default), the volume object also contains:

• a fs. resource, with mnt=/srv/<volume fqdn>

Keywords

• Reference

symmetrix

Capabilities

roo, rwo, shared, blk, fc

Layout

A volume object from this type of pool contains:

• a disk.disk resource named, with name=<volume fqdn>

If the consumer has format=true (default), the volume object also contains:

• a fs.<pool fs_type> resource, with mnt=/srv/<volume fqdn>

Keywords

• Reference

vg

Capabilities

rox, rwx, roo, rwo, blk, snap

Layout

A volume object from this type of pool contains:

• a disk.lv resource, with name=<volume fqdn>

If the consumer has format=true (default), the volume object also contains:

• a fs. resource, with mnt=/srv/<volume fqdn>

Keywords

• Reference

share

Capabilities

rox, rwx, roo, rwo, shared

Layout

A volume object from this type of pool contains:

• a fs.directory resource, with path=<pool head>/<volume fqdn>.

Keywords

• Reference

zpool

Capabilities

rox, rwx, roo, rwo, blk, snap

Layout

A volume object from this type of pool contains:

• a fs.zfs resource, with name=<pool>/<volume fqdn> and mnt=/srv/<volume fqdn>.

Keywords

• Reference

Virtual Pool Driver

A virtual pool allow administrators to create complex layouts based on volumes from other pools.

A typical use-case in a virtual pool allocating volumes mirrored over two other volumes allocated from arrays on two different sites.

A virtual pool volume is created from a template volume object the administrator can design at wish to meet its specific needs.

Capabilities

Capabilities are user defined.

Keywords

• Reference

Pool Commands

Pool list

# om pool ls
default
freenas
mpool

Pool Status

# om pool status
name        type       caps                      head                             vols  size   used   free   
|- default  directory  rox,rwx,roo,rwo           /opt/opensvc/var/pool/directory  0     29.0g  3.57g  24.0g  
|- freenas  freenas    roo,rwo,shared,blk,iscsi  array://freenas/osvcdata         6     195g   9.37g  185g   
`- mpool    virtual    roo,rox,rwo,rwx,shared    templates/mpool                  1     -      -      -      

Examples

loop pool

Pool configuration

om cluster config update \
	--set pool#loop.type=loop \
	--set pool#loop.path=/bigfs \
	--set pool#loop.mkfs_opt="-n ftype=1" \
	--set pool#loop.fs_type=xfs

[pool#loop]
type = loop
path = /bigfs
mkfs_opt = -n ftype=1
fs_type = xfs

The volume resource in the service

[volume#1]
size = 100m
pool = loop

Resulting configuration of the volume object

[disk#1]
size = 104857600
type = loop
file = /bigfs/<fqdn>.img

[fs#1]
type = xfs
dev = {disk#1.exposed_devs[0]}
mnt = /srv/<fqdn>
mkfs_opt = -n ftype=1

zfs pool

Pool configuration

om cluster config update \
	--set pool#tank.type=zpool \
	--set pool#tank.name=tank \
	--set pool#tank.mkfs_opt="-o mountpoint=legacy -o dedup=on -o compression=on"

[pool#tank]
type = zpool
name = tank
mkfs_opt = -o mountpoint=legacy -o dedup=on -o compression=on

The volume resource in the service

[volume#1]
size = 100m
pool = tank

Resulting configuration of the volume object

[fs#1]
type = zfs
dev = tank/<fqdn>
mnt = /srv/<fqdn>
mkfs_opt = -o mountpoint=legacy -o dedup=on -o compression=on

virtual pool, mirrored zpool over 2 SAN disks

Pools configuration

om cluster config update \
	--set pool#freenas1.type=array \
	--set pool#freenas1.array=freenas1 \
	--set pool#freenas1.sparse=true \
	--set pool#freenas1.diskgroup=cluster1 \
	--set pool#freenas2.type=array \
	--set pool#freenas2.array=freenas2 \
	--set pool#freenas2.sparse=true \
	--set pool#freenas2.diskgroup=cluster1 \
	--set pool#mpool.type=virtual \
	--set pool#mpool.template=templates/mpool \
	--set pool#mpool.capabilities="rox rwx roo rwo shared"

[pool#freenas1]
type = array
array = freenas1
diskgroup = cluster1
sparse = true

[pool#freenas2]
type = array
array = freenas2
diskgroup = cluster1
sparse = true

[pool#mpool]
type = virtual
template = templates/mpool
capabilities = rox rwx roo rwo shared

The volume object template referenced by the vpool

[DEFAULT]
kind = vol
nodes = *
disable = true

[disk#1]
name = {namespace}-{svcname}
type = zpool
vdev = mirror {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]}
shared = true

[fs#1]
dev = {disk#1.name}
mnt = /srv/{namespace}/{svcname}
type = zfs
shared = true

[fs#2]
dev = {disk#1.name}/data
mnt = {fs#1.mnt}/data
type = zfs
shared = true

[fs#3]
dev = {disk#1.name}/log
mnt = {fs#1.mnt}/log
type = zfs
shared = true

[volume#1]
format = false
name = {svcname}-1
pool = freenas1
size = {env.size}
shared = true

[volume#2]
format = false
name = {svcname}-2
pool = freenas2
size = {env.size}
shared = true

virtual pool, mirrored lv over 2 SAN disks

Pools configuration

om cluster config update \
	--set pool#freenas1.type=array \
	--set pool#freenas1.array=freenas1 \
	--set pool#freenas1.sparse=true \
	--set pool#freenas1.diskgroup=cluster1 \
	--set pool#freenas2.type=array \
	--set pool#freenas2.array=freenas2 \
	--set pool#freenas2.sparse=true \
	--set pool#freenas2.diskgroup=cluster1 \
	--set pool#mvg.type=virtual \
	--set pool#mvg.template=templates/mvg \
	--set pool#mvg.capabilities="rox rwx roo rwo shared"

[pool#freenas1]
type = array
array = freenas1
diskgroup = cluster1
sparse = true

[pool#freenas2]
type = array
array = freenas2
diskgroup = cluster1
sparse = true

[pool#mvg]
type = virtual
template = templates/mvg
capabilities = rox rwx roo rwo shared

The volume object template referenced by the vpool

[DEFAULT]
kind = vol
nodes = *
disable = true

[volume#1]
shared = true
size = {env.size}
name = {svcname}-1
pool = freenas
format = false

[volume#2]
shared = true
size = {env.size}
name = {svcname}-2
pool = freenas
format = false

[disk#1]
shared = true
type = vg
name = {namespace}-{svcname}
pvs = {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]}

[fs#1]
shared = true
mnt = /srv/{namespace}/{svcname}
dev = /dev/{disk#1.name}/root
type = ext4
size = 10m
create_options = -m 1
vg = {namespace}-{svcname}

[fs#2]
shared = true
mnt = {fs#1.mnt}/data
dev = /dev/{disk#1.name}/data
type = ext4
size = 60%FREE
create_options = -m 1
vg = {namespace}-{svcname}

[fs#3]
shared = true
mnt = {fs#1.mnt}/log
dev = /dev/{disk#1.name}/log
type = ext4
size = 40%FREE
create_options = -m 1
vg = {namespace}-{svcname}

virtual pool, mirrored md over 2 SAN disks

Pools configuration

om cluster config update \
	--set pool#freenas1.type=array \
	--set pool#freenas1.array=freenas1 \
	--set pool#freenas1.sparse=true \
	--set pool#freenas1.diskgroup=cluster1 \
	--set pool#freenas2.type=array \
	--set pool#freenas2.array=freenas2 \
	--set pool#freenas2.sparse=true \
	--set pool#freenas2.diskgroup=cluster1 \
	--set pool#md.type=virtual \
	--set pool#md.template=templates/md \
	--set pool#md.capabilities="rox rwx roo rwo shared"

[pool#freenas1]
type = array
array = freenas1
diskgroup = cluster1
sparse = true

[pool#freenas2]
type = array
array = freenas2
diskgroup = cluster1
sparse = true

[pool#md]
type = virtual
template = templates/md
capabilities = rox rwx roo rwo shared

The volume object template referenced by the vpool

[DEFAULT]
kind = vol
disable = true
nodes = *

[disk#1]
shared = true
devs = {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]}
type = md
level = raid1

[disk#2]
shared = true
pvs = {disk#1.exposed_devs[0]}
type = vg
name = {namespace}-{svcname}

[fs#1]
vg = {namespace}-{svcname}
mnt = /srv/{namespace}/{svcname}
dev = /dev/{disk#1.name}/root
shared = true
type = ext4
size = 10m

[fs#2]
vg = {namespace}-{svcname}
mnt = {fs#1.mnt}/data
dev = /dev/{disk#1.name}/data
shared = true
type = ext4
size = 60%FREE

[fs#3]
vg = {namespace}-{svcname}
mnt = {fs#1.mnt}/log
dev = /dev/{disk#1.name}/log
shared = true
type = ext4
size = 40%FREE

[volume#2]
shared = true
size = {env.size}
name = {svcname}-2
pool = freenas
format = false

[volume#1]
shared = true
size = {env.size}
name = {svcname}-1
pool = freenas
format = false

drbd pool

Pool configuration

om cluster config update \
  --set pool#drbdloop.type=drbd

om cluster config update \
  --set pool#drbdvg.type=drbd \
  --set pool#drbdvg.vg=centos

[pool#drbdloop]
type = drbd

[pool#drbdvg]
type = drbd
vg = centos

Example postgres service using a volume from a pool.

[DEFAULT]
nodes = *
orchestrate = ha

[volume#1]
shared = true
size = 200m
name = {name}

[container#1]
type = oci
image = postgres
volume_mounts = {name}/data:/var/lib/postgresql/data
secrets_environment = POSTGRES_PASSWORD=pg/password
rm = true
shared = true

Cluster API

The Cluster API provides remote access to the OpenSVC cluster management and monitoring functionality. It is accessible through the agent listener on any running cluster node.

Access and Network Resolution

The Cluster API URL (server name) can be configured to resolve to various network endpoints to ensure high availability and redundancy.

Possible Resolutions:

• Single Floating IP

  A single virtual IP (VIP) address, typically managed by a failover service like system/svc/vip. This is the most common and robust approach.

• Multiple Floating IP Addresses

  A set of VIPs for complex network routing or multi-site setups.

• All Cluster Node IPs

  Every physical or virtual IP address of all nodes in the cluster.

• Subset of Cluster Node IPs

  A selection of node IP addresses, often used in large clusters or specific network segments.

Authentication Methods

The agent listener supports several industry-standard authentication methods to secure access to the API.

Basic Authentication

• Mechanism: The client provides the username and password in the request header of every API call and the password is compared to the password key of the system/usr/<username> object.
• Authorization: The RBAC grants are read from the system/usr/<username> grant key.
• User Object: The system/usr/<username> object must exist on the cluster.

X.509 Certificate Authentication

• Mechanism: Authentication is based on a client-side X.509 certificate.
• Username: The username is derived from the Common Name (cn) field of the client certificate.
• Authorization: The RBAC grants are read from the system/usr/<username> grant key.
• User Object: The system/usr/<username> object must exist on the cluster.

JSON Web Token (JWT)

• Mechanism: Authentication is based on a JWT passed as a bearer token.
• Username: The username is derived from the sub token claim.
• Authorization: The RBAC grants are derived from the grant token claim.
• User Object: The system/usr/<username> object does not need to exist on the cluster if the tokens are issued and managed by an external OpenID Connect (OIDC) server.
• Availability: This method was added in OpenSVC v3 agents.

Managing API Users

Creating Local Users (system/usr/)

Skip this step if you plan to use only OIDC-issued tokens for a user. Local user objects define the user’s cluster grants for Basic and X.509 authentication.

# Create a cluster-wide administrator user
om system/usr/root create --kw grant=root

# Create user `usr1` with
#  `admin` rights on namespace `ns1`
#  `guest` (read-only) rights on namespace `ns2`
om system/usr/usr1 create --kw grant="admin:ns1 guest:ns2"

Generating X.509 Certificates for Users

This step is only necessary if you require X.509 authentication for the user.

# Generate Certificate:
om system/usr/root certificate create
om system/usr/usr1 certificate create

# Download Keys (for client use):
om system/usr/usr1 key decode --name certificate_chain
om system/usr/usr1 key decode --name certificate
om system/usr/usr1 key decode --name private_key

These commands will print the PEM-encoded keys to standard output.

Testing the API

The API manifest (documentation) is exposed by a demonstration agent for reference.

A practical test using a JWT:

# Generate a Temporary Token:
$ TOKEN=$(sudo om daemon auth token --subject usr1 --duration 10m)

# Call the `whoami` Endpoint:
$ curl -o- -k -s -H "Authorization: Bearer $TOKEN" https://localhost:1215/whoami

Example Output:

{"auth":"jwt","grant":{"guest":["ns2"], "admin": ["ns1"]},"name":"usr1","namespace":"system","raw_grant":"admin:ns1 guest:ns2"}

Configuring the Listener TLS Certificate

The agent listener requires a TLS certificate to accept remote connections securely.

Upon initial agent installation and daemon startup:

• A self-signed Certificate Authority (CA) is automatically created as system/sec/ca.
• A listener certificate, signed by this internal CA, is automatically created as system/sec/cert.

The following steps are only required if you need to re-sign the internal CA or switch to an external PKI.

Option 1: Using the Internal PKI

The initial configuration is done automatically upon agent installation and daemon startup, creating:

• The CA certificate at system/sec/ca.
• The listener certificate at system/sec/cert.

1. Import a valid certificate in system/sec/ca:

   om system/sec/ca key change --name certificate --from ~/$CLUSTERNAME.pem
   om system/sec/ca key change --name certificate_chain --from ~/$CLUSTERNAME.chain
   om system/sec/ca key change --name private_key --from ~/$CLUSTERNAME.key
   

2. Recreate the listener certificate:

   om system/sec/cert certificate create
   

Option 2: Using an External PKI

1. Adjust the Listener Certificate Subject:

   Only the cn is strictly required by OpenSVC, but the signing certificate authority will surely require more.

   om system/sec/cert config update --set cn=vip.$CLUSTERNAME.mycorp \
       --set alt_names="node1.$CLUSTERNAME.mycorp node2.$CLUSTERNAME.mycorp" \
       --set c=FR \
       --set o=MyCorp \
       --set ou=Support \
       --set l=Paris \
       --set st=IDF \
       --set email=support@mycorp
   

2. Create the Listener Certificate Request:

   om system/sec/cert certificate signing-request
   

3. Perform External Signing: (This is an external procedure to be done with your PKI tool using the generated csr.)

4. Load the Signed Certificate:

   om system/sec/cert key change --name certificate_chain --from ~/$CLUSTERNAME_crt_chain.pem
   

   The certificate is always first, then the CA certificate chain.

5. Configure Certificate Revocation List (Optional):

   om cluster config update --set cluster.crl=http://crl.mycorp
   

6. Restart the Daemons

   om daemon restart --node="*"
   

➡️ See Also

• RBAC For detailed information on Role-Based Access Control and grant management.

Cluster API Client

The ox program uses only the agent API and mirrors the om commandset, which makes it suitable for managing one or more clusters from a tiers linux box.

Configure remotes

The remotes configuration is described in YAML format in the ~/.config/opensvc/contexts.yaml file.

Example:

users:
  john:
    password: xxx
  mary:
    password: xxx
clusters:
  dreamy-leopard:
    server: https://dreamy-leopard.example.com:1215
    insecure: true
  bold-rat:
    server: https://bold-rat:1215
contexts:
  john@dreamy-leopard:
    user: john
    cluster: dreamy-leopard
  mary@bold-rat:
    user: mary
    cluster: bold-rat

Terminal UI

At this point, executing ox with no argument launches the Terminal User Interface, and offers a context selector dialog.

The h keypress displays a help page.

Commandline UI

# Set a context
# -------------
$ export OSVC_CONTEXT=john@dreamy-leopard


# Manage like om
# --------------
$ ox cluster get --kw cluster.name
dreamy-leopard 

$ ox node ls
NAME                  AGENT STATE
dreamy-leopard-node-1 3.0.0 idle
dreamy-leopard-node-2 3.0.0 idle
dreamy-leopard-node-3 3.0.0 idle

$ ox svc ls
OBJECT AVAIL OVERALL 
svc2   down  down    
svc1   down  down   

Cluster Backend Networks

These networks are only required for services private ip auto-allocation. If configured, the cluster DNS exposes the allocated ip addresses as predictible names, and the cluster Ingress Gateways or portmapping can expose the services to clients outside the cluster.

OpenSVC relies on CNI for this subsystem. Any CNI plugin can be used but some plugins can have dependencies like etcd or consul, which OpenSVC does not require for himself. The bridge plugin, having no such dependencies, is simpler to setup.

Install CNI

From package

Some distributions ship CNI packages.

On Red Hat or CentOS 7, for example, CNI is served by the EPEL repositories:

# to activate epel repositories:
# yum install -y epel-release

yum install -y containernetworking-cni

Then tell OpenSVC where to find the CNI plugins and network configurations:

om cluster config update --set cni.plugins=/usr/libexec/cni \
                         --set cni.config=/var/lib/opensvc/cni/net.d

From upstream

cd /tmp
wget https://github.com/containernetworking/cni/releases/download/v0.6.0/cni-amd64-v0.6.0.tgz
wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz
sudo mkdir -p /opt/cni/bin
cd  /opt/cni/bin
sudo tar xvf /tmp/cni-amd64-v0.6.0.tgz
sudo tar xvf /tmp/cni-plugins-amd64-v0.6.0.tgz
sudo mkdir -p /opt/cni/net.d

Here the plugins and network configurations directories are aligned with the OpenSVC defaults.

Configure networks

Networks are declared in the OpenSVC node or cluster configuration.

The agent create the CNI configuration files as needed.

Local Bridge

A local bridge network is always present and named default.

To create another network of this type, named local1, available on every cluster node:

$ om cluster config update --set network#local1.type=bridge \
                           --set network#local1.network=10.10.10.0/24

To create another network of this type, named local1, available on the current cluster node only:

$ om node config update --set network#local1.type=bridge \
                        --set network#local1.network=10.10.10.0/24

Routed Bridge

This network type split the subnet into per-node segments. Trafic is routed from node-to-node via static routes to each segment, and ipip tunnels are created if necessary.

The simple bridge CNI plugin is used for IPAM and plumbing in network namespaces, and OpenSVC is responsible for node-to-node routing and tunneling.

To create a network of this type, named backend1, spanned on every cluster node:

$ om cluster config update --set network#backend1.type=routed_bridge \
                           --set network#backend1.network=10.11.0.0/16 \
                           --set network#backend1.ips_per_node=1024

In this example, the network is split like:

• node 1 : 10.11.0.0/22
• node 2 : 10.11.4.0/22
• node 3 : 10.11.8.0/22
• …

Tunnel endpoints addresses are guessed using a lookup of the nodenames. Different addresses can be setup if necessary, using:

$ om cluster config update --set network#backend1.addr@node1=1.2.3.4 \
                           --set network#backend1.addr@node2=1.2.3.5 \
                           --set network#backend1.addr@node3=1.2.4.4

Some hosting providers, like OVH, don’t support static network routes from node to node, even if they have an ip address in a common subnet. For this situation, you can force OpenSVC to always use tunnels for this backend network::

$ om cluster config update --set network#backend1.tunnel=always

The default tunnel mode is ipip if the network is ipv4, or ip6ip6 if the network is ipv6. The tunnel_mode keyword of the routed_bridge driver also accepts gre. The GRE tunnels can transport both ipv4 and ipv6 and may work in some hosting situations where ipip does not work (OVH).

Use in service configurations

Here is a typical ip resource configuration, using the “weave” CNI network configured above.

[ip#0]
type = cni
network = backend1
netns = container#0
expose = 80/tcp

The container pointed by netns can be a docker or lxc container. netns can also be left empty, causing the weave ip address to be assigned to the service cgroup.

The expose keyword is optional. If set, a SRV record is served by the cluster DNS (in this example _http._tcp.<svcname>.<namespace>.svc.<clustername>). If expose is set to portmapping expression, for example 80:8001/tcp, the portmap CNI plugin is will configure the portmapping and expose the 80/tcp backend server on the 8001 port of the node public ip addresses.

Useful commands

# om net ls
NAME      TYPE           NETWORK        SIZE   USED  FREE   
backend1  routed_bridge  fdfe::/112     65536  0     65536  
backend2  routed_bridge  fdff::/112     65536  0     65536  
backend3  routed_bridge  10.100.0.0/22  1024   2     1022   
lo        lo             127.0.0.1/32   1      0     1      
default   bridge         10.22.0.0/16   65536  0     65536  

List the IP addresses allocated in networks associated with their respective requester object and resource:

# om net ip ls
OBJECT               NODE    RID   IP          NET_NAME  NET_TYPE       
testigw/svc/haproxy  dev2n1  ip#1  10.100.0.2  backend3  routed_bridge  
testigw/svc/haproxy  dev2n2  ip#1  10.100.1.2  backend3  routed_bridge  
...

Cluster Domain Name Server

The OpenSVC agent daemon can act as a remote backend for PowerDNS, serving dynamic records for services deployed within the cluster. This functionality is particularly useful when services are assigned IP addresses on private backends with internal IPAM.

If enabled, the agent configures the container’s resolver (nameserver and search) to use the internal name server when starting a container.

This feature is not enabled by default.

Records

• A record: <hostname>.<svcname>.<namespace>.svc.<clustername> for each resource that includes ipaddr and hostname in the info map in its states.
• Round-Robin A Record: <svcname>.<namespace>.svc.<clustername> where each resource that includes ipaddr in the info map in its states is included in the round-robin.
• Round-Robin SRV Record: _<service>._<protocol>.<svcname>.<namespace>.svc.<clustername> where each resource with an expose keyword matching <port>/<service> is included in the round-robin.

Implementation

• A farmed (flex) service.
• Each instance runs a authoritative PowerDNS server, a PowerDNS recursor and a recursor cache janitoring daemon.
• Each component runs as a privileged docker instance to have r/w access to shared unix domain sockets.
• The DNS server and recursor share the node network namespace.
• The PowerDNS server uses the dns thread of the OpenSVC daemon as a remote backend. Communications go through the /var/lib/opensvc/dns/pdns.sock unix domain socket.

Docker images

• ghcr.io/opensvc/pdns_server
• ghcr.io/opensvc/pdns_recursor
• ghcr.io/opensvc/pdns_janitor

Configure

Preliminary steps

• Make sure the cluster configuration :kw:cluster.name is set to a meaningful, unique site-wide, value. It can be a fqdn like cluster1.my.org, or just a basename like cluster1.
• Choose at least 2 cluster nodes that will act as DNS backends.
• Choose a free port for the DNS to listen on (default is 5300).
• Identify the ip addresses you want the DNS to listen on (public or private). In the following examples, 192.168.100.11 and 192.168.100.14.
• Make sure these ip addresses are resolved to the node name as declared in the :kw:cluster.nodes keyword (edit /etc/hosts if necessary).
• OpenSVC agent installed, minimum version 2.1-1651
• Make sure docker or podman is installed and running on selected dns nodes.
• Make sure CNI is installed
• Make sure you have access to pull from docker.io on selected dns nodes (you can pre-pull or save/load the images if not).

Declare DNS backends

om cluster config update --set cluster.dns+=192.168.100.11 --set cluster.dns+=192.168.100.14

Deploy the DNS service

om system/cfg/dns create
om system/cfg/dns key add --name server --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/pdns.conf.template
om system/cfg/dns key add --name recursor --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/recursor.conf.template
om system/cfg/dns key add --name configure --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/configure
om system/svc/dns deploy --config https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/dns.conf

Configure the nodes resolver

On every node, execute:

# create the resolved configlet directory if it doesn't exist yet
$ mkdir -p /etc/systemd/resolved.conf.d

# install a configlet routing all requests to the cluster domain to the cluster nameservers
$ cat - <<EOF >/etc/systemd/resolved.conf.d/opensvc.conf
[Resolve]
Domains=$(om cluster config get --kw cluster.name)
DNS=$(om cluster config get --kw cluster.dns)
EOF

# activate the new configuration
$ systemctl restart systemd-resolved.service

Verify

Verify the backend

Dump the records served by opensvc to the PowerDNS server

om daemon dns dump

Test the unix socket served by opensvc for the PowerDNS server

echo '{"method": "list", "parameters": {"zonename": "cluster1."}}' | sudo socat - unix://var/lib/opensvc/dns/pdns.sock | jq

Verify the DNS server

Dump the zone contents asking the PowerDNS server

dig +noall +answer cluster1. AXFR @192.168.100.11 -p 5300

Verify the DNS recursor

dig +short cluster1. SOA @192.168.100.11

Administration

Add forwarding for the reverse zones

Either switch to --forward-zones-file or add new elements to forward-zones in the recursor key of system/cfg/dns.

Ingress Gateway

Services configured to obtain an IP address from a backend network are not naturally accessible to clients outside the cluster.

To expose them, the user or a cluster administrator can deploy a ingress gateway configured with a public IP address.

HAProxy is our recommended program to route layer 4 and layer 7 communications from the frontend to the backend servers.

Behaviour

The backend composition is kept up to date by HAProxy the resolvers mechanism.

To declare the cluster dns in the HAProxy configuration:

resolvers clusterdns
    parse-resolv-conf
    accepted_payload_size 8192

As the HAProxy server runs in a container resource started by OpenSVC, the /etc/resolv.conf file contains the cluster nameservers IP address. The parse-resolv-conf tells HAProxy to read the nameservers from there.

This resolvers configuration can be referenced in every backend definition like:

backend svc1
    option httpchk GET /health
    server-template svc1_ 1 svc1.ns1.svc.${CLUSTERNAME}:8080 resolvers clusterdns check init-addr none

Configurations

• Intra-Cluster Load-Balancing: Run only one HAproxy on the cluster, in a failover topology svc.
• Extra-Cluster Load-Balancing: Every node runs a HAProxy exposing the same servers. The upstream load-balancer picks one.

Intra-Cluster Load-Balancing Configuration

Listen on port 443, with a self-signed certificate.

Deploy a haproxy service using the basic example from the igw_haproxy template page on github.

# Create a self signed key and certificate
sudo om testigw/sec/haproxy create
sudo om testigw/sec/haproxy certificate create

# Create a haproxy configuration as a cfg key
sudo om testigw/cfg/haproxy create
sudo om testigw/cfg/haproxy key add --name haproxy.cfg --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/igw_haproxy/basic-cfg-haproxy.cfg

# Deploy the Ingress Gateway svc
# * change the network to a cluster spaning network if you have one setup
# * make sure requests from this network are allowed by the nameservers
sudo om testigw/svc/haproxy deploy --config https://raw.githubusercontent.com/opensvc/opensvc_templates/main/igw_haproxy/basic-svc.conf --kw ip#1.network=default

A ip#1 failover-capable public IP address should be added and started for this service to be useful to extra-cluster clients, but it can be tested from a cluster node already.

# Store the haproxy IP address allocated on start
eval IP=$(sudo om testigw/svc/haproxy resource ls -o json --rid ip --node $HOSTNAME| jq .items[].data.status.info.ipaddr)

# Test, faking a DNS resolution of svc1.opensvc.com to the haproxy ip address
curl -o- -k --resolve svc1.opensvc.com:443:$IP https://svc1.opensvc.com

# Deploy a test webserver to populate the svc1.opensvc.com backend:
# * change the network to a cluster spaning network if you have one setup
# * make sure requests from this network are allowed by the nameservers
sudo om testigw/svc/svc1 deploy --config https://raw.githubusercontent.com/opensvc/opensvc_templates/main/igw_haproxy/nginx.conf --kw ip#1.network=default --wait

# Retest until available
curl -o- -k --resolve svc1.opensvc.com:443:$IP https://svc1.opensvc.com

Automated Certificate Management Environment

The igw_haproxy template page on github also documents the deployment of a HAProxy cluster ingress gateway service implementing ACME.

Upgrade

The agent supports upgrading with zero service down-time.

Upgrading does not require a node reboot.

Debian, Ubuntu

sudo apt update
sudo apt install --only-upgrade opensvc-server opensvc-client

Red Hat Enterprise Linux 7

sudo yum makecache
sudo yum upgrade opensvc-server opensvc-client

Red Hat Enterprise Linux 8+

sudo dnf makecache
sudo dnf upgrade opensvc-server opensvc-client

SuSE Linux Enterprise Server

sudo zypper refresh
sudo zypper update opensvc-server opensvc-client

Deploy Apps

Applications are composed of one or more objects (services, configs, secrets, volumes, service accounts). These objects can be deployed and operated individually or as a group.

Naming

A fully qualified object name is formatted as <namespace>/<kind>/<name>.

Namespace

Namespaces allow users to create objects with the same name in different naming spaces.

Namespace names must conform to RFC952:

• only alphanum characters or dash
• start with an alpha
• end with an alphanum

Kind

• svc A service, with a mix of ip, app, container, volume, disk, fs and task resources.
• vol A data volume from a pool, with a mix of volume, disk and fs resources.
• cfg A configuration map, storing unencrypted key/value pairs for use by other kinded objects.
• sec A secret, storing encrypted key/value pairs for use by other kinded objects.
• ccfg The special kind for the cluster configuration object.

Name

Names must conform to RFC952, with a tolerance for dots.

A name must be unique in its namespace and kind.

Create, Deploy

The following actions only modify files in /etc/opensvc. No operating system configuration file is modified, so they are safe to experiment with.

The agent support object creation via two commands:

• create The object is created but not provisioned nor started.

• deploy The object is created, provisioned and started.

Both actions support the same arguments. The following examples use only create commands.

From Scratch, non Interactive

Create a new object with minimal configuration. No resources are described.

om <path> create

Resources and default keywords can be set right from the create command, using --kw <keyword>=<value> options

om <path> create
	--kw container#0.type=oci \
	--kw orchestrate=ha \
	--kw nodes={clusternodes}

From Another Object

om <dst path> create --config=<src path>

From Manifest, Single Object

The manifest must be json formatted, structured like om <path> print config --format=json.

om <path> create --config=<manifest uri>

This method can also be used to clone objects

om <src path> print config --format json | \
	om <dst path> create --config=- [--interactive] [--provision]

From Manifest, Multiple Objects

The manifest must be json formatted, structured like om <selector> print config --format=json.

In this case, the <dst path> can not be specified, but the destination namespace where to create the objects can. The new objects will adopt the names set in the manifest.

om svc create --namespace=newns --config=<manifest uri>

om 'test/svc/*' print config --format=json | \
	om svc create --namespace=testclone --config=-

From Existing Local Configuration File

Experienced users may find it easier to start from a copy of the conf file of an existing similar object.

om <path> create --config <path to config file> [--interactive] [--provision]

The configuration file can be remote, referenced by URI.

From Collector Template

Templates can be served by the collector.

om <path> create --template <id|name> [--interactive] [--provision]

➡️ See Also

• Provisioning

Update

Configuration files are stored in /etc/opensvc/.

• /etc/opensvc/<name>.conf

  Root objects configuration file:

• /etc/opensvc/namespaces/<namespace>/<kind>/<name>.conf

  Namespaced objects configuration file:

Do not edit these files directly. Use one of the following method instead.

Interactive

om <path> config edit

The configuration file syntax is checked upon editor exit. The new configuration is installed if the syntax is found correct, or saved in a temporary location if not. Two options are then possible:

• Discard the erroneous configuration:

    om <path> config edit --discard
  

• Re-edit the erroneous configuration:

    om <path> config edit --recover
  

Non-Interactive Resource Addition

om <path> config update --set fs#2.type=ext4 --set fs#2.mnt=/srv/{fqdn}

The resource identifier (rid) must not be specified. The resource type must be specified (rtype). A free rid will be allocated.

Non-Interactive Resource Modification

om <path> config update --set fs#2.type=ext4 --set fs#2.mnt=/srv/{fqdn}

The resource identifier must be specified.

Non-Interactive Resource Deletion

om <path> config update --delete fs#2

This command does not stop the resource before removing its definition. If desired, this can be done with

om <path> stop --rid fs#2

Purge

om <path> purge

This command asks the cluster to orchestrate a stop, unprovision and delete. Non-leader instances are sequenced first.

Purging a service does not purge its referenced volumes.

Purging a volume actually removes all volume data.

Delete

om <path> delete

This command does not stop nor unprovision the object, so it can leave unreferenced mounts, containers and processes on the nodes.

This command should be used by administrators only.

Operate Apps

Usual operator actions covered in this chapter are:

• Assess the state of the cluster and its hosted applications, troubleshoot warnings, post-mortem analysis.
• Start, stop, relocate one service or a selection of services to fix warning causes, prepare hardware or operating system maintenance

Services Status

Cluster Overview (om mon)

The om mon command provides a real-time, human-readable overview of the cluster and service states.

Human Readable

om monitor

Requirements:

• The agent daemon must be up and running.
• The displayed information is near synchronous.

Status and Alert Markers

Markers are used to optimize information density.

Machine Readable

Use the --format option for structured data output.

om daemon status --format json

Watch

Continuously refresh the status display.

om monitor --watch

Select Objects

Commands use the om <selector> <action> syntax to operate on a selection of objects. The om <selector> ls command can test a selector before submitting a dangerous action.

Note the selector expression may need to be quoted for the shell not to interpret the ! and * characters.

All Objects

List all cluster objects (services, volumes, etc.).

om '**' ls

All Services

List all service objects.

om '*' ls
om '*/svc/*' ls

Single Object

List a specific object.

om <path> ls
om ns1/svc/web1 ls

List of objects

With a /tmp/svc.list containing:

# Ignored object paths
; svc1
;svc2

# Ignored empty lines

# Honored object paths
# Leading and trailing whitespaces are trimmed
svc3
  svc4

An action can be executed using:

cat /tmp/svc.list | om - ls

or

om - ls </tmp/svc.list

The template output renderer of many om command can produce lists of object paths matching criteria.

# Start all objects in avail=down state
om svc ls -o 'template={{ range . }}{{ if ne .data.avail "up" }}{{println .meta.object}}{{ end }}{{ end }}' | om - start

# Freeze all objects not frozen
om svc ls -o 'template={{ range . }}{{ if ne .data.frozen "frozen" }}{{println .meta.object}}{{ end }}{{ end }}' | om - freeze

Unions

List multiple specific objects.

om <path1>,<path2> ls
om ns1/svc/web1,ns1/vol/web1 ls

Intersections

List objects matching multiple criteria.

om 'ns1/*/web1+*/svc/*' ls

is equivallent to

om ns1/svc/web1 ls

Mixing Unions and Intersections

The unioned expressions are evaluated from comma to comma, intersections are evaluated in the context of the current union.

For example,

om 'ns1/svc/*+ns1/*/web1,ns2/svc/web2'

is parsed as:

(ns1/svc/* intersected with ns1/*/web1) unioned with (ns2/svc/web2)

and evaluates as:

ns1/svc/web1
ns2/svc/web1

Negation

The negation marker is !. This symbol needs quoting for the shell not to interpret it.

Example:

# All object of namespace `ns1` except those named `web1`:
om 'ns1/**+!**/web1' ls

Services by State

Filter services based on their overall status.

• List all services in the down state:

  om '*' ls --status down
  

• List all services in up and warn states:

  om '*' ls --status up,warn
  

Service Selector Expressions

Use powerful expressions to filter objects based on configuration parameters.

om <expr> ls

Where $\text{}$ is a pattern or condition: $\text{}$ or $\text{[!]\text{}\text{}\text{}}$.

Note: Matching is case-sensitive, except for boolean values.

Examples

1. Services with name ending with dns OR starting with ha, AND which have an app resource with a timeout greater than 1:

   om '*dns,web*' config get -o 'template={{range .}}{{if and (hasPrefix .keyword "app#") (hasSuffix .keyword ".timeout") (ge .value 1)}}{{end}}'
   

2. Services with at least one ip resource AND one task resource:

   om 'ip:+task:' ls
   # ha1, ha2, ha3, registry
   

3. Services with at least one monitored resource AND with monitor_schedule not set:

   om '!monitor_schedule+.monitor=true' ls
   # ha1, ha4
   

Action

Base Actions

Start

Local Start (Bypasses Orchestrator)

Start the service instance on the local node directly.

om <path> instance start

Resource Start Order: ip, disk, fs, share, container, app.

Orchestrated Start

Instruct the orchestrator to start the service on the node(s) selected by the placement policy.

om <path> start [--wait] [--time <duration expr>] [--watch]

• By default, the command returns upon daemon acknowledgment.
• --wait holds the command until the action completes.
• --time sets a maximum wait duration.

Stop

Local Stop (Bypasses Orchestrator)

Stop the service instance on the local node directly.

om <path> instance stop

Resource Stop Order: app, container, share, fs, disk, ip.

Orchestrated Stop and Freeze

Instruct the orchestrator to stop the service wherever it runs and freeze it to prevent automatic restarts.

om <path> stop [--wait] [--time <duration expr>] [--watch]

Relocation

Switch

Stop the service on its current node(s) and start it on the specified target node. All instances are thawed afterward.

om <path> switch --node <nodename> [--wait] [--time <duration expr>] [--watch] [--live]

The container.kvm supports live migration. Live migration requires the VM storage to be read-write from all nodes during the switch. SAN disks and drbd pass-through, NFS, Ceph, ClusterFS can satisfy this requirement.

Takeover

Stop the service instances on peer nodes and start it on the local node. All instances are thawed afterward.

om <path> takeover [--wait] [--time <duration expr>] [--watch]

Giveback

Thaw all nodes/instances, stop the service on non-leader nodes, and let the orchestrator start instances on the designated leaders. All instances are thawed afterward.

om <path> giveback [--wait] [--time <duration expr>] [--watch]

Handling Failures

If an action fails, the orchestrator is blocked, and the failure is reported in om mon and om <path> print status.

• Clear Failure: Allows the daemon to retry the execution plan.

  om <path> clear
  

• Abort Action: Aborts the currently blocked orchestrated action.

  om <path> abort
  

Sync

Sync All

Run resource replication to all configured targets (e.g., production (prd) or disaster recovery (drp)).

om <path> update

Sync Nodes

Trigger file synchronization to secondary cluster nodes. No-op if run from a node not running the service.

om <path> sync update --target nodes

Sync DRP

Trigger file synchronization to disaster recovery nodes. No-op if run from a node not running the service.

om <path> sync update --target drp

Run

Execute tasks defined within the service configuration.

om <path> instance run [--rid ...]

Resource Filtering

Filter actions to be executed only on specific resources using --rid, --tags, or --subsets.

Logging

All action logs are multiplexed to multiple destinations:

• Stdout/Stderr
• Journald or Syslog
• Collector Database (Optional, via asynchronous XML-RPC calls)

Examples

Starting a Service (Local)

Shows the ordered execution of resource start-up.

# om svc0 instance start
10:51:50.590 INF svc0: >>> do start [om svc0 instance start] (origin user, sid ae60fdd4-6629-40eb-a993-4f1380d66516)
10:51:52.061 INF svc0: fs#1: install flag file /dev/shm/opensvc/svc/svc0/fs#1.flag
10:51:52.062 INF svc0: app#0: applied pg /opensvc.slice/opensvc-svc.svc0.slice
10:51:52.063 INF svc0: app#0: applied pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice
10:51:52.090 INF svc0: app#0: run: /usr/bin/om exec --pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice -- touch test -f /tmp/svc0.root.svc.reliable-leopard
10:51:52.194 INF svc0: <<< done start [om svc0 instance start] in 1.603582081s, instance status is now up

Stopping a Service (Local)

Shows the ordered execution of resource shut-down.

# om svc0 instance stop
10:51:55.282 INF svc0: >>> do stop [om svc0 instance stop] (origin user, sid 807b55a5-8a5e-4a38-920c-94b6956da6a7)
10:51:55.380 INF svc0: app#0: run: /usr/bin/om exec --pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice -- rm -f /tmp/svc0.root.svc.reliable-leopard
10:51:55.461 INF svc0: fs#1: uninstall flag file /dev/shm/opensvc/svc/svc0/fs#1.flag
10:51:55.494 INF svc0: <<< done stop [om svc0 instance stop] in 211.596639ms, instance status is now down

Design Apps

Scoping

Most keywords in a service configuration support a scoping syntax, allowing each node agent to interpret the value differently.

Syntax

A scoped keyword is written as:

<keyword>@<scope> = <value>

Supported Scopes

• <nodename>: The hostname of the node where the keyword value is interpreted as <value>.
• nodes: The keyword value is interpreted as <value> on all service nodes.
• drpnodes: The keyword value is interpreted as <value> on all service DRP nodes.
• encapnodes: The keyword value is interpreted as <value> on all service encapsulated nodes.
• flex_primary: The keyword value is interpreted as <value> on the flex service primary node.
• drp_flex_primary: The keyword value is interpreted as <value> on the flex service disaster recovery primary node.

Examples

Use a Different FS Type on DRP Nodes

[DEFAULT]
nodes = n1 n2
drpnode = n3

[fs#1]
type = ext4
type@drpnodes = xfs

Use a Different Nodes List at Encapsulated Level

[DEFAULT]
nodes = n1 n2
encapnodes = vm1
nodes@encapnodes = vm1

Disable a Resource on a Node

[DEFAULT]
nodes = n1 n2
drpnodes = n3

[ip#backup]
disable@n3 = true

Precedence

When a section has multiple definitions of the same keyword, the most specific takes precedence. If multiple definitions of the same rank are found, the last one takes precedence.

Examples

[DEFAULT]
drpnodes = n3

[share#1]
disable = true
disable@drpnodes = false

This resource is enabled on n3 because the generic disable is overridden by the more specific disable@drpnodes scoped definition.

[DEFAULT]
drpnodes = n3

[share#1]
disable = true
disable@drpnodes = false
disable@n3 = true

This resource is disabled on n3 because the generic disable and disable@drpnodes are overridden by the more specific disable@n3 scoped definition.

[DEFAULT]
drpnodes = n3

[share#1]
disable@n3 = true
disable@n3 = false

This resource is disabled on n3 because the last of the two same-ranked scoped definitions takes precedence.

Outputs

The om query commands display the data using a human-readable renderer by default. The -o command flag can be used to select another renderer.

$ om svc resource list
OBJECT      NODE    RID     TYPE       STATUS  IS_MONITORED  IS_DISABLED  IS_STANDBY  RESTART  RESTART_REMAINING  
stonith8    dev2n2  fs#1    fs.flag    down    false         false        false       0        0                  
testsched   dev2n2  fs#1    fs.flag    down    true          false        false       0        0                  
testsched   dev2n2  task#1  task.host  n/a     false         false        false       0        0                  
...

Custom tables

Custom column selection, with header

$ om svc resource ls -o tab=OBJ:.meta.object,NODE:.meta.node,RID:.meta.rid,STATUS:.data.status.status
OBJ         NODE    RID     STATUS       
cva1        dev2n2  fs#1    down        
cva1        dev2n2  sync#1  n/a         
testflex    dev2n2  fs#1    down        
stonith236  dev2n2  fs#1    down        
...

Single column selection, without header

$ om svc resource ls -o tab=.meta.rid
fs#1
sync#1
fs#1
fs#1
...

Machine-readable

$ om svc resource ls -o json
{
    "items": [
        {
            "data": {
                "config": {
                    "is_disabled": false,
                    "is_monitored": false,
                    "is_standby": false,
                    "restart": 0,
                    "restart_delay": 500000000
...

$ om svc resource ls -o yaml
items:
- data:
    config:
      is_disabled: false
      is_monitored: false
      is_standby: false
      restart: 0
      restart_delay: 500000000
    monitor:
      restart:

Key-Value

$ om svc resource ls -o yaml
items[0].data.config.is_disabled = false
items[0].data.config.is_monitored = false
items[0].data.config.is_standby = false
items[0].data.config.restart = 0
items[0].data.monitor.restart.last_at = "0001-01-01T00:00:00Z"
items[0].data.monitor.restart.remaining = 0
items[0].data.status.label = "zfssnap 1d of foo"
items[0].data.status.provisioned.mtime = "2025-10-03T19:04:15.963080783+02:00"
items[0].data.status.provisioned.state = "n/a"
items[0].data.status.status = "n/a"
...

Template

This output uses the golang template syntax, with the following extra functions:

Example:

$ om svc resource ls -o template='{{range .}}{{if eq (resName .meta.rid) "hdoc"}}{{printf "%s@%s\n" .meta.object .meta.node}}{{end}}{{end}}'
test/svc/hdoc@dev2n2
test/svc/hdoc@dev2n3
test/svc/hdoc@dev2n1

Internals

Installed Items

Directories

• /etc/opensvc

  The cluster, node and objects configuration files.

• /var/lib/opensvc

  The state files. Deleting or creating files in this directory can have undesired side-effects.

• /var/tmp/opensvc

  Temporary files. Deleting or creating files in this directory can have undesired side-effects.

Executable files

• /usr/bin/om

  This executable, installed by the opensvc-server package, implements:

  • The Cluster Resource Manager
  • The Cluster Monitor and API daemon
  • The local management commandline interface

• /usr/bin/ox

  This executable, installed by the opensvc-client package, implements:

  • The remote management commandline interface

Configuration files

The agent configuration is the result of the merge of two ini configuration files:

• /etc/opensvc/cluster.conf

  This file is replicated on all cluster nodes.

• /etc/opensvc/node.conf

  This file is not replicated.

Heartbeats

Heartbeats serve the following purposes:

• Exchange data between cluster nodes.
• Detect stale nodes.
• Execute the quorum race when a peer becomes stale.

OpenSVC supports multiple parallel running heartbeats. Exercising different code paths and infrastructure data paths (network and storage switches and site interconnects) helps limit split-brain situations.

Configuration

Heartbeats are declared in /etc/opensvc/cluster.conf, each in a dedicated section named [hb#<n>]. A heartbeat definition should work on all nodes, using scoped keywords if necessary, as the definitions are served by the joined node to the joining nodes.

Reconfiguration

Any command that changes the timestamp of the following configuration files triggers a reconfiguration of heartbeats:

• /etc/opensvc/node.conf
• /etc/opensvc/cluster.conf

Actions Taken During Reconfiguration:

• Any updated parameters are applied to the heartbeats.
• Heartbeats removed from the configuration are stopped.
• Heartbeats newly defined in the configuration are started.

Set a Heartbeat Timeout

To set a timeout for the hb#1 heartbeat, use this command:

om cluster config update --set hb#1.timeout=20

Drop a Heartbeat

To delete the hb#1 heartbeat from the configuration:

om cluster config update --delete hb#1

Monitoring

Each heartbeat runs two threads: tx and rx.

The om mon command display the heartbeats status, statistics, and each peer state.

Threads                                n1        n2        n3        
 ...
 hb                                  |                                           
  hb#1.rx          running unicast   | /         O         O             
  hb#1.tx          running unicast   | /         O         O             
  hb#2.rx          running relay     | /         O         O             
  hb#2.tx          running relay     | /         O         O             
 ...

The agent daemon automatically restarts heartbeat threads if they exit unexpectedly.

Heartbeat Thread Pair

Tx (Transmit)

The Tx thread handles the transmission of the node data:

• Regularly transmit data or send it as soon as changes occur.
• Data is encrypted.

Rx (Receive)

The Rx thread manages data reception and integration into cluster data:

• Regularly read data from disk or receive it in response to transmissions (unicast/multicast).
• Update peer data in the cluster.
• Timeout if no heartbeat is received within the configured <hb#n>.timeout. The default timeout is 15 seconds.

Actions Performed by Rx:

• On receive data:
  • Merge updated peer data to maintain accurate cluster data.
  • Publish the received events on the local event bus.
• On receive timeout:
  • Publish a HbStale event
  • Purge stale peer data if:
    • No Maintenance Advertised: Immediately purge stale peer data.
    • Maintenance Advertised: Wait for the node.maintenance grace_period before purging.

➡️ See Also

• Cluster Data

hb.unicast

The hb.unicast driver sends and receives using TCP unicast packets.

Basic Configuration

[hb#1]
type = unicast

Behavior with Basic Configuration

• The Rx thread listens on 0.0.0.0:10000
• The Tx thread sends to <nodename>:10000

Advanced Configuration

A more precise definition allows specifying network interfaces, addresses, and ports for each node:

[hb#1]
type = unicast
intf@node1 = eth0
intf@node2 = eth2
addr@node1 = 1.2.3.4
addr@node2 = 1.2.3.5
port@node1 = 10001
port@node2 = 10002
timeout = 15s

Note the driver accepts to use the same port for every node:

port = 10001

Proper configuration of the hb.unicast driver ensures reliable communication between cluster nodes by leveraging TCP unicast.

hb.multicast

The hb.multicast driver sends and receives using TCP multicast packets.

Basic Configuration

[hb#2]
type = multicast

Behavior with Basic Configuration

• The Rx thread listens on all interfaces on port 10000
• The Tx thread sends to 224.3.29.71:10000

Advanced Configuration

A more precise definition allows specifying network interfaces, addresses, and ports for each node:

[hb#2]
type = multicast
intf@node1 = eth0
intf@node2 = eth2
addr = 224.3.29.71
port = 10001
timeout = 15

The addr and port keywords are not scopable.

hb.disk

This driver reads and writes on a dedicated disk, using O_DIRECT|O_SYNC|O_DSYNC open flags on a block device on Linux.

Configuration

[hb#2]
type = disk
dev = /dev/mapper/3123412312412414214
timeout = 15

Behavior

• The Rx thread loops over peer nodes and for each reads its heartbeat data at its reserved slot device offset
• The Tx thread writes to its reserved slot offset on the device

On-disk format

When the tx and rx threads are started or reconfigured, they parse a metadata segment at the head of the device and prepare a <nodename>:<slot index> hash.

The metadata zone maximum size is 4MB.

A node metadata slot size is 4k, and contains the cluster node name.

Limits:

• 1000 nodes (metadata zone size/slot meta data size)
• nodenames are limited to 4k characters (slot meta data size)
• A -nodes cluster requires a (<n>+1)*4MB device
• The heartbeat data (which is gziped) must not exceed 4MB (slot size). A 10 services cluster usually produces ~3k messages.

If a the local nodename is not found in any slot, the thread allocates one.

hb.relay

This driver implements a heartbeat mechanism by reading and writing on the memory of a dedicated OpenSVC agent relay host.

Purpose

The hb.relay heartbeat is designed to prevent split-brain scenarios in a cluster by establishing a neutral, off-site communication check.

• A relay should ideally be located in a third site that hosts no other node of the cluster.
• This setup allows the cluster to make a correct quorum decision when the sites hosting the cluster nodes become disconnected from each other, but can still independently reach the relay’s site.
• The same relay host can be shared and used as a heartbeat mechanism by multiple different clusters.

Configuration

The heartbeat is defined in the cluster configuration file (cluster.conf) within a [hb#N] section.

Example

[hb#2]
type = relay
relay = relay3.opensvc.com
timeout = 15
username = relay
password = system/sec/relay

Authentication Note

The OpenSVC v3 relay configuration no longer supports the secret keyword. Authentication credentials must be specified exclusively using the username and password keywords.

Requirements

• Reachability: The relay listener (typically at <address>:<port>) must be reachable from all cluster nodes during normal operations.
• Version Compatibility: OpenSVC v3 clusters must use an OpenSVC v3 relay agent.
• Arbitration: The relay host can also be configured and used as a cluster arbitrator.

Key Operations

The driver utilizes two primary threads for communication:

• The Transmission Thread (TX):
  • This thread is responsible for sending the local cluster node’s heartbeat data to the remote relay agent, updating its status in the relay’s memory.
• The Reception Thread (RX):
  • This thread constantly loops over all peer nodes in the cluster and for each one, requests its current heartbeat data from the relay agent. This data retrieval determines the peer’s reachability.

Quorum

When a peer is flagged as stale by all heartbeats, the daemon assumes the cluster is in a split-brain situation, as it cannot determine whether the stale peer has failed or is isolated.

OpenSVC minimizes the likelihood of a split-brain scenario by leveraging multiple independent heartbeats.

Enabling Quorum Enforcement

Users who prefer to have a cluster segment shut down in such situations can enable quorum by setting cluster.quorum to true:

om cluster config update --set cluster.quorum=true

By default, the system allows split nodes to take over services, which may result in services running on multiple isolated segments. To revert to the default behavior, use:

om cluster config update --unset cluster.quorum

To check the current quorum configuration:

om cluster config get --kw cluster.quorum

Quorum Behavior

If the cluster is configured for quorum and a split-brain situation occurs, a node will shut down if the number of reachable nodes (including itself) plus arbitrators is less than half of the total cluster and arbitrator nodes.

Frozen nodes do no evaluate quorum. They will not shut down on split-brain.

Frozen nodes still vote for peer nodes quorum evaluation.

Example Arbitrator Requirements

To survive a interconnect outage:

• In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split.
• In a 3-node cluster, a single node requires 2 arbitrator votes.
• In a 4-node cluster, a single node requires 3 arbitrator votes.
• In a 5-node cluster, a single node requires 3 arbitrator votes.

To survive a interconnect outage, plus all peers outage in the same availability zone:

• In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split.
• In a 3-node cluster, a single node requires 2 arbitrator votes.
• In a 4-node cluster, a single node requires 3 arbitrator votes.
• In a 5-node cluster, a single node requires 4 arbitrator votes.

Configuring Arbitrators

Any OpenSVC agent can act as an arbitrator, and multiple arbitrators can be configured. For example, to configure an arbitrator:

Use a https server as an arbitrator

[arbitrator#a1]
uri = https://dev2n1:1215/metrics
#insecure = true

Use a tcp server as an arbitrator

[arbitrator#a2]
uri = dev2n2:22

Testing Arbitrators

Alive test of an arbitrator:

    $ om node ping --node a1

The om mon output show all arbitrator alive state from the point of view of every node.

    $ om mon
    ...
    Arbitrators                       n1   n2
     a1                warn         | X    X          
     a2                warn         | X    X          
     a3                             | O    O          
    ...

Best Practices

• Configure minus 1 arbitrators
• Host all arbitrators on the same 3rd site
• Use one of the arbitrators as a relay for the relay heartbeat driver
• Disable quorum or freeze all nodes when doing a relayout of the cluster

Example: odd-nodes cluster

    +-------------------------------------------+                                
    |  site3                                    |                                
    |                                           |                                
    |  +-------------+   +-------------+        |
    |  |             |   |             |        |
    |  | arbitrator1 |   | arbitrator2 |        |
    |  |             |   |             |        |
    |  +-------------+   +-------------+        |
    |                                           |                                
    +-------------------------------------------+                                
    
    +-------------------------------------------+     +------------------------------+
    | site1                                     |     |   site2                      |
    |                                           |     |                              |
    |    +--------------------------------------|-----|-----------------------+      |
    |    | cluster                              |     |                       |      |
    |    |                                      |     |                       |      |
    |    |    +-----------+    +-----------+    |     |   +-----------+       |      |
    |    |    |           |    |           |    |     |   |           |       |      |
    |    |    |   node1   |    |   node2   |    |     |   |   node3   |       |      |
    |    |    |           |    |           |    |     |   |           |       |      |
    |    |    +-----------+    +-----------+    |     |   +-----------+       |      |
    |    |                                      |     |                       |      |
    |    +--------------------------------------|-----|-----------------------+      |
    |                                           |     |                              |
    +-------------------------------------------+     +------------------------------+

• Total: 5 votes
• Majority: 3 votes

Site1 Isolated

node1 standpoint:

• live nodes: 2 (node1, node2)
• arbitrators votes: 0
• votes: 2

=> node does not have quorum, commits suicide

node2 standpoint

• live nodes: 2 (node1, node2)
• arbitrators votes: 0
• votes: 2

=> node does not have quorum, commits suicide

node3 standpoint

• live nodes: 1 (node3)
• arbitrators votes: 2
• votes: 3

=> node has quorum, does not commit suicide

Site2 Isolated

node1 standpoint

• live nodes: 2 (node1, node2)
• arbitrators votes: 2
• votes: 4

=> node has quorum, does not commit suicide

node2 standpoint

• live nodes: 2 (node1, node2)
• arbitrators votes: 2
• votes: 4

=> node has quorum, does not commit suicide

node3 standpoint

• live nodes: 1 (node3)
• arbitrators votes: 0
• votes: 1

=> node does not have quorum, commits suicide

Node2 Dies

node1 standpoint

• live nodes: 2 (node1, node3)
• arbitrators votes: 2
• votes: 4

=> node has quorum, does not commit suicide

node3 standpoint

• live nodes: 2 (node1, node3)
• arbitrators votes: 2
• votes: 4

=> node has quorum, does not commit suicide

Node2 and Node3 Die

node1 standpoint

• live nodes: 1 (node1)
• arbitrators votes: 2
• votes: 3

=> node has quorum, does not commit suicide

Example: even-nodes cluster

    +-------------------------------------------+                                
    |  site3                                    |                                
    |                                           |                                
    |  +-------------+                          |
    |  |             |                          |
    |  | arbitrator1 |                          |
    |  |             |                          |
    |  +-------------+                          |
    |                                           |                                
    +-------------------------------------------+                                
    
    +--------------------------+     +------------------------------+
    | site1                    |     |   site2                      |
    |                          |     |                              |
    |    +---------------------|-----|-----------------------+      |
    |    | cluster             |     |                       |      |
    |    |                     |     |                       |      |
    |    |    +-----------+    |     |   +-----------+       |      |
    |    |    |           |    |     |   |           |       |      |
    |    |    |   node1   |    |     |   |   node2   |       |      |
    |    |    |           |    |     |   |           |       |      |
    |    |    +-----------+    |     |   +-----------+       |      |
    |    |                     |     |                       |      |
    |    +---------------------|-----|-----------------------+      |
    |                          |     |                              |
    +--------------------------+     +------------------------------+

• Total: 3 votes
• Majority: 2 votes

Site1 Isolated

node1 standpoint

• live nodes: 1 (node1)
• arbitrators votes: 0
• votes: 1

=> node does not have quorum, commits suicide

node2 standpoint

• live nodes: 1 (node2)
• arbitrators votes: 1
• votes: 2

=> node has quorum, does not commit suicide

Node1 dies

node2 standpoint

• live nodes: 1 (node2)
• arbitrators votes: 1
• votes: 2

=> node has quorum, does not commit suicide

Scheduler

The OpenSVC agent includes a scheduler that manages jobs for both the node and each individual service.

Basic Schedule Definition

The schedule constraints are defined by allowed time ranges and minimum execution interval. An example schedule definition is 00:00-02:00@121m. In this example:

• Time Range: From midnight to 2:00 AM.
• Interval: 121 minutes.

Multiple schedule definitions can be specified using the syntax:

["00:00-02:00@121", "12:00-14:00@121"]

Execution is permitted if any one of the defined constraints is satisfied.

Policies

If an allowed time range is longer than the interval, multiple executions happen in the time range.

If not specified, the default interval is the duration of the time range, so there is only one execution of the job during the time range.

If not specified, the default time range is unrestricted. In this case a period must be specified.

If the definition begins with a ~, the execution is delayed randomly in the allowed time range. The probability of execution increases linearly as time progresses within the allowed time range. For instance:

• At the beginning of the time range (00:00 in 00:00-02:00), the probability might be around 10%.
• Near the end of the time range (01:50), the probability reaches 100%.

This behavior ensures that the execution of job reporting information to the collector is spread across all nodes throughout the entire time range, leveling the load on the central collector. This approach prevents sudden spikes in load.

Node Scheduler

$ om node schedule list -o +KEY:data.key
NODE  ACTION           LAST_RUN_AT                NEXT_RUN_AT           SCHEDULE      KEY                       
n1    pushasset        2025-01-27T05:57:06+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  asset.schedule            
n1    checks           2025-01-27T01:54:15+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  checks.schedule           
n1    compliance_auto  2025-01-27T02:00:00+01:00  0001-01-01T00:00:00Z  02:00-06:00   compliance.schedule       
n1    pushdisks        2025-01-27T04:56:30+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  disks.schedule            
n1    pushpkg          2025-01-27T18:59:54+01:00  0001-01-01T00:00:00Z  @1m           packages.schedule         
n1    pushpatch        2025-01-27T04:58:22+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  patches.schedule          
n1    sysreport        0001-01-01T00:00:00Z       0001-01-01T00:00:00Z  ~00:00-06:00  sysreport.schedule        
n1    dequeue_actions  0001-01-01T00:00:00Z       0001-01-01T00:00:00Z                dequeue_actions.schedule  
n2    pushasset        2025-01-29T00:35:49+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  asset.schedule            
n2    checks           2025-01-29T00:10:39+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  checks.schedule           
n2    compliance_auto  2025-01-29T02:00:00+01:00  0001-01-01T00:00:00Z  02:00-06:00   compliance.schedule       
n2    pushdisks        2025-01-29T05:14:15+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  disks.schedule            
n2    pushpkg          2025-01-29T05:33:22+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  packages.schedule         
n2    pushpatch        2025-01-29T00:42:55+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  patches.schedule          
n2    sysreport        2025-01-29T03:08:18+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  sysreport.schedule        
n2    dequeue_actions  0001-01-01T00:00:00Z       0001-01-01T00:00:00Z                dequeue_actions.schedule  
n3    pushasset        2025-01-29T04:50:18+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  asset.schedule            
n3    checks           2025-01-29T05:17:24+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  checks.schedule           
n3    compliance_auto  2025-01-29T02:00:00+01:00  0001-01-01T00:00:00Z  02:00-06:00   compliance.schedule       
n3    pushdisks        2025-01-29T05:10:43+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  disks.schedule            
n3    pushpkg          2025-01-29T03:07:57+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  packages.schedule         
n3    pushpatch        2025-01-29T05:36:14+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  patches.schedule          
n3    sysreport        2025-01-29T00:34:02+01:00  0001-01-01T00:00:00Z  ~00:00-06:00  sysreport.schedule        
n3    dequeue_actions  0001-01-01T00:00:00Z       0001-01-01T00:00:00Z                dequeue_actions.schedule  

The scheduled jobs can be configured in /etc/opensvc/node.conf with a configlet like:

[<section>]
<parameter> = <definition>

The KEY column in the above command output is formatted as:

<section>.<parameter>

And the current definition, explicit or implicit, is visible in the SCHEDULE column. Empty means never scheduled.

The om node command action executed when the job fires is displayed in the ACTION column.

The node supports the following jobs:

• Node inventoring tasks : pushasset pushpatch pushpkg pushdisks
• Node performance metrics inventoring : pushstats
• Node performance metrics collection : collect_stats
• Node file content tracking task : sysreport
• Node configuration audit and/or remediation task : compliance_auto
• Health checking task : checks
• Scheduled node reboot task : auto_reboot
• Scheduled root password rotation task : auto_rotate_root_pw
• Execution of node actions queued by the collector : dequeue_actions
• SAN switches inventoring tasks : pushbrocade
• Storage arrays inventoring tasks : pushcentera pushdcs pushemcvnx pusheva pushfreenas pushhds pushhp3par pushibmds pushibmsvc pushnecism pushnetapp pushsym pushvioserver
• Backup servers saves index inventoring tasks : pushnsr

Service Scheduler

$ om tflex schedule list
OBJECT  NODE    ACTION           KEY               LAST_RUN_AT                NEXT_RUN_AT                SCHEDULE    
tflex   dev2n1  status           status_schedule   2025-01-30T11:54:55+01:00  2025-01-30T12:04:55+01:00  @10m        
tflex   dev2n1  compliance_auto  comp_schedule     2025-01-27T00:09:18+01:00  0001-01-01T00:00:00Z       ~00:00-06:00
tflex   dev2n1  run              task#1.schedule   2025-01-28T16:27:16+01:00  2025-01-30T16:27:16+01:00  @2d         
tflex   dev2n1  run              task#2.schedule   2025-01-29T16:27:08+01:00  2025-01-30T16:27:08+01:00  @1d         
tflex   dev2n1  run              task#3.schedule   2025-01-29T16:27:08+01:00  2025-01-30T16:27:08+01:00  @1d         
tflex   dev2n1  push_resinfo     resinfo_schedule  2025-01-27T18:56:47+01:00  0001-01-01T00:00:00Z       @60m        

The scheduled jobs can be configured in the service configurations with a configlet like:

[<section>]
<parameter> = <definition>

The KEY column in the above command output is formatted as:

<section>.<parameter>

And the current definition, explicit or implicit, is visible in the SCHEDULE column. Empty means never scheduled.

The om <path> command action executed when the job fires is displayed in the ACTION column.

The supported jobs are:

• Service configuration audit and/or remediation : compliance_auto
• Service resources kvstores inventoring : push_env
• Service status evaluation : status
• Service data sync : sync_all

Advanced Schedule Definition

[!] <timeranges> [<days> [<weeks> [<months>]]]

!
  desc: exclusion pattern. ommiting the ! implies an inclusion

<timeranges> := <timerange>[,<timerange>]
  <timerange> := <begin>:<end>@<interval>
    <begin> <end> := <hour>:<minute>
    <interval>
      type: integer
      unit: minutes

<days> := <day>[-<day>][,<day>[-<day>]]
  <day> := <day_of_week>[:<day_of_month>]
    <day_of_week>
       * iso week day format
         type: integer between 0 and 6
       * literal format
         type: string in ("mon", "tue", "wed", "thu", "fri", "sat",
               "sun", "monday", "tuesday", "wednesday", "thursday",
               "friday", "saturday", "sunday")
    <day_of_month> := <literal> | +<nth> | -<nth> | <nth>
       <nth>
         type: integer
       <literal>
         type: string in ("first", "1st", "second", "2nd", "third",
               "3rd", "fourth", "4th", "fifth", "5th", "last")

<weeks> := <week>[-<week>][,<week>[-<week>]]
  <week>
    type: integer between 1 and 53

<months> := <monthrange>[,<monthrange>]
  <monthrange> := <month>[-<month>] | <month_filter>
    <month>
      * numeric month format
        type: integer between 1 and 12
      * literal format
        type: string in ("jan", "feb", "mar", "apr", "may", "jun",
              "jul", "aug", "sep", "oct", "nov", "dec", "january",
              "february", "march", "april", "may", "june", "july",
              "august", "september", "october", "november",
              "december")
    <month_filter> := %<modulo>[+<shift>]
      <modulo>
        type: integer
      <shift>
        type: integer

Examples

• Never schedule

  Either , or @0

• Always schedule

  *

• Schedule every 60 minutes

  @60

• Schedule at first occasion after 9am

  09:00

• Schedule every hour between midnight and 6am, every day

  00:00-06:00@60

• Schedule once between midnight and 2am, every day

  00:00-02:00

• Schedule once between midnight and 2am every last day of month

  00:00-02:00@121 *:last or 00:00-02:00@121 *:-1

• Schedule once between midnight and 2am every last friday of month

  00:00-02:00@121 fri:last or 00:00-02:00@121 fri:-1

• Schedule once between midnight and 2am every week day

  00:00-02:00@121 mon-fri

• Schedule once between midnight and 2am every week day from january to february

  00:00-02:00@121 mon-fri * jan-feb

• Schedule once between midnight and 2am every odd day (1, 3, 5)

  00:00-02:00@121 *:%2+1

• Schedule once between midnight and 2am every monday of even weeks

  00:00-02:00@121 mon %2

Rosetta Stone

Release Notes

Changelog

KVM High Availability Clustering Using DRBD and OpenSVC on Ubuntu 24.04

Introduction

This document provides a step-by-step guide for deploying a high-availability (HA) KVM cluster using DRBD 9 from Linbit and OpenSVC on Ubuntu 24.04. DRBD provides network-based block replication, effectively acting as shared storage comparable to RAID1 over TCP/IP.

This setup enables:

• Automatic failover of KVM virtual machines when a node becomes unavailable
• Live migration of virtual machines with zero downtime

Prerequisites

This guide uses the following environment:

• Two Ubuntu 24.04 servers: n1 and n2
• Same network segment: 10.30.0.0/24
• IP addressing:
  • n1 → 10.30.0.11
  • n2 → 10.30.0.12
  • kvm1 → 10.30.0.20
  • Gateway → 10.30.0.1
• CPU virtualization extensions enabled (VT-x for Intel, AMD-V for AMD)
• Sudo or root access
• Sufficient disk space available on both nodes

Installation

DRBD

DRBD software is a product from Linbit company. It is shipped by default on Ubuntu distributions as version 8.x. The current guide requires DRBD version 9.x minimum.

Linbit exposes a drbd public ppa repository, where DRBD release candidates are regularly pushed. We will use this repository to install DRBD version 9.

Add Linbit DRBD9 PPA

Let’s add Linbit DRBD9 public ppa to our distributions, and install DRBD.

sudo add-apt-repository ppa:linbit/linbit-drbd9-stack
sudo apt update
sudo apt -y install drbd-dkms drbd-utils
modinfo drbd | grep ^version

⚠️ Warning: The Linbit public PPA is not recommended for production. For production-grade repositories and support, contact Linbit directly.

Verify DRBD version

opensvc@n1:~$ modinfo drbd | grep ^version
version:        9.2.15

Ensure DRBD systemd unit is disabled

DRBD systemd unit must remain disabled on cluster nodes.

⚠️ Warning: OpenSVC must be the only component responsible of DRBD resources management.

opensvc@n1:~$ systemctl status drbd
○ drbd.service - DRBD -- please disable. Instead, use the per resource drbd@.target template unit.
     Loaded: loaded (/usr/lib/systemd/system/drbd.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:drbd.service(7)

System Upgrade

Ensure your operating system is up to date

sudo apt update
sudo apt -y dist-upgrade
sudo reboot

Time Synchronization (NTP)

Ensure all nodes have NTP enabled. Ubuntu defaults are usually sufficient.

opensvc@n1:~$ sudo timedatectl status
               Local time: Fri 2025-10-31 11:52:34 CET
           Universal time: Fri 2025-10-31 10:52:34 UTC
                 RTC time: Fri 2025-10-31 10:52:34
                Time zone: Europe/Paris (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Check server name resolution

It is very important that both nodes are correctly configured from a name service resolution point of view

Check that:

• n1 can resolve n2 to ip addr
• n2 can resolve n1 to ip addr
• both nodes can resolve kvm1 to ip addr

KVM Installation

Verify CPU Virtualization Support

Check that your processors support hardware virtualisation extensions

You must see either vmx or svm from the egrep --color "(vmx|svm)" /proc/cpuinfo command

Install Hypervisor Components

You need to install the following packages to run KVM virtual machines

sudo apt install -y qemu-kvm virtinst virt-manager libguestfs-tools bridge-utils

Once done, ensure that libvirtd systemd unit is enabled and started

sudo systemctl enable --now libvirtd.service

Local Storage Setup

DRBD requires a block device. For this guide, we use an LVM volume group on /dev/vdb.

opensvc@n1:~$ sudo pvcreate -f /dev/vdb
opensvc@n1:~$ sudo vgcreate data /dev/vdb
opensvc@n1:~$ sudo vgs
  VG   #PV #LV #SN Attr   VSize   VFree 
  data   1   0   0 wz--n-  <5.00g <5.00g
  root   1   1   0 wz--n- <26.95g     0 

Setup firewall

If your network enforces filtering, ensure the following flows are allowed:

• n1 <=> 22/tcp <=> n2 ssh communication between nodes
• n1 <=> 1215/tcp <=> n2 api communication between nodes
• n1 <=> 10000/tcp <=> n2 unicast hearbeat communication between nodes
• n1 <=> 7289/tcp <=> n2 DRBD communication between nodes for first DRBD resource
• n1, n2 <=> icmp <=> 10.30.0.0/24 OpenSVC has security checks, one of them uses icmp ping

⚠️ Warning: Each DRBD resource need a dedicated TCP port to communicate with peer node. If you plan to deploy 10 different DRBD resources, then you have to open ports starting from 7289 to 7289+9=7298

Network Bridge Setup

As we prepare a KVM virtual machine deployment, it will require a direct network connection to our subnet 10.30.0.0/24.

A network bridge has to be configured on both cluster nodes.

The netplan configuration is described below

opensvc@n1:~$ sudo cat /etc/netplan/br-prd.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      match:
        macaddress: "22:23:24:1e:00:11"
      set-name: enp1s0
      dhcp4: no
      dhcp6: no
  bridges:
    br-prd:
      interfaces: 
        - enp1s0
      parameters:
        stp: false
        forward-delay: 0
      accept-ra: no
      addresses: 
        - 10.30.0.11/24
      nameservers:
        search:
          - demo.com
        addresses:
          - 8.8.8.8
          - 8.8.4.4
      routes:
        - to: 0.0.0.0/0
          via: 10.30.0.1
          metric: 100

Adjust the bridge configuration to your environment, and then apply it

sudo netplan generate --debug
sudo netplan apply --debug

OpenSVC

Installation

The OpenSVC server installation is described here

Import opensvc gpg signing keys

curl -s -o- https://packages.opensvc.com/gpg.public.key.asc | \
    sudo gpg --dearmor --output /etc/apt/trusted.gpg.d/opensvc-package-pub.gpg --yes

Add the opensvc repository to apt sources

cat - <<EOF | sudo tee /etc/apt/sources.list.d/opensvc.list 
deb https://packages.opensvc.com/apt/ubuntu uat-opensvc-v3-noble main
deb-src https://packages.opensvc.com/apt/ubuntu uat-opensvc-v3-noble main
EOF

Install the opensvc server

sudo apt update
sudo apt -y install opensvc-server

🛈 Info: Although not mandatory, the package opensvc-client should also be installed. It installs the ox command, which is a fancy text user interface tool to manage OpenSVC clusters

Once the installation is done, ensure systemd unit opensvc-server is enabled and started

sudo systemctl enable --now opensvc-server.service

Finally, check that OpenSVC is running fine on both nodes

opensvc@n1:~$ sudo om mon
Threads                              n1           
 daemon                            | O            
 collector                         | /            
 hb                                |              
  hb#1.rx          running unicast | /            
  hb#1.tx          running unicast | /            
 listener                          | 1215         
 scheduler         running         |              
                                                  
Nodes                                n1           
 score                             | 97           
  load15m                          | 0.0          
  mem                              | 20%2.83g<98% 
  swap                             | 0%3.56g<90%  
 state                             |              
                                                  
Objects matching *                   n1           

Set a custom cluster name (optional)

It is a best practise to set a custom cluster name, especially if you plan to deploy multiple different clusters

On node n1:

opensvc@n1:~$ sudo om cluster config update --set cluster.name=drbd4kvm
committed
opensvc@n1:~$ sudo om cluster config get --kw cluster.name
drbd4kvm

Join cluster nodes

We join both nodes from our laptop:

On laptop, ask for a join token, and store it into a variable:

token=$(ssh opensvc@n1 sudo om daemon auth --role join)

and then ask n2 to join n1 using token from variable:

ssh opensvc@$n2 "sudo om cluster join --node $n1 --token $token"

The log should looks like below

john@laptop:~$ token=$(ssh opensvc@n1 sudo om daemon auth --role join)
john@laptop:~$ ssh opensvc@n2 "sudo om cluster join --node n1 --token $token"
Fetch cluster config from n1
Add localhost node to the remote cluster configuration on n1
Daemon join
Cluster nodes updated
Fetch cluster from n1
Fetch system/sec/ca from n1
Fetch system/sec/cert from n1
Fetch system/sec/hb from n1
Draining node
Stop daemon
Dump all configs
Save configs to /var/lib/opensvc/backup/.pre-daemon-join-2025-11-04T10:40:51.json
Install fetched config system/sec/cert
Install fetched config system/sec/hb
Install fetched config cluster
Install fetched config system/sec/ca
Start daemon
Joined

And the om mon command executed on one of the cluster nodes should looks like:

root@n1:~# om mon
Daemon          n1           n2           
 uptime       | 3m23         3m24         
 state        | dns          dns          
 hb queue     | 0            0            
 hb rx        | O            O            
 hb tx        | O            O            
                                          
Nodes           n1           n2           
 uptime       | -            -            
 score        | 95           95           
  load15m     | 0.2          0.2          
  mem         | 30%2.83g<98% 27%2.83g<98% 
  swap        | 1%3.56g<90%  1%3.56g<90%  
 state        |                           
                                          
Objects ~ *     n1           n2           

🛈 Info: We can see that a unicast heartbeat hb rx and hb tx has been autoconfigured. By default, it exchanges information on TCP port 10000.

Hearbeats details can be seen using om daemon hb status command.

root@n1:~# om daemon hb status
RUNNING  BEATING  ID       NODE  PEER  TYPE     DESC         CHANGED_AT                           
O        O        hb#1.rx  n1    n2    unicast  :10000 ← n2  2025-11-28T09:35:08.603796279+01:00  
O        O        hb#1.tx  n1    n2    unicast  → n2:10000   2025-11-28T09:35:08.444365461+01:00  
O        O        hb#1.rx  n2    n1    unicast  :10000 ← n1  2025-11-28T09:35:11.706616707+01:00  
O        O        hb#1.tx  n2    n1    unicast  → n1:10000   2025-11-28T09:35:11.463338726+01:00  

⚠️ Warning: The default heartbeat configuration is ++minimal++. For a real production deployment, it is highly recommended to add other heartbeats kinds (relay, disk, multicast, unicast), and enable quorum arbitration.

Configure SSH trust

Enable automatic SSH trust between nodes (used for internal safety mechanisms).

This configuration is automated using the command below.

om cluster ssh trust

The configuration creates an opensvc dedicated ssh key pair on each node, and trust the public keys on all cluster nodes

root@n1:~# ls -l ~/.ssh/opensvc*
-rw------- 1 root root 452 Nov  4 11:06 /root/.ssh/opensvc
-rw-r--r-- 1 root root 133 Nov  4 11:06 /root/.ssh/opensvc.pub
root@n1:~# cat ~/.ssh/authorized_keys 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMgCrZCehGhMPJK5j1ImjfzEKCaoSTiUBweLHteT8Hxd opensvc@n1 sshkey=opensvc 2025-11-04T11:06:31+01:00
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKRBR+niCfqJPDINuIeZjQo17FhYK0FpQMutb3474p48 opensvc@n2 sshkey=opensvc 2025-11-04T11:06:31+01:00

Test:

root@n1:~# ssh -i /root/.ssh/opensvc n2 hostname

Configure DRBD storage pool

To make smooth DRBD configuration, we declare a DRBD storage pool in the cluster. The pool will be backed by the LVM volume group named data, which was created previously.

root@n1:~# om cluster config update --set pool#drbdvm.type=drbd --set pool#drbdvm.vg=data
committed

root@n1:~# om cluster config show --section pool#drbdvm
[pool#drbdvm]
type = drbd
vg = data

Once done, we can query the cluster about free space in the pool

root@n1:~# om pool list --name drbdvm
NAME    TYPE  CAPABILITIES                     HEAD  VOLUME_COUNT  BIN_SIZE  BIN_USED  BIN_FREE  
drbdvm  drbd  rox,rwx,roo,rwo,snap,blk,shared  data  0             10g       0         10g       

By default the pool is cluster wide, so the reported space is aggregated accross nodes. To see local space, we have to specify the target node

root@n1:~# om pool list --name drbdvm --node n1
NODE  NAME    TYPE  CAPABILITIES                     HEAD  VOLUME_COUNT  BIN_SIZE  BIN_USED  BIN_FREE  
n1    drbdvm  drbd  rox,rwx,roo,rwo,snap,blk,shared  data  0             5g        0         5g        

Configure OpenSVC service

We first deal with the DRBD storage part, and then with VM provisioning

Create service: storage part

First we create a service named kvm1 in the demo namespace, with automated orchestration, and accross all cluster nodes

root@n1:~# om demo/svc/kvm1 create --kw nodes={clusternodes} --kw orchestrate=ha

Then we add storage resources to the service

om demo/svc/kvm1 config update --set volume#data.name={name} --set volume#data.shared=true --set volume#data.size=4G --set volume#data.format=false --set volume#data.pool=drbdvm

We have the following config at the moment

root@n1:~# om demo/svc/kvm1 config show
[DEFAULT]
nodes = {clusternodes}
orchestrate = ha
id = a6867a1e-e8b3-46eb-919f-78d4c60ca9c9

[volume#data]
name = {name}
shared = true
size = 4G
format = false
pool = drbdvm

🛈 Info: The volume#data config snippet will request a storage device to the cluster, which will provision it into the drbd pool because of the shared=true parameter. In case you have many pools in the cluster config, it is possible to add the pool=drbdkvm parameter to force the target pool used for volume creation.

Now provision the storage part

root@n1:~# om demo/svc/kvm1 provision --wait
OBJECT         ORCHESTRATION_ID                      ERROR  
demo/svc/kvm1  e332c71f-1f6b-4aed-b7fe-345db3fb00a9  -      

During the DRBD device initial synchronisation, we see the transitional states below:

root@n1:~# drbdadm status
kvm1.demo.vol.drbd4kvm role:Primary
  disk:UpToDate open:no
  n2 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:30.84

root@n1:~# om demo/svc/kvm1 instance status -r
demo/svc/kvm1                    up    warn                                     
└ instances            
  ├ n2                           down  idle                                     
  └ n1                           up    warn idle started                        
    └ resources                                                                 
      └ volume#data    ........  up    kvm1                                     
                                       warn: Volume demo/vol/kvm1 has warnings  

Once the synchronisation is finished, the states looks like:

root@n1:~# drbdadm status
kvm1.demo.vol.drbd4kvm role:Primary
  disk:UpToDate open:no
  n2 role:Secondary
    peer-disk:UpToDate

root@n1:~# om demo/svc/kvm1 instance status -r
demo/svc/kvm1                    up                  
└ instances            
  ├ n2                           down  idle          
  └ n1                           up    idle started  
    └ resources                                      
      └ volume#data    ........  up    kvm1          

The DRBD volume creation is fully automated. The DRBD configuration is located in /etc/drbd.d/kvm1.demo.vol.drbd4kvm.res

root@n1:~# cat /etc/drbd.d/kvm1.demo.vol.drbd4kvm.res
resource kvm1.demo.vol.drbd4kvm {
    
    on n1 {
        device    /dev/drbd0;
        disk      /dev/data/kvm1.demo.vol.drbd4kvm;
        meta-disk internal;
        address   ipv4 10.30.0.11:7289;
        node-id   0;
    }
    
    on n2 {
        device    /dev/drbd0;
        disk      /dev/data/kvm1.demo.vol.drbd4kvm;
        meta-disk internal;
        address   ipv4 10.30.0.12:7289;
        node-id   1;
    }
    
    connection-mesh {
        hosts n1 n2;
    }
}

The command below displays the devices involved in the service. We can see that the resulting exposed DRBD device is /dev/drbd0

root@n1:~# om demo/svc/kvm1 instance device list
OBJECT         RESOURCE     DRIVER_GROUP  DRIVER_NAME  ROLE     DEVICE                            
demo/svc/kvm1  volume#data  volume                     exposed  /dev/drbd0
demo/svc/kvm1  volume#data  volume                     sub      /dev/data/kvm1.demo.vol.drbd4kvm
demo/svc/kvm1  volume#data  volume                     base     /dev/data/kvm1.demo.vol.drbd4kvm

Create service: vm part

Now that the DRBD block device is ready, we can focus on VM deployment. 4 steps are needed:

1. Download a qcow2 image and customize it
2. Dump image content into DRBD device
3. Create the kvm guest
4. Declare the kvm resource in OpenSVC service

Download a qcow2 image and customize it

We start by downloading a VM image, Ubuntu 25.10 (Questing Quokka) for the demo. We store it in /var/lib/libvirt/images/questing.kvm1.img.

curl -L -o /var/lib/libvirt/images/questing.kvm1.img https://cloud-images.ubuntu.com/questing/current/questing-server-cloudimg-amd64.img

For customization purposes, the virt_customize command brings the following features

• set root password to drbdrocks
• set hostname to kvm1
• copy file enp1s0.yaml into image file at /etc/netplan/enp1s0.yaml

First create the file enp1s0.yaml to configure network at boot

cat >| enp1s0.yaml <<EOF
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
      addresses: 
        - 10.30.0.95/24
      routes:
        - to: 0.0.0.0/0
          via: 10.30.0.1
          metric: 100
EOF

Then actually customize the image file

root@n1:~# virt-customize -a /var/lib/libvirt/images/questing.kvm1.img \
--root-password password:drbdrocks \
--hostname kvm1 \
--copy-in enp1s0.yaml:/etc/netplan

🛈 Info: Try with --no-network if virt-customize fails.

Dump image content into DRBD device

Let’s convert the qcow2 image to raw format, and write it into the DRBD device

IMAGE_LOC=/var/lib/libvirt/images/questing.kvm1.img
DRBD_DEV=/dev/drbd0
qemu-img convert -f qcow2 -O raw $IMAGE_LOC $DRBD_DEV

Create the KVM guest

Initiate the virt-install command to instantiate and boot the vm

OS_VARIANT=ubuntu25.10
VM_BRIDGE=br-prd
virt-install \
--name kvm1 \
--ram 2048 \
--vcpus 2 \
--cpu=host \
--disk path=$DRBD_DEV \
--os-variant $OS_VARIANT \
--network bridge:$VM_BRIDGE,model=virtio \
--graphics none \
--audio none \
--console pty,target_type=serial \
--boot hd \
--noautoconsole \
--import

🛈 Info: The OS_VARIANT value can be identified using command virt-install --osinfo list. The VM_BRIDGE value corresponds to the bridge that we created earlier in the demo.

The virtual machine boots in the background. The console is available using command virsh console kvm1, and can be exited with control+]. If you have an X server available, it is possible to use the graphical interface using the command virt-manager.

After a few seconds, the virtual machine is available

root@n1:~# ping -c3 kvm1
PING kvm1 (10.30.0.20) 56(84) bytes of data.
64 bytes from kvm1 (10.30.0.20): icmp_seq=1 ttl=64 time=0.255 ms
64 bytes from kvm1 (10.30.0.20): icmp_seq=2 ttl=64 time=0.794 ms
64 bytes from kvm1 (10.30.0.20): icmp_seq=3 ttl=64 time=0.545 ms

--- kvm1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2021ms
rtt min/avg/max/mdev = 0.255/0.531/0.794/0.220 ms

Declare the kvm resource in OpenSVC service

Once the virtual machine is operational, we have to add it to the OpenSVC service configuration, as a new resource:

root@n1:~# om demo/svc/kvm1 config update --set container#1.type=kvm --set container#1.name=kvm1
committed

root@n1:~# om demo/svc/kvm1 instance status -r
demo/svc/kvm1                    up                          
└ instances            
  ├ n2                           down  mix-provisioned idle  
  └ n1                           up    idle started          
    └ resources                                              
      ├ volume#data    ........  up    kvm1                  
      └ container#1    ........  up    kvm kvm1              

⚠️ Warning: As the VM provisioning was done manually on n1, the mix-provisioned state appear on the n2 instance. We have to inform n2 OpenSVC agent that the virtual machine resource is actually provisioned.

root@n1:~# om demo/svc/kvm1 instance provision --state-only --node n2
OBJECT         NODE  SID                                   
demo/svc/kvm1  n2    d24fb390-8d62-4fa8-80cb-77da3980778c  

root@n1:~# om demo/svc/kvm1 print status -r
demo/svc/kvm1                    up                  
└ instances            
  ├ n2                           down  idle          
  └ n1                           up    idle started  
    └ resources                                      
      ├ volume#data    ........  up    kvm1          
      └ container#1    ........  up    kvm kvm1      

The service state is now normal on both nodes.

Tests

Test OpenSVC service

It is now possible to test service failover between nodes. Two methods are possible:

• VM switch with downtime
• VM switch with kvm live migration (no downtime)

VM switch with downtime

The switch action can be submitted with the command om demo/svc/kvm1 switch

The following steps are orchestrated:

• n1: stop the VM
• n1: bring down drbd device (secondary)
• n2: bring up drbd device (primary)
• n2: start the vm

root@n1:~# om demo/svc/kvm1 instance ls
OBJECT         NODE  AVAIL  
demo/svc/kvm1  n1    up   
demo/svc/kvm1  n2    down 

root@n1:~# om demo/svc/kvm1 switch --wait
OBJECT         ORCHESTRATION_ID                      ERROR  
demo/svc/kvm1  983c32b5-96c4-4930-8eb2-670c5fae0cc8  -      

root@n1:~# om demo/svc/kvm1 instance ls
OBJECT         NODE  AVAIL  
demo/svc/kvm1  n1    down   
demo/svc/kvm1  n2    up 

🛈 Info: This process can take several seconds or even minutes depending on kvm domain use.

VM switch with kvm live migration (no downtime)

The switch action can be submitted with the command om demo/svc/kvm1 switch --live

The following steps are orchestrated:

• n1: enable drbd dual primary
• n1: launch virsh migrate –live –persistent
• n1: disable drbd dual primary

root@n1:~# om demo/svc/kvm1 instance ls
OBJECT         NODE  AVAIL  
demo/svc/kvm1  n1    up   
demo/svc/kvm1  n2    down 

root@n1:~# om demo/svc/kvm1 switch --live --wait
OBJECT         ORCHESTRATION_ID                      ERROR  
demo/svc/kvm1  c898ddc6-f827-48b5-8686-21ddf500e56e  -      

root@n1:~# om demo/svc/kvm1 instance ls
OBJECT         NODE  AVAIL  
demo/svc/kvm1  n1    down   
demo/svc/kvm1  n2    up 

🛈 Info: This process is transparent, without any network disruption.

arbitrator

Minimal configlet:

[arbitrator]
uri = http://www.opensvc.com

Minimal setup command:

om node set --kw="uri=http://www.opensvc.com"

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Set to true to disable the arbitrator SSL certificate verification on the https uri.

This should only be enabled for testing.

uri

required:    true
scopable:    false

Example:

uri=http://www.opensvc.com

The arbitrator uri used by cluster node to ask for a vote when the cluster is split.

When the uri scheme is http or https, the vote checker is based on a GET request, else it is based on a TCP connect.

For backward compatibility, when the port is not specified in a TCP connect uri, the 1214 port is implied.

Arbitrators are tried in sequence, each reachable arbitrator gives a vote.

In case of a real split, all arbitrators are expected to be unreachable from the lost segment. At least one of them is expected to be reachable from the surviving segment.

Arbitrators of a cluster must thus be located close enough to each other, so a subset of arbitrators can’t be reachable from a split cluster segment, while another subset of arbitrators is reachable from the other split cluster segment.

But not close enough so they can all fail together. Usually, this can be interpreted as: same site, not same rack and power lines.

Arbitrators are verified every 60s to alert admins of the arbitrator failures.

weight

required:    false
scopable:    false
default:     1
convert:     int

During a quorum vote, this reachable arbitrator contributes votes.

array.centera

Minimal configlet:

[array#1]
type = centera
java_bin = /opt/java/bin/java
jcass_dir = /opt/centera/LIB
password = from system/sec/array1 key password
server = centera1
username = root

Minimal setup command:

om node set \
	--kw="type=centera" \
	--kw="java_bin=/opt/java/bin/java" \
	--kw="jcass_dir=/opt/centera/LIB" \
	--kw="password=from system/sec/array1 key password" \
	--kw="server=centera1" \
	--kw="username=root"

java_bin

required:    true
scopable:    false

Example:

java_bin=/opt/java/bin/java

The path to the java executable to use to run the Centera management program.

jcass_dir

required:    true
scopable:    false

Example:

jcass_dir=/opt/centera/LIB

The path of the directory hosting the JCASScript.jar.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

server

required:    true
scopable:    false

Example:

server=centera1

The storage server to connect.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.dorado

Minimal configlet:

[array#1]
type = dorado
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=dorado" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

name

required:    false
scopable:    false

Example:

name=a09

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.emcvnx

Minimal configlet:

[array#1]
type = emcvnx
spa = array1-a
spb = array1-b

Minimal setup command:

om node set \
	--kw="type=emcvnx" \
	--kw="spa=array1-a" \
	--kw="spb=array1-b"

method

required:    false
scopable:    false
candidates:  secfile, credentials
default:     secfile

Example:

method=secfile

The authentication method to use.

password

required:    false
scopable:    false

Example:

password=system/sec/array1

The password to use to log in, expressed as a datastore reference:

    from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

scope

required:    false
scopable:    false
default:     0

Example:

scope=1

The VNC scope to work in.

spa

required:    true
scopable:    false

Example:

spa=array1-a

The name of the Service Processor A.

spb

required:    true
scopable:    false

Example:

spb=array1-b

The name of the Service Processor B.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.eva

Minimal configlet:

[array#1]
type = eva
manager = evamanager.mycorp
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=eva" \
	--kw="manager=evamanager.mycorp" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

bin

required:    false
scopable:    false

Example:

bin=/opt/sssu/bin/sssu

The EVA manager executable to use.

manager

required:    true
scopable:    false

Example:

manager=evamanager.mycorp

The EVA manager to connect.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.freenas

Minimal configlet:

[array#1]
type = freenas
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=freenas" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

schedule

required:    false
scopable:    false

Schedule parameter for the pusharray node action.

See usr/share/doc/schedule for the schedule syntax.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.hds

Minimal configlet:

[array#1]
type = hds
password = from system/sec/array1 key password
url = https://hdsmanager/
username = root

Minimal setup command:

om node set \
	--kw="type=hds" \
	--kw="password=from system/sec/array1 key password" \
	--kw="url=https://hdsmanager/" \
	--kw="username=root"

bin

required:    false
scopable:    false

Example:

bin=/opt/hds/bin/HiCommandCLI

The HDS manager executable to use.

jre_path

required:    false
scopable:    false

Example:

jre_path=/opt/java

The path hosting the java installation to use to execute the HiCommandCLI.

name

required:    false
scopable:    false

Example:

name=HUSVM.1234

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

url

required:    true
scopable:    false

Example:

url=https://hdsmanager/

The url passed to HiCommandCli, pointing the manager in charge of the array.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.hoc

Minimal configlet:

[array#1]
type = hoc
api = https://array.opensvc.com/api/v1.0
model = VSP G350
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=hoc" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="model=VSP G350" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

delay

required:    false
scopable:    false
default:     10s
convert:     duration

The delay between request attempts on retryable errors.

http_proxy

required:    false
scopable:    false

Example:

http_proxy=http://proxy.mycorp:3158

The proxy server to use for http requests to the api.

https_proxy

required:    false
scopable:    false

Example:

https_proxy=https://proxy.mycorp:3158

The proxy server to use for https requests to the api.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

model

required:    true
scopable:    false
candidates:  VSP G370, VSP G700, VSP G900, VSP F370, VSP F700, VSP F900, VSP G350, VSP F350, VSP G800, VSP F800, VSP G400, VSP G600, VSP F400, VSP F600, VSP G200, VSP G1000, VSP G1500, VSP F1500, Virtual Storage Platform, HUS VM

Example:

model=VSP G350

The array model.

name

required:    false
scopable:    false

Example:

name=a09

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

retry

required:    false
scopable:    false
default:     30
convert:     int

The number of request attempts on retryable errors.

schedule

required:    false
scopable:    false

Schedule parameter for the pusharray node action.

See usr/share/doc/schedule for the schedule syntax.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

wwid_prefix

required:    false
scopable:    false

Hitachi APIs do not report the disks NAA wwids, but it can be forged from a array-specifix prefix concatenated with the LDev id. This keyword allow the cluster admin to define this prefix. Do not include the NAA Type digit prefix (define 62400000ec12ac73541d instead of 362400000ec12ac73541d).

array.hp3par

Minimal configlet:

[array#1]
type = hp3par

Minimal setup command:

om node set --kw="type=hp3par"

cli

required:    false
scopable:    false
default:     3parcli

Example:

cli=/path/to/pwf

The path of the executable hp3par CLI.

key

required:    false
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

manager

required:    false
scopable:    false
default:     The name of the array

Example:

manager=mymanager.mycorp

The array manager host name.

method

required:    false
scopable:    false
candidates:  proxy, cli, ssh
default:     ssh

Example:

method=ssh

The connection method to use.

pwf

required:    false
scopable:    false

Example:

pwf=/path/to/pwf

The path to the 3par password file to use to log in.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.ibmds

Minimal configlet:

[array#1]
type = ibmds
hmc1 = hmc1.mycorp
hmc2 = hmc2.mycorp
username = root

Minimal setup command:

om node set \
	--kw="type=ibmds" \
	--kw="hmc1=hmc1.mycorp" \
	--kw="hmc2=hmc2.mycorp" \
	--kw="username=root"

hmc1

required:    true
scopable:    false

Example:

hmc1=hmc1.mycorp

The host name of the primary HMC.

hmc2

required:    true
scopable:    false

Example:

hmc2=hmc2.mycorp

The host name of the secondary HMC.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.ibmsvc

Minimal configlet:

[array#1]
type = ibmsvc
key = /path/to/key
username = root

Minimal setup command:

om node set \
	--kw="type=ibmsvc" \
	--kw="key=/path/to/key" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.netapp

Minimal configlet:

[array#1]
type = netapp
key = /path/to/key
server = centera1
username = root

Minimal setup command:

om node set \
	--kw="type=netapp" \
	--kw="key=/path/to/key" \
	--kw="server=centera1" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

server

required:    true
scopable:    false

Example:

server=centera1

The storage server to connect.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.nexenta

Minimal configlet:

[array#1]
type = nexenta
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=nexenta" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

port

required:    false
scopable:    false
default:     2000
convert:     int

Example:

port=2000

The nexenta administration listener port.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.pure

Minimal configlet:

[array#1]
type = pure
api = https://array.opensvc.com/api/v1.0
client_id = bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7
issuer = opensvc
key_id = df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f
secret = system/sec/array1
username = opensvc

Minimal setup command:

om node set \
	--kw="type=pure" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="client_id=bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7" \
	--kw="issuer=opensvc" \
	--kw="key_id=df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f" \
	--kw="secret=system/sec/array1" \
	--kw="username=opensvc"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

client_id

required:    true
scopable:    false

Example:

client_id=bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7

The client id to use as the aud key in the payload of the login jwt.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

issuer

required:    true
scopable:    false

Example:

issuer=opensvc

The issuer to use as the iss key in the payload of the login jwt token.

key_id

required:    true
scopable:    false

Example:

key_id=df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f

The key id to use as the kid key in the header of the login jwt.

schedule

required:    false
scopable:    false

Schedule parameter for the pusharray node action.

See usr/share/doc/schedule for the schedule syntax.

secret

required:    true
scopable:    false

Example:

secret=system/sec/array1

The secret to use to store the information required to create the login jwt, expressed as a reference to a secret. The secret must be in the system namespace and must have the following keys: private_key.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=opensvc

The username to use as the sub key in the payload of the login jwt.

array.symmetrix

Minimal configlet:

[array#1]
type = symmetrix

Minimal setup command:

om node set --kw="type=symmetrix"

name

required:    false
scopable:    false

Example:

name=00012345

The name of the array. If not provided, fallback to the section name suffix.

password

required:    false
scopable:    false

Example:

password=system/sec/array1

The password to use to log in, expressed as a datastore reference:

    from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

schedule

required:    false
scopable:    false

Schedule parameter for the pusharray node action.

See usr/share/doc/schedule for the schedule syntax.

symcli_connect

required:    false
scopable:    false

Example:

symcli_connect=MY_SYMAPI_SERVER

Set the SYMCLI_CONNECT environment variable to this value.

If not set, the SCSI communication channels are used.

The value set must be declared in the /var/symapi/config/netcnfg file.

symcli_path

required:    false
scopable:    false
default:     /usr/symcli

Example:

symcli_path=/opt/symcli

Force use of a symcli programs installation, pointing the path of its head directory.

For the case multiple symcli versions are installed and the default selector does not select the version preferred for the array.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.truenas

Minimal configlet:

[array#1]
type = truenas
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=truenas" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

schedule

required:    false
scopable:    false

Schedule parameter for the pusharray node action.

See usr/share/doc/schedule for the schedule syntax.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.vioserver

Minimal configlet:

[array#1]
type = vioserver
key = /path/to/key
username = root

Minimal setup command:

om node set \
	--kw="type=vioserver" \
	--kw="key=/path/to/key" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.xtremio

Minimal configlet:

[array#1]
type = xtremio
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om node set \
	--kw="type=xtremio" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

name

required:    false
scopable:    false

Example:

name=array1

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

asset

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushasset node action.

See usr/share/doc/schedule for the schedule syntax.

checks

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushchecks node action.

See usr/share/doc/schedule for the schedule syntax.

cluster

ca

required:    false
scopable:    false
default:     `system/sec/ca`

convert:     list

A whitespace-separated list of sec paths.

The listener accepts a x509 client certificate if it is trusted by any CA certificate found in these sec objects.

cert

required:    false
scopable:    false
default:     `system/sec/cert`

The path of the secret hosting the certificate that the listener use for its TLS socket.

dns

required:    false
scopable:    true
convert:     list

The list of nodes to set as nameserver in the resolv.conf of the containers the CRM starts.

If set, the search will also be set to:

1/ <name>.<namespace>.svc.<clustername> 2/ <namespace>.svc.<clustername> 3/ <clustername>.

drpnodes

required:    false
scopable:    false
convert:     list

This list is fetched from the join command payload received from the joined node.

The service configuration {clusterdrpnodes} is resolved to this keyword value.

envs

required:    false
scopable:    false
default:     CERT DEV DRP FOR INT PRA PRD PRJ PPRD QUAL REC STG TMP TST UAT
convert:     list

id

required:    false
scopable:    true
default:     An autogenerated random UUID.

This unique identifier is auto-generated on install and should never be change by the cluster administrators.

It is changed when the node joins a cluster, so the remote cluster id replaces the joiners’ cluster id.

name

required:    false
scopable:    false
default:     A random generated clustername.

The cluster name is used,

• as the zone name in the cluster dns records
• in the {fqdn} configuration reference
• in the AES secret encryption metadata

The cluster name should be unique site-wide. Missing cluster name will be automatically created with random value during daemon startup.

It is always lowercased, so better to set it to a lowercase value to avoid confusion.

The cluster name is provided to joining nodes, so they can replace their own.

nodes

required:    false
scopable:    false
convert:     list

This list of node names contains only the local node name on install.

When the node joins a cluster, the joined node provides the new list, with the new node added. The joiner then replace its nodes list with the one received.

When a node receives a join request, it adds the new node to its cluster nodes list, then provide the new list to the joiner.

quorum

required:    false
scopable:    false
default:     false
convert:     bool

If true, when the cluster is split a vote happens on each cluster node.

Each reachable node and each reachable arbitrator give their vote. If the votes is less than half the total number of nodes plus arbitrators, the node trigger a node fencing method defined by node.split_action (crash, reboot or disabled).

secret

required:    false
scopable:    true
default:     A random string autogenerated on first use

The cluster shared secret used to encrypt and decrypt heartbeat payloads and sec values, with AES256

This secret is auto-generated on install, then merged from the joined nodes when joining a cluster.

The cluster name should be unique site-wide and be set right before starting to add sec keys.

cni

config

required:    false
scopable:    false
default:     /var/lib/opensvc/cni/net.d

Example:

config=/var/lib/opensvc/cni/net.d

The directory hosting the CNI network configuration files.

plugins

required:    false
scopable:    false
default:     /usr/lib/cni

Example:

plugins=/var/lib/opensvc/cni/bin

The directory hosting the CNI plugins.

compliance

auto_update

required:    false
scopable:    false
default:     false
convert:     bool

If set to true, execute om node updatecomp upon every scheduler-executed om node compliance check.

These updates keep the compliance modules in sync with the reference repository.

Warning: the module repository security is critical. Attackers could insert malicious code in served modules.

schedule

required:    false
scopable:    false
default:     02:00-06:00

Schedule parameter for the compliance auto node action, which check all attached modules and fix only those flagged autofix.

See usr/share/doc/schedule for the schedule syntax.

console

insecure

required:    false
scopable:    false
convert:     bool

If set, don’t verify the console server certificate.

max_greet_timeout

required:    false
scopable:    false
default:     20s
convert:     duration

This keyword sets the absolute upper limit (maximum duration) that an API user can request via the greet_timeout query parameter when calling the handler to start a service instance resource console.

The console handler accepts a greet_timeout query parameter, which specifies how long the service instance should wait for the first client connection to the generated console URL.

The max_greet_timeout acts as a security safeguard, preventing API users from setting excessively long greet_timeout values.

If max_greet_timeout is set to 30s, any user request for greet_timeout=60s will be refused.

Recommendation:

• Keep Short: The effective timeout should generally be kept to a few seconds (e.g., 5s to 10s). A shorter timeout minimizes the risk that a malicious actor could successfully guess the console URL through a brute-force attack before the link expires.

• Manual Testing: Users may occasionally need to set a slightly higher greet_timeout (e.g., 60s) for manual testing or debugging purposes, but this value will always be constrained by the max_greet_timeout defined here.

max_seats

required:    false
scopable:    false
default:     1
convert:     int

This keyword sets the absolute upper limit that an API user can request via the seats query parameter when calling the handler to start a service instance resource console.

server

required:    false
scopable:    false

The tty-proxy console server address. A TLS capable server is expected, so the value should be a TLS terminator reverse proxy.

dequeue_actions

schedule

required:    false
scopable:    false

Schedule parameter for the dequeue actions node action.

See usr/share/doc/schedule for the schedule syntax.

disks

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushdisks node action.

See usr/share/doc/schedule for the schedule syntax.

hb.disk

Minimal configlet:

[hb#1]
type = disk
dev = /dev/mapper/36589cfc000000e03957c51dabab8373a

Minimal setup command:

om node set \
	--kw="type=disk" \
	--kw="dev=/dev/mapper/36589cfc000000e03957c51dabab8373a"

dev

required:    true
scopable:    true

Example:

dev=/dev/mapper/36589cfc000000e03957c51dabab8373a

The device to write the heartbeats to and read from.

It must be,

• Dedicated to the daemon use.
• Sized 1MB for metadata + 1MB/node.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

max_slots

required:    false
scopable:    false
default:     1024
convert:     int

Example:

max_slots=1024

The maximum number of slots that can be written to the HB disk device.

It should be set to at least twice the number of cluster nodes using the HB disk device.

Do not modify this value after first activation. Do not modify default value if the hb disk is used by multiple clusters.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hb.multicast

Minimal configlet:

[hb#1]
type = multicast

Minimal setup command:

om node set --kw="type=multicast"

addr

required:    false
scopable:    true
default:     224.3.29.71

The multicast address to send to and listen on.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

intf

required:    false
scopable:    true
default:     The natural interface for `<addr>`

Example:

intf=eth0

The interface to bind.

port

required:    false
scopable:    true
default:     10000
convert:     int

The port for each node to send to or listen on.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hb.relay

Minimal configlet:

[hb#1]
type = relay
relay = relaynode1

Minimal setup command:

om node set \
	--kw="type=relay" \
	--kw="relay=relaynode1"

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Set to true to disable the relay SSL certificate verification.

This should only be enabled for testing.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

password

required:    false
scopable:    false
default:     system/sec/relay

The name of a sec object containing a password key, which value is used as password for log in the relay api.

relay

required:    true
scopable:    false

Example:

relay=relaynode1

The relay resolvable node name.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

username

required:    false
scopable:    false
default:     relay

The username for login the relay api.

hb.unicast

Minimal configlet:

[hb#1]
type = unicast

Minimal setup command:

om node set --kw="type=unicast"

addr

required:    false
scopable:    true
default:     0.0.0.0 for listening and to the resolved nodename for sending.

Example:

addr=1.2.3.4

Use the scoping syntax to assign non-default ip addresses nodes.

Warning: Using the unscoped syntax would assign the same address to all nodes and the heartbeat would go down.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

intf

required:    false
scopable:    true
default:     The natural interface for `<addr>`

Example:

intf=eth0

The interface to bind.

nodes

required:    false
scopable:    true
default:     All nodes.

convert:     list

The nodes participating to the heartbeat.

This keyword can be used to setup a partial redundancy like:

       n1   n2   n3   n4
hb#1   O    O    O    O
hb#2   O    O          
hb#1             O    O

Which can be relevant if n[12] are in the same bladecenter b1, and n[34] are in the same bladecenter b2. The bladecenter having an internal network completely hardware independent of the network used to go outside of the bladecenters.

port

required:    false
scopable:    true
default:     10000
convert:     int

The port for each node to send to or listen on.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hook

command

required:    false
scopable:    false
convert:     shlex

The command to execute on selected events.

The program is fed the json-formatted event data through stdin.

events

required:    false
scopable:    false
convert:     list

The list of events to execute the hook command on.

The special value all is also supported.

listener

addr

required:    false
scopable:    true

Example:

addr=1.2.3.4

The ip addr the daemon tls listener must listen on.

crl

required:    false
scopable:    false
default:     /var/lib/opensvc/certs/ca_crl

Example:

crl=https://crl.opensvc.com

The URL serving the certificate revocation list.

The default points to the path of the cluster CA CRL in {var}/certs/ca_crl.

dns_sock_gid

required:    false
scopable:    false
default:     953

The gid owning the unix socket serving the remote backend to the pdns authoritative server.

dns_sock_uid

required:    false
scopable:    false
default:     953

The uid owning the unix socket serving the remote backend to the pdns authoritative server.

openid_client_id

required:    false
scopable:    false
default:     om3

The openid client id used by om3-webapp.

openid_issuer

required:    false
scopable:    false

Example:

openid_issuer=https://keycloak.opensvc.com/auth/realms/clusters

The base URL of the identity issuer aka provider. It is used to detect the metadata location: openid_issuer/.well-known/openid-configuration.

If set, the http listener will try to validate the Bearer token provided in the requests headers.

If the token is valid,

• the user name is fetched from the preferred_username claim (fallback on name)

• the user grant list is obtained by joining the multiple entitlements claims.

The keyword replaced deprecated openid_well_known.

port

required:    false
scopable:    true
default:     1215
convert:     int

The port the daemon tls listener must listen on.

In pull action mode, the collector post request to notify there are actions to unqueue. The opensvc daemon executes the dequeue actions node action upon receive.

The listener.port value is sent to the collector on pushasset.

rate_limiter_burst

required:    false
scopable:    true
default:     100
convert:     int

The maximum number of inet listener requests to pass at the same moment. It additionally allows a number of requests to pass when rate limit is reached.

rate_limiter_expires

required:    false
scopable:    true
default:     60s
convert:     duration

The duration after that a inet listener rate limiter is cleaned up.

rate_limiter_rate

required:    false
scopable:    true
default:     20
convert:     int

The rate of inet listener requests allowed to pass per seconds.

network.bridge

Minimal configlet:

[network#1]
type = bridge

Minimal setup command:

om node set --kw="type=bridge"

dev

required:    false
scopable:    false

The network bridge device name. If not set the name will be obr_<network name>. Use this keyword if you need the network to use an already existing bridge.

mask_per_node

required:    false
scopable:    false
default:     0
convert:     int

The prefix length of the subnets distributed to each node. For example if the network is a x.x.x.x/16 you can distribute

• x.x.x.x/17 subnets to 2 nodes
• x.x.x.x/18 to 4 nodes
• etc…

If both mask_per_node and ips_per_node are set, ips_per_node is ignored. If only ips_per_node is set, it is honored for backward compatibility.

The ips_per_node keyword is deprecated because its value is hard to manage for large ipv6 subnets (e.g a x.x.x.x/48 subnet has 281474976710656 ips).

network

required:    false
scopable:    true

The cluster backend network.

The routed_bridge driver fragments this network into subnets with a prefix length given bymask_per_nodes.

public

required:    false
scopable:    false
convert:     bool

Set to true if the network ip range is public and we must not configure masquerading rules.

type

required:    false
scopable:    false
candidates:  bridge, routed_bridge
default:     bridge

The type of network.

network.lo

Minimal configlet:

[network#1]
type = lo

Minimal setup command:

om node set --kw="type=lo"

mask_per_node

required:    false
scopable:    false
default:     0
convert:     int

The prefix length of the subnets distributed to each node. For example if the network is a x.x.x.x/16 you can distribute

• x.x.x.x/17 subnets to 2 nodes
• x.x.x.x/18 to 4 nodes
• etc…

If both mask_per_node and ips_per_node are set, ips_per_node is ignored. If only ips_per_node is set, it is honored for backward compatibility.

The ips_per_node keyword is deprecated because its value is hard to manage for large ipv6 subnets (e.g a x.x.x.x/48 subnet has 281474976710656 ips).

type

required:    false
scopable:    false
candidates:  bridge, routed_bridge
default:     bridge

The type of network.

network.routed_bridge

Minimal configlet:

[network#1]
type = routed_bridge

Minimal setup command:

om node set --kw="type=routed_bridge"

addr

required:    false
scopable:    true
default:     Detect using a name resolution of `<nodename>`.

Beware, if the nodename resolves to 127.0.1.1 or 127.0.0.1 the ipip tunnel can not work.

The ip address used as local endpoint for the ipip tunnel configured by the network setup command to access the backend subnet of peer nodes not reachable on the same subnet.

dev

required:    false
scopable:    false

The network bridge device name. If not set the name will be obr_<network name>. Use this keyword if you need the network to use an already existing bridge.

gateway

required:    false
scopable:    true

The gateway to use to reach the network segment of the node specified as scope.

ips_per_node

required:    false
scopable:    false
default:     1024
convert:     int

The number of ips each node must be able to allocate in the network. This number is translated into the prefix length of the subnets distributed to each node. For example if the network is a x.x.x.x/16 you can distribute

• x.x.x.x/17 subnets to 2 nodes
• x.x.x.x/18 to 4 nodes
• etc…

If both mask_per_node and ips_per_node are set, ips_per_node is ignored. If only ips_per_node is set, it is honored for backward compatibility.

The ips_per_node keyword is deprecated because its value is hard to manage for large ipv6 subnets (e.g a x.x.x.x/48 subnet has 281474976710656 ips).

mask_per_node

required:    false
scopable:    false
default:     0
convert:     int

The prefix length of the subnets distributed to each node. For example if the network is a x.x.x.x/16 you can distribute

• x.x.x.x/17 subnets to 2 nodes
• x.x.x.x/18 to 4 nodes
• etc…

If both mask_per_node and ips_per_node are set, ips_per_node is ignored. If only ips_per_node is set, it is honored for backward compatibility.

The ips_per_node keyword is deprecated because its value is hard to manage for large ipv6 subnets (e.g a x.x.x.x/48 subnet has 281474976710656 ips).

network

required:    false
scopable:    false

The cluster backend network.

The routed_bridge driver fragments this network into subnets with a prefix length given bymask_per_nodes.

public

required:    false
scopable:    false
convert:     bool

Set to true if the network ip range is public and we must not configure masquerading rules.

subnet

required:    false
scopable:    true

The cidr subnet handled by this node.

This parameter must be scoped for each node.

Usually, the subnets are allocated automatically upon initial network setup, each node being attributed a subnet based on its index in the cluster.nodes list.

tables

required:    false
scopable:    false
default:     main
convert:     list

Example:

tables=main custom1 custom2

The list of routing tables to add the backend network routes to.

The list of available tables is in /etc/iproute2/rt_tables.

tunnel

required:    false
scopable:    false
candidates:  auto, always, never
default:     auto

Create and route traffic through tunnels to peer nodes policy.

• auto

  Tunnel if the peer is not in the same subnet

• always

  Tunnel even if the peer seems to be in the same subnet. Some hosting providers require this as traffic goes through routers even between adjacent nodes.

tunnel_mode

required:    false
scopable:    false
candidates:  gre, ipip, ip6ip6
default:     ipip

The ip tunnel mode. gre can tunnel mcast ip and ipv6 at the price of a 24B header, ipip can only tunnel ipv4 but with a 20B header. Note, some OVH servers combinations don’t support ipip but work with gre.

type

required:    false
scopable:    false
candidates:  bridge, routed_bridge
default:     bridge

The type of network.

node

allowed_networks

required:    false
scopable:    false
default:     10.0.0.0/8 172.16.0.0/24 192.168.0.0/16
convert:     list

The list of cidr blocks the agents allows creation of backend network into.

Should be restricted to match your site constraints.

asset_env

required:    false
scopable:    false

Example:

asset_env=Production

An asset information to push to the collector on pushasset, overriding the currently stored value.

bios_version

required:    false
scopable:    false

Example:

bios_version=1.025

Override for the corresponding pushasset discovery probe.

branch

required:    false
scopable:    false

Example:

branch=1.9

Set the targeted opensvc agent branch.

The downloaded upgrades will honor that branch.

If not set, the repopkg imposes the target branch via the current link.

It is recommended to set branch when repopkg points to a repository you are not responsible for.

connect_to

required:    false
scopable:    false

Example:

connect_to=1.2.3.4

An asset information pushed to the collector on pushasset.

If not set, the collector picks one of the node ip addresses inventoried on pushasset too.

On GCE instances, defaults to the instance ip address.

cpu_cores

required:    false
scopable:    false
convert:     int

Example:

cpu_cores=2

Override for the corresponding pushasset discovery probe.

cpu_dies

required:    false
scopable:    false
convert:     int

Example:

cpu_dies=1

Override for the corresponding pushasset discovery probe.

cpu_freq

required:    false
scopable:    false

Example:

cpu_freq=3.2 Ghz

Override for the corresponding pushasset discovery probe.

cpu_model

required:    false
scopable:    false

Example:

cpu_model=Alpha EV5

Override for the corresponding pushasset discovery probe.

cpu_threads

required:    false
scopable:    false
convert:     int

Example:

cpu_threads=4

Override for the corresponding pushasset discovery probe.

dbcompliance

required:    false
scopable:    false
default:     Same protocol, server and port as `dbopensvc`, but with a different path.

Example:

dbcompliance=https://collector.opensvc.com

Set the uri of the collector’s main rpc server.

The path part of the uri can be left unspecified.

dbinsecure

required:    false
scopable:    false
convert:     bool

Set to true to disable the collector x509 certificate verification.

This should only be used for testing.

dblog

required:    false
scopable:    false
default:     true
convert:     bool

If true and dbopensvc is set, the objects action logs are reported to the collector.

Set to false to disable log reporting to the collector, even if dbopensvc is set.

dbopensvc

required:    false
scopable:    false

Example:

dbopensvc=https://collector.opensvc.com

Set the uri of the collector’s feed rpc server.

The path part of the uri can be left unspecified.

If dbopensvc is not set, the agent does not try to communicate with a collector.

enclosure

required:    false
scopable:    false

Example:

enclosure=1

Override for the corresponding pushasset discovery probe.

env

required:    false
scopable:    false
default:     TST

A code like PRD, DEV, etc… the agent can use to enforce data protection policies:

• A non-PRD object instance can not be started on a PRD node
• A PRD object instance can be started on a non-PRD node (typically in a DRP situation)

loc_addr

required:    false
scopable:    false

Example:

loc_addr=7 rue blanche

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_building

required:    false
scopable:    false

Example:

loc_building=Crystal

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_city

required:    false
scopable:    false

Example:

loc_city=Paris

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_country

required:    false
scopable:    false

Example:

loc_country=fr

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_floor

required:    false
scopable:    false

Example:

loc_floor=21

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_rack

required:    false
scopable:    false

Example:

loc_rack=R42

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_room

required:    false
scopable:    false

Example:

loc_room=102

An asset information to push to the collector on pushasset, overriding the currently stored value.

loc_zip

required:    false
scopable:    false

Example:

loc_zip=75017

An asset information to push to the collector on pushasset, overriding the currently stored value.

maintenance_grace_period

required:    false
scopable:    false
default:     60
convert:     duration

A duration expression, like 1m30s, defining how long the daemon keeps remote node data while it is known to be in maintenance.

The maintenance state is announced to peers at the beginning of a daemon stop and daemon restart, but not on daemon shutdown.

As long as the remote node data is kept, the local daemon won’t takeover the instances running on the node in maintenance.

This parameter should be adjusted to span the daemon restart time.

manufacturer

required:    false
scopable:    false

Example:

manufacturer=Digital

Override for the corresponding pushasset discovery probe.

max_key_size

required:    false
scopable:    false
default:     1mb
convert:     size

max_parallel

required:    false
scopable:    false
default:     10
convert:     int

Allow a maximum of max_parallel CRM commands to run simultaneously.

Applies to both:

• om <selector> <action> commands.
• commands executed by the daemon for orchestrations

mem_banks

required:    false
scopable:    false
convert:     int

Example:

mem_banks=4

Override for the corresponding pushasset discovery probe.

mem_bytes

required:    false
scopable:    false
convert:     size

Example:

mem_bytes=256mb

Override for the corresponding pushasset discovery probe.

mem_slots

required:    false
scopable:    false
convert:     int

Example:

mem_slots=4

Override for the corresponding pushasset discovery probe.

min_avail_mem_pct

required:    false
scopable:    false
default:     2
convert:     int

The minimum required available memory to allow orchestration.

min_avail_swap_pct

required:    false
scopable:    false
default:     10
convert:     int

The minimum required available swap to allow orchestration.

model

required:    false
scopable:    false

Example:

model=ds20e

Override for the corresponding pushasset discovery probe.

oci

required:    false
scopable:    false

The default micro-container driver.

If not set, prefer podman if installed, else fallback to docker.

os_arch

required:    false
scopable:    false

Example:

os_arch=5.1234

Override for the corresponding pushasset discovery probe.

os_kernel

required:    false
scopable:    false

Example:

os_kernel=5.1234

Override for the corresponding pushasset discovery probe.

os_release

required:    false
scopable:    false

Example:

os_release=5

Override for the corresponding pushasset discovery probe.

os_vendor

required:    false
scopable:    false

Example:

os_vendor=Digital

Override for the corresponding pushasset discovery probe.

prkey

required:    false
scopable:    false
default:     Autogenerated on first use.

The scsi3 persistent reservation key used by the pr resources.

ready_period

required:    false
scopable:    false
default:     5s
convert:     duration

A duration expression, like 10s, defining how long the daemon waits before starting a service instance in ready state.

A peer node can preempt the start during this period.

Usually set to allow at least a couple of heartbeats to be received.

rejoin_grace_period

required:    false
scopable:    false
default:     90s
convert:     duration

A duration expression, like 1m30s, defining how long a starting daemon waits in rejoin state.

The daemon normally exits the rejoin state when it has received a heartbeat from all its peer nodes.

During this phase, the orchestration is not allowed, to give a chance to place the services optimally when multiple daemon were restarted at the same time.

But if a peer stays down, the other daemons have to stop waiting at some point to let the service start, even if not on their natural placement leader.

This should be adjusted to:

2s + <longest reboot duration>

The worse case of multiple nodes reboot is when the longest reboot node is rebooted near the end of the reboot of the second longest rebooting node.

|==========>
    n1 reboot
                |--------------------|
                n1 rejoin_grace_period
                |================>
                n1 in rejoin state
                                  |=====================
                                  n1 in idle state
              |==================>
              n2 reboot
                                  |--------------------|
                                  n2 rejoin_grace_period
                                  |=====================
                                  n2 in idle state

As a consequence, to minimize the rejoin_grace_period, prefer fast boot nodes.

repo

required:    false
scopable:    false

Example:

repo=http://opensvc.repo.corp

Set the uri of the opensvc agent package repository and compliance modules gzipped tarball repository.

This parameter is used by the om node updatepkg and om node updatecomp commands.

Expected repository structure:

ROOT +- compliance |+- compliance-100.tar.gz |+- compliance-101.tar.gz |- current -> compliance-101.tar.gz +- packages +- deb +- depot +- pkg +- sunos-pkg +- rpms |+- current -> 2.0/current |+- 1.9 | +- current -> opensvc-1.9-50.rpm | +- opensvc-1.9-49.rpm | - opensvc-1.9-50.rpm |+- 2.0 | +- current -> opensvc-2.0-90.rpm | - opensvc-2.0-90.rpm - tbz

repocomp

required:    false
scopable:    false

Example:

repocomp=http://compliance.repo.corp

Set the uri of the opensvc compliance modules repository.

A gzipped tarball is expected to be found there by the om node updatecomp command.

Expected repository structure:

ROOT
+- compliance-100.tar.gz
+- compliance-101.tar.gz
`- current -> compliance-101.tar.gz

repopkg

required:    false
scopable:    false

Example:

repopkg=http://repo.opensvc.com

Set the uri of the opensvc agent package repository.

This parameter is used by the om node updatepkg command.

Expected repository structure:

ROOT +- deb +- depot +- pkg +- sunos-pkg +- rpms |+- current -> 2.0/current |+- 1.9 | +- current -> opensvc-1.9-50.rpm | +- opensvc-1.9-49.rpm | - opensvc-1.9-50.rpm |+- 2.0 | +- current -> opensvc-2.0-90.rpm | - opensvc-2.0-90.rpm `- tbz

ruser

required:    false
scopable:    false
default:     root

Example:

ruser=root opensvc@node1

Set the remote user to use to login to a remote node with ssh and rsync.

The remote user must have the privileges to run as root the following commands on the remote node:

• om
• rsync

The default ruser is root for all nodes.

ruser accepts a list of user[@node]. If @node is omitted, user is considered the new default user.

sec_zone

required:    false
scopable:    false

Example:

sec_zone=dmz1

An asset information to push to the collector on pushasset, overriding the currently stored value.

secure_fetch

required:    false
scopable:    false
default:     true
convert:     bool

If set to false, disable ssl authentication checks on all uri fetches.

serial

required:    false
scopable:    false

Example:

serial=abcdef0123456

Override for the corresponding pushasset discovery probe.

sp_version

required:    false
scopable:    false

Example:

sp_version=1.026

Override for the corresponding pushasset discovery probe.

split_action

required:    false
scopable:    true
candidates:  crash, reboot, disabled
default:     crash

The node suicide method to use when a cluster split occurs and the node does not have the quorum.

This opting-out is meant to avoid double-start situations when the cluster is split.

Possible values are:

• crash

  Default.

• reboot

  May be preferred when the node power-on is not easy. No remote access via IPMI or equivalent for example.

• disabled

  May be used for test or training only (it does nothing).

sshkey

required:    false
scopable:    false
default:     opensvc

The basename of the ssh public key served by the GET /node/name/:nodename/ssh/key. For example, the opensvc default value serves ~/.ssh/opensvc.pub.

team_integ

required:    false
scopable:    false

Example:

team_integ=TINT

An asset information to push to the collector on pushasset, overriding the currently stored value.

team_support

required:    false
scopable:    false

Example:

team_support=TSUP

An asset information to push to the collector on pushasset, overriding the currently stored value.

tz

required:    false
scopable:    false

Example:

tz=+0200

Override for the corresponding pushasset discovery probe.

uuid

required:    false
scopable:    false

The authentication token provided by the collector on om node register.

packages

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushpkg node action.

See usr/share/doc/schedule for the schedule syntax.

patches

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushpatch node action.

See usr/share/doc/schedule for the schedule syntax.

pool.directory

Minimal configlet:

[pool#1]
type = directory

Minimal setup command:

om node set --kw="type=directory"

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

path

required:    false
scopable:    false
default:     {var}/pool/directory

The fullpath of the directory hosting the pool volumes directories or loop files.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.dorado

Minimal configlet:

[pool#1]
type = dorado
array = 
diskgroup = 

Minimal setup command:

om node set \
	--kw="type=dorado" \
	--kw="array=" \
	--kw="diskgroup="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

compression

required:    false
scopable:    false
default:     false
convert:     bool

Activate compression on created luns.

dedup

required:    false
scopable:    false
default:     false
convert:     bool

Activate data deduplcation on created luns.

diskgroup

required:    true
scopable:    false

The name of the array disk group to allocate volumes from.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

hypermetrodomain

required:    false
scopable:    false

Example:

hypermetrodomain=HyperMetroDomain_000

Create LUN as HyperMetro replicated pairs, using this domain.

pool.drbd

Minimal configlet:

[pool#1]
type = drbd

Minimal setup command:

om node set --kw="type=drbd"

addr

required:    false
scopable:    true
default:     The ipaddr resolved for the nodename.

Example:

addr=1.2.3.4

The addr to use to connect a peer. Use scoping to define each non-default address.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

path

required:    false
scopable:    false

The fullpath of the directory hosting the pool volumes loop files.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

template

required:    false
scopable:    false

Example:

template=live-migration

The value of the template keyword to set in the drbd resource of the created volumes

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

vg

required:    false
scopable:    false

The name of the volume group to allocate the pool volumes logical volumes into.

zpool

required:    false
scopable:    false

The name of the zpool to allocate the pool volumes zvol into.

pool.freenas

Minimal configlet:

[pool#1]
type = freenas
array = 
diskgroup = 

Minimal setup command:

om node set \
	--kw="type=freenas" \
	--kw="array=" \
	--kw="diskgroup="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

blocksize

required:    false
scopable:    false
default:     512
convert:     size

Allow initiators to xcopy without authenticating to foreign targets.

compression

required:    false
scopable:    false
candidates:  inherit, none, lz4, gzip-1, gzip-2, gzip-3, gzip-4, gzip-5, gzip-6, gzip-7, gzip-8, gzip-9, zle, lzjb
default:     inherit

Compression level.

dedup

required:    false
scopable:    false
default:     off

Activate data deduplication on created dataset and zvol. Example values: on, off, verify

diskgroup

required:    true
scopable:    false

The name of the array disk group to allocate volumes from.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

insecure_tpc

required:    false
scopable:    false
default:     false
convert:     bool

Allow initiators to xcopy without authenticating to foreign targets.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

sparse

required:    false
scopable:    false
default:     false
convert:     bool

Create zvol in sparse mode.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.hoc

Minimal configlet:

[pool#1]
type = hoc
array = 
diskgroup = 

Minimal setup command:

om node set \
	--kw="type=hoc" \
	--kw="array=" \
	--kw="diskgroup="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

compression

required:    false
scopable:    false
default:     false
convert:     bool

Activate compression on created luns.

dedup

required:    false
scopable:    false
default:     false
convert:     bool

Activate data deduplcation on created luns.

diskgroup

required:    true
scopable:    false

The name of the array disk group to allocate volumes from.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

label_prefix

required:    false
scopable:    false

The prefix to add to the label assigned to the created disks.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

pool_id

required:    false
scopable:    false

The Hitachi Ops Center storage machine pool name. Volumes are created in this storage pool.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

volume_id_range_from

required:    false
scopable:    false

The start of the range of ldev ids to allocate from.

volume_id_range_to

required:    false
scopable:    false

The end of the range of ldev ids to allocate from.

vsm_id

required:    false
scopable:    false

The name of the virtual storage machine id to allocate volume into.

pool.loop

Minimal configlet:

[pool#1]
type = loop

Minimal setup command:

om node set --kw="type=loop"

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

path

required:    false
scopable:    false
default:     {var}/pool/loop

The path to create the pool loop files in.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.pure

Minimal configlet:

[pool#1]
type = pure
array = 
diskgroup = 

Minimal setup command:

om node set \
	--kw="type=pure" \
	--kw="array=" \
	--kw="diskgroup="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

delete_now

required:    false
scopable:    false
default:     true
convert:     bool

If set to false the pure volumes are not immediately deleted on unprovision, so a following provision action could fail.

diskgroup

required:    true
scopable:    false

The name of the array disk group to allocate volumes from.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

label_prefix

required:    false
scopable:    false

The prefix to add to the label assigned to the created disks.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

pod

required:    false
scopable:    false

The pod to create volume into.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

volumegroup

required:    false
scopable:    false

The volumegroup to create volume disks into.

pool.rados

Minimal configlet:

[pool#1]
type = rados
rbd_pool = 

Minimal setup command:

om node set \
	--kw="type=rados" \
	--kw="rbd_pool="

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

rbd_namespace

required:    false
scopable:    false

The ceph pool namespace where to create images.

rbd_pool

required:    true
scopable:    false

The ceph pool where to create images.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.share

Minimal configlet:

[pool#1]
type = share

Minimal setup command:

om node set --kw="type=share"

path

required:    false
scopable:    false
default:     {var}/pool/share

The fullpath of the shared directory hosting the pool volumes directories or loop files.

pool.shm

Minimal configlet:

[pool#1]
type = shm

Minimal setup command:

om node set --kw="type=shm"

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.symmetrix

Minimal configlet:

[pool#1]
type = symmetrix
array = 
srp = 

Minimal setup command:

om node set \
	--kw="type=symmetrix" \
	--kw="array=" \
	--kw="srp="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

rdfg

required:    false
scopable:    false

Replication Group to use for SRDF.

slo

required:    false
scopable:    false

The name of the Service Level Agreement of the selected Storage Group.

srdf

required:    false
scopable:    false
default:     false
convert:     bool

Use SRDF replication.

srp

required:    true
scopable:    false

The name of the array resource pool to allocate volumes from.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.truenas

Minimal configlet:

[pool#1]
type = truenas
array = 
diskgroup = 

Minimal setup command:

om node set \
	--kw="type=truenas" \
	--kw="array=" \
	--kw="diskgroup="

array

required:    true
scopable:    true

The name of the array, known as array#<name> in the node or cluster configuration.

blocksize

required:    false
scopable:    false
default:     512
convert:     size

Allow initiators to xcopy without authenticating to foreign targets.

compression

required:    false
scopable:    false
candidates:  inherit, none, lz4, gzip-1, gzip-2, gzip-3, gzip-4, gzip-5, gzip-6, gzip-7, gzip-8, gzip-9, zle, lzjb
default:     inherit

Compression level.

dedup

required:    false
scopable:    false
default:     off

Activate data deduplication on created dataset and zvol. Example values: on, off, verify

diskgroup

required:    true
scopable:    false

The name of the array disk group to allocate volumes from.

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

insecure_tpc

required:    false
scopable:    false
default:     false
convert:     bool

Allow initiators to xcopy without authenticating to foreign targets.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

sparse

required:    false
scopable:    false
default:     false
convert:     bool

Create zvol in sparse mode.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.vg

Minimal configlet:

[pool#1]
type = vg
name = 

Minimal setup command:

om node set \
	--kw="type=vg" \
	--kw="name="

fs_type

required:    false
scopable:    false
default:     xfs

The filesystem to format the pool devices with.

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

name

required:    true
scopable:    false

The name of the volume group to allocate the pool volumes logical volumes into.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

pool.virtual

Minimal configlet:

[pool#1]
type = virtual

Minimal setup command:

om node set --kw="type=virtual"

capabilities

required:    false
scopable:    false
default:     roo rwo rox rwx
convert:     list

The capabilities exposed by the virtual pool.

Supported capabilities:

• shared
• roo
• rox
• rwo
• rwx
• blk

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

optional_volume_env

required:    false
scopable:    false
convert:     list

Example:

optional_volume_env=container#1.name:container_name env.foo:foo

The list of the vol consumer service config keywords which values are mapped as env keys in the allocated volume service.

If the keyword is not set at the source, the default value in the template env section applies.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

template

required:    false
scopable:    false

Example:

template=templates/vol/mpool-over-loop

The path of a vol to use as a template for new volumes.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

volume_env

required:    false
scopable:    false
convert:     list

Example:

volume_env=container#1.name:container_name env.foo:foo

The list of the vol consumer service config keywords which values are mapped as env keys in the allocated volume service.

If the keyword is not set at the source, an error is raised.

pool.zpool

Minimal configlet:

[pool#1]
type = zpool
name = 

Minimal setup command:

om node set \
	--kw="type=zpool" \
	--kw="name="

mkblk_opt

required:    false
scopable:    false

The zvol, lv, and other block device creation command options to use to prepare the pool devices.

mkfs_opt

required:    false
scopable:    false

Example:

mkfs_opt=-O largefile

The mkfs command options to use to format the pool devices.

mnt_opt

required:    false
scopable:    true

The mount options of the fs created over the pool devices.

name

required:    true
scopable:    false

The name of the zpool to allocate the pool volumes zvol or datasets into.

status_schedule

required:    false
scopable:    false

The value to set to the status_schedule keyword of the vol objects allocated from the pool.

See usr/share/doc/schedule for the schedule syntax.

type

required:    false
scopable:    false
candidates:  directory, loop, vg, zpool, freenas, share, shm, symmetrix, truenas, virtual, dorado, hoc, drbd, pure, rados
default:     directory

The pool type.

switch.brocade

Minimal configlet:

[switch#1]
type = brocade
username = admin

Minimal setup command:

om node set \
	--kw="type=brocade" \
	--kw="username=admin"

key

required:    false
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in the switch.

method

required:    false
scopable:    false
candidates:  telnet, ssh
default:     ssh

Example:

method=ssh

The method to use to connect to the switch.

• ssh Use key to provide a ssh key, or use the sshpass program.

• telnet Set username and password with this method.

name

required:    false
scopable:    false

Example:

name=sansw1.my.corp

The name connect to the switch (dns name or ip address).

If not set, fallback to the section name suffix.

password

required:    false
scopable:    false

Example:

password=mysec/password

The password to use to log in, expressed as a sec name (not path).

The secret must be in the system namespace and must have a password key.

Either username or key must be specified.

username

required:    true
scopable:    false

Example:

username=admin

The username to use to log in the switch.

syslog

facility

required:    false
scopable:    false
default:     daemon

The syslog facility to log to.

host

required:    false
scopable:    false
default:     `localhost` if port is set.

The syslog server host to send logs to.

If neither host nor port are specified and if /dev/log exists, the messages are posted to /dev/log.

level

required:    false
scopable:    false
candidates:  critical, error, warning, info, debug
default:     info

The minimum message criticity to feed to syslog.

Setting to critical actually disables the syslog logging, as the agent does not emit messages at this level.

port

required:    false
scopable:    false
default:     514

The syslog server port to send logs to.

If neither host nor port are specified and if /dev/log exists, the messages are posted to /dev/log.

sysreport

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the sysreport node action, which collects into an archive all files and command outputs defined in /etc/opensvc/sysreport and sends that archive to the collector.

The collector stores the unpacked files in a per-node git repository.

See usr/share/doc/schedule for the schedule syntax.

DEFAULT

id

required:    false
scopable:    false
default:     A autogenerated random uuid

A RFC 4122 random uuid generated by the agent.

arbitrator

Minimal configlet:

[arbitrator]
uri = http://www.opensvc.com

Minimal setup command:

om test/ccfg/foo set --kw="uri=http://www.opensvc.com"

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Set to true to disable the arbitrator SSL certificate verification on the https uri.

This should only be enabled for testing.

uri

required:    true
scopable:    false

Example:

uri=http://www.opensvc.com

The arbitrator uri used by cluster node to ask for a vote when the cluster is split.

When the uri scheme is http or https, the vote checker is based on a GET request, else it is based on a TCP connect.

For backward compatibility, when the port is not specified in a TCP connect uri, the 1214 port is implied.

Arbitrators are tried in sequence, each reachable arbitrator gives a vote.

In case of a real split, all arbitrators are expected to be unreachable from the lost segment. At least one of them is expected to be reachable from the surviving segment.

Arbitrators of a cluster must thus be located close enough to each other, so a subset of arbitrators can’t be reachable from a split cluster segment, while another subset of arbitrators is reachable from the other split cluster segment.

But not close enough so they can all fail together. Usually, this can be interpreted as: same site, not same rack and power lines.

Arbitrators are verified every 60s to alert admins of the arbitrator failures.

weight

required:    false
scopable:    false
default:     1
convert:     int

During a quorum vote, this reachable arbitrator contributes votes.

array.centera

Minimal configlet:

[array#1]
type = centera
java_bin = /opt/java/bin/java
jcass_dir = /opt/centera/LIB
password = from system/sec/array1 key password
server = centera1
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=centera" \
	--kw="java_bin=/opt/java/bin/java" \
	--kw="jcass_dir=/opt/centera/LIB" \
	--kw="password=from system/sec/array1 key password" \
	--kw="server=centera1" \
	--kw="username=root"

java_bin

required:    true
scopable:    false

Example:

java_bin=/opt/java/bin/java

The path to the java executable to use to run the Centera management program.

jcass_dir

required:    true
scopable:    false

Example:

jcass_dir=/opt/centera/LIB

The path of the directory hosting the JCASScript.jar.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

server

required:    true
scopable:    false

Example:

server=centera1

The storage server to connect.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.dorado

Minimal configlet:

[array#1]
type = dorado
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=dorado" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

name

required:    false
scopable:    false

Example:

name=a09

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.emcvnx

Minimal configlet:

[array#1]
type = emcvnx
spa = array1-a
spb = array1-b

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=emcvnx" \
	--kw="spa=array1-a" \
	--kw="spb=array1-b"

method

required:    false
scopable:    false
candidates:  secfile, credentials
default:     secfile

Example:

method=secfile

The authentication method to use.

password

required:    false
scopable:    false

Example:

password=system/sec/array1

The password to use to log in, expressed as a datastore reference:

    from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

scope

required:    false
scopable:    false
default:     0

Example:

scope=1

The VNC scope to work in.

spa

required:    true
scopable:    false

Example:

spa=array1-a

The name of the Service Processor A.

spb

required:    true
scopable:    false

Example:

spb=array1-b

The name of the Service Processor B.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.eva

Minimal configlet:

[array#1]
type = eva
manager = evamanager.mycorp
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=eva" \
	--kw="manager=evamanager.mycorp" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

bin

required:    false
scopable:    false

Example:

bin=/opt/sssu/bin/sssu

The EVA manager executable to use.

manager

required:    true
scopable:    false

Example:

manager=evamanager.mycorp

The EVA manager to connect.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.freenas

Minimal configlet:

[array#1]
type = freenas
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=freenas" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.hds

Minimal configlet:

[array#1]
type = hds
password = from system/sec/array1 key password
url = https://hdsmanager/
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=hds" \
	--kw="password=from system/sec/array1 key password" \
	--kw="url=https://hdsmanager/" \
	--kw="username=root"

bin

required:    false
scopable:    false

Example:

bin=/opt/hds/bin/HiCommandCLI

The HDS manager executable to use.

jre_path

required:    false
scopable:    false

Example:

jre_path=/opt/java

The path hosting the java installation to use to execute the HiCommandCLI.

name

required:    false
scopable:    false

Example:

name=HUSVM.1234

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

url

required:    true
scopable:    false

Example:

url=https://hdsmanager/

The url passed to HiCommandCli, pointing the manager in charge of the array.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.hoc

Minimal configlet:

[array#1]
type = hoc
api = https://array.opensvc.com/api/v1.0
model = VSP G350
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=hoc" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="model=VSP G350" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

delay

required:    false
scopable:    false
default:     10s
convert:     duration

The delay between request attempts on retryable errors.

http_proxy

required:    false
scopable:    false

Example:

http_proxy=http://proxy.mycorp:3158

The proxy server to use for http requests to the api.

https_proxy

required:    false
scopable:    false

Example:

https_proxy=https://proxy.mycorp:3158

The proxy server to use for https requests to the api.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

model

required:    true
scopable:    false
candidates:  VSP G370, VSP G700, VSP G900, VSP F370, VSP F700, VSP F900, VSP G350, VSP F350, VSP G800, VSP F800, VSP G400, VSP G600, VSP F400, VSP F600, VSP G200, VSP G1000, VSP G1500, VSP F1500, Virtual Storage Platform, HUS VM

Example:

model=VSP G350

The array model.

name

required:    false
scopable:    false

Example:

name=a09

The name of the array. If not provided, fallback to the section name suffix.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

retry

required:    false
scopable:    false
default:     30
convert:     int

The number of request attempts on retryable errors.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

wwid_prefix

required:    false
scopable:    false

Hitachi APIs do not report the disks NAA wwids, but it can be forged from a array-specifix prefix concatenated with the LDev id. This keyword allow the cluster admin to define this prefix. Do not include the NAA Type digit prefix (define 62400000ec12ac73541d instead of 362400000ec12ac73541d).

array.hp3par

Minimal configlet:

[array#1]
type = hp3par

Minimal setup command:

om test/ccfg/foo set --kw="type=hp3par"

cli

required:    false
scopable:    false
default:     3parcli

Example:

cli=/path/to/pwf

The path of the executable hp3par CLI.

key

required:    false
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

manager

required:    false
scopable:    false
default:     The name of the array

Example:

manager=mymanager.mycorp

The array manager host name.

method

required:    false
scopable:    false
candidates:  proxy, cli, ssh
default:     ssh

Example:

method=ssh

The connection method to use.

pwf

required:    false
scopable:    false

Example:

pwf=/path/to/pwf

The path to the 3par password file to use to log in.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.ibmds

Minimal configlet:

[array#1]
type = ibmds
hmc1 = hmc1.mycorp
hmc2 = hmc2.mycorp
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=ibmds" \
	--kw="hmc1=hmc1.mycorp" \
	--kw="hmc2=hmc2.mycorp" \
	--kw="username=root"

hmc1

required:    true
scopable:    false

Example:

hmc1=hmc1.mycorp

The host name of the primary HMC.

hmc2

required:    true
scopable:    false

Example:

hmc2=hmc2.mycorp

The host name of the secondary HMC.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.ibmsvc

Minimal configlet:

[array#1]
type = ibmsvc
key = /path/to/key
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=ibmsvc" \
	--kw="key=/path/to/key" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.netapp

Minimal configlet:

[array#1]
type = netapp
key = /path/to/key
server = centera1
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=netapp" \
	--kw="key=/path/to/key" \
	--kw="server=centera1" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

server

required:    true
scopable:    false

Example:

server=centera1

The storage server to connect.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.nexenta

Minimal configlet:

[array#1]
type = nexenta
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=nexenta" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

port

required:    false
scopable:    false
default:     2000
convert:     int

Example:

port=2000

The nexenta administration listener port.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.pure

Minimal configlet:

[array#1]
type = pure
api = https://array.opensvc.com/api/v1.0
client_id = bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7
issuer = opensvc
key_id = df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f
secret = system/sec/array1
username = opensvc

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=pure" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="client_id=bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7" \
	--kw="issuer=opensvc" \
	--kw="key_id=df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f" \
	--kw="secret=system/sec/array1" \
	--kw="username=opensvc"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

client_id

required:    true
scopable:    false

Example:

client_id=bd2c75d0-f0d5-11ee-a362-8b0f2d1b83d7

The client id to use as the aud key in the payload of the login jwt.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

issuer

required:    true
scopable:    false

Example:

issuer=opensvc

The issuer to use as the iss key in the payload of the login jwt token.

key_id

required:    true
scopable:    false

Example:

key_id=df80ae3a-f0d5-11ee-94c9-b7c8d2f57c4f

The key id to use as the kid key in the header of the login jwt.

secret

required:    true
scopable:    false

Example:

secret=system/sec/array1

The secret to use to store the information required to create the login jwt, expressed as a reference to a secret. The secret must be in the system namespace and must have the following keys: private_key.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=opensvc

The username to use as the sub key in the payload of the login jwt.

array.symmetrix

Minimal configlet:

[array#1]
type = symmetrix

Minimal setup command:

om test/ccfg/foo set --kw="type=symmetrix"

name

required:    false
scopable:    false

Example:

name=00012345

The name of the array. If not provided, fallback to the section name suffix.

password

required:    false
scopable:    false

Example:

password=system/sec/array1

The password to use to log in, expressed as a datastore reference:

    from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

symcli_connect

required:    false
scopable:    false

Example:

symcli_connect=MY_SYMAPI_SERVER

Set the SYMCLI_CONNECT environment variable to this value.

If not set, the SCSI communication channels are used.

The value set must be declared in the /var/symapi/config/netcnfg file.

symcli_path

required:    false
scopable:    false
default:     /usr/symcli

Example:

symcli_path=/opt/symcli

Force use of a symcli programs installation, pointing the path of its head directory.

For the case multiple symcli versions are installed and the default selector does not select the version preferred for the array.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    false
scopable:    false

Example:

username=root

The username to use to log in, if configured.

array.truenas

Minimal configlet:

[array#1]
type = truenas
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=truenas" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Example:

insecure=true

Disable secure socket verification.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

timeout

required:    false
scopable:    false
default:     120s
convert:     duration

Example:

timeout=10s

The api request timeout.

type

required:    true
scopable:    false
candidates:  freenas, hds, eva, nexenta, vioserver, centera, symmetrix, emcvnx, netapp, hp3par, ibmds, ibmsvc, xtremio, dorado, hoc, truenas

The storage array driver name.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.vioserver

Minimal configlet:

[array#1]
type = vioserver
key = /path/to/key
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=vioserver" \
	--kw="key=/path/to/key" \
	--kw="username=root"

key

required:    true
scopable:    false

Example:

key=/path/to/key

The path to the private key to use to log in.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

array.xtremio

Minimal configlet:

[array#1]
type = xtremio
api = https://array.opensvc.com/api/v1.0
password = from system/sec/array1 key password
username = root

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=xtremio" \
	--kw="api=https://array.opensvc.com/api/v1.0" \
	--kw="password=from system/sec/array1 key password" \
	--kw="username=root"

api

required:    true
scopable:    false

Example:

api=https://array.opensvc.com/api/v1.0

The array rest api url.

password

required:    true
scopable:    false

Example:

password=from system/sec/array1 key password

The password to use to log in, expressed as a datastore reference:

from <namespace>/<kind>/<name> key <name>

Array passwords are usually stored in sec datastores like system/sec/<array name>.

username

required:    true
scopable:    false

Example:

username=root

The username to use to log in.

asset

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushasset node action.

See usr/share/doc/schedule for the schedule syntax.

checks

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushchecks node action.

See usr/share/doc/schedule for the schedule syntax.

cluster

ca

required:    false
scopable:    false
default:     `system/sec/ca`

convert:     list

A whitespace-separated list of sec paths.

The listener accepts a x509 client certificate if it is trusted by any CA certificate found in these sec objects.

cert

required:    false
scopable:    false
default:     `system/sec/cert`

The path of the secret hosting the certificate that the listener use for its TLS socket.

dns

required:    false
scopable:    true
convert:     list

The list of nodes to set as nameserver in the resolv.conf of the containers the CRM starts.

If set, the search will also be set to:

1/ <name>.<namespace>.svc.<clustername> 2/ <namespace>.svc.<clustername> 3/ <clustername>.

drpnodes

required:    false
scopable:    false
convert:     list

This list is fetched from the join command payload received from the joined node.

The service configuration {clusterdrpnodes} is resolved to this keyword value.

envs

required:    false
scopable:    false
default:     CERT DEV DRP FOR INT PRA PRD PRJ PPRD QUAL REC STG TMP TST UAT
convert:     list

id

required:    false
scopable:    true
default:     An autogenerated random UUID.

This unique identifier is auto-generated on install and should never be change by the cluster administrators.

It is changed when the node joins a cluster, so the remote cluster id replaces the joiners’ cluster id.

name

required:    false
scopable:    false
default:     A random generated clustername.

The cluster name is used,

• as the zone name in the cluster dns records
• in the {fqdn} configuration reference
• in the AES secret encryption metadata

The cluster name should be unique site-wide. Missing cluster name will be automatically created with random value during daemon startup.

It is always lowercased, so better to set it to a lowercase value to avoid confusion.

The cluster name is provided to joining nodes, so they can replace their own.

nodes

required:    false
scopable:    false
convert:     list

This list of node names contains only the local node name on install.

When the node joins a cluster, the joined node provides the new list, with the new node added. The joiner then replace its nodes list with the one received.

When a node receives a join request, it adds the new node to its cluster nodes list, then provide the new list to the joiner.

quorum

required:    false
scopable:    false
default:     false
convert:     bool

If true, when the cluster is split a vote happens on each cluster node.

Each reachable node and each reachable arbitrator give their vote. If the votes is less than half the total number of nodes plus arbitrators, the node trigger a node fencing method defined by node.split_action (crash, reboot or disabled).

secret

required:    false
scopable:    true
default:     A random string autogenerated on first use

The cluster shared secret used to encrypt and decrypt heartbeat payloads and sec values, with AES256

This secret is auto-generated on install, then merged from the joined nodes when joining a cluster.

The cluster name should be unique site-wide and be set right before starting to add sec keys.

cni

config

required:    false
scopable:    false
default:     /var/lib/opensvc/cni/net.d

Example:

config=/var/lib/opensvc/cni/net.d

The directory hosting the CNI network configuration files.

plugins

required:    false
scopable:    false
default:     /usr/lib/cni

Example:

plugins=/var/lib/opensvc/cni/bin

The directory hosting the CNI plugins.

compliance

auto_update

required:    false
scopable:    false
default:     false
convert:     bool

If set to true, execute om node updatecomp upon every scheduler-executed om node compliance check.

These updates keep the compliance modules in sync with the reference repository.

Warning: the module repository security is critical. Attackers could insert malicious code in served modules.

schedule

required:    false
scopable:    false
default:     02:00-06:00

Schedule parameter for the compliance auto node action, which check all attached modules and fix only those flagged autofix.

See usr/share/doc/schedule for the schedule syntax.

console

insecure

required:    false
scopable:    false
convert:     bool

If set, don’t verify the console server certificate.

max_greet_timeout

required:    false
scopable:    false
default:     20s
convert:     duration

This keyword sets the absolute upper limit (maximum duration) that an API user can request via the greet_timeout query parameter when calling the handler to start a service instance resource console.

The console handler accepts a greet_timeout query parameter, which specifies how long the service instance should wait for the first client connection to the generated console URL.

The max_greet_timeout acts as a security safeguard, preventing API users from setting excessively long greet_timeout values.

If max_greet_timeout is set to 30s, any user request for greet_timeout=60s will be refused.

Recommendation:

• Keep Short: The effective timeout should generally be kept to a few seconds (e.g., 5s to 10s). A shorter timeout minimizes the risk that a malicious actor could successfully guess the console URL through a brute-force attack before the link expires.

• Manual Testing: Users may occasionally need to set a slightly higher greet_timeout (e.g., 60s) for manual testing or debugging purposes, but this value will always be constrained by the max_greet_timeout defined here.

max_seats

required:    false
scopable:    false
default:     1
convert:     int

This keyword sets the absolute upper limit that an API user can request via the seats query parameter when calling the handler to start a service instance resource console.

server

required:    false
scopable:    false

The tty-proxy console server address. A TLS capable server is expected, so the value should be a TLS terminator reverse proxy.

dequeue_actions

schedule

required:    false
scopable:    false

Schedule parameter for the dequeue actions node action.

See usr/share/doc/schedule for the schedule syntax.

disks

schedule

required:    false
scopable:    false
default:     ~00:00-06:00

Schedule parameter for the pushdisks node action.

See usr/share/doc/schedule for the schedule syntax.

hb.disk

Minimal configlet:

[hb#1]
type = disk
dev = /dev/mapper/36589cfc000000e03957c51dabab8373a

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=disk" \
	--kw="dev=/dev/mapper/36589cfc000000e03957c51dabab8373a"

dev

required:    true
scopable:    true

Example:

dev=/dev/mapper/36589cfc000000e03957c51dabab8373a

The device to write the heartbeats to and read from.

It must be,

• Dedicated to the daemon use.
• Sized 1MB for metadata + 1MB/node.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

max_slots

required:    false
scopable:    false
default:     1024
convert:     int

Example:

max_slots=1024

The maximum number of slots that can be written to the HB disk device.

It should be set to at least twice the number of cluster nodes using the HB disk device.

Do not modify this value after first activation. Do not modify default value if the hb disk is used by multiple clusters.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hb.multicast

Minimal configlet:

[hb#1]
type = multicast

Minimal setup command:

om test/ccfg/foo set --kw="type=multicast"

addr

required:    false
scopable:    true
default:     224.3.29.71

The multicast address to send to and listen on.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

intf

required:    false
scopable:    true
default:     The natural interface for `<addr>`

Example:

intf=eth0

The interface to bind.

port

required:    false
scopable:    true
default:     10000
convert:     int

The port for each node to send to or listen on.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hb.relay

Minimal configlet:

[hb#1]
type = relay
relay = relaynode1

Minimal setup command:

om test/ccfg/foo set \
	--kw="type=relay" \
	--kw="relay=relaynode1"

insecure

required:    false
scopable:    false
default:     false
convert:     bool

Set to true to disable the relay SSL certificate verification.

This should only be enabled for testing.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

password

required:    false
scopable:    false
default:     system/sec/relay

The name of a sec object containing a password key, which value is used as password for log in the relay api.

relay

required:    true
scopable:    false

Example:

relay=relaynode1

The relay resolvable node name.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

username

required:    false
scopable:    false
default:     relay

The username for login the relay api.

hb.unicast

Minimal configlet:

[hb#1]
type = unicast

Minimal setup command:

om test/ccfg/foo set --kw="type=unicast"

addr

required:    false
scopable:    true
default:     0.0.0.0 for listening and to the resolved nodename for sending.

Example:

addr=1.2.3.4

Use the scoping syntax to assign non-default ip addresses nodes.

Warning: Using the unscoped syntax would assign the same address to all nodes and the heartbeat would go down.

interval

required:    false
scopable:    true
default:     5s
convert:     duration

The maximum interval between 2 heartbeat payload sends.

The actual interval is not fixed, because the daemon tries to send the message as soon as it has something to notify. A minimum interval protects the node from saturating the network and cpu with the daemon synchronization workload.

intf

required:    false
scopable:    true
default:     The natural interface for `<addr>`

Example:

intf=eth0

The interface to bind.

nodes

required:    false
scopable:    true
default:     All nodes.

convert:     list

The nodes participating to the heartbeat.

This keyword can be used to setup a partial redundancy like:

       n1   n2   n3   n4
hb#1   O    O    O    O
hb#2   O    O          
hb#1             O    O

Which can be relevant if n[12] are in the same bladecenter b1, and n[34] are in the same bladecenter b2. The bladecenter having an internal network completely hardware independent of the network used to go outside of the bladecenters.

port

required:    false
scopable:    true
default:     10000
convert:     int

The port for each node to send to or listen on.

timeout

required:    false
scopable:    true
default:     15s
convert:     duration

The delay since the last received heartbeat from a node before considering this node is gone.

type

required:    true
scopable:    false
candidates:  unicast, multicast, disk, relay

The heartbeat driver name.

hook

command

required:    false
scopable:    false
convert:     shlex

The command to execute on selected events.

The program is fed the json-formatted event data through stdin.

events

required:    false
scopable:    false
convert:     list