Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Cluster Quorum

When a peer is flagged as stale by all heartbeats, the daemon assumes the cluster is in a split-brain situation, as it cannot determine whether the stale peer has failed or is isolated.

OpenSVC minimizes the likelihood of a split-brain scenario by leveraging multiple independent heartbeats.

Enabling Quorum Enforcement

The API tabs assume the $TOKEN and the listener endpoint set up in Cluster API. cluster.quorum is a cluster keyword, so it is written on any node with PATCH /api/cluster/config, which takes the same set and unset operations as om cluster config update.

Users who prefer to have a cluster segment shut down in such situations can enable quorum by setting cluster.quorum to true:

om cluster config update --set cluster.quorum=true

curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \
  --data-urlencode 'set=cluster.quorum=true' \
  "https://<node>:1215/api/cluster/config"

By default, the system allows split nodes to take over services, which may result in services running on multiple isolated segments. To revert to the default behavior, use:

om cluster config update --unset cluster.quorum

curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \
  --data-urlencode 'unset=cluster.quorum' \
  "https://<node>:1215/api/cluster/config"

To check the current quorum configuration:

om cluster config get --kw cluster.quorum

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<node>:1215/api/cluster/config?kw=cluster.quorum"

Quorum Behavior

If the cluster is configured for quorum and a split-brain situation occurs, a node will shut down if the number of reachable nodes (including itself) plus arbitrators is less than half of the total cluster and arbitrator nodes.

Frozen nodes do no evaluate quorum. They will not shut down on split-brain.

Frozen nodes still vote for peer nodes quorum evaluation.

Example Arbitrator Requirements

To survive an interconnect outage:

  • In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split.
  • In a 3-node cluster, a single node requires 2 arbitrator votes.
  • In a 4-node cluster, a single node requires 3 arbitrator votes.
  • In a 5-node cluster, a single node requires 3 arbitrator votes.

To survive an interconnect outage, plus all peers outage in the same availability zone:

  • In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split.
  • In a 3-node cluster, a single node requires 2 arbitrator votes.
  • In a 4-node cluster, a single node requires 3 arbitrator votes.
  • In a 5-node cluster, a single node requires 4 arbitrator votes.

Configuring Arbitrators

Any OpenSVC agent can act as an arbitrator, and multiple arbitrators can be configured. For example, to configure an arbitrator:

Use an https server as an arbitrator

[arbitrator#a1]
uri = https://dev2n1:1215/metrics
#insecure = true

Use a tcp server as an arbitrator

[arbitrator#a2]
# 22, the ssh port of dev2n2: with no port, listener.port is implied
uri = dev2n2:22

The vote of a tcp arbitrator is a TCP connect to its uri. The port of the uri is optional: with none, the port of the cluster listener is implied, listener.port, 1215 by default. An arbitrator still listening on the 1214 port of a v2 agent must name it in the uri.

Testing Arbitrators

Every node votes on its own, so an arbitrator is alive from the point of view of a node, not of the cluster. The monitor shows that grid:

om mon

...
Arbitrators                       n1   n2
 a1                warn         | X    X          
 a2                warn         | X    X          
 a3                             | O    O          
...

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<node>:1215/api/cluster/status" |
  jq '.cluster.node | map_values(.status.arbitrators)'

Each node status carries an arbitrators object, keyed by arbitrator name, which is what the grid renders one column per node.

Best Practices

  • Configure <number of nodes> minus 1 arbitrators
  • Host all arbitrators on the same 3rd site
  • Use one of the arbitrators as a relay for the relay heartbeat driver
  • Disable quorum or freeze all nodes when doing a relayout of the cluster

➡️ See Also