On the node to enroll node2, create a token with the join role, and store it in a file:

sudo om daemon auth --role join --duration 10m >/root/node2.token

The token is valid for 60 seconds by default, so --duration gives time to copy the file to a cluster node.

On any cluster node, for example node1, enroll node2 using the copied token file:

sudo om cluster enroll --node node2 --token /root/node2.token --wait

Node node2 accepted the join order
Cluster nodes updated
Waiting for node2 to beat
Node node2 is alive

• --node is the location of the node to enroll, in the [<scheme>://]<addr>[:<port>] format.
• --token is the path of the token file. The OSVC_JOIN_TOKEN environment variable is used when the flag is not set.
• --join-addr is the location node2 must use to reach node1, in the same format. By default, a name the cluster certificate is valid for is used. Set it when this name does not resolve to an address reachable from node2.
• --wait waits for the node2 heartbeat to beat in the cluster. Without it, follow the join with om daemon events --filter 'NodeAlive,.node=node2'.
• --timeout is the maximum duration of the join, 1h by default.

The enroll requires the root grant.

The same enroll, driven from a host that is a member of neither cluster.

1. Create a user on the node to enroll

POST /api/auth/token only accepts the requests authenticated by the unix socket or by Basic Authentication, and a node freshly installed has no api user yet. So this first step is run locally on node2, once:

om system/usr/enroller create --kw grant=join
om system/usr/enroller key add --name=password --from /dev/urandom
om system/usr/enroller key decode --name=password

2. Ask that node for a token with the join role

curl -sSk -u enroller:$ENROLLER_PASSWORD -X POST \
  "https://node2:1215/api/auth/token?role=join&access_duration=10m" \
  | jq -r .access_token >/root/node2.token

• role=join is filtered against the grants of the user: the token carries the join role only if the user holds it.
• access_duration defaults to 60 seconds, which gives little time for the next call.
• -k is needed here: node2 is still its own cluster, so its listener certificate is signed by its own certificate authority, which nothing outside of it trusts yet. That is also why the join role token carries a ca claim: it is what lets the cluster trust that certificate for the rest of the enroll.

3. Post the enroll order to a cluster node

curl -s -H "Authorization: Bearer $TOKEN" -X POST \
  "https://node1:1215/api/cluster/enroll" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile t /root/node2.token '{node:"node2", token:($t|rtrimstr("\n")), timeout:"1h"}')"

{"node":"node2"}

• node is the location of the node to enroll, in the [<scheme>://]<addr>[:<port>] format. The scheme defaults to https and the port to the daemon listener port.
• token is the token obtained at the previous step.
• join_addr is the location node2 must use to reach node1, in the same format. By default, a name the cluster certificate is valid for is used. Set it when this name does not resolve to an address reachable from node2.
• timeout is the lifetime of the join role token the cluster hands to node2, 1h by default. It must outlive the node drain.

The requester must be granted the root role.

The response is sent as soon as node2 has accepted the join order, and carries the nodename it reads from its own configuration. The join itself runs asynchronously.

4. Follow the join

curl -s -N -H "Authorization: Bearer $TOKEN" \
  "https://node1:1215/api/node/name/node1/daemon/event?duration=1h&filter=JoinSuccess,added_node=node2&filter=JoinIgnored,candidate_node=node2&filter=JoinError,candidate_node=node2&filter=NodeAlive,.node=node2"

• JoinSuccess says the api node has added node2 to its cluster.nodes. The node has not drained, reconfigured, or restarted yet at this point.
• JoinIgnored says node2 already was a cluster node.
• JoinError says the cluster nodes update failed.
• NodeAlive says the node2 heartbeat beats in the cluster: it has fully joined. This is what om cluster enroll --wait waits for.

On any other cluster node, for example node1, drain then evict node2:

sudo om node drain --node node2 --wait
sudo om cluster evict --node node2 --credential /root/node2.cred --wait

Node node2 accepted the leave order
Cluster nodes updated
Node node2 is restarting alone, its instances stay down

• --node is the name of the cluster node to evict. It can not be the local node: run the command on one of its peers.
• --credential is the path of a file holding the <username>:<password> of a user to create on the evicted node, in the system namespace with the root grant. The OSVC_CREDENTIAL environment variable is used when the flag is not set.
• --wait waits for the cluster nodes to be updated. Without it, follow the eviction with om daemon events --filter 'LeaveSuccess,removed_node=node2'.
• --timeout is the maximum duration of the leave, 1h by default.

The evict requires the root grant.

1. Drain the node to evict

curl -s -H "Authorization: Bearer $TOKEN" -X POST \
  "https://node1:1215/api/node/name/node2/action/drain"

{"orchestration_id":"e2ba0d0e-...."}

The drain is asynchronous, and the eviction is refused with a 409 while node2 is not drained, so wait for the drain to end before the next call. A drained node is frozen and runs no instance, which GET /api/cluster/status reports.

2. Post the evict order to another cluster node

curl -s -H "Authorization: Bearer $TOKEN" -X POST \
  "https://node1:1215/api/cluster/evict" \
  -H "Content-Type: application/json" \
  -d '{"nodename":"node2","credential":"admin:secret","timeout":"1h"}'

{"detail":"node node2 is leaving the cluster","status":200,"title":"background cluster leave has been called"}

• nodename is the cluster node to evict. It can not be the node serving the request: post this to one of its peers.
• credential is the <username>:<password> of a user to create on the evicted node, in the system namespace with the root grant. It is optional, and checked before anything is ordered.
• timeout is the maximum duration of the leave forked on the evicted node, 1h by default.

The requester must be granted the root role.

The response is sent as soon as node2 has accepted the leave order. The leave itself runs asynchronously.

3. Follow the eviction

curl -s -N -H "Authorization: Bearer $TOKEN" \
  "https://node1:1215/api/node/name/node1/daemon/event?duration=1h&filter=LeaveSuccess,removed_node=node2&filter=LeaveIgnored,candidate_node=node2&filter=LeaveError,candidate_node=node2"

• LeaveSuccess says the cluster has dropped node2 from its cluster.nodes. This is what om cluster evict --wait waits for.
• LeaveIgnored says node2 was not a cluster node.
• LeaveError says the cluster nodes update failed.

The daemon restart and the user creation happen after LeaveSuccess, on a node this cluster no longer observes, so no event reports them.