# Agent Daemon Lifecycle

## Enable

    sudo systemctl enable --now opensvc-server

🛈 Newly installed daemons are not enabled by default.

## Boot

    om daemon start

* Executed on server boot.
* After reaching the multi-user target.

Side Effects:

* HA services in down state start a local instance.
* Standby resources of all services are started.

🛈 Administrators can prevent these side effects on boot setting the `osvc.freeze` option in the kernel boot command line.

## Shutdown

    om daemon shutdown

* Executed on server shutdown.
* The daemon drains all service instances running on the node.
* The peer nodes are free to takeover a HA service as soon as its local instance is drained.

## Stop

    om daemon stop

* Announces a maintenance period.
* The service instances running on the node are not stopped.
* Peer nodes wait for `node.maintenance_grace_period` before taking over, expecting the daemon to restart.

🛈 The mainteance period ends as soon as the daemon restarts.

## Restart

    om daemon restart

A restart is a simple stop-start sequence:

* Maintenance period is announced
* Peer nodes wait for the daemon to restart without taking over.

A restart does not freeze the node, nor its peers, even when the daemons of
all the nodes restart at once, as with `om daemon restart --node='*'`: the
maintenance period is what keeps the peers from acting while a restart rolls.

## Rejoin

A starting daemon waits for its peers, `node.rejoin_grace_period` at most,
before orchestrating. It freezes the node on its own in two cases only, both
saying something went wrong:

* A peer is still out of reach when the rejoin grace period expires: the node
  may be split from the others, and does not orchestrate blind.
* The server booted with the `osvc.freeze` option on the kernel command line.

A node that was down misses the freezes asked meanwhile. When it comes back,
it adopts a freeze of the whole cluster, `om cluster freeze`, and of a whole
object, `om <path> freeze`, if a peer took it while the node was down:

    the cluster was frozen while this daemon was down (peer n1 frozen): local node has been frozen

It adopts no other freeze. A peer frozen alone, with `om node freeze` or
`om <path> instance freeze --node <peer>`, stays the only one frozen, as does a
peer the daemon froze on its own, for a drain, at boot, or at the end of its
rejoin grace period.

The node and instance statuses say which a freeze is, next to `frozen_at`:
`frozen_scope` is `cluster` or `node` for a node, `object` or `instance` for an
instance. A freeze adopted is a freeze of the node or of the instance: it is
not adopted again from there.

🛈 OpenSVC v2 adopted any freeze a peer took while the node was down, so a
node frozen for maintenance froze its peers as they rebooted.

## Run

    om daemon run

* Runs a daemon in foreground
* Logs are printed on the console
* Stacks are printed on the console

🛈 Useful for debugging.

## Watchdog

* The daemon sends a periodic probe to systemd.
* If the daemon hangs, the probe flow stops, and systemd kill-restart the daemon.

🛈 The daemon event bus hangs are detected internally and cause the watchdog probe flow to stop.