Keyword reference: OpenSVC agent v3.0.0-rc56. # Introduction ## What is OpenSVC OpenSVC is an open-source software product and the name of the company developing and supporting it. In production since 2009, OpenSVC consists of: - **Agent**: A supervisor, clusterware, container orchestrator, and configuration manager (Apache 2.0 licensed since v3). - **Collector**: Optionally, aggregates agent data and manages configuration, infrastructure, and resources (Apache 2.0 licensed). ## Why OpenSVC - **Service Mobility**: Ensures high availability and scalability. - **Configuration Management**: Handles thousands of servers and clusters with minimal dependencies. - **Inventorying**: Tracks assets, storage, networks, and services. - **Orchestration**: RESTful API, self-service portal, provisioning, fine-grained delegation. - **Auditing**: Monitors performance, alerts, and audits infrastructure and services. - **Productivity Boost**: Reduces infrastructure maintenance and allows for task delegation. ## Requirements OpenSVC runs on Linux servers. | Subsystem | Agent | Collector | |-------------------------------------------------------------|-------|-----------| | Service mobility, clustering, container orchestration | O | | | Configuration management | O | O | | Inventoring | O | O | | Orchestration and interoperability | O | O | | Continuous infrastructure, systems, and services auditing | O | O | # Install ## Branch We feed packages in 3 different branches: * `dev` is unstable. Every candidate Pull Request causes a new package to be spawned here for OpenSVC QA purposes. * `uat` is for testing. OpenSVC will push there pre-release packages and packages that contain a candidate fixes for known issues that client are encouraged to validate. * `prod` is stable. It is the default and recommended branch. Execute one of the following variable settings in a shell, and the code block corresponding to the operating system. BRANCH=dev BRANCH=uat BRANCH=prod ## Debian, Ubuntu # Set ID (ubuntu, debian) and VERSION_CODENAME (bookworm, bullseye, noble) source /etc/os-release BRANCH=${BRANCH:-prod} DISTRIB=${VERSION_CODENAME:-noble} # Import opensvc gpg signing keys # ------------------------------- curl -s -o- https://packages.opensvc.com/gpg.public.key.asc | \ sudo gpg --dearmor --output /etc/apt/trusted.gpg.d/opensvc-package-pub.gpg --yes # # Add the opensvc repository to apt sources # ----------------------------------------- cat - </etc/yum.repos.d/opensvc.repo [opensvc] name=OpenSVC Packages RHEL \$releasever - \$basearch baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/ enabled=1 gpgcheck=0 EOF # # Install the opensvc server # -------------------------- sudo yum makecache sudo yum install opensvc-server # # Enable the systemd unit and start the server # -------------------------------------------- sudo systemctl enable --now opensvc-server ## Red Hat Enterprise Linux 8+ # Set ID (rhel) and VERSION_ID (8.10, ...) source /etc/os-release BRANCH=${BRANCH:-prod} DISTRIB=${ID}${VERSION_ID%.*} # # Add the opensvc repository to yum sources # ----------------------------------------- cat << EOF >/etc/yum.repos.d/opensvc.repo [opensvc] name=OpenSVC Packages RHEL \$releasever - \$basearch baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/ enabled=1 gpgcheck=1 gpgkey=https://packages.opensvc.com/gpg.public.key.asc EOF # # Install the opensvc server # -------------------------- sudo dnf makecache sudo dnf install opensvc-server # # Enable the systemd unit and start the server # -------------------------------------------- sudo systemctl enable --now opensvc-server ## SuSE Linux Enterprise Server # Set ID (rhel) and VERSION_ID (8.10, ...) source /etc/os-release BRANCH=${BRANCH:-prod} DISTRIB=${ID}${VERSION_ID%.*} # # Add the opensvc repository to zypper sources # ----------------------------------------- cat << EOF >/etc/zypp/repos.d/opensvc.repo [opensvc] name=OpenSVC Packages SLES \$releasever - \$basearch baseurl=https://packages.opensvc.com/rpm/$BRANCH-opensvc-v3-$DISTRIB/\$basearch/ enabled=1 autorefresh=1 gpgcheck=1 gpgkey=https://packages.opensvc.com/gpg.public.key.asc EOF # # Install the opensvc server # -------------------------- sudo zypper --gpg-auto-import-keys --non-interactive refresh sudo zypper install opensvc-server # # Enable the systemd unit and start the server # -------------------------------------------- sudo systemctl enable --now opensvc-server > ➡️ See Also > * [Installed Files](internals.installed_files.md) # Your First Service This page takes about five minutes and needs nothing but the agent [installed](install.md) on one machine. No shared storage, no second node, no cluster. At the end you will have created a service, started it, proven it is serving, and stopped it. We will wrap a plain HTTP server, because it is the shortest thing whose running you can prove rather than take on trust. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). `hello` is a `svc` of the `root` namespace, > so its path is `/api/object/path/root/svc/hello`. They are here to show that > nothing on this page is command line only: a reader following along has the > agent installed and no token yet, and does not need one. ## Create it An object is created from the command line, resource by resource. Here is a single `app` resource, told how to start:
```bash om hello create \ --kw app#1.type=simple \ --kw 'app#1.start=/usr/bin/python3 -m http.server 8099' ```
Creating an object through the api is posting its configuration file: ```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary $'[app#1]\ntype = simple\nstart = /usr/bin/python3 -m http.server 8099\n' \ "https://:1215/api/object/path/root/svc/hello/config/file" ```
Nothing was started, and nothing outside `/etc/opensvc` was touched. The command only wrote a configuration, which you can read back:
```bash om hello config show ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/svc/hello/config/file" ```
[DEFAULT] id = 0e745444-6724-48f9-8167-e865b4f9ec94 [app#1] type = simple start = /usr/bin/python3 -m http.server 8099 `[app#1]` is a resource. `app` is its driver group, `1` its index, and `type = simple` picks the driver that supervises a long-running process. ## Start it
```bash om hello instance start ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/root/svc/hello/action/start" ``` The api answers the `session_id` and `exec_id` of the run it forked, where the cli prints the run as it happens.
INF hello: >>> do start [om hello instance start] (origin user, sid 7b9772c8) INF hello: app#1: run: om exec --pg /opensvc.slice/... -- /usr/bin/python3 -m http.server 8099 INF hello: <<< done start in 98.079925ms, instance status is now up The agent tells you what it ran and what the instance status became. ## Prove it ```bash curl -o /dev/null -w "%{http_code}\n" http://localhost:8099/ ``` 200 ## Look at it
```bash om hello instance status ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/instance?path=root/svc/hello&node=" ``` The api answers the configuration, status and monitor state of the instance as json, which the cli renders as the tree below.
hello └ instances └ node1 up no-monitor └ resources └ app#1 ...../.. up simple The instance is `up` because its only resource is `up`. Had the resource failed, both lines would say so. This is the command you will reach for most often. ## Stop it
```bash om hello instance stop ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/root/svc/hello/action/stop" ```
INF hello: app#1: send termination signal to process 2615892 INF hello: app#1: process 2615892 is now terminated INF hello: <<< done stop in 86.862589ms, instance status is now down The port is closed and the status agrees: ```bash curl --max-time 3 http://localhost:8099/ om hello instance status ``` curl: (7) Failed to connect to localhost port 8099: Connection refused hello └ instances └ node1 down no-monitor └ resources └ app#1 ...../.. down simple ## Clean up
```bash om hello delete ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/svc/hello/action/delete" ```
OBJECT ORCHESTRATION_ID STATUS hello 068b4fb0-ba3f-4d62-81e7-92ed88039239 accepted The configuration is gone, and with it the object. ## Two ways to start You used `om hello instance start`, which starts the instance **on this node, now**, and prints each step as it happens. Its cluster counterpart is `om hello start`, which does not act directly. It asks the cluster to reach the started state, and answers with an orchestration id: OBJECT ORCHESTRATION_ID STATUS hello 45c9de82-ed12-48df-bdb2-21a251fb039d accepted The daemon then picks a node and starts the instance there, which is what you want once the object runs on more than one node. Watch it happen with `om mon`. Note that the matching `om hello stop` flags the instance stopped on purpose, so the daemon does not immediately start it again. The next `om hello start` clears the flag. ## What you just did You described a process in a configuration file, and the agent started it, tracked its state, and stopped it. Everything else the agent does is this same loop with more interesting resources: an ip that moves between nodes, a filesystem mounted from a shared disk, a container, a database volume. Where to go next: * [Naming](apps.deploy.naming.md) to place objects in namespaces before you have many of them. * [Create, Deploy](apps.deploy.create.md) for the other ways to create an object, including from a manifest. * [Cluster Configuration](configure.cluster.md) to add a second node, which is where failover starts to matter. # Agent Configuration ## Concepts The agent uses `ini` configuration files. Considering a configuration like: ```ini [env] bar = 1 bar@n2 = 2 ``` * `env` is a section * `bar` is an option * `env.bar` is a keyword. * `env.bar=1` is a keyword operation. * `1` is the `env.bar` keyword value. * `@n2` is a node [scope](./apps.design.scoping.md) for the keyword `env.bar` ## Policies * If a keyword appears in both `node.conf` and `cluster.conf`, the value from node.conf takes precedence. * Sections only accept recognized keywords, with the exception of the `[env]` and `[labels]` sections, which are open. * More specific scoped values override less specific ones. With the above section in a `svc1` object configuration: # on n1: $ om svc1 config eval --kw env.bar 1 # on n2: $ om svc1 config eval --kw env.bar 2 ## Syntax validation A syntax check is performed before finalizing any modifications made with either the set or edit commands. om cluster config edit om cluster config set --kw hb#test.type=unsupported A direct modification to the configuration file is not validated and may disrupt the cluster. In such cases, you can perform a post-hoc validation using: # verify the syntax of cluster.conf om cluster config validate # verify the syntax of node.conf om node config validate # verify the syntax of a svc configuration om svc1 config validate # Node Configuration ## Set the Node Environment Set a default environment for all cluster nodes: sudo om cluster config update --set node.env=PRD Optionally set an override in the node configuration: sudo om node config update --set node.env=DEV `node.env` marks a node production or not, which the agent then enforces two ways: * Only production services may start on a production node. * Only production nodes may push data to a production node. Supported `node.env` values: Env | Behaves As | Description ---------|------------|--------------------- PRD | PRD | Production PPRD | PRD | Pre Production REC | not PRD | Prod-like testing INT | not PRD | Integration DEV | not PRD | Development TST | not PRD | Testing (Default) TMP | not PRD | Temporary DRP | not PRD | Disaster recovery FOR | not PRD | Training PRA | not PRD | Disaster recovery PRJ | not PRD | Project STG | not PRD | Staging ## Set Node Jobs Schedules The agent executes periodic tasks. Display the scheduler configuration and states: $ sudo om node schedule list NODE ACTION LAST_RUN_AT NEXT_RUN_AT SCHEDULE eggplant pushasset 2025-01-20T01:31:17+01:00 0001-01-01T00:00:00Z ~00:00-06:00 eggplant checks 2025-01-20T16:40:20+01:00 0001-01-01T00:00:00Z @10m eggplant compliance_auto 2025-01-20T05:34:49+01:00 0001-01-01T00:00:00Z 02:00-06:00 eggplant pushdisks 2025-01-20T02:42:29+01:00 0001-01-01T00:00:00Z ~00:00-06:00 eggplant pushpkg 2025-01-20T00:16:38+01:00 0001-01-01T00:00:00Z ~00:00-06:00 eggplant pushpatch 2025-01-20T01:50:37+01:00 0001-01-01T00:00:00Z ~00:00-06:00 eggplant sysreport 2025-01-20T00:58:22+01:00 0001-01-01T00:00:00Z ~00:00-06:00 eggplant dequeue_actions 2023-08-03T14:05:50+02:00 0001-01-01T00:00:00Z eggplant pushhcs 2025-01-15T18:00:59+01:00 0001-01-01T00:00:00Z @1d eggplant pushswitch 0001-01-01T00:00:00Z 0001-01-01T00:00:00Z Schedule configuration:
```bash # Set a job schedule om node config update --set "switch#sansw1.schedule=02:00-04:00@120 sat,sun" # Disable a job schedule om node config update --set "switch#sansw1.schedule=@0" ```
```bash # Set a job schedule curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=switch#sansw1.schedule=02:00-04:00@120 sat,sun' \ "https://:1215/api/node/name//config" # Disable a job schedule curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=switch#sansw1.schedule=@0' \ "https://:1215/api/node/name//config" ```
> The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). A node keyword is patched on the node that > holds it, `PATCH …/node/name//config`, and a cluster keyword on any > node, `PATCH /api/cluster/config`. Both take the same `set`, `unset` and > `delete` operations as the matching `config update` command. > ➡️ See Also > * [Agent Scheduler](internals.daemon.scheduler.md) ## Register on a Collector ### Set a Collector Url By default, the agent does not communicate with a collector. To enable communications with an OpenSVC Collector v3, set the `collector.url` cluster configuration parameter:
```bash om cluster config update --set collector.url=https://collector.opensvc.com ```
```bash curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=collector.url=https://collector.opensvc.com' \ "https://:1215/api/cluster/config" ```
The node feeds the collector at `/feeder`, and calls its server api at `/server`. #### Advanced Url Formats Each api url can be set apart, when the collector does not serve them under the same url: om cluster config update --set collector.feeder=https://feeder.collector.opensvc.com om cluster config update --set collector.server=https://server.collector.opensvc.com `collector.feeder` and `collector.server` take precedence over the url derived from `collector.url`. ### Register the Node The collector requires the nodes to provide an authentication token (shared secret) with each request. The token is forged by the collector and stored on the node in the `node.uuid` keyword of `/etc/opensvc/node.conf`. The token is obtained with the credential of a collector user able to register a node. Store its `:` in a file, so the password does not show on a command line: ```bash install -m 600 /dev/null /root/collector.cred echo 'my.self@my.com:mypassword' >/root/collector.cred ``` Then register the node, or every node of the cluster:
```bash om node register --credential /root/collector.cred [--app MYAPP] om cluster register --credential /root/collector.cred [--app MYAPP] ``` The `OSVC_COLLECTOR_CREDENTIAL` environment variable is used when `--credential` is not set.
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"user": "my.self@my.com", "password": "mypassword", "app": "MYAPP"}' \ "https://:1215/api/node/name//action/register" curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"credential": "my.self@my.com:mypassword", "app": "MYAPP"}' \ "https://:1215/api/cluster/register" ``` Both are asynchronous: the response is sent when the registration is started, not when it is done.
If `--app` is not specified the collector automatically chooses one the user is responsible of. The registration also sends the cluster id, so the collector places the node in its cluster before its first daemon status. This requires a collector server api 1.0.4 or later. A successful register is followed by a node discovery, so the collector has detailed information about the node and can serve contextualized compliance rulesets up front. The discovery is also scheduled daily, and can be manually replayed with: om node push asset om node push pkg om node push disk om node sysreport om node checks A node enrolled into a cluster whose nodes are registered does not need to be registered again: see [Enroll a Node](configure.cluster.md#enroll-a-node). To disable collector communications, use: om cluster config update --unset collector.url Or if the setting was added to node.conf om node config update --unset collector.url ### Inventory the SAN Switches A couple of nodes of the infrastructure usually inventory the SAN switches, and report their configuration to the collector, which indexes their ports, zones and aliases. A switch is a `switch#` section of the node or cluster configuration: ```ini [switch#sansw1] type = brocade name = sansw1.my.corp username = admin password = from system/sec/sansw1 key password schedule = 02:00-04:00 ``` The node logs in over ssh, with the private key `key` points to or with the password of the secret, and runs `switchshow`, `nsshow` and `zoneshow`. The telnet method v2 offered is refused. The key of the switch is trusted on the first connection and recorded in the known hosts of root: a switch presenting another key later is refused, until its old key is removed from `/root/.ssh/known_hosts`. ```bash # Push every switch, or the one named om node push switch om node push switch sansw1 ``` The report goes to the switch feed of the collector. A collector that does not serve it yet is sent the report the way v2 sent it. ### Track the System Configuration The node reports its configuration files, and the output of a few commands, to the collector, which keeps their history: what changed on a node, and when, is at hand when a service misbehaves there. The first report sends every tracked file, and the following ones what changed and what was deleted. The agent configuration files are always tracked, the cluster secret masked. The other files and commands are listed in files of the `/etc/opensvc/sysreport.conf.d` directory, one item per line: | Item | Tracks | | :--- | :--- | | `FILE ` | a file | | `DIR ` | a directory, recursively | | `GLOB ` | the files and directories matching the pattern | | `EXC ` | none of the files matching the pattern, among the ones above | | `CMD ` | the output of a command, run without a shell | ``` # /etc/opensvc/sysreport.conf.d/system FILE /etc/hosts FILE /etc/fstab DIR /etc/netplan FILE /etc/multipath.conf FILE /etc/lvm/lvm.conf DIR /etc/sysctl.d CMD ip -br addr CMD lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT CMD multipath -ll CMD systemctl list-unit-files --state=enabled --no-pager --no-legend ``` The files of the directory must belong to root and not be writable by others, or they are ignored. Track no file holding a credential, as an `iscsid.conf` with CHAP passwords can. Prefer the commands whose output only changes when the configuration does: a counter or a timestamp in an output makes every report a change. The report runs on the `sysreport.schedule`, once a day by default. Every hour: ```bash om cluster config update --set sysreport.schedule=@60m ``` Run a report, or a full one, which sends every tracked file for the collector to replace what it holds of the node with:
```bash om node sysreport om node sysreport --force ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//action/sysreport" curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//action/sysreport?force=true" ```
A report the collector refused is followed by a full one, so no change is lost to a collector out of reach. ## Extra System Configurations ### Linux LVM2 OpenSVC controls volume group activation and deactivation. Old Linux distributions activate all visible volume groups at boot, some even re-activate them upon de-activation events. These mechanisms can be disabled using the following setup. It also provides another protection against unwanted volume group activation from a secondary cluster node. This setup tells LVM2 commands to activate only the objects tagged with the hostname. Opensvc makes sure the tags are set on start and unset on stop. Opensvc also purges all tags before adding the one it needs to activate a volume group, so opensvc can satisfy a start request on a service uncleanly shut down. #### /etc/lvm/lvm.conf Add the following root-level configuration node: tags { hosttags = 1 local {} } And add the ``local`` tag to all local volume groups. For example: sudo vgchange --addtag local rootvg Finally you need to rebuild the initrd/initramfs to prevent shared vg activation at boot. #### /etc/lvm/lvm_$HOSTNAME.conf cat >/etc/lvm/lvm_$HOSTNAME.conf < # Cluster Configuration Upon agent installation, the node is considered part of its own 1-node cluster. In `/etc/opensvc/cluster.conf`: * `cluster.secret` is initialized to a random value. * `cluster.name` is initialized to a random value. ## Bootstrap a new cluster If the node is to be enrolled in an existing cluster, skip this section. ### Add Heartbeats If the cluster seed node has no heartbeat setup, a `unicast` heartbeat with default settings is automatically added on daemon start. This default heartbeat requires every nodename to be resolved to an ip address reachable on 1215/tcp. If these requirements are not met, you can setup one or more custom heartbeats on the seed node before enrolling nodes. For example, a custom heartbeat configuration would be: om cluster config update --set hb#2.type=unicast --set hb#2.port=10002 The heartbeats status is summarized in the top section of the monitoring command output: om mon And the detailed status is shown by: om daemon hb status > ➡️ See Also > * [Heartbeats](internals.daemon.heartbeats.md) > * [Roles](configure.rbac.md) ### Add Stonith methods Stonith is optional. Skip to the next section if not concerned. On a new cluster, the stonith configuration can be applied on the first node. The enrolled nodes fetch this configuration from the cluster. For example, a dummy stonith configuration would be ``` om cluster config update --set stonith#node2.command=/bin/true ``` This configuration will execute `/bin/true` on the node taking over a service which was previously running on the now stalled `node2`. Good, isolated fencing packages are freely available. For one, https://github.com/ClusterLabs/fence-agents ## Enroll a Node A node is added to the cluster from the cluster side: a cluster node enrolls the new node, which leaves its own 1-node cluster and joins. The node to enroll must be a 1-node cluster. A node that is still a member of a cluster with peers is refused, because its peers would keep it in their `cluster.nodes`. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). That token is a cluster token: it speaks to > `node1` only. The node to enroll is not a member of > this cluster yet, so the calls addressed to it carry credentials of its own.
On the node to enroll `node2`, create a token with the `join` role, and store it in a file: ```bash sudo om daemon auth --role join --duration 10m >/root/node2.token ``` The token is valid for 60 seconds by default, so `--duration` gives time to copy the file to a cluster node. On any cluster node, for example `node1`, enroll `node2` using the copied token file: ```bash sudo om cluster enroll --node node2 --token /root/node2.token --wait ``` Node node2 accepted the join order Cluster nodes updated Waiting for node2 to beat Node node2 is alive * `--node` is the location of the node to enroll, in the `[://][:]` format. * `--token` is the path of the token file. The `OSVC_JOIN_TOKEN` environment variable is used when the flag is not set. * `--join-addr` is the location `node2` must use to reach `node1`, in the same format. By default, a name the cluster certificate is valid for is used. Set it when this name does not resolve to an address reachable from `node2`. * `--wait` waits for the `node2` heartbeat to beat in the cluster. Without it, follow the join with `om daemon events --filter 'NodeAlive,.node=node2'`. * `--timeout` is the maximum duration of the join, `1h` by default. The enroll requires the `root` grant.
The same enroll, driven from a host that is a member of neither cluster. **1. Create a user on the node to enroll** `POST /api/auth/token` only accepts the requests authenticated by the unix socket or by Basic Authentication, and a node freshly installed has no api user yet. So this first step is run locally on `node2`, once: ```bash om system/usr/enroller create --kw grant=join om system/usr/enroller key add --name=password --from /dev/urandom om system/usr/enroller key decode --name=password ``` **2. Ask that node for a token with the `join` role** ```bash curl -sSk -u enroller:$ENROLLER_PASSWORD -X POST \ "https://node2:1215/api/auth/token?role=join&access_duration=10m" \ | jq -r .access_token >/root/node2.token ``` * `role=join` is filtered against the grants of the user: the token carries the `join` role only if the user holds it. * `access_duration` defaults to 60 seconds, which gives little time for the next call. * `-k` is needed here: `node2` is still its own cluster, so its listener certificate is signed by its own certificate authority, which nothing outside of it trusts yet. That is also why the `join` role token carries a `ca` claim: it is what lets the cluster trust that certificate for the rest of the enroll. **3. Post the enroll order to a cluster node** ```bash curl -s -H "Authorization: Bearer $TOKEN" -X POST \ "https://node1:1215/api/cluster/enroll" \ -H "Content-Type: application/json" \ -d "$(jq -n --rawfile t /root/node2.token '{node:"node2", token:($t|rtrimstr("\n")), timeout:"1h"}')" ``` {"node":"node2"} * `node` is the location of the node to enroll, in the `[://][:]` format. The scheme defaults to `https` and the port to the daemon listener port. * `token` is the token obtained at the previous step. * `join_addr` is the location `node2` must use to reach `node1`, in the same format. By default, a name the cluster certificate is valid for is used. Set it when this name does not resolve to an address reachable from `node2`. * `timeout` is the lifetime of the `join` role token the cluster hands to `node2`, `1h` by default. It must outlive the node drain. The requester must be granted the `root` role. The response is sent as soon as `node2` has accepted the join order, and carries the nodename it reads from its own configuration. The join itself runs asynchronously. **4. Follow the join** ```bash curl -s -N -H "Authorization: Bearer $TOKEN" \ "https://node1:1215/api/node/name/node1/daemon/event?duration=1h&filter=JoinSuccess,added_node=node2&filter=JoinIgnored,candidate_node=node2&filter=JoinError,candidate_node=node2&filter=NodeAlive,.node=node2" ``` * `JoinSuccess` says the api node has added `node2` to its `cluster.nodes`. The node has not drained, reconfigured, or restarted yet at this point. * `JoinIgnored` says `node2` already was a cluster node. * `JoinError` says the cluster nodes update failed. * `NodeAlive` says the `node2` heartbeat beats in the cluster: it has fully joined. This is what `om cluster enroll --wait` waits for.
The enrolled node is drained before it leaves its cluster. As a 1-node cluster has nowhere to relocate its instances, they are stopped, stay down, and their configurations are removed. Its configuration directory is moved to the backup directory, then the cluster, heartbeat and certificate configurations are fetched from the cluster.
### Collector Registration The collector registration id of the enrolled node, its `node.uuid` keyword, is in the configuration directory moved aside. When the node was registered on the collector, the join keeps it registered: * Before its daemon starts, the enrolled node asks the cluster node it joins through to register it on the collector, with the registration id it held before the join. * That cluster node, authenticated on the collector by its own registration id, registers it. The collector moves the enrolled node to the app and the cluster of that node, and hands it a new registration id, which the enrolled node stores in its `node.uuid`. * Once its daemon has started, the enrolled node pushes the inventory `om node register` pushes: asset, packages, disks, sysreport and checks. This requires a collector server api 1.0.4 or later, allowing a node to register another one (the collector `allow_node_enroll` setting, on by default), and a cluster node registered on it. The join does not depend on the collector: a node that was not registered is left unregistered, and a node the collector refuses to register is warned. Register it with: om node register --credential /root/collector.cred > ➡️ See Also > * [Register on a Collector](configure.base.md#register-on-a-collector) ## Evict a Node A node is removed from the cluster by another cluster node. The evicted node leaves the cluster, then restarts its daemon alone, in a new 1-node cluster. The node to evict must be drained: frozen, with no running instance. Else its instances would stay up on a node the cluster no longer knows about, free to start a second time elsewhere.
On any other cluster node, for example `node1`, drain then evict `node2`: ```bash sudo om node drain --node node2 --wait sudo om cluster evict --node node2 --credential /root/node2.cred --wait ``` Node node2 accepted the leave order Cluster nodes updated Node node2 is restarting alone, its instances stay down * `--node` is the name of the cluster node to evict. It can not be the local node: run the command on one of its peers. * `--credential` is the path of a file holding the `:` of a user to create on the evicted node, in the `system` namespace with the `root` grant. The `OSVC_CREDENTIAL` environment variable is used when the flag is not set. * `--wait` waits for the cluster nodes to be updated. Without it, follow the eviction with `om daemon events --filter 'LeaveSuccess,removed_node=node2'`. * `--timeout` is the maximum duration of the leave, `1h` by default. The evict requires the `root` grant.
**1. Drain the node to evict** ```bash curl -s -H "Authorization: Bearer $TOKEN" -X POST \ "https://node1:1215/api/node/name/node2/action/drain" ``` {"orchestration_id":"e2ba0d0e-...."} The drain is asynchronous, and the eviction is refused with a `409` while `node2` is not drained, so wait for the drain to end before the next call. A drained node is frozen and runs no instance, which `GET /api/cluster/status` reports. **2. Post the evict order to another cluster node** ```bash curl -s -H "Authorization: Bearer $TOKEN" -X POST \ "https://node1:1215/api/cluster/evict" \ -H "Content-Type: application/json" \ -d '{"nodename":"node2","credential":"admin:secret","timeout":"1h"}' ``` {"detail":"node node2 is leaving the cluster","status":200,"title":"background cluster leave has been called"} * `nodename` is the cluster node to evict. It can not be the node serving the request: post this to one of its peers. * `credential` is the `:` of a user to create on the evicted node, in the `system` namespace with the `root` grant. It is optional, and checked before anything is ordered. * `timeout` is the maximum duration of the leave forked on the evicted node, `1h` by default. The requester must be granted the `root` role. The response is sent as soon as `node2` has accepted the leave order. The leave itself runs asynchronously. **3. Follow the eviction** ```bash curl -s -N -H "Authorization: Bearer $TOKEN" \ "https://node1:1215/api/node/name/node1/daemon/event?duration=1h&filter=LeaveSuccess,removed_node=node2&filter=LeaveIgnored,candidate_node=node2&filter=LeaveError,candidate_node=node2" ``` * `LeaveSuccess` says the cluster has dropped `node2` from its `cluster.nodes`. This is what `om cluster evict --wait` waits for. * `LeaveIgnored` says `node2` was not a cluster node. * `LeaveError` says the cluster nodes update failed. The daemon restart and the user creation happen after `LeaveSuccess`, on a node this cluster no longer observes, so no event reports them.
The evicted node ends up with a cluster secret of its own, so no user, token or certificate of the cluster can reach its api anymore. Without a credential, its api is only reachable locally, as root. Its configuration directory is moved to the backup directory.
> ➡️ See Also > * [Scheduler](internals.daemon.scheduler.md) > * [Heartbeats](internals.daemon.heartbeats.md) > * [Roles](configure.rbac.md) # Cluster Storage Pools Services can use `volume` resources to: * Abstract the disks and filesystems layout, which are hosting specificities, from the service deployment. A development cluster can for example define pools on a ceph cluster, while a production cluster can define pools on fc arrays. * Enable service redeployment while retaining the data. In this case the translation from volumes to disks and filesystems is delegated to the storage pool drivers. Pools are defined in the node configuration. Each pool is identified by its name (the section suffix). For example, a `pool#tank` section defines a pool named `tank`. The `default` pool always exist, even if not defined in the node configuration. If not explicitly changed, the `default` pool driver is `directory`. ## Volumes * A volume resource drives a volume object, automatically created upon service provisioning if not already existing. * The volume is hosted in the same namespace than its users. * If not explicitly set, the volume object name is `-vol-`. For example, a `svc1` service with a `volume#1` resource will create a `svc1-vol-1` volume object. * A volume object can be referenced by multiple services in the same namespace. * On provision, a service adds itself as a child of the volume objects mapped via volume resources. Due to this parent/child relation, stopping a volume object is delayed until all its consumers are stopped. * On unprovision, a service removes itself from the children list of the volume objects mapped via volume resources. * A consumer service instance stop also stops its node-affine volume object instances if the consumer service is the only child of the volume service. * A consumer service instance start always tries to start its node-affine volume object instances. ## Volume Resources Keywords - [Reference](/agent.reference.keywords/svc/volume.md) ### Access Modes * `roo` Read Only Once * `rwo` Read Write Once (default) * `rox` Read Only from multiple instances * `rwx` Read Write from multiple instances ### Access Mode to Volume Topology * `..x` access modes imply the volume is configured in a flex topology (active on all service nodes). * `..o` access modes imply the volume is configured in a failover topology (active on only one service node). ### Volume Resource Parameter Requirements * `..x + shared=true format=false` requires a shared block storage (SAN array, a rados blockdev gateway, ...) * `..x + shared=true format=true` requires either a shared block storage (SAN array, a rados blockdev gateway, ...) plus a cluster filesystem (gfs2, ocfs, ...), or a distributed cluster filesystem (CephFS, GlusterFS, NAS, ...) ## Pool Selector A volume resource requires a size and capabilities from the pool, via its `size`, `access`, `shared` and `format` keywords. If `pool` is not set explicitly to a pool name, the pool selector will return the available pool matching those criteria with the most free space. ## Pool Drivers Each pool is backed by a driver, which decides what a volume is made of: a directory, a logical volume, a zfs dataset, a lun on an array. The driver also decides which access modes the pool can serve, and so which volumes the selector may place on it. The drivers, their capabilities and their keywords are listed in [Pool Drivers](configure.pool.drivers.md). ## Pool Commands > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). ### Pool list
```bash om pool ls ``` NAME TYPE CAPABILITIES HEAD VOLUME_COUNT BIN_SIZE BIN_USED BIN_FREE default directory rox,rwx,roo,rwo /opt/opensvc/var/pool/directory 0 29.0g 3.57g 24.0g freenas freenas roo,rwo,shared,blk,iscsi array://freenas/osvcdata 6 195g 9.37g 185g mpool virtual roo,rox,rwo,rwx,shared templates/mpool 1 - - -
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/pool" ``` Each item carries the `name`, `type`, `capabilities`, `head`, `volume_count`, `size`, `used` and `free` the columns render, the cli only formatting the byte counts. The `name` and `node` query parameters narrow the answer, as `--name` and `--node` narrow the listing.
The sizes are what the pool can still hand out to volumes, which is the size a volume is asked for and the size a claim rations. A pool whose nodes each hold a copy of every volume hands out what the node with the least room can take, not the sum of what its nodes hold. `--physical` shows the storage behind the pool instead, and `--node` shows one line per node. ### Pool volumes
```bash om pool volume ls ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/pool/volume" ```
This lists the volumes each pool holds, and takes the same `--name` to narrow on one pool. ## Examples > A pool is a cluster keyword, so every `om cluster config update` below is > a `PATCH /api/cluster/config` carrying the same `set` operations, with the > `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). The first example shows both forms; the > others show the cli form only, the api one being the same substitution. ### loop pool Pool configuration
``` om cluster config update \ --set pool#loop.type=loop \ --set pool#loop.path=/bigfs \ --set pool#loop.mkfs_opt="-n ftype=1" \ --set pool#loop.fs_type=xfs ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=pool#loop.type=loop' \ --data-urlencode 'set=pool#loop.path=/bigfs' \ --data-urlencode 'set=pool#loop.mkfs_opt=-n ftype=1' \ --data-urlencode 'set=pool#loop.fs_type=xfs' \ "https://:1215/api/cluster/config" ```
```ini [pool#loop] type = loop path = /bigfs mkfs_opt = -n ftype=1 fs_type = xfs ``` The volume resource in the service ```ini [volume#1] size = 100m pool = loop ``` Resulting configuration of the volume object ```ini [disk#1] size = 104857600 type = loop file = /bigfs/.img [fs#1] type = xfs dev = {disk#1.exposed_devs[0]} mnt = /srv/ mkfs_opt = -n ftype=1 ``` ### zfs pool Pool configuration ``` om cluster config update \ --set pool#tank.type=zpool \ --set pool#tank.name=tank \ --set pool#tank.mkfs_opt="-o mountpoint=legacy -o dedup=on -o compression=on" ``` ```ini [pool#tank] type = zpool name = tank mkfs_opt = -o mountpoint=legacy -o dedup=on -o compression=on ``` The volume resource in the service ```ini [volume#1] size = 100m pool = tank ``` Resulting configuration of the volume object ```ini [fs#1] type = zfs dev = tank/ mnt = /srv/ mkfs_opt = -o mountpoint=legacy -o dedup=on -o compression=on ``` ### virtual pool, mirrored zpool over 2 SAN disks Pools configuration ``` om cluster config update \ --set pool#freenas1.type=array \ --set pool#freenas1.array=freenas1 \ --set pool#freenas1.sparse=true \ --set pool#freenas1.diskgroup=cluster1 \ --set pool#freenas2.type=array \ --set pool#freenas2.array=freenas2 \ --set pool#freenas2.sparse=true \ --set pool#freenas2.diskgroup=cluster1 \ --set pool#mpool.type=virtual \ --set pool#mpool.template=templates/mpool \ --set pool#mpool.capabilities="rox rwx roo rwo shared" ``` ```ini [pool#freenas1] type = array array = freenas1 diskgroup = cluster1 sparse = true [pool#freenas2] type = array array = freenas2 diskgroup = cluster1 sparse = true [pool#mpool] type = virtual template = templates/mpool capabilities = rox rwx roo rwo shared ``` The volume object template referenced by the vpool ```ini [DEFAULT] kind = vol nodes = * disable = true [disk#1] name = {namespace}-{svcname} type = zpool vdev = mirror {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]} shared = true [fs#1] dev = {disk#1.name} mnt = /srv/{namespace}/{svcname} type = zfs shared = true [fs#2] dev = {disk#1.name}/data mnt = {fs#1.mnt}/data type = zfs shared = true [fs#3] dev = {disk#1.name}/log mnt = {fs#1.mnt}/log type = zfs shared = true [volume#1] format = false name = {svcname}-1 pool = freenas1 size = {env.size} shared = true [volume#2] format = false name = {svcname}-2 pool = freenas2 size = {env.size} shared = true ``` ### virtual pool, mirrored lv over 2 SAN disks Pools configuration ``` om cluster config update \ --set pool#freenas1.type=array \ --set pool#freenas1.array=freenas1 \ --set pool#freenas1.sparse=true \ --set pool#freenas1.diskgroup=cluster1 \ --set pool#freenas2.type=array \ --set pool#freenas2.array=freenas2 \ --set pool#freenas2.sparse=true \ --set pool#freenas2.diskgroup=cluster1 \ --set pool#mvg.type=virtual \ --set pool#mvg.template=templates/mvg \ --set pool#mvg.capabilities="rox rwx roo rwo shared" ``` ```ini [pool#freenas1] type = array array = freenas1 diskgroup = cluster1 sparse = true [pool#freenas2] type = array array = freenas2 diskgroup = cluster1 sparse = true [pool#mvg] type = virtual template = templates/mvg capabilities = rox rwx roo rwo shared ``` The volume object template referenced by the vpool ```ini [DEFAULT] kind = vol nodes = * disable = true [volume#1] shared = true size = {env.size} name = {svcname}-1 pool = freenas format = false [volume#2] shared = true size = {env.size} name = {svcname}-2 pool = freenas format = false [disk#1] shared = true type = vg name = {namespace}-{svcname} pvs = {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]} [disk#2] shared = true type = lv name = root vg = {disk#1.name} size = 10m create_options = -m 1 [fs#1] shared = true mnt = /srv/{namespace}/{svcname} dev = {disk#2.exposed_devs[0]} type = ext4 [disk#3] shared = true type = lv name = data vg = {disk#1.name} size = $(60% * {disk#1.free}) create_options = -m 1 [fs#2] shared = true mnt = {fs#1.mnt}/data dev = {disk#3.exposed_devs[0]} type = ext4 [disk#4] shared = true type = lv name = log vg = {disk#1.name} size = $(40% * {disk#1.free}) create_options = -m 1 [fs#3] shared = true mnt = {fs#1.mnt}/log dev = {disk#4.exposed_devs[0]} type = ext4 ``` A logical volume is a resource of its own: om knows how big it is, can grow it, and can count it against the claim its namespace holds on the pool. Each size is read when that volume is made, so the shares are of what is left at that point: data takes 60% of the free space the root volume left, and log 40% of what data left. ### virtual pool, mirrored md over 2 SAN disks Pools configuration ``` om cluster config update \ --set pool#freenas1.type=array \ --set pool#freenas1.array=freenas1 \ --set pool#freenas1.sparse=true \ --set pool#freenas1.diskgroup=cluster1 \ --set pool#freenas2.type=array \ --set pool#freenas2.array=freenas2 \ --set pool#freenas2.sparse=true \ --set pool#freenas2.diskgroup=cluster1 \ --set pool#md.type=virtual \ --set pool#md.template=templates/md \ --set pool#md.capabilities="rox rwx roo rwo shared" ``` ```ini [pool#freenas1] type = array array = freenas1 diskgroup = cluster1 sparse = true [pool#freenas2] type = array array = freenas2 diskgroup = cluster1 sparse = true [pool#md] type = virtual template = templates/md capabilities = rox rwx roo rwo shared ``` The volume object template referenced by the vpool ```ini [DEFAULT] kind = vol disable = true nodes = * [disk#1] shared = true devs = {volume#1.exposed_devs[0]} {volume#2.exposed_devs[0]} type = md level = raid1 [disk#2] shared = true pvs = {disk#1.exposed_devs[0]} type = vg name = {namespace}-{svcname} [disk#3] shared = true type = lv name = root vg = {disk#2.name} size = 10m [fs#1] mnt = /srv/{namespace}/{svcname} dev = {disk#3.exposed_devs[0]} shared = true type = ext4 [disk#4] shared = true type = lv name = data vg = {disk#2.name} size = $(60% * {disk#2.free}) [fs#2] mnt = {fs#1.mnt}/data dev = {disk#4.exposed_devs[0]} shared = true type = ext4 [disk#5] shared = true type = lv name = log vg = {disk#2.name} size = $(40% * {disk#2.free}) [fs#3] mnt = {fs#1.mnt}/log dev = {disk#5.exposed_devs[0]} shared = true type = ext4 [volume#2] shared = true size = {env.size} name = {svcname}-2 pool = freenas format = false [volume#1] shared = true size = {env.size} name = {svcname}-1 pool = freenas format = false ``` ### drbd pool Pool configuration ``` om cluster config update \ --set pool#drbdloop.type=drbd om cluster config update \ --set pool#drbdvg.type=drbd \ --set pool#drbdvg.vg=centos ``` ```ini [pool#drbdloop] type = drbd [pool#drbdvg] type = drbd vg = centos ``` Example postgres service using a volume from a pool. ```ini [DEFAULT] nodes = * orchestrate = ha [volume#1] shared = true size = 200m name = {name} [container#1] type = oci image = postgres volume_mounts = {name}/data:/var/lib/postgresql/data secrets_environment = POSTGRES_PASSWORD=pg/password rm = true shared = true ``` # Pool Drivers Reference of the pool drivers, their capabilities and their keywords. To choose between them, see [Cluster Storage Pools](configure.pool.md). ## directory ### Capabilities rox, rwx, roo, rwo ### Layout A volume object from this type of pool contains: * a fs.directory resource, with `path=/`. ### Keywords - [Reference](/agent.reference.keywords/node/pool.directory.md) ## drbd ### Capabilities rox, rwx, shared, blk, roo, rwo ### Layout A volume object from this type of pool contains: If a vg is defined in the pool configuration, * a fs resource, with `dev=` * a drbd resource, layered over a logical volume of the pool vg * a lv resource If a zpool is defined in the pool configuration, * a fs resource, with `dev=` * a drbd resource, layered over a zvol of the pool zpool * a zvol resource If the pool configuration has neither vg nor zpool set, * a fs resource, with `dev=` * a drbd resource, layered over a logical volume * a lv resource * a vg resource * a loop resource, with image file hosted in the pool defined `path` or in `/pool//` ### Keywords - [Reference](/agent.reference.keywords/node/pool.drbd.md) ## freenas ### Capabilities roo, rwo, shared, blk, iscsi ### Layout A volume object from this type of pool contains: * a disk.disk resource named, with `name=` If the consumer has `format=true` (default), the volume object also contains: * a `fs.` resource, with `mnt=/srv/` ### Keywords - [Reference](/agent.reference.keywords/node/pool.freenas.md) ## loop ### Capabilities rox, rwx, roo, rwo, blk ### Layout A volume object from this type of pool contains: * a disk.loop resource, with `file=/.img` If the consumer has `format=true` (default), the volume object also contains: * a `fs.` resource, with `mnt=/srv/` ### Keywords - [Reference](/agent.reference.keywords/node/pool.loop.md) ## symmetrix ### Capabilities roo, rwo, shared, blk, fc ### Layout A volume object from this type of pool contains: * a disk.disk resource named, with `name=` If the consumer has `format=true` (default), the volume object also contains: * a `fs.` resource, with `mnt=/srv/` ### Keywords - [Reference](/agent.reference.keywords/node/pool.symmetrix.md) ## vg ### Capabilities rox, rwx, roo, rwo, blk, snap ### Layout A volume object from this type of pool contains: * a disk.lv resource, with `name=` If the consumer has `format=true` (default), the volume object also contains: * a `fs.` resource, with `mnt=/srv/` ### Keywords - [Reference](/agent.reference.keywords/node/pool.vg.md) ## share ### Capabilities rox, rwx, roo, rwo, shared ### Layout A volume object from this type of pool contains: * a fs.directory resource, with `path=/`. ### Keywords - [Reference](/agent.reference.keywords/node/pool.share.md) ## zpool ### Capabilities rox, rwx, roo, rwo, blk, snap ### Layout A volume object from this type of pool contains: * a fs.zfs resource, with `name=/` and `mnt=/srv/`. ### Keywords - [Reference](/agent.reference.keywords/node/pool.zpool.md) ## Virtual Pool Driver A virtual pool allow administrators to create complex layouts based on volumes from other pools. A typical use-case in a virtual pool allocating volumes mirrored over two other volumes allocated from arrays on two different sites. A virtual pool volume is created from a template volume object the administrator can design at wish to meet its specific needs. ### The template The `template` keyword names a volume object, and every volume the pool serves is a copy of its configuration. ```ini [pool#v1] type = virtual template = system/vol/v1 capabilities = roo rwo rox rwx blk ``` ```ini # system/vol/v1, a mirror over two volumes from the default pool [DEFAULT] size = 100mi devices_from = disk#1 [disk#1] type = md level = raid1 devs = {volume#1.exposed_devs} {volume#2.exposed_devs} [volume#1] pool = default size = {DEFAULT.size} [volume#2] pool = default size = {DEFAULT.size} ``` The template is designed, not provisioned. It describes a layout; the volumes the pool serves are the things that exist. A consumer asking this pool for 105Mi gets a copy whose `DEFAULT.size` is 105Mi, whose two sub-volumes are allocated from the `default` pool, and whose array is created over them. Write the template with references rather than names wherever a name would be the template's own. `{volume#1.exposed_devs}` above resolves in each copy to that copy's devices, where a device path written out would be the template's. ### What a copy does not inherit A copy is another volume, so what the template recorded of itself does not come with it. Those keywords are marked in the keyword reference: recorded: written when what it names is made, and reset when the object is cloned The object id is one. The uuid of an md array is another: it is written when the array is created, and is what assembles that array again afterwards. A copy keeping it would assemble the template's array under its own name instead of creating one of its own. They are reset in the copy, and written again for it when the copy makes what they name. The same reset is what `om create --config ` does, and what its `--restore` flag turns off. ### A provisioned template Provisioning the template object makes the very things its copies are supposed to make. The array it then holds writes its uuid into the template configuration, where it is visible: ```bash $ om system/vol/v1 config show [disk#1] type = md level = raid1 uuid@dev2n1 = 37a0a0a5:a10342a6:6f4fddb8:7de5d01d ``` That uuid is an administrator's to remove. The copies do not carry it, so it breaks nothing for them, but a template holding an array is a template that is also a volume, and the layout it is meant to describe is harder to read for it. ### Capabilities Capabilities are user defined. Nothing derives them from the template, so they are what the administrator declares the copies to be good for. ### Keywords - [Reference](/agent.reference.keywords/node/pool.virtual.md) # Cluster API The Cluster API provides remote access to the OpenSVC cluster management and monitoring functionality. It is accessible through the **agent listener** on any running cluster node. ## Access and Network Resolution The Cluster API URL (server name) can be configured to resolve to various network endpoints to ensure high availability and redundancy. Possible Resolutions: * **Single Floating IP** A single virtual IP (VIP) address, typically managed by a failover service like `system/svc/vip`. This is the most common and robust approach. * **Multiple Floating IP Addresses** A set of VIPs for complex network routing or multi-site setups. * **All Cluster Node IPs** Every physical or virtual IP address of all nodes in the cluster. * **Subset of Cluster Node IPs** A selection of node IP addresses, often used in large clusters or specific network segments. ## Authentication Methods The agent listener supports several industry-standard authentication methods to secure access to the API. ### Basic Authentication * **Mechanism:** The client provides the **username** and **password** in the request header of every API call and the password is compared to the `password` key of the `system/usr/` object. * **Authorization:** The RBAC grants are read from the `system/usr/` `grant` key. * **User Object:** The `system/usr/` object **must exist** on the cluster. ### X.509 Certificate Authentication * **Mechanism:** Authentication is based on a client-side X.509 certificate. * **Trust:** The certificate must be signed by one of the Certificate Authorities listed in the cluster `ca` keyword, `system/sec/ca` by default, and carry the **TLS client authentication** extended key usage. A certificate signed by any other authority is refused, whatever its `cn`. * **Username:** The username is derived from the **Common Name (`cn`)** field of the client certificate. A certificate with an empty `cn` is refused. * **Authorization:** The RBAC grants are read from the `system/usr/` `grant` key. * **User Object:** The `system/usr/` object **must exist** on the cluster. ### JSON Web Token (JWT) * **Mechanism:** Authentication is based on a JWT passed as a bearer token. * **Username:** The username is derived from the `sub` token claim. * **Authorization:** The RBAC grants are derived from the `grant` claim of a token the cluster issued itself, or from the `entitlements` claim of a token issued by an OIDC server. * **User Object:** The `system/usr/` object **does not need to exist** on the cluster **if** the tokens are issued and managed by an external OpenID Connect (OIDC) server. * **Issuance:** A token is obtained from the `POST /api/auth/token` endpoint, which accepts only the requests authenticated by the **unix socket** (a local `om` command) or by **Basic Authentication**. A request authenticated by a token or by a certificate is refused with `403 not allowed to create token`. A remote client therefore presents its password once to obtain a token, and asks for a refresh token at the same time if it needs to renew without presenting the password again. * **Availability:** This method was **added in OpenSVC v3** agents. ### Local Accounts and Sidecars (User Socket) * **Mechanism:** A process of the node connects to the **user socket**, `/run/opensvc-ux/http.sock`, and presents no credential: the daemon reads who connected from the connection, as the kernel recorded it, which the process can not choose. `ox` uses this socket when it runs as a user other than root with no context set. * **Access:** The socket is mode `0660`, in a directory of mode `0750`, both owned by root and the group the `listener.ux_owner_group` keyword names, `opensvc` by default. A node that does not know the group serves no socket, and logs why. * **Local accounts:** The grants are the union of the `grant` keyword of the `[uxgroup#]` sections of the cluster configuration naming the unix groups of the account, as the name service lists them, and of the `grant` keyword of the `system/usr/` object whose `uxuser` keyword names the login or the uid of the account. A `usr` object without `uxuser` is no local account, even one of the same name. * **Sidecars:** A process of a container started by the agent, the socket bind mounted into it, gets the grants of the `uxgrant` keyword of its container resource. The container is told by its cgroup, under the process group of the resource, never by its uid: root in a container is root on the cluster only if `uxgrant = root` says so. This needs a kernel giving the pidfd of the peer of a unix socket (Linux 6.5 and later). * **Root:** Root is **not** root on the user socket: a local `om` command uses the root socket. * **Writes:** Setting `uxuser`, a `[uxgroup#]` section, or `uxgrant` requires the root grant. * **Name service:** The login and the groups of an account are looked up with a deadline of 3 seconds, at most 8 lookups at once, and cached. A directory server not answering refuses the accounts it is asked about, never the root socket nor the sidecars, and a `uxuser` naming a uid keeps working without the name service. * **Availability:** This method was **added in OpenSVC v3** agents. Grant the members of the `ops` unix group the operator role on the `ns1` namespace:
``` om cluster config update --set uxgroup#ops.grant=operator:ns1 ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'set=uxgroup#ops.grant=operator:ns1' \ -G "https://:1215/api/cluster/config" ```
Let a sidecar act as root, the socket directory bind mounted into its container:
``` om ns1/svc/app config update \ --set container#sidecar.volume_mounts=/run/opensvc-ux:/run/opensvc-ux:ro \ --set container#sidecar.uxgrant=root ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'set=container#sidecar.volume_mounts=/run/opensvc-ux:/run/opensvc-ux:ro' \ --data-urlencode 'set=container#sidecar.uxgrant=root' \ -G "https://:1215/api/object/path/ns1/svc/app/config" ```
> **Note:** A `sudo` rule letting a user run `om` is a root grant, whatever arguments the rule allows: om runs commands as root by design, as the start command of an app, the editor of `om config edit`, or a task. Give the user a group mapped by a `[uxgroup#]` section, and `ox`, instead. ## Managing API Users ### Using an OpenID server The OpenID server will need to be configured to provide tokens with an `entitlements` claim, holding the grant expressions this cluster is to give the user. The claim is not named `grant`: that is the claim of the tokens the cluster issues itself. ``` [listener] openid_issuer = https://auth.mycorp.com/realms/clusters openid_client_id = om3 ``` The daemon reads `/.well-known/openid-configuration` to discover the provider, and fetches the signing keys from the `jwks_uri` it advertises. A token is accepted only if: * Its `aud` claim contains the `openid_client_id` value. * Its `iss` claim equals the `issuer` value **advertised by the discovery document**, which is not necessarily the `openid_issuer` url used to reach it. * Its signature verifies against a key of the `jwks_uri` key set, named by the token `kid` header. > ➡️ See Also > * [The cluster `listener.openid_client_id` keyword](../agent.reference.keywords/cluster/listener.html#openid_client_id) > * [The cluster `listener.openid_issuer` keyword](../agent.reference.keywords/cluster/listener.html#openid_issuer) > * [The user `grant` keyword](../agent.reference.keywords/usr/DEFAULT.html#grant) ### Creating Local Users (`system/usr/`) Skip this step if you plan to use only OIDC-issued tokens for a user. Local user objects define the user's cluster grants for Basic and X.509 authentication. ```bash # Create a cluster-wide administrator user om system/usr/root create --kw grant=root # Create user `usr1` with # `admin` rights on namespace `ns1` # `guest` (read-only) rights on namespace `ns2` om system/usr/usr1 create --kw grant="admin:ns1 guest:ns2" ``` ### Generating X.509 Certificates for Users This step is only necessary if you require X.509 authentication for the user. The certificate is named after the object and signed by the cluster certificate authority, which is what the api expects of it. The `cn` and `ca` keywords hold those two and rarely need to be set. ```bash # Generate Certificate: om system/usr/root certificate create om system/usr/usr1 certificate create # Download Keys (for client use): om system/usr/usr1 key decode --name certificate_chain om system/usr/usr1 key decode --name certificate om system/usr/usr1 key decode --name private_key ``` These commands will print the PEM-encoded keys to standard output. ## Testing the API The API manifest (documentation) is exposed by a demonstration agent for reference. A practical test using a JWT: ```bash # Generate a Temporary Token: $ TOKEN=$(sudo om daemon auth token --subject usr1 --duration 10m) # Call the `whoami` Endpoint: $ curl -o- -k -s -H "Authorization: Bearer $TOKEN" https://localhost:1215/api/auth/whoami ``` Example Output: ```json {"auth":"jwt","grant":{"guest":["ns2"], "admin": ["ns1"]},"name":"usr1","namespace":"system","raw_grant":"admin:ns1 guest:ns2"} ``` ## Configuring the Listener TLS Certificate The agent listener requires a TLS certificate to accept remote connections securely. Upon initial agent installation and daemon startup: * A self-signed Certificate Authority (CA) is automatically created as `system/sec/ca`. * A listener certificate, signed by this internal CA, is automatically created as `system/sec/cert`. The following steps are only required if you need to **re-sign the internal CA** or **switch to an external PKI**. ### Option 1: Using the Internal PKI The initial configuration is done automatically upon agent installation and daemon startup, creating: * The CA certificate at `system/sec/ca`. * The listener certificate at `system/sec/cert`. 1. **Import a valid certificate in `system/sec/ca`:** ```bash om system/sec/ca key change --name certificate --from ~/$CLUSTERNAME.pem om system/sec/ca key change --name certificate_chain --from ~/$CLUSTERNAME.chain om system/sec/ca key change --name private_key --from ~/$CLUSTERNAME.key ``` 2. **Recreate the listener certificate:** ```bash om system/sec/cert certificate create ``` ### Option 2: Using an External PKI 1. **Adjust the Listener Certificate Subject:** Only the `cn` is strictly required by OpenSVC, but the signing certificate authority will surely require more. ```bash om system/sec/cert config update --set cn=vip.$CLUSTERNAME.mycorp \ --set alt_names="node1.$CLUSTERNAME.mycorp node2.$CLUSTERNAME.mycorp" \ --set c=FR \ --set o=MyCorp \ --set ou=Support \ --set l=Paris \ --set st=IDF \ --set email=support@mycorp ``` 2. **Create the Listener Certificate Request:** ```bash om system/sec/cert certificate signing-request ``` 3. **Perform External Signing:** *(This is an external procedure to be done with your PKI tool using the generated csr.)* 4. **Load the Signed Certificate:** ```bash om system/sec/cert key change --name certificate_chain --from ~/$CLUSTERNAME_crt_chain.pem ``` The certificate is always first, then the CA certificate chain. 5. **Configure Certificate Revocation List (Optional):** ```bash om cluster config update --set cluster.crl=http://crl.mycorp ``` 6. **Restart the Daemons** ```bash om daemon restart --node="*" ``` ### Option 3: A Public Certificate from an ACME Directory `system/sec/cert` certifies the names the nodes reach each other by, which no public authority certifies. A name of the cluster published on the internet, as the name of its virtual address, can have a second certificate, which browsers and remote clients trust, obtained and renewed from an ACME directory like Let's Encrypt. The listener presents it to the clients asking that name, and `system/sec/cert` to the others. 1. **Create the sec of the public certificate:** ```bash om system/sec/public create --kw cn=vip.mycorp.com --kw email=support@mycorp.com \ --kw acme.directory=letsencrypt-staging ``` 2. **Have the listener present it, and answer its challenges:** ```bash om cluster set --kw listener.tls_secs=system/sec/public --kw listener.acme_port=80 ``` `listener.tls_secs` lists the secs whose certificates the listener presents to the clients asking one of their names. `listener.acme_port` has every node answer the http-01 challenges of these certificates in plain http, on a port the nodes must leave free for it, `80` being the one an ACME directory reads them from. The listener answers `/.well-known/acme-challenge/` there, and nothing else. 3. **Obtain the certificate:** ```bash om system/sec/public certificate renew ``` The renewal stores the challenge token in the sec, which the daemon replicates, and waits for every node to answer it before the directory reads it. The name may resolve to any node, or to a virtual address that moves between them. Once issued, the listener presents the certificate from the next connection, on every node, without a restart. 4. **Switch to the trusted directory** once the staging certificate is in place, keeping the ACME keys of the sec, which record the directory its certificate comes from: ```bash om system/sec/public set --kw acme.directory=letsencrypt om system/sec/public certificate renew ``` 5. **Schedule the renewal**, daily, from a cron of a node for example: it does nothing until the certificate expires within 30 days. The http-01 challenge certifies a name, not a wildcard: a `*.mycorp.com` certificate needs the dns challenge, which is not supported. [Certificates](apps.design.certificates.md) describes the renewal. > ➡️ See Also > * [Access Control](configure.rbac.md) for roles and grant management. # Role-Based Access Control A cluster reached over its API requires authentication, and every request is checked against the privileges of the authenticated user. Privileges live in the `grant` keyword of a `usr` object, as a whitespace-separated list of grant expressions. A grant is a role, optionally scoped to namespaces: ```ini [DEFAULT] grant = guest:prod* admin:test* ``` That user is a guest in every namespace whose name starts with `prod`, and an admin in those starting with `test`. A role given without a scope applies cluster-wide: ```ini [DEFAULT] grant = operator ``` ## Namespace roles These roles are the ones you scope to namespaces. Each inherits the previous. ### guest Read-only. List objects, read their configuration, status, schedule, resource info and logs. ### operator Inherits `guest`, and may change the state of what already exists: start, stop, restart, freeze, unfreeze, shutdown, switch, giveback, abort, run, clear, refresh status and resource info. An operator runs services. An operator does not create, delete or provision them. ### admin Inherits `operator`, and may change what exists: create and delete objects, edit their configuration, provision and unprovision their resources, purge them, and manage the keys of `cfg` and `sec` objects. ## Cluster roles These roles are not namespace-scoped. ### root All privileges. A user granted `root` can read and change the system's files and run commands as the root system user, so it is the grant to give last and audit first. ### squatter May create namespaces, and may write a namespace configuration. The creator is automatically granted `admin` on the namespace it creates. A namespace configuration says what the objects of that namespace may take of what the cluster shares: how much of a pool, how many addresses of a network, how much of the node the slice they all run under may use. That is given from outside the namespace, which is why it is this role and not `admin`. Administering the objects of a namespace and rationing the cluster between namespaces are two jobs, and holding the first does not give the second: no `admin` grant, scoped or not, writes a namespace configuration. ```bash $ om ns1/ config update --set claim#1.limit=500m # as admin:ns1 403 need one of [squatter root] grant ``` ### prioritizer May set object priority. Priority arbitrates between objects across namespace boundaries, which is why it is a cluster role rather than something an admin holds in its own namespace. ### blacklistadmin Reserved for clearing the client blacklist. The role is accepted in a grant, but no v3 endpoint requires it yet, so granting it currently confers nothing. ### heartbeat May use the relay handlers. A relay heartbeat authenticates on a foreign OpenSVC daemon with a user holding this role. om daemon relay status ### join, leave May add a node to the cluster, or remove one from it, plus the reads a node performs while joining, and the collector registration of the joining node by the node it joins through. These back the token minted on a node to enroll, which `om cluster enroll --token` uses to reach it: om daemon auth --role join --duration 10m `om cluster enroll` and `om cluster evict` themselves require the `root` grant. ## Keyword rules A role says which objects a user may write. Some keywords ask for more than the role, whatever the object, because of what setting them does to the node rather than to the object. `om config doc` says so, keyword by keyword: ```bash $ om config doc --kw container#1.run_args rbac: Requires the root grant. $ om config doc --kw fs#1.type rbac: Requires the root grant, except for the values flag. ``` The shape of the rules: * **A trigger requires `root`**, on every driver. `pre_start` and its siblings run a command of the user's choosing on the node. * **A driver group whose resources take something from the node requires `root`**, opening only the keywords that say what the object does with what it was given. An `ip` resource is the example: which address and which link are the node administrator's, while `network`, `netns` and `expose` are the object's. * **A driver group the policy says nothing about requires `root`.** A driver added to om later is refused until a rule allows it, rather than allowed until someone remembers to refuse it. * **Some rules read the value.** A container's `type` is open for the types that run an image under the container engine, and `root` for the ones that run on the node itself. A volume's `install` is `root` only when its source is a file of the server. * **A keyword can mean different things on different kinds.** A `pg_` limit on a namespace caps every object of that namespace and needs `squatter`; the same keyword on a service caps that service alone and is the object administrator's. ### A volume belongs to root, except its size A volume object is the storage itself, where a service is what consumes it. Its resources were written when the volume was served — which pool served it, which device it exposes, what filesystem is on it — so none of it is the namespace administrator's to edit, not even the flag filesystems that are open on a service. `DEFAULT.size` is the exception. A namespace grows its own volume within the claim it holds on the pool, and rationing that growth is what the claim is for, so the size is written by the namespace and bounded by the claim. Asking for different storage is a keyword of the service that consumes the volume, not of the volume. ### A write answers for what it changes A configuration you may not have written is not a configuration you may not change. The keywords already in it were written by whoever was allowed to write them, and only what a write changes is checked against the policy. An object administrator can therefore fix a comment on a service that mounts a filesystem, without holding the `root` grant that mounting one asks for. Creating an object has nothing to compare against, so every keyword of it is a change and answers for itself. Unsetting a keyword is a change like setting one: a user who may not set a keyword may not unset it either. What counts as a change is what the value evaluates to, not what is written. A keyword can hold a reference, and moving what it refers to rewrites it without touching it: ```ini [env] cmd = /bin/evil # written by the namespace administrator [app#1] pre_start = {env.cmd} # written by root, and now running something else ``` That write is refused. References are followed however deep they go, and on every node the object runs on, because a keyword is written once per node and a reference that resolves the same here can resolve to something else on a peer. > ➡️ See Also > * [Cluster API](configure.api.md) > * [Cluster Configuration](configure.cluster.md) for the enroll and evict procedures. # Cluster API Client The `ox` program uses only the agent API and mirrors the `om` commandset, which makes it suitable for managing one or more clusters from a tiers linux box. ## Configure remotes The remotes configuration is stored in JSON format in the `~/.config/opensvc/contexts.json` file. A context names a cluster to talk to and a user to talk as. Clusters and users are defined once and referenced by name, so several contexts can share them. Example: { "clusters": { "dreamy-leopard": { "server": "https://dreamy-leopard.example.com:1215", "insecure": true }, "bold-rat": { "server": "https://bold-rat.example.com:1215" } }, "users": { "john": {}, "mary": {} }, "contexts": { "john@dreamy-leopard": { "cluster": "dreamy-leopard", "user": "john" }, "mary@bold-rat": { "cluster": "bold-rat", "user": "mary", "namespace": "prod", "access_token_duration": "1h", "refresh_token_duration": "1d" } } } The file holds no password. Authentication is by JSON Web Token: `ox context login` exchanges a password for tokens once, and the tokens are cached outside this file. The commands below write the same file, and create it if it does not exist yet: $ ox context cluster add --name dreamy-leopard --server https://dreamy-leopard.example.com:1215 --insecure $ ox context user add --name john $ ox context add --name john@dreamy-leopard --cluster dreamy-leopard --user john ### Context keys | Key | Description | | --- | --- | | `cluster` | The name of the cluster definition to connect to. Required. | | `user` | The name of the user definition to connect as. Required. | | `namespace` | Restrict the session to this namespace. The tokens are then granted for it alone. | | `access_token_duration` | How long an access token stays valid. The access token is the one sent with each request, so a short duration limits what a leaked one is worth. | | `refresh_token_duration` | How long the session can be renewed without typing the password again. When it expires, `ox context login` has to be run again. | Durations are written as a number and a unit, for example `1h` or `1d`. The units are `d`, `h`, `m` and `s`, and several can be combined, as in `1d12h`. ## Login `ox context login` requests the tokens and caches them. $ ox context login --context john@dreamy-leopard Password for john@dreamy-leopard: Login successful. Switch to this context with : export OSVC_CONTEXT=john@dreamy-leopard Called with no context, it offers a menu of the configured ones. When the standard input is not a terminal, the password is read from it instead of being asked for. This is how a script logs in without putting the password on a command line, where the process table would show it to every user of the machine, or in an environment variable, which every child process inherits: # from a password manager $ pass show opensvc/john | ox context login --context john@dreamy-leopard # from a file only this user can read $ ox context login --context john@dreamy-leopard < ~/.config/opensvc/john.password # from a variable already held by the shell, without a file $ ox context login --context john@dreamy-leopard < <(printf %s "$password") The password is the first line, and a trailing newline is not part of it. The context cannot be picked from a menu in that case, since the menu is read on the terminal the password is no longer typed on: name it with `--context` or with the `OSVC_CONTEXT` environment variable. `ox context list` shows the configured contexts and the state of their tokens: $ ox context list NAME AUTHENTICATED ACCESS_EXPIRE REFRESH_EXPIRE AUTHENTICATED_AT john@dreamy-leopard false - - - mary@bold-rat false - - - `ox context logout` drops the cached tokens. ## Terminal UI At this point, executing `ox` with no argument launches the Terminal User Interface, and offers a context selector dialog. The `h` keypress displays a help page. ## Commandline UI # Set a context # ------------- $ export OSVC_CONTEXT=john@dreamy-leopard # Manage like om # -------------- $ ox cluster get --kw cluster.name dreamy-leopard $ ox node ls NAME AGENT STATE dreamy-leopard-node-1 3.0.0 idle dreamy-leopard-node-2 3.0.0 idle dreamy-leopard-node-3 3.0.0 idle $ ox svc ls OBJECT AVAIL OVERALL svc2 down down svc1 down down # Cluster Networks A cluster network gives objects private addresses that om hands out, tracks, and publishes in the cluster DNS. An object names a network, and gets an address, a device, a netmask and a gateway from it, with nothing to install and no external store to run: the addresses are allocated by the agent itself. The resource that takes an address from a network is `ip.netns`. It plumbs the address into the network namespace of a container of the object. ## Network types | Type | Scope | Use | | :--- | :--- | :--- | | `bridge` | one node | addresses reachable from the node and its containers only | | `routed_bridge` | the cluster | addresses routed from node to node, each node drawing from a subnet of its own | | `lo` | one node | the loopback, which om allocates nothing in | A network named `default`, of the `bridge` type on `10.22.0.0/16`, exists on every node of a fresh installation, so an object can take an address before any network is declared. ## Declaring networks Networks are sections of the cluster configuration, or of the node configuration for a network that exists on one node only. ### Bridge A bridge network is node local: its addresses are not routed between nodes, so every node draws from the whole subnet, and the same address on two nodes never meets.
``` om cluster config update --set network#local1.type=bridge \ --set network#local1.network=10.10.10.0/24 ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/cluster/config?set=network%23local1.type%3Dbridge&set=network%23local1.network%3D10.10.10.0%2F24" ```
Use `om node config update` instead to declare it on the current node only. ### Routed bridge A routed bridge spans the cluster. Its subnet is split into one segment per node, each node allocating in its own segment, and om routes each segment to its node, through a tunnel when the peer is not on the same subnet:
``` om cluster config update --set network#backend1.type=routed_bridge \ --set network#backend1.network=10.11.0.0/16 \ --set network#backend1.mask_per_node=22 ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/cluster/config?set=network%23backend1.type%3Drouted_bridge&set=network%23backend1.network%3D10.11.0.0%2F16&set=network%23backend1.mask_per_node%3D22" ```
`mask_per_node` is the prefix length of the segment each node gets. Here every node gets a `/22`, 1024 addresses, and the segments are assigned in the order of `cluster.nodes`, recorded as `subnet@`: * node 1: `10.11.0.0/22` * node 2: `10.11.4.0/22` * node 3: `10.11.8.0/22` `ips_per_node`, the former way to size the segments, is still read, but `mask_per_node` wins when both are set: a count of addresses is unwieldy for an ipv6 network. The tunnel endpoints are the addresses the nodenames resolve to. Name others when the nodes reach each other on another network: ``` om cluster config update --set network#backend1.addr@node1=192.168.1.1 \ --set network#backend1.addr@node2=192.168.1.2 ``` Some hosting providers route traffic between servers even on a common subnet. Force the tunnels there: ``` om cluster config update --set network#backend1.tunnel=always ``` The tunnel mode is `ipip` for an ipv4 network and `ip6ip6` for an ipv6 one. `gre`, set with `tunnel_mode`, carries both, and works where some providers refuse `ipip`. What a node reaches its peers by, the addresses of both ends and the tunnel, is what a [template](apps.design.datastores.md#templates) reads with the `network`, `networks` and `tunnels` functions. A program configured from it, as the strongSwan of the [strongSwan Mesh](../howtos/strongswan.mesh.md) howto encrypting the tunnels, follows the peers with a task running on `NetworkPeersUpdated`. ### Masquerading Traffic leaving a `bridge` or `routed_bridge` network is masqueraded behind the node address. Set `public=true` on a network whose addresses are routable as they are, to leave them unmasqueraded. ### Applying a network `om network setup` creates the bridges, routes, tunnels and nftables rules of the networks on the node it runs on. The daemon runs it on start and on every cluster configuration change, so a network declared on a running cluster is set up on every node without asking. It has no api counterpart: it acts on the node it runs on. A setup that changed the peers of a network, a node joining or leaving it, the address of a node or the tunnel to it, publishes a `NetworkPeersUpdated` event on the node, which a task can [run on](internals.daemon.scheduler.md#event-triggered-runs). ## How addresses are allocated Each node allocates for itself, which is what makes the allocation safe without a lock held across the cluster: a `routed_bridge` node draws from its own segment only, and a `bridge` address never leaves its node. * **An address belongs to a resource.** The reservation is named after the object and the resource id, so an object holding several `ip.netns` resources in one network gets an address for each. * **The same resource gets the same address.** The search for a free address starts from a point derived from the object and the resource id, so a resource that stops and starts again gets its address back as long as nobody took it meanwhile, and its DNS name keeps pointing to the same place. * **A stop releases the address**, and so does a start that fails and rolls back, so an object that is down holds no address. * **The reservations are files** under `/ipam//`, one per address, holding the object path and the resource id that took it. The netmask of the resource is the one of the node's range, and its gateway is the first address of that range, which the node sets on the bridge. An earlier agent left the allocation of these networks to the `host-local` CNI plugin, which allocates the first free address, so an address moved as the neighbours of an object came and went. The first network setup of an upgraded node adopts the addresses the running resources hold, so no address is handed out twice during the transition. ## Using a network in an object A container of the object, usually a pause container holding the network namespace, and an `ip.netns` resource naming the network: ```ini [container#0] type = podman image = ghcr.io/opensvc/pause rm = true [ip#0] type = netns netns = container#0 network = backend1 ``` Naming the network is enough. The resource takes from it: * `name`, the address, allocated as described above when left empty. * `dev`, the bridge of the network, `obr_` unless the network names another with its `dev` keyword. * `netmask` and `gateway`, from the node's range. Any of them set in the object configuration wins over the network. The other containers of the object join the namespace of the pause container with `netns = container#0`, and reach each other over `127.0.0.1`. `ip.netns` also plugs an address in modes other than a bridge: `macvlan`, `ipvlan-l2`, `ipvlan-l3`, `ipvlan-l3s`, `ovs` and `dedicated`, set with `mode`. See the [`mode` keyword](../agent.reference.keywords/svc/ip.netns.md#keyword-mode) for what each one lets the host and the containers reach. A `disk.drbd` resource sends its replication traffic over a network with its `network` keyword. The address of each node is then the first of the segment the node was assigned, so the network is a `routed_bridge`: a network giving the nodes no segment of their own, as a `lan`, is refused. Name the addresses with `addr@` on the resource there. ### Names The address is published in the cluster DNS under the name of the object: ``` $ getent hosts web.test.svc.mycluster 10.11.0.78 web.test.svc.mycluster ``` A second address of the same object is told apart with `dns_name_suffix`, which is appended to the published name. `expose` publishes SRV records for the services the address serves, for example `expose = 443/tcp` publishes `_443._tcp.web.test.svc.mycluster`. ### Namespace claims A namespace can be capped on the addresses it takes from a network, with a `claim` section of its configuration. An allocation taking the namespace over its limit is refused, while an address a resource already holds is never re-claimed, so an object at the limit still restarts. See [network claims](apps.design.namespaces.md#network-claims). ## Inspecting networks The networks, and how full they are:
``` $ om network ls NAME TYPE NETWORK SIZE USED FREE backend1 routed_bridge 10.11.0.0/16 64ki 3 64ki lo lo 127.0.0.1/32 1 0 1 default bridge 10.22.0.0/16 64ki 0 64ki ```
``` curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/network" ```
The addresses allocated, with the object and resource holding each, here in one network:
``` $ om network ip ls --name backend1 OBJECT NODE RID IP NET_NAME NET_TYPE test/svc/web node1 ip#0 10.11.0.78 backend1 routed_bridge test/svc/db node2 ip#0 10.11.4.12 backend1 routed_bridge ```
``` curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/network/ip?name=backend1" ```
## CNI The `ip.cni` driver plugs an address through CNI plugins instead. It is the one to use for a network defined by a third-party CNI plugin, or to publish a port of the address on the node addresses: an `expose` entry with a host port, as `80:8001/tcp`, is configured by the `portmap` plugin. `ip.cni` needs the CNI plugins installed, where the `cni.plugins` and `cni.config` keywords of the node configuration say. Plugged into an om network, `bridge` or `routed_bridge`, it draws its address from the allocation described above, like `ip.netns`. Plugged into a network om does not define, the address is the plugin's to allocate. > ➡️ See Also > * [`ip.netns` keywords](../agent.reference.keywords/svc/ip.netns.md) > * [`ip.cni` keywords](../agent.reference.keywords/svc/ip.cni.md) > * [`network.routed_bridge` keywords](../agent.reference.keywords/cluster/network.routed_bridge.md) > * [Namespaces](apps.design.namespaces.md) # Cluster Domain Name Server The OpenSVC agent daemon can act as a remote backend for PowerDNS, serving dynamic records for services deployed within the cluster. This functionality is particularly useful when services are assigned IP addresses on private backends with internal IPAM. If enabled, the agent configures the container's resolver (`nameserver` and `search`) to use the internal name server when starting a container. This feature is not enabled by default. ## Records * **A record:** `...svc.` for each resource that includes `ipaddr` and `hostname` in the `info` map in its states. * **Round-Robin A Record:** `..svc.` where each resource that includes `ipaddr` in the `info` map in its states is included in the round-robin. * **Round-Robin SRV Record:** `_._...svc.` where each resource with an `expose` keyword matching `/` is included in the round-robin.
Note: A service created without a specific namespace defaults to the `root` namespace.
## Implementation * A farmed (flex) service. * Each instance runs an authoritative PowerDNS server, a PowerDNS recursor and a recursor cache janitoring daemon. * Each component runs as a privileged docker instance to have r/w access to shared unix domain sockets. * The DNS server and recursor share the node network namespace. * The PowerDNS server uses the dns thread of the OpenSVC daemon as a remote backend. Communications go through the `/var/lib/opensvc/dns/pdns.sock` unix domain socket. ### Docker images * ghcr.io/opensvc/pdns_server * ghcr.io/opensvc/pdns_recursor * ghcr.io/opensvc/pdns_janitor ## Configure ### Preliminary steps * Make sure the cluster configuration :kw:`cluster.name` is set to a meaningful, unique site-wide, value. It can be a fqdn like `cluster1.my.org`, or just a basename like `cluster1`. * Choose at least 2 cluster nodes that will act as DNS backends. * Choose a free port for the DNS to listen on (default is `5300`). * Identify the ip addresses you want the DNS to listen on (public or private). In the following examples, `192.168.100.11` and `192.168.100.14`. * Make sure these ip addresses are resolved to the node name as declared in the :kw:`cluster.nodes` keyword (edit /etc/hosts if necessary). * OpenSVC agent installed, minimum version 2.1-1651 * Make sure docker or podman is installed and running on selected dns nodes. * Make sure CNI is installed * Make sure you have access to pull from docker.io on selected dns nodes (you can pre-pull or save/load the images if not). > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). `cluster.dns` is a cluster keyword, so it is > written on any node with `PATCH /api/cluster/config`, and `system/cfg/dns` is > `/api/object/path/system/cfg/dns`. ### Declare DNS backends
```bash om cluster config update --set cluster.dns+=192.168.100.11 --set cluster.dns+=192.168.100.14 ```
```bash curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=cluster.dns+=192.168.100.11' \ --data-urlencode 'set=cluster.dns+=192.168.100.14' \ "https://:1215/api/cluster/config" ```
### Deploy the DNS service
```bash om system/cfg/dns create om system/cfg/dns key add --name server --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/pdns.conf.template om system/cfg/dns key add --name recursor --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/recursor.conf.template om system/cfg/dns key add --name configure --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/configure om system/svc/dns deploy --config https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns/dns.conf ```
```bash base=https://raw.githubusercontent.com/opensvc/opensvc_templates/main/dns curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary '[DEFAULT]' \ "https://:1215/api/object/path/system/cfg/dns/config/file" for key in server:pdns.conf.template recursor:recursor.conf.template configure:configure do curl -sL "$base/${key#*:}" | curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary @- \ "https://:1215/api/object/path/system/cfg/dns/data/key?name=${key%%:*}" done curl -sL "$base/dns.conf" | curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary @- \ "https://:1215/api/object/path/system/svc/dns/config/file" curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/system/svc/dns/action/provision" ``` The api fetches nothing for you: `--from ` and `--config ` are the cli reading the template and sending its bytes, which is what the pipes above do. `deploy` has no endpoint of its own either, being a create followed by a provision: post the configuration file, then post the provision action, which answers the `orchestration_id` of the queued orchestration.
Note: Make sure `allow-from` in the `recursor` key of `system/cfg/dns` contains all the cluster backend networks allowed to request the DNS (the default is `127.0.0.1/32,10.0.0.0/8,fd00::/112,::1`).
### Configure the nodes resolver On every node, execute: # create the resolved configlet directory if it doesn't exist yet $ mkdir -p /etc/systemd/resolved.conf.d # install a configlet routing all requests to the cluster domain to the cluster nameservers $ cat - </etc/systemd/resolved.conf.d/opensvc.conf [Resolve] Domains=$(om cluster config get --kw cluster.name) DNS=$(om cluster config get --kw cluster.dns) EOF # activate the new configuration $ systemctl restart systemd-resolved.service ## Verify ### Verify the backend #### Dump the records served by opensvc to the PowerDNS server
```bash om daemon dns dump ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//daemon/dns/dump" ```
#### Test the unix socket served by opensvc for the PowerDNS server ``` echo '{"method": "list", "parameters": {"zonename": "cluster1."}}' | sudo socat - unix://var/lib/opensvc/dns/pdns.sock | jq ``` ### Verify the DNS server #### Dump the zone contents asking the PowerDNS server ``` dig +noall +answer cluster1. AXFR @192.168.100.11 -p 5300 ``` ### Verify the DNS recursor ``` dig +short cluster1. SOA @192.168.100.11 ``` ## Administration ### Add forwarding for the reverse zones Either switch to `--forward-zones-file` or add new elements to `forward-zones` in the `recursor` key of `system/cfg/dns`. # Ingress Gateway Services obtaining their IP address from a backend network are not reachable from outside the cluster. An ingress gateway is a service with a public address that routes the requests of the outside world to them. HAProxy is our recommended program for it, routing layer 4 and layer 7 traffic. This page deploys one, with certificates the agent renews. ## How the gateway finds the backends The gateway names the backend servers by their [cluster dns](configure.dns.md) names, `..svc.`, and HAProxy resolves them again as they move, with its `resolvers` mechanism: resolvers clusterdns parse-resolv-conf accepted_payload_size 8192 HAProxy runs in a container started by the agent, whose `/etc/resolv.conf` names the cluster nameservers, and `parse-resolv-conf` reads them from there. A backend then refers to these resolvers: backend svc1 server-template svc1_ 1 svc1.ns1.svc.${CLUSTERNAME}:80 resolvers clusterdns check init-addr none `init-addr none` lets HAProxy start while a backend is not resolvable yet, its service being down or not deployed. ## Topologies * **One gateway for the cluster:** a single HAProxy, in a failover service. The public address moves with it. This is the deployment below. * **One gateway per node:** every node runs a HAProxy exposing the same servers, in a `topology = flex` service, and an upstream load balancer picks one. ## Deploy The gateway is three objects of a namespace, here `igw`: * `igw/sec/haproxy`, the certificate of the https frontend * `igw/cfg/haproxy`, the haproxy configuration * `igw/svc/haproxy`, the service running haproxy, which installs the other two in a volume ### The certificate A self-signed certificate is enough to start with. The [Certificates](#certificates) section below replaces it with one the browsers trust. ```bash om igw/sec/haproxy create --kw cn=svc1.example.com om igw/sec/haproxy certificate create ``` ### The haproxy configuration ``` global tune.lua.bool-sample-conversion normal lua-load /usr/local/etc/haproxy/acme-http01-webroot.lua # a reload waits no longer than this for the open connections to end hard-stop-after 30s resolvers clusterdns parse-resolv-conf accepted_payload_size 8192 defaults mode http timeout connect 5s timeout client 30s timeout server 30s frontend http bind :80 acl acme path_beg /.well-known/acme-challenge/ http-request use-service lua.acme-http01 if METH_GET acme redirect scheme https code 301 if !acme frontend https bind :443 ssl crt /usr/local/etc/haproxy/certs/ use_backend svc1 if { hdr(host) -i svc1.example.com } backend svc1 server-template svc1_ 1 svc1.${NAMESPACE}.svc.${CLUSTERNAME}:80 resolvers clusterdns check init-addr none ``` * The `http` frontend redirects to https, except the ACME challenge path, which a small lua responder serves from the webroot the certificate renewal writes in. * The `https` frontend loads every certificate of the `certs/` directory, and presents the one whose names match the server name the client asks for. Serving another domain is installing one more certificate there. * `hard-stop-after` bounds a reload, as the one following a renewed certificate: the old process waits for the open connections to end, and a websocket can stay open for hours. Save it as `haproxy.cfg`, and store it in the cfg, with the lua responder: ```bash om igw/cfg/haproxy create om igw/cfg/haproxy key add --name haproxy.cfg --from haproxy.cfg om igw/cfg/haproxy key add --name acme-http01-webroot.lua \ --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/igw_haproxy/common/acme-http01-webroot.lua ``` ### The service ```ini [DEFAULT] nodes = * orchestrate = ha [volume#1] type = shm name = {name} size = 1m user = 99 group = 99 install = ` /haproxy/haproxy.cfg from ./cfg/{name} key haproxy.cfg signal HUP:container#1 /haproxy/acme-http01-webroot.lua from ./cfg/{name} key acme-http01-webroot.lua signal HUP:container#1 /haproxy/certs/{name}.pem from ./sec/{name} key fullpem signal HUP:container#1 /haproxy/acme-challenges/ ` [ip#1] type = netns network = default netns = container#0 shared = true [container#0] image = ghcr.io/opensvc/pause rm = true shared = true [container#1] image = docker.io/haproxy:3.2-alpine rm = true environment = CLUSTERNAME={clustername} NAMESPACE={namespace} volume_mounts = ` volume#1:/haproxy:/usr/local/etc/haproxy volume#1:/haproxy/acme-challenges:/acme-challenges ` netns = container#0 restart = 3 shared = true [task#acme] type = acme secs = {name} webroot = volume#1:/haproxy/acme-challenges schedule = @1d run_requires = container#1(up) ``` * `volume#1` is a small memory-backed volume holding the files haproxy reads, installed from the cfg and the sec. Nothing in it needs keeping or replicating: it is installed anew on every start, so the service fails over without any storage. * The `signal` of the install lines makes haproxy reload when a key changes, on every node running it: a configuration edited, or a certificate renewed, takes effect without a restart. * `user` and `group` are the ones haproxy runs as in its image. A `sec` key installs readable by its owner only. * `ip#1` takes an address from a [cluster network](configure.networks.md), in the network namespace of the `container#0` pause container, which the haproxy container joins. * `volume_mounts` names the paths of the volume by the rid of its resource. The lua responder reads the challenge tokens from `/acme-challenges`. * `task#acme` renews the certificates, see [Certificates](#certificates). Save it as `haproxy.conf`, and deploy: ```bash # Use a network spanning the cluster nodes, if you have one, and make sure # the cluster nameservers answer requests from it. om igw/svc/haproxy deploy --config haproxy.conf --kw ip#1.network=default --wait ``` ### Test Deploy a web server as the `svc1` backend, saved as `svc1.conf`: ```ini [DEFAULT] nodes = * orchestrate = ha [ip#1] type = netns network = default netns = container#0 [container#0] image = ghcr.io/opensvc/pause rm = true [container#1] image = docker.io/nginx:alpine rm = true netns = container#0 ``` ```bash om igw/svc/svc1 deploy --config svc1.conf --kw ip#1.network=default --wait ``` The gateway answers on its backend address already, from a cluster node: ```bash IP=$(om igw/svc/haproxy resource ls ip#1 --node $HOSTNAME -o json | jq -r '.items[].data.status.info.ipaddr') # fake the dns resolution of svc1.example.com to the gateway address curl -k --resolve svc1.example.com:443:$IP https://svc1.example.com ``` ## Public address For the clients outside the cluster, add a public address to the service, in the network namespace of `container#0`, so it moves with the gateway: ```ini [ip#2] type = netns name = 203.0.113.10 netmask = 24 gateway = 203.0.113.1 dev = br-prd mode = bridge netns = container#0 shared = true ``` `dev` is the bridge of the nodes on the public network. The dns records of the domains served point to `name`. [Cluster Networks](configure.networks.md) describes the other modes. ## Certificates The https frontend presents the certificate of `igw/sec/haproxy`, self-signed so far. Once the domain resolves to the public address, obtain it from Let's Encrypt instead, trying the setup out against its staging directory first: ```bash om igw/sec/haproxy set --kw acme.directory=letsencrypt-staging om igw/svc/haproxy instance run acme --node --follow ``` The run proves the domain through the `http` frontend, stores the certificate issued in the sec, and installs it, haproxy reloading. Once the staging certificate is in place, switch to the directory browsers trust, and run again: ```bash om igw/sec/haproxy set --kw acme.directory=letsencrypt om igw/svc/haproxy instance run acme --node --follow ``` From then on, the daily `task#acme` renews the certificate within 30 days of its expiry. To serve another domain, create a sec for it, install its `fullpem` in `certs/`, add it to the task, and run the task: ```bash om igw/sec/www create --kw cn=www.example.com --kw acme.directory=letsencrypt om igw/svc/haproxy set \ --kw volume#1.install+="/haproxy/certs/www.pem from ./sec/www key fullpem signal HUP:container#1" \ --kw task#acme.secs+=www om igw/svc/haproxy instance run acme --node --follow ``` INF igw/sec/www: set key fullpem INF igw/svc/haproxy: volume#1: install key fullpem from igw/sec/www to /srv/haproxy.igw.vol.cluster1/haproxy/certs/www.pem with owner 99:99 perm -rw------- INF igw/svc/haproxy: container#1: send SIGHUP signal to container igw..haproxy.container.1 (pid 3869733) INF igw/svc/haproxy: task#acme: igw/sec/www: certificate issued for www.example.com: no certificate A namespace administrator may configure every piece of this: the task writes only in the volume of its service, and renews only the secs the namespace may use. [Certificates](apps.design.certificates.md) has the details: when a renewal is due, the secs of other namespaces, and the renewal from the command line. # Cluster Relay A relay is typically installed on a third site to provide two services to clusters: * Act as a key-value store for relay-type heartbeats to exchange their encrypted datasets using different network paths. * Act as a voter in quorum votations. ## Compatibility A relay is a simple standalone OpenSVC server. * A v2 server can serve as a relay for v2 clusters * A v3 server can serve as a relay for v3 clusters. * A v2 server can not serve as relay for v3 clusters * A v3 server can not serve as relay for v2 clusters Changes: * The v2 server relay API handlers require the cluster nodes communicate with the relay node secret, granting `root` privilege. * The v3 server relay API handlers require the cluster nodes communicate with a user account with the `heartbeat` privilege. * The v3 server relay API handlers store the cluster nodes datasets in per-user namespaces, so a single relay can serve multiple client populations. ## Configure a v3 relay server > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). A user is an object of kind `usr`, so > `system/usr/relay` is `/api/object/path/system/usr/relay`. Install the opensvc-server package, then create the user account the clusters will use:
```bash om system/usr/relay create --kw grant=heartbeat ```
Creating an object through the api is posting its configuration file: ```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary $'[DEFAULT]\ngrant = heartbeat\n' \ "https://:1215/api/object/path/system/usr/relay/config/file" ```
Set a password for the user. Example with a random generated password:
```bash om system/usr/relay key add --name=password --from /dev/urandom om system/usr/relay key decode --name=password ```
```bash head -c 32 /dev/urandom | curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary @- \ "https://:1215/api/object/path/system/usr/relay/data/key?name=password" curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/system/usr/relay/data/key?name=password" ``` The key value is the request body, so the bytes are read where the caller runs: `--from /dev/urandom` is the cli doing the same read for you. The decode is the same endpoint read back, which answers the value as the response body.
> ➡️ See Also > * [How to install the agent](install.md) > * [How to add a relay heartbeat to a cluster](internals.daemon.heartbeats.relay.md) > * [How to add an arbitrator to a cluster](configure.quorum.md) # Cluster Quorum When a peer is flagged as stale by all heartbeats, the daemon assumes the cluster is in a split-brain situation, as it cannot determine whether the stale peer has failed or is isolated. OpenSVC minimizes the likelihood of a split-brain scenario by leveraging multiple independent heartbeats. ## Enabling Quorum Enforcement > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). `cluster.quorum` is a cluster keyword, so > it is written on any node with `PATCH /api/cluster/config`, which takes the > same `set` and `unset` operations as `om cluster config update`. Users who prefer to have a cluster segment shut down in such situations can enable quorum by setting `cluster.quorum` to `true`:
```bash om cluster config update --set cluster.quorum=true ```
```bash curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=cluster.quorum=true' \ "https://:1215/api/cluster/config" ```
By default, the system allows split nodes to take over services, which may result in services running on multiple isolated segments. To revert to the default behavior, use:
```bash om cluster config update --unset cluster.quorum ```
```bash curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'unset=cluster.quorum' \ "https://:1215/api/cluster/config" ```
To check the current quorum configuration:
```bash om cluster config get --kw cluster.quorum ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/cluster/config?kw=cluster.quorum" ```
## Quorum Behavior If the cluster is configured for quorum and a split-brain situation occurs, a node will shut down if the number of reachable nodes (including itself) plus arbitrators is less than half of the total cluster and arbitrator nodes. Frozen nodes do no evaluate quorum. They will not shut down on split-brain. Frozen nodes still vote for peer nodes quorum evaluation. ### Example Arbitrator Requirements To survive an interconnect outage: - In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split. - In a 3-node cluster, a single node requires 2 arbitrator votes. - In a 4-node cluster, a single node requires 3 arbitrator votes. - In a 5-node cluster, a single node requires 3 arbitrator votes. To survive an interconnect outage, plus all peers outage in the same availability zone: - In a 2-node cluster, a single node requires 1 arbitrator vote to survive the split. - In a 3-node cluster, a single node requires 2 arbitrator votes. - In a 4-node cluster, a single node requires 3 arbitrator votes. - In a 5-node cluster, a single node requires 4 arbitrator votes. ## Configuring Arbitrators Any OpenSVC agent can act as an arbitrator, and multiple arbitrators can be configured. For example, to configure an arbitrator: ### Use an https server as an arbitrator [arbitrator#a1] uri = https://dev2n1:1215/metrics #insecure = true ### Use a tcp server as an arbitrator [arbitrator#a2] # 22, the ssh port of dev2n2: with no port, listener.port is implied uri = dev2n2:22 The vote of a tcp arbitrator is a TCP connect to its uri. The port of the uri is optional: with none, the port of the cluster listener is implied, `listener.port`, 1215 by default. An arbitrator still listening on the 1214 port of a v2 agent must name it in the uri. ## Testing Arbitrators Every node votes on its own, so an arbitrator is alive from the point of view of a node, not of the cluster. The monitor shows that grid:
```bash om mon ``` ... Arbitrators n1 n2 a1 warn | X X a2 warn | X X a3 | O O ...
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/cluster/status" | jq '.cluster.node | map_values(.status.arbitrators)' ``` Each node status carries an `arbitrators` object, keyed by arbitrator name, which is what the grid renders one column per node.
## Best Practices * Configure `` minus 1 arbitrators * Host all arbitrators on the same 3rd site * Use one of the arbitrators as a relay for the relay heartbeat driver * Disable quorum or freeze all nodes when doing a relayout of the cluster > ➡️ See Also > * [Example 1: Odd-nodes cluster](internals.daemon.quorum.example1.md) > * [Example 2: Even-nodes cluster](internals.daemon.quorum.example2.md) # Upgrade The agent supports upgrading with zero service down-time. Upgrading does not require a node reboot. ## Debian, Ubuntu sudo apt update sudo apt install --only-upgrade opensvc-server opensvc-client ## Red Hat Enterprise Linux 7 sudo yum makecache sudo yum upgrade opensvc-server opensvc-client ## Red Hat Enterprise Linux 8+ sudo dnf makecache sudo dnf upgrade opensvc-server opensvc-client ## SuSE Linux Enterprise Server sudo zypper refresh sudo zypper update opensvc-server opensvc-client # Deploy Apps Applications are composed of one or more objects (services, configs, secrets, volumes, service accounts). These objects can be deployed and operated individually or as a group. This chapter covers the life of an object's configuration, from the name you give it to the day you remove it: * [Naming](apps.deploy.naming.md) — the `//` path, and what each kind is for. * [Create, Deploy](apps.deploy.create.md) — the ways to bring an object into existence: from scratch, from another object, or from a manifest. * [Provisioning](apps.deploy.provisioning.md) — creating what the configuration describes: disks, filesystems, addresses, images. * [Update](apps.deploy.update.md) — changing a configuration safely, rather than editing the file behind the agent's back. * [Purge, Delete](apps.deploy.delete.md) — removing the configuration, with or without the resources it provisioned. If you have not created an object yet, [Your First Service](firstservice.md) walks through one end to end in a few minutes. # Naming A fully qualified object name is formatted as ``//``. ## Namespace Namespaces allow users to create objects with the same name in different naming spaces. Namespace names must conform to RFC952: * only alphanum characters or dash * start with an alpha * end with an alphanum ## Kind * ``svc`` A service, with a mix of ip, app, container, volume, disk, fs and task resources. * ``vol`` A data volume from a pool, with a mix of volume, disk and fs resources. * ``cfg`` A configuration map, storing unencrypted key/value pairs for use by other kinded objects. * ``sec`` A secret, storing encrypted key/value pairs for use by other kinded objects. * ``ccfg`` The special kind for the cluster configuration object. ## Name Names must conform to RFC952, with a tolerance for dots. A name must be unique in its namespace and kind. # Create, Deploy The following actions only modify files in `/etc/opensvc`. No operating system configuration file is modified, so they are safe to experiment with. The agent support object creation via two commands: * ``create`` The object is created but not provisioned nor started. * ``deploy`` The object is created, provisioned and started. Both actions support the same arguments. The following examples use only create commands. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). Creating an object through the api is > posting its configuration file: `POST …/config/file` creates the object, > `PUT …/config/file` replaces the configuration of an object that already > exists, and both answer the `OM-Last-Modified` timestamp the written > configuration now carries. ## From Scratch, non Interactive Create a new object with minimal configuration. No resources are described.
``` om create ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary '[DEFAULT]' \ "https://:1215/api/object/path////config/file" ```
Resources and default keywords can be set right from the create command, using ``--kw =`` options
``` om create \ --kw container#0.type=oci \ --kw orchestrate=ha \ --kw nodes={clusternodes} ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" \ --data-binary $'[DEFAULT]\norchestrate = ha\nnodes = {clusternodes}\n\n[container#0]\ntype = oci\n' \ "https://:1215/api/object/path////config/file" ```
Scripts should describe the object as completely as possible in this single command. The keywords given to ``create`` are part of the first and only commit, so the object is never announced to the cluster in a half-configured state. Creating a bare object then completing it with a series of ``config update`` commands publishes every intermediate state to the peers, and the orchestrator acts on them as soon as ``orchestrate`` is set. The api has the same property: one `POST …/config/file` carrying the whole configuration is one commit, where a bare create followed by `PATCH …/config` calls is several. ## From Another Object
``` om create --config= ```
``` curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////config/file" | curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary @- \ "https://:1215/api/object/path////config/file" ```
> The api writes the configuration it is given, as given. The reset of the > recorded keywords described below is the client's work, so a clone driven > straight through the api has to drop them from the body itself. ## From Manifest The manifest must be ini formatted, like the output of ``om config show``.
``` om create --config= ```
``` curl -s | curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary @- \ "https://:1215/api/object/path////config/file" ```
This method can also be used to clone objects ``` om create --config= ``` or ``` om config show | om create --config=- ``` ## What a Clone Resets A clone is another object, so what the source recorded of itself does not come with it. Those keywords are marked in the keyword reference: ``` recorded: written when what it names is made, and reset when the object is cloned ``` The object id is one of them, and the clone is given a new one. The uuid of an md array is another: it is written when the array is created, and is what assembles that array again afterwards, so a clone keeping it would assemble the source's array rather than create one of its own. They are written again, for the clone, when the clone makes what they name. `--restore` keeps them, which is what restoring a saved configuration means: the object being created is the one the configuration was saved from, not another like it. ``` om create --config= --restore ``` > ➡️ See Also > * [Provisioning](apps.deploy.provisioning.md) > * [Batching configuration changes](apps.deploy.update.md#batching) # Provisioning Creating an object writes a configuration. It does not create anything the configuration describes. Provisioning is the step that does: allocate the disks, create the volume group and its logical volumes, format the filesystems, plumb the ip address, pull the container image. The reverse, unprovisioning, destroys them again. That distinction matters: deleting an object removes its configuration, unprovisioning destroys its data. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). An object is addressed by the three > segments of its path, so `myapp` is `/api/object/path/root/svc/myapp`. ## Provisioners A resource driver may implement a provisioner, and most of the drivers that represent something allocatable do: the `disk`, `fs`, `ip`, `volume`, `container`, `share` and `app` families. A provisioner often needs keywords the resource does not need in order to run, such as a size or a pool. Those are listed with the driver in the keyword reference. A provisioner may also write back into the configuration. Most ip provisioners set the resource's `ipname` once they know the address they obtained. ## Provisioning an object
```bash om myapp provision ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/svc/myapp/action/provision" ```
This is orchestrated. The daemon provisions the placement leader first and leaves the instance `up`, so that later provisioners can rely on what the earlier ones produced: a disk must stay attached for a filesystem provisioner to format it. Once the leader is provisioned, the other instances are provisioned in parallel and rolled back, leaving the object in its optimal state. `om myapp deploy` does the same in one step, creating the object and provisioning it. To act on the local instance only, without involving the daemon:
```bash om myapp instance provision --leader ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/root/svc/myapp/action/provision?leader=true" ```
The provisioners run in the resource start order. `--leader` matters here. Driving this by hand, you are telling the agent which node provisions the resources that only one node may provision. Without it, the local instance is treated as a follower: its own resources are provisioned and the shared ones are silently skipped. Pass `--leader` on the node that should own them, and omit it on the others. The orchestrated `om myapp provision` above works this out for you, which is the reason to prefer it whenever the daemon is available. ## Shared resources A SAN disk visible from several nodes, or the filesystem on it, must be provisioned once, not once per node. Flag those resources: ```ini [disk#1] type = vg name = datavg shared = true ``` The provisioned state of a shared resource is synchronized across the object's nodes. The state of a non-shared resource is node-affine, and each node provisions its own. Which node provisions them is the leader question above: the orchestrator picks the placement leader, and a manual provision needs `--leader` to say so. ## Resources you provisioned yourself Resources that already existed when the object was created are reported as not provisioned, which `om mon` marks with a `P`. Successfully starting such a resource marks it provisioned: if it starts, it is sane to consider it provisioned. When starting it is not possible or not wanted, flag it without touching the system:
```bash om myapp instance provision --state-only --rid disk#1 ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/root/svc/myapp/action/provision?state_only=true&rid=disk%231" ```
This only sets the flag. Nothing is formatted, allocated or created. The flag matters because unprovision skips resources that were never provisioned, so an unflagged resource survives an unprovision that should have destroyed it. ## Unprovisioning
```bash om myapp unprovision ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/svc/myapp/action/unprovision" ```
This destroys the resources, and it is the dangerous half of the pair. Shared resources are unprovisioned once, by the leader, in reverse order. > ➡️ See Also > * [Create, Deploy](apps.deploy.create.md) > * [Purge, Delete](apps.deploy.delete.md) # Update Configuration files are stored in `/etc/opensvc/`. * `/etc/opensvc/.conf` Root objects configuration file: * `/etc/opensvc/namespaces///.conf` Namespaced objects configuration file: Do not edit these files directly. Use one of the following method instead. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). `PATCH …/config` takes the same `set`, > `unset` and `delete` operations as `om config update`, and commits > them as one transaction. ## Interactive ``` om config edit ``` The configuration file syntax is checked upon editor exit. The new configuration is installed if the syntax is found correct, or saved in a temporary location if not. Two options are then possible: * Discard the erroneous configuration: ``` om config edit --discard ``` * Re-edit the erroneous configuration: ``` om config edit --recover ``` The api has no interactive counterpart, the editor being the client's. Fetch the configuration file, edit it, and write it back: ```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////config/file" > .conf $EDITOR .conf curl -s -X PUT -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary @.conf \ "https://:1215/api/object/path////config/file" ``` The write is refused if the configuration it carries does not validate, so a typo is caught the same way the editor check catches it. ## Non-Interactive Resource Addition
``` om config update --set fs#2.type=ext4 --set fs#2.mnt=/srv/{fqdn} ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'set=fs#2.type=ext4' \ --data-urlencode 'set=fs#2.mnt=/srv/{fqdn}' \ -G "https://:1215/api/object/path////config" ```
The resource identifier (rid) must not be specified. The resource type must be specified (rtype). A free rid will be allocated. ## Non-Interactive Resource Modification ``` om config update --set fs#2.type=ext4 --set fs#2.mnt=/srv/{fqdn} ``` The resource identifier must be specified. ## Non-Interactive Resource Deletion
``` om config update --delete fs#2 ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'delete=fs#2' \ -G "https://:1215/api/object/path////config" ```
This command does not stop the resource before removing its definition. If desired, this can be done with ``` om stop --rid fs#2 ``` ## Batching `om config update` applies all its `--delete`, `--unset` and `--set` arguments as a single transaction: the batch is staged, validated, and committed once. Scripts should prefer one command carrying many changes:
``` om config update \ --set app#1.type=forking \ --set app#1.start=/srv/app/bin/start \ --set app#1.check=/srv/app/bin/check ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'set=app#1.type=forking' \ --data-urlencode 'set=app#1.start=/srv/app/bin/start' \ --data-urlencode 'set=app#1.check=/srv/app/bin/check' \ -G "https://:1215/api/object/path////config" ```
Splitting this into one command per keyword commits three times. Each commit rewrites the configuration file, and every node in the object scope then fetches the whole file. Worse, each intermediate state is a state the peers and the orchestrator see and act upon, including the states where the resource definition is still incomplete. Scripts driving the daemon API have the same choice: the `PATCH /api/object/path/{namespace}/{kind}/{name}/config` handler accepts repeated `set`, `unset` and `delete` parameters, and commits them as one transaction. ## Waiting for the Configuration to Propagate A configuration update is acknowledged by the node that committed it, and reaches the other nodes of the object a moment later, each fetching the new file. A script going on to act on the instances, restarting them to apply a change for example, can wait for that moment instead of racing it:
``` om config update --set env.foo=bar --wait [--time 30s] ```
``` curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ --data-urlencode 'set=env.foo=bar' \ --data-urlencode 'wait=30s' \ -G "https://:1215/api/object/path////config" ```
The answer is held until every live node of the object holds the new configuration, which usually takes well under a second: * The nodes waited for are the ones of the configuration written, so a node the update adds to `nodes` is waited for too. * A node is live while the heartbeats reach it. A node that is down is not waited for: it fetches the configuration when it comes back. A wait that expires, after `--time` (5 minutes by default) or the `wait` duration, ends with an error, and the api answers `408`, naming the nodes the configuration has not reached yet. The configuration is committed all the same. Do not answer a `408` by updating again: a `+=` operation would be applied twice. The answer carries the `OM-Last-Modified` timestamp of the configuration written, which an action asked of an instance can require with its `config_updated_at` parameter, to refuse running on an older configuration. # Purge, Delete > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). An object is addressed by the three > segments of its path, so `svc1` is `/api/object/path/root/svc/svc1`. ## Purge
``` om purge ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/purge" ```
This command asks the cluster to orchestrate a stop, unprovision and delete. Non-leader instances are sequenced first. Purging a service does not purge its referenced volumes. Purging a volume actually removes all volume data. The api answers the `orchestration_id` of the queued orchestration. See [Follow an Action](apps.operate.sessions.md) for waiting on it. ## Delete
``` om delete ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/delete" ```
This command does not stop nor unprovision the object, so it can leave unreferenced mounts, containers and processes on the nodes. This command should be used by administrators only. # Operate Apps Usual operator actions covered in this chapter are: * Assess the state of the cluster and its hosted applications, troubleshoot warnings, post-mortem analysis. * Start, stop, relocate one service or a selection of services to fix warning causes, prepare hardware or operating system maintenance The chapter follows that order: * [Assess](apps.operate.status.md) — reading cluster and service state, and the markers `om mon` uses to pack it densely. * [Resource Info](apps.operate.info.md) — the key-values each resource publishes about what it is made of. * [Select](apps.operate.select.md) — naming the objects you want to act on, from one path to an expression matching many. * [Action](apps.operate.action.md) — running the action, and what to do when it fails. # Services Status > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). ## Cluster Overview (`om mon`) The `om mon` command provides a real-time, human-readable overview of the cluster and service states. ### Human Readable ```bash om monitor ``` > **Requirements:** > > * The agent daemon must be **up and running**. > * The displayed information is **near synchronous**. ### Status and Alert Markers Markers are used to optimize information density. | Heartbeat Markers | On `hb.tx` Target | On `hb.rx` Source | | :--- | :--- | :--- | | **`O`** | Data has been sent in time | Data has been received in time | | **`X`** | Data has not been sent in time | Data has not been received in time | | **`/`** | Not applicable | Not applicable | | General Markers | On Service Instance | On Service | On Node Status | | :--- | :--- | :--- | :--- | | **`O`** | `up` | | | | **`o`** | `standby up` instance | | | | **`X`** | `down` instance or heartbeat failure | | | | **`x`** | `standby down` instance | | | | **`/`** | Not applicable, undefined | | | | **`^`** | Placement leader | Placement alert | | | **`!`** | Warning | Warning raised by any instance | | | **`!!`** | Not fully available instance | | | | **`*`** | Frozen instance | | Frozen node | | **`P`** | Not fully provisioned instance | | | ### Machine Readable Use the `--output` option for structured data output.
```bash om cluster status --output json ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/cluster/status" ```
### Watch Continuously refresh the status display. The api counterpart is the event stream the display is fed by: a `text/event-stream` the daemon keeps open, which the `filter` and `selector` query parameters narrow.
```bash om monitor --watch ```
```bash curl -sN -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//daemon/event" ```
## Interactive (`ox tui`) `om monitor` prints and returns, or redraws with `--watch`. To move around the cluster instead of reading it, the `ox` client carries a terminal interface: ```bash ox tui ``` It shows the same data, and lets you select an object or an instance and act on it without leaving the view, which suits watching a service move between nodes while you drive it. `ox` is the client command. It talks to the API rather than the local node, so the same session can watch a remote cluster: ```bash ox tui --server https://mycluster.example.com:1215 ``` ### Contexts On an admin station watching several clusters, passing the server on every command gets old, and says nothing about who you are. A context bundles an endpoint, a user and an optional default namespace under a name, in `~/.config/opensvc/contexts`. Declare the cluster, the user, and the context tying them together: ```bash ox context cluster add --name prod --server https://prod.example.com:1215 ox context user add --name alice --username alice --client-certificate ~/.opensvc/alice.crt --client-key ~/.opensvc/alice.key ox context add --name prod --cluster prod --user alice --namespace myapp ``` Then select it with `OSVC_CONTEXT`, and every `ox` command follows: ```bash $ export OSVC_CONTEXT=prod $ ox tui $ ox '**' ls ``` Switching clusters is switching the variable, so two terminals can watch two clusters at once without either forgetting who it is. ```bash ox context list # the configured contexts, and whether each is authenticated ox context show # the resolved endpoint, user and namespace ox context login # request and cache tokens ox context logout # drop them ``` > **Requirements:** > > * `ox` is packaged in `opensvc-client`. > * The daemon must be reachable, and your user needs at least the `guest` > grant on what you want to see. See [Access Control](configure.rbac.md). # Resource Info Each resource of an instance publishes a set of key-values describing what it is made of: the device a disk resource activates, the mount point a fs resource uses, the command an app resource runs. Together they form a per-resource inventory, refreshed on a schedule and centralized on the collector. This is deliberately not the resource *status*. Status answers "is it up right now?", changes constantly, and is evaluated every 10 minutes by default. Resource info answers "what is it made of?", changes rarely, and is refreshed hourly. Keeping the two apart lets each have the schedule it needs. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). Reading is an object-wide call, answering > the table of every instance; refreshing is a per-node call, the refresh > being a run on the node holding the resources. ## Read
```bash om myapp instance info ```
```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/svc/myapp/resource/info" ```
OBJECT NODE RID KEY VALUE myapp dev2n1 disk#1 driver disk.vg myapp dev2n1 disk#1 name datavg myapp dev2n1 app#1 driver app.simple myapp dev2n1 app#1 start /opt/myapp/bin/start This reports from a cache. It does not run anything on the resources, so it is cheap and safe to call as often as needed. Every resource contributes a common set of keys, `driver`, `standby`, `optional`, `disable`, `monitor`, `shared`, `encap`, and the restart policy when it has one. Each driver adds its own on top. The command is also available on every resource group, so a group name narrows the report, and a trailing pattern narrows it further: ```bash om myapp disk info # only the disk resources om myapp disk info 1 # only disk#1 om myapp resource info # same as instance info ``` ## Refresh
```bash om myapp instance info --refresh ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/root/svc/myapp/action/info" ```
This walks the resources, asks each one for its key-values, and rewrites the cache. It is what the scheduler runs hourly, driven by the `info_schedule` keyword: ```ini [DEFAULT] info_schedule = @60m ``` Refreshing a subset merges into the cache rather than replacing it, so `om myapp disk info --refresh 1` leaves the key-values of the other resources alone. A full refresh rewrites everything, and drops the resources the configuration no longer declares. ## Centralization The refresh does not talk to the collector. It only writes the local cache and signals the cluster. One node, the collector speaker, subscribes to that signal, fetches the key-values from whichever node refreshed them, and reports them to the collector on its own throttled schedule. A cluster therefore feeds the collector through a single node, and a burst of refreshes across many instances is coalesced instead of turning into a burst of collector requests. The practical consequences: * The reporting is deferred. A refresh shows up on the collector within minutes, not instantly. * An instance whose node is unreachable when the speaker tries to fetch it is retried later, not lost. * Nothing is reported for an instance that never refreshed successfully. ## Plug your own key-values An `app` resource can feed arbitrary key-values into this subsystem with the `info` keyword, which makes the inventory extensible with data the agent has no way to know about: an application version, a schema revision, a licence expiry, a tenant name. Point it at a command printing one `key:value` per line: ```ini [app#1] type = simple start = /opt/myapp/bin/start info = /opt/myapp/bin/inventory ``` With `/opt/myapp/bin/inventory` printing: version:4.2.1 schema:19 tenant:acme Those three keys are then reported next to the standard ones, and land on the collector like the rest. Set `info = true` instead to call the resource `script` with the `info` argument, which keeps a single executable for the start, stop, check and info entry points: ```ini [app#1] type = simple script = /opt/myapp/bin/app.sh start = true info = true ``` > **Writing an info command** > > * Lines that are not exactly `key:value` are silently dropped. A value > containing a `:`, like a url or a timestamp, has more than one separator > and is dropped with them. Encode it, or split it across keys. > * The command must exit 0. A failure aborts the refresh of the whole > instance, which then reports no key-value at all, not even for the other > resources. > * Keep it fast and side-effect free. It runs on a schedule, and is bounded > by `info_timeout`. # Select Objects Commands use the `om ` syntax to operate on a selection of objects. The `om ls` command can test a selector before submitting a dangerous action. > Note the selector expression may need to be **quoted** for the shell not to interpret the `!` and `*` characters. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). The selector expression is the same on > both sides: `GET /api/object/path?path=` answers the paths it > matches, and `GET /api/object?path=` answers what `om ls` > renders. The examples below show the cli form only where the point is the > expression itself. ## All Objects List all cluster objects (services, volumes, etc.).
```bash om '**' ls ```
```bash curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=**' \ "https://:1215/api/object/path" ```
## All Services List all service objects.
```bash om '*' ls om '*/svc/*' ls ```
```bash curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=*/svc/*' \ "https://:1215/api/object/path" ```
## Single Object List a specific object.
```bash om ls om ns1/svc/web1 ls ```
```bash curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=ns1/svc/web1' \ "https://:1215/api/object/path" ```
## List of objects With a `/tmp/svc.list` containing: ```bash # Ignored object paths ; svc1 ;svc2 # Ignored empty lines # Honored object paths # Leading and trailing whitespaces are trimmed svc3 svc4 ``` An action can be executed using: ```bash cat /tmp/svc.list | om - ls ``` or ```bash om - ls
```bash om , ls om ns1/svc/web1,ns1/vol/web1 ls ```
```bash curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=ns1/svc/web1,ns1/vol/web1' \ "https://:1215/api/object/path" ```
## Intersections List objects matching multiple criteria. ```bash om 'ns1/*/web1+*/svc/*' ls ``` is equivallent to ```bash om ns1/svc/web1 ls ``` ## Mixing Unions and Intersections The unioned expressions are evaluated from comma to comma, intersections are evaluated in the context of the current union. For example, ```bash om 'ns1/svc/*+ns1/*/web1,ns2/svc/web2' ``` is parsed as: ``` (ns1/svc/* intersected with ns1/*/web1) unioned with (ns2/svc/web2) ``` and evaluates as: ```bash ns1/svc/web1 ns2/svc/web1 ``` ## Negation The negation marker is `!`. This symbol needs quoting for the shell not to interpret it. Example:
```bash # All object of namespace `ns1` except those named `web1`: om 'ns1/**+!**/web1' ls ```
```bash # All object of namespace `ns1` except those named `web1`: curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=ns1/**+!**/web1' \ "https://:1215/api/object/path" ```
## Services by State Filter services based on their overall status. * List all services in the **`down`** state: ```bash om '*' ls --status down ``` * List all services in **`up`** and **`warn`** states: ```bash om '*' ls --status up,warn ``` The api has no `--status` filter. `GET /api/object?path=` carries the availability of each object, and the client keeps the rows it wants. ## Service Selector Expressions Use powerful expressions to filter objects based on configuration parameters. ```bash om ls ``` Where `` is a pattern or condition, either a `` or `[!]`. | Parameter (``) | Description | | :--- | :--- | | `.` | A key within a specific resource ID in the service config. | | `.` | A key within a driver group (e.g., `disk`, `fs`, `app`). | | `` | A key in the service configuration file header. | | Operator (``) | Description | | :--- | :--- | | `<` `>` `<=` `>=` `=` | Standard comparison operators. | | `:` | **Existence test** operator (value is empty). | | `~` | **Regular expression** operator. | | Separators / Modifiers | Description | | :--- | :--- | | `!` | **Negation** operator. | | `+` | **AND** expression separator. | | `,` | **OR** expression separator. | > **Note:** Matching is **case-sensitive**, except for boolean values. ### Examples 1. Services with name ending with `dns` OR starting with `ha`, **AND** which have an `app` resource with a `timeout` greater than 1: ```bash om '*dns,web*' config get -o 'template={{range .}}{{if and (hasPrefix .keyword "app#") (hasSuffix .keyword ".timeout") (ge .value 1)}}{{end}}' ``` 2. Services with at least one `ip` resource **AND** one `task` resource: ```bash om 'ip:+task:' ls # ha1, ha2, ha3, registry ``` 3. Services with at least one monitored resource **AND** with `monitor_schedule` not set: ```bash om '!monitor_schedule+.monitor=true' ls # ha1, ha4 ``` # Action Every action below is available both from the command line and through the cluster api. Pick the tab you want: the choice sticks for the whole book. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). An object is addressed by the three > segments of its path, so `svc1` is `/api/object/path/root/svc/svc1`, and > `ns1/svc/svc1` is `/api/object/path/ns1/svc/svc1`. ## Base Actions ### Start **Local Start (Bypasses Orchestrator)** Start the service instance on the local node directly.
``` om instance start ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/start" ```
> **Resource Start Order:** `ip`, `disk`, `fs`, `share`, `container`, `app`. The api answers a `session_id` and an `exec_id` naming the run this node forked. `GET /api/node/name//daemon/exec/id/?wait=5m` holds until that run ends, which is the counterpart of the cli waiting for the command to return. **Orchestrated Start** Instruct the orchestrator to start the service on the node(s) selected by the placement policy.
``` om start [--wait] [--time ] [--watch] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/start" ```
* By default, the command returns upon daemon acknowledgment. * `--wait` holds the command until the action completes. * `--time` sets a maximum wait duration. The api answers the `orchestration_id` of the queued orchestration. Waiting for its end is a separate call, which any node of the cluster answers, even one carrying no instance of the object: ``` curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//daemon/orchestration/id/?wait=5m" ``` The call answers 200 with the orchestration state, 408 if it is still running when `wait` expires, and 410 if the id is no longer known. > **Frozen instances are started, and stay frozen.** Freezing tells the daemon > not to act by itself. It does not make an object refuse a start you asked > for, so a frozen instance is started and its freeze is left as you set it. > The daemon still never starts a frozen instance on its own, and still passes > over frozen nodes when choosing where to start. > > A stop does not freeze. It flags every instance of the object stopped on > purpose, and the start you ask for clears that flag, so a stop followed by a > start leaves the object up and under orchestration, with whatever freeze you > set yourself still exactly where you set it. > > Earlier versions, and OpenSVC v2, unfroze the object as part of starting it, > and froze it as part of stopping it. ### Stop **Local Stop (Bypasses Orchestrator)** Stop the service instance on the local node directly.
``` om instance stop ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/stop" ```
> **Resource Stop Order:** `app`, `container`, `share`, `fs`, `disk`, `ip`. **Orchestrated Stop** Instruct the orchestrator to stop the service wherever it runs, and to leave it down.
``` om stop [--wait] [--time ] [--watch] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/stop" ```
> The stop flags every instance of the object stopped on purpose, which is what > keeps the daemon from starting it back: without the flag the next ha decision > would undo the stop. `om print status` shows the flag as `stopped`, > and `om mon` as a `=` next to the instance. > > The flag is cleared by a start, a restart or a switch you ask for, on every > instance of the object, and by the instance being seen up again. It survives > a daemon restart and a reboot, so a node coming back up does not start what > you asked to be down. > > Earlier versions froze the instances to get the same result, which said more > than it meant: you could no longer tell your own freeze from one a stop had > left behind, and the object went on running without failover once started > again. ### Relocation **Switch** Stop the service on its current node(s) and start it on the specified target node. The freeze you set on an instance is left as you set it: a switch is a request to place the object, not to thaw it.
``` om switch --node [--wait] [--time ] [--watch] [--live] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"destination": [""], "live": false}' \ "https://:1215/api/object/path////action/switch" ```
> The `container.kvm` supports live migration. The VM storage is either read-write from all nodes during the switch, as SAN disks, drbd pass-through, NFS, Ceph or ClusterFS, or held by `fs.zfs` and `disk.zvol` resources, whose datasets the switch copies to the destination and whose disks the migration mirrors there: see [KVM Live Migration on ZFS](../howtos/kvm.live.migration.zfs.md). **Takeover** Stop the service instances on peer nodes and start it on the local node. The freeze you set on an instance is left as you set it.
``` om takeover [--wait] [--time ] [--watch] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"destination": [""], "live": false}' \ "https://:1215/api/object/path////action/switch" ```
> A takeover is a switch whose destination is the node you run it from, so the > api has no separate endpoint for it. **Giveback** Stop the service on non-leader nodes, and let the orchestrator start instances on the designated leaders. The freeze you set on a node or an instance is left as you set it, and the daemon still passes over what is frozen when it chooses where to start.
``` om giveback [--wait] [--time ] [--watch] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/giveback" ```
### Handling Failures If an action fails, the orchestrator is blocked, and the failure is reported in `om mon` and `om instance status`. **Clear Failure** Allows the daemon to **retry** the execution plan. The failure is held by the instance monitor of each node, so the api clears it one node at a time, where the cli command clears it on every node of the object.
``` om clear ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////clear" ```
**Abort Action** Aborts the currently blocked orchestrated action.
``` om abort ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path////action/abort" ```
### Data Replication > The disk, fs and sync resources support data replication. **Replicate All** Run resource replication to all configured targets (e.g., production (`prd`) or disaster recovery (`drp`)).
``` om instance update ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/update" ```
> This command can run on a schedule. **Replicate to peer Nodes** Run resource replication to secondary cluster nodes. No-op if run from a node not running the service.
``` om instance update --target nodes ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/update?target=nodes" ```
**Replicate to DRP Nodes** Trigger file synchronization to disaster recovery nodes. No-op if run from a node not running the service.
``` om instance update --target drp ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/update?target=drp" ```
### Run Execute tasks defined within the service configuration.
``` om instance run [--rid ...] ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path////action/run?rid=task%231" ```
> This command can run on a schedule. ## Resource Filtering Filter actions to be executed only on specific resources using `--rid`, `--tags`, or `--subsets`. | Option | Syntax | Description | | :--- | :--- | :--- | | **`--rid` (ID List)** | `om --rid [,,...] ` | Execute action on resources specified by resource IDs. | | **`--rid` (Group List)** | `om --rid [,,...] ` | Execute action on resources belonging to specified driver groups (`ip`, `disk`, `fs`, `share`, `container`, `app`, `sync`, `task`). | | **`--tag` (OR)** | `om --tag tag1,tag2 ` | Execute action on resources tagged with **either** `tag1` **or** `tag2`. | | **`--tag` (AND/OR)** | `om --tag tag1+tag2,tag3 ` | Execute action on resources tagged with **both** `tag1` **and** `tag2`, **OR** with `tag3`. | | **`--subset`** | `om --subset s1,s2 ` | Execute action on resources belonging to subset `s1` **or** `s2`. | The instance action endpoints accept the same filters as the `rid`, `tag` and `subset` query parameters. The `#` of a resource id has to be percent-encoded as `%23`. ## Logging All action logs are multiplexed to multiple destinations: * **Stdout/Stderr** * **Journald** or **Syslog** * **Collector Database** (Optional, via asynchronous XML-RPC calls) ## Examples ### Starting a Service (Local) Shows the ordered execution of resource start-up. ``` # om svc0 instance start 10:51:50.590 INF svc0: >>> do start [om svc0 instance start] (origin user, sid ae60fdd4-6629-40eb-a993-4f1380d66516) 10:51:52.061 INF svc0: fs#1: install flag file /dev/shm/opensvc/svc/svc0/fs#1.flag 10:51:52.062 INF svc0: app#0: applied pg /opensvc.slice/opensvc-svc.svc0.slice 10:51:52.063 INF svc0: app#0: applied pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice 10:51:52.090 INF svc0: app#0: run: /usr/bin/om exec --pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice -- touch test -f /tmp/svc0.root.svc.reliable-leopard 10:51:52.194 INF svc0: <<< done start [om svc0 instance start] in 1.603582081s, instance status is now up ``` ### Stopping a Service (Local) Shows the ordered execution of resource shut-down. ``` # om svc0 instance stop 10:51:55.282 INF svc0: >>> do stop [om svc0 instance stop] (origin user, sid 807b55a5-8a5e-4a38-920c-94b6956da6a7) 10:51:55.380 INF svc0: app#0: run: /usr/bin/om exec --pg /opensvc.slice/opensvc-svc.svc0.slice/opensvc-svc.svc0-app.0.slice -- rm -f /tmp/svc0.root.svc.reliable-leopard 10:51:55.461 INF svc0: fs#1: uninstall flag file /dev/shm/opensvc/svc/svc0/fs#1.flag 10:51:55.494 INF svc0: <<< done stop [om svc0 instance stop] in 211.596639ms, instance status is now down ``` # Grow a Volume A volume rests on a chain of resources: a filesystem on a replicated device on a logical volume, a mirror over two volumes from another pool. Growing the volume means growing every link of that chain, from the bottom up, and a resize is the one command that does it. A resize only grows. There is no shrink. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). `POST …/action/resize` takes the size in > its body and answers the `orchestration_id` of the queued orchestration. > The per-node commands below have no api counterpart: they drive the local > instance, which is what the orchestration does for you. ## The two commands | command | acts on | ask it when | | :--- | :--- | :--- | | `om resize SIZE` | every node holding an instance | the normal case | | `om instance resize SIZE` | this node | you are driving the nodes yourself | The first writes `SIZE` into the volume configuration and asks the cluster to converge to it. The configured size is what every node reads, and what a listing like `om pool volume ls` reports, so it is the size the volume is from then on. `SIZE` is a size to reach, as `11g`, `11GB` or `12Gi`, or an amount to add, as `+1g`.
```bash om vol/data resize 20g om vol/data resize +5g ox vol/data resize 20g ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"size": "20g"}' \ "https://:1215/api/object/path/root/vol/data/action/resize" ```
## Seeing the plan first `--dry-run` reports what a resize would do and changes nothing: ```bash $ om vol/data instance resize 600mi --dry-run grow, in this order: stage 0, on every node: disk#1 disk.zvol 454mi -> 604mi stage 1, once every node has finished stage 0: disk#2 disk.drbd 454mi -> 600mi asks 604mi of the link below fs#1 fs.xfs 454mi -> 600mi ``` Read it from the bottom of the chain up. Each link is asked what it needs from the link below it, which is not always what it was asked for: the drbd above keeps its metadata out of what it hands up, so it asks 604Mi of the zvol to offer 600Mi to the filesystem. A raid5 over `n` members asks `to/(n-1)` of each. Every link is asked before any of them is changed, so a chain holding one link that cannot do it is refused whole. Finding out that a filesystem cannot grow after the device under it already has is the failure this avoids. `-o json` carries the same plan for a client that would rather read fields. ## Stages A replicated resource offers what its smallest replica holds, so it cannot be grown before every node has grown what is under it. That splits the work into stages, and the plan names them. Every node runs stage 0. Once every node has finished it, the node holding the volume up runs stage 1, which grows the replicated link and whatever rests on it. A chain with two replicated links has three stages; that is the most. The instance monitor says which stage a node has reached: | state | meaning | | :--- | :--- | | `resizing` | a stage is running here | | `resized:0` | this node has finished stage 0, and waits for its peers | | `resized` | this node has nothing left to grow | | `resize failed` | a stage failed here | ## Asking again is how a resize is finished A failed resize is final on the instance it failed on. Nothing is retried, on purpose: a chain left part way is grown by asking for it again.
```bash om vol/data resize # no size: converge to the configured size ```
```bash # no size: converge to the configured size curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -d '{}' \ "https://:1215/api/object/path/root/vol/data/action/resize" ```
The configured size is the target, and every link that already holds it is skipped, so asking again costs nothing on the nodes that finished and finishes the ones that did not. This is why the configuration holds the size asked for rather than the size reached. The size a link reached is written back to that link's own `size` keyword, so what a resource holds and what its keyword says agree; only the target it was growing towards disagrees, which is what says the work is unfinished. ## What a namespace may take Growing is claiming more of the pool, so it is rationed the way an allocation is. A namespace claiming more than its `claim` allows is refused, and the refusal says by how much: ```bash $ om vol/data resize 1g 403 vol/data is served by the drbdz pool, and the root namespace may claim 130mi of it and already claims 100mi, so it cannot claim 924mi more ``` See [Namespaces](apps.design.namespaces.md) for the claim itself. ## What is refused, and why **A shrink.** Asking for less than the configured size is refused. Asking for exactly it is not: that is how an unfinished resize is finished. **A lone replica.** Growing one replica of a replicated object on its own strands the space until the others catch up. `--force` does it anyway. **A raid0 or linear array.** It holds what its members give it, with no size of its own on each, and grows only by taking another member — a reshape, which a resize is not. The array says so: a raid0 array uses no size of its own on each member, so it cannot be grown onto members that grew **A logical volume sized as a share of its volume group.** `size = 100%FREE` is resolved by lvm2, once, when the volume is created, and om never learns what it came out as. A resize would have to write a count of bytes over the share, answering a question nobody asked. Write the size as a size, or as an expression om computes, and it can be grown: size = $(50% * {disk#vg.capacity}) **An array that is not whole.** A degraded array grows without complaint if asked, and the space that adds has no redundancy and no member to rebuild it from. An array rebuilding, resyncing or reshaping is busy with the space it already has. Both are refused until the array is clean. ## A volume that holds less than it is configured to The instance status warns when a resource holds less than its configured size, which is what a resize that stopped part way looks like: ``` warn: holds 366907392 bytes of the 367001600 bytes it is configured to hold, so a resize has not finished ``` Asking for the resize again is the answer. When the difference is one no resize can make up, the warning says that instead, and why: ``` warn: holds 96mi of the 100mi it is configured to hold, and cannot grow to it: a raid0 array uses no size of its own on each member … ``` That second one is a configuration asking for a size its layout can never hold, and it is the layout or the configured size that has to change. ## Growing one resource The resource commands grow one link and everything under it, rather than the volume's own head: ```bash om fs resize 20g --rid fs#1 om disk resize 20g --rid disk#1 om volume resize 20g --rid volume#1 ``` These act on the local instance. They are for a chain the volume head does not lead to, and for repairing one link of a chain by hand. > ➡️ See Also > * [Cluster Storage](configure.pool.md) > * [Namespaces](apps.design.namespaces.md) > * [Follow an Action](apps.operate.sessions.md) # Follow an Action An action submitted to the daemon returns before it is done. What comes back is a pair of identifiers, and the daemon remembers what became of them, so a client that asked for something can ask how it went. > The **API** tabs assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). The daemon answers for what it ran, so > an exec listing is per node and a client wanting the whole of a session > asks every node and folds the answers, which is what the cli commands do. > An orchestration needs no such union: any node answers for it. ## Three commands, three scales | command | one row is | ask it when | | :--- | :--- | :--- | | `om daemon session list` | one submitted command, whole | "how did what I asked for go?" | | `om daemon exec list` | one run of one object on one node | "which part of it failed, and why?" | | `om daemon ps` | one run that is still running | "what is running now, and what is its pid?" | `om daemon ps` is `om daemon exec list --state running` under a shorter name and with the pid in front. What is running and what has run are the same records at different ages, so they are one listing. ## The identifiers you are given An action addressed to a node returns the **session** id and the **exec** id. The session is the whole of what you submitted; the exec is the one run: ```bash $ om 'pod[36]' instance stop --node '*' OBJECT NODE SESSION_ID EXEC_ID pod6 dev2n1 e9440381-9507-42b3-bef3-2c815aceb169 d29abc32-2c0b-45ff-b639-508d5c19fbee pod3 dev2n1 e9440381-9507-42b3-bef3-2c815aceb169 9178f4f8-cd37-4ab5-874f-6d72b1d825f5 pod6 dev2n2 e9440381-9507-42b3-bef3-2c815aceb169 f40da200-7b6f-41aa-b6aa-dc2568b8cde2 … ``` An action addressed to the object returns an **orchestration** id instead. The cluster decides which nodes act, and one orchestration runs execs on each of them: ```bash $ om pod3 switch OBJECT ORCHESTRATION_ID STATUS pod3 934a42f9-b7eb-4d7e-a3ac-18347522f9f9 accepted ``` [Session, Execution and Orchestration Ids](internals.ids.md) says how the three relate. ## Asking after a session One row, whatever it took: ```bash $ om daemon session list e9440381-9507-42b3-bef3-2c815aceb169 SESSION_ID STATE EXECS FAILED NODES OBJECTS ORIGIN STARTED_AT DURATION COMMAND e9440381… succeeded 6 0 3 2 api … 11s om pod3 instance stop (+5) ``` `STATE` is the state of the whole: running while any of its execs runs, failed if any of them failed, and succeeded only when every one of them did. That is the question you came with, and the reason not to count rows yourself. `DURATION` is how long you waited, not the sum of the parts: the execs run at the same time. Every node is asked. A session spans the nodes it reached, and the count of part of one is not the session's count, so this command does not default to the local node the way the others do.
```bash om daemon session list # every session this cluster remembers om daemon session list --state failed # sessions with a failed exec in them om daemon session list --origin scheduler ```
```bash # the execs, to fold into sessions yourself, on each node in turn curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//daemon/exec" curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'state=failed' \ "https://:1215/api/node/name//daemon/exec" curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'origin=scheduler' \ "https://:1215/api/node/name//daemon/exec" ```
## Asking after one run When a session failed, the exec listing says which part did: ```bash $ om daemon exec list --session-id 1ed811b1-088d-4531-bd32-c4be35efe29e --node '*' NODE STATE EXEC_ID SESSION_ID PATH ORIGIN STARTED_AT DURATION COMMAND dev2n1 failed ff0c2d2e… 1ed811b1… pod3 api … 35ms om pod3 instance start --rid nosuch ``` and `-o json` carries the exit code and the error: ```bash $ om daemon exec list ff0c2d2e-e6a5-47ef-800b-304610179fc6 -o json [ { "node": "dev2n1", "state": "failed", "exec_id": "ff0c2d2e-e6a5-47ef-800b-304610179fc6", "session_id": "1ed811b1-088d-4531-bd32-c4be35efe29e", "path": "pod3", "origin": "api", "started_at": "2026-09-11T12:31:23+02:00", "ended_at": "2026-09-11T12:31:23+02:00", "duration": "35ms", "exit_code": 1, "command": "/usr/bin/om pod3 instance start --rid nosuch", "error": "/usr/bin/om exit code 1 not in success codes: [0]" } ] ``` The api carries the two ends and not the length. `duration` is derived where it is shown, the same way for an exec, a session and an orchestration, so the three numbers cannot drift apart: an exec's end is its start plus what the daemon measured while the process ran, never the instant the news reached the store. A span that has not ended is measured against the clock of whoever is reading. `exit_code` is what the shell would report: the process status, or 128 plus the signal number when a signal ended it (`143` for SIGTERM, `137` for SIGKILL). It is `-1` only when the process never ran at all. A running exec has no exit code, which is what tells it apart from one that exited zero. ## What is running now ```bash $ om daemon ps PID NODE EXEC_ID PATH RID ORIGIN DURATION COMMAND 3871250 dev2n1 1a4a5266-49d8-4983-8613-073b1d40744c test1 imon 18ms om test1 instance status -r 3871248 dev2n1 be384858-f3d8-4fab-9fd4-f147427cb6d2 svc111 imon 20ms om svc111 instance status -r ``` Only the processes listed here can be signaled. `ORIGIN` says what submitted the run: `api` for a command a client sent, `imon` and `nmon` for one the monitors decided on, `scheduler` for one that came due. `RID` is set on the scheduler's runs, naming the resource whose schedule fired. ## Stopping what is running `om daemon kill` signals the execs a filter selects, and the filter is the one `om daemon exec list` takes, so what you listed is what you signal:
```bash om daemon kill --session-id 723906bb-… --node '*' # stop what I submitted, everywhere om daemon kill --orchestration-id 934a42f9-… # stop an orchestration's runs om daemon kill 823c9944-… # one exec om daemon kill -s 'pod*' --signal=term # everything running on these objects ```
```bash # each call signals the execs of one node curl -s -X DELETE -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'session_id=723906bb-…' \ "https://:1215/api/node/name//daemon/exec" curl -s -X DELETE -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'orchestration_id=934a42f9-…' \ "https://:1215/api/node/name//daemon/exec" curl -s -X DELETE -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'exec_id=823c9944-…' \ "https://:1215/api/node/name//daemon/exec" curl -s -X DELETE -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'selector=pod*' --data-urlencode 'signal=TERM' \ "https://:1215/api/node/name//daemon/exec" ```
There is no pid option, on purpose. Naming the exec rather than the pid is what makes signaling the wrong process impossible: a pid you read from a listing may have exited and been recycled by the time you send the signal, where an exec id is resolved to its pid by the daemon at the moment it signals, under the lock that keeps the two in step. The pid is reported all the same, and killing a pid is what the system's own `kill` is for. Something must narrow the selection. Signaling every exec of a node is not something you do by leaving the options out, and an empty filter is refused rather than obeyed. `--dry-run` reports what would be signaled and signals nothing: ```bash $ om daemon kill --session-id 723906bb-… --node '*' --dry-run PID NODE EXEC_ID PATH RID ORIGIN DURATION COMMAND 3955881 dev2n1 0131c61e-3a8a-412d-8ea1-8bcecc0ea9f5 pod1 api 1s om pod1 instance start ``` The answer is the execs signaled, in the same shape the listing uses, so a kill that matched nothing says so by answering nothing. An exec that has already ended selects nothing, which is the outcome you asked for. ## Asking after an orchestration ```bash $ om daemon orchestration list 934a42f9-b7eb-4d7e-a3ac-18347522f9f9 STATE ORCHESTRATION_ID PATH GLOBAL_EXPECT ACCEPTED_BY STARTED_AT DURATION succeeded 934a42f9 pod3 placed@ dev2n1 2026-09-10T20:34:27+02:00 4s ``` **Any node answers.** Every node of the object learns of the orchestration from the instance monitors the daemon replicates, so it does not matter which one you reach. That is what lets a client behind a floating address follow an orchestration: the address may move to another node while the orchestration runs, and a switch is precisely what moves it. `ACCEPTED_BY` is the node that was asked for the orchestration, and is empty on a node that only learned of it from the monitors. It is the one detail that differs between nodes; the state, the object, the target and the start do not. Through the api, the same id is asked of any node, and the call can be told to hold until the orchestration ends rather than answer its current state: ```bash curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//daemon/orchestration/id/934a42f9-…?wait=5m" ``` It answers 200 with the orchestration once it has ended, 408 if it is still running when `wait` expires, and 410 if the daemon no longer knows the id. A client that submitted an action and wants to know how it went has nothing else to keep: neither a connection held open since the submission, nor a node it has to come back to. The execs an orchestration caused are found by its id:
```bash om daemon exec list --orchestration-id 934a42f9-… --node '*' ```
```bash curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'orchestration_id=934a42f9-…' \ "https://:1215/api/node/name//daemon/exec" ```
## Reading what an id logged The listings say how an action went. To read what it said while it went, give the same id to the matching `logs` command: ```bash om daemon orchestration logs 934a42f9-b7eb-4d7e-a3ac-18347522f9f9 om daemon session logs e9440381-9507-42b3-bef3-2c815aceb169 om daemon exec logs d29abc32-2c0b-45ff-b639-508d5c19fbee ``` The three ids nest, and so do the three commands. An exec is one run of one command, on one node, for one object. The session it belongs to has the entries of its siblings too. The orchestration above that has every node's: ```bash $ om daemon orchestration logs fcd85a86-152a-4065-953d-eb5a681c929c … dev2n1: daemon: imon: vol/testdrbd: change global expect none -> resized … dev2n3: daemon: imon: vol/testdrbd: change global expect none -> resized … dev2n2: daemon: imon: vol/testdrbd: change global expect none -> resized … dev2n1: daemon: imon: vol/testdrbd: -> exec [om vol/testdrbd instance resize --stage 0] … ``` **Every node is asked**, because what an id names is not confined to one. A session reaches the nodes of the objects it named, an orchestration reaches every node of the object, and an exec runs on one node that the id does not tell you. Narrowing to a node you have to name yourself is how following an action across a cluster is got wrong. These are the node logs with the filter already written. The ids are log fields, so the long form works too and is what the short form does:
```bash om node logs --filter ORCHESTRATION_ID=934a42f9-… --node '*' ```
```bash curl -sN -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'filter=ORCHESTRATION_ID=934a42f9-…' \ "https://:1215/api/node/name//log" ```
The options of the node logs are all still there, and narrow within the id rather than beside it: ```bash om daemon exec logs d29abc32-… --grep 'exit code' om daemon orchestration logs 934a42f9-… --follow om daemon session logs e9440381-… --lines 200 ``` An exec that logged nothing answers nothing. A resize stage with no work to do runs, succeeds and says nothing, and the empty answer is that, not a failure: the listing is where you learn whether it ran. ## When the daemon has forgotten What the daemon remembers is bounded, by age and by count, so a node that has been running for months does not carry every command it ever ran. An identifier it no longer holds says so: ```bash $ om daemon exec list ff0c2d2e-e6a5-47ef-800b-304610179fc6 Error: exec ff0c2d2e-… is no longer known on dev2n1: it ended long enough ago to have been dropped, or never ran there ``` That is deliberately not the same answer as an exec that is still running, and not the same as one that never existed. A client polling for the end of an action must be able to tell "not finished" from "I no longer know", which is why forgetting is admitted rather than reported as absence. What is running is never forgotten for the count. Only what has ended is dropped. > ➡️ See Also > * [Action](apps.operate.action.md) > * [Session, Execution and Orchestration Ids](internals.ids.md) > * [Events](internals.daemon.events.md) # Design Apps Applications are composed of one or more objects (services, configs, secrets, volumes, service accounts). The first design decision is the application split into different services. The main reasons for splitting are: * Different lifecycle (volume vs svc) * Different topologies (failover, flex) * Security ## Topologies The `topology` keyword decides how many nodes may run the object at once, and is the choice that shapes everything else. A `failover` object runs on one node at a time. The other nodes hold a standby instance, ready to take over. A floating ip follows the active instance, so clients keep reaching the service at the same address wherever it runs. ![Failover service with a disaster recovery node](../images/agent.service.ip.failover.png) A `flex` object runs on several nodes at once, `flex_target` of them, within the `[flex_min, flex_max]` range. Every instance is active and carries its own ip, so a load balancer in front of them spreads the clients. Pick `failover` for something only one node may own at a time, like a database writing to a shared disk. Pick `flex` for something that scales by running more copies, like a stateless web front end. Configuration and secrets are objects too, installed into a service's storage before it starts. See [Configs and Secrets](apps.design.datastores.md). When the split is decided, each object must be named. Each object configuration can be designed on a development cluster or namespace. These configurations can be tracked alongside the application code base. Finally, the configurations can be deployed on test, up to production clusters or namespaces. Usually, these deployments are handled by a CI/CD pipeline. # Namespaces Namespaces allow users to create objects with the same name in different naming spaces. The `//` notation is called a path. The object selector (`--service` option for `svcmgr` and `svcmon`, first argument for `om`) accepts the path notation. Namespaced objects have their config stored in `/etc/opensvc/namespaces///`. This feature is available since version 1.9-2748. ## Operations with namespaces > A namespace configuration is an object of kind `nscfg`, named `namespace`, > so `test/` is `/api/object/path/test/nscfg/namespace`. The **API** tabs > assume the `$TOKEN` and the listener endpoint set up in > [Cluster API](configure.api.md). ### The object path The namespace is part of the path, so naming the object names its namespace. This is the way to create an object in a namespace, and it works for every command:
```sh om test/svc/svc1 create om test/svc/svc1 start om 'test/**' ls ```
```sh curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary '[DEFAULT]' \ "https://:1215/api/object/path/test/svc/svc1/config/file" curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/test/svc/svc1/action/start" curl -s -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'path=test/**' \ "https://:1215/api/object/path" ```
### OSVC_NAMESPACE environment variable `OSVC_NAMESPACE` narrows which objects a command sees, so a selector that would match the whole cluster is confined to one namespace: ```sh $ export OSVC_NAMESPACE=test $ om '**' ls test/svc/svc1 test/svc/svc2 ``` ```sh $ unset OSVC_NAMESPACE $ om '**' ls svc0 test/svc/svc1 test/svc/svc2 ``` It selects among the objects that exist. It does not decide where a new one is created: with `OSVC_NAMESPACE=test` exported, `om svc1 create` still creates `svc1` in the root namespace, because the path it was given says root. Name the namespace in the path instead. ### Cloning into a namespace When creating an object from another one, `--namespace` overrides the destination namespace, which saves rewriting the path: ```sh om svc2 create --config=svc1 --namespace=test ``` This creates `test/svc/svc2` from the configuration of `svc1`. A configuration that refers to other objects survives the clone only if those references are relative. See [Relative object paths](#relative-object-paths). ## Claims on cluster resources The objects of a namespace consume things the cluster owns and its peers share: the space of a pool, the addresses of a network, the cpu and memory of the nodes. A namespace can be capped on what it takes of one, so that it cannot drain a resource the others are meant to share. A cap is declared in the namespace configuration, as a `claim` section naming the kind of resource, the resource when there are several of that kind, and the limit: ```ini [claim#1] type = pool name = tank limit = 250m [claim#2] type = network name = backend2 limit = 10 ``` Edit it like any other object configuration, naming the namespace with a trailing `/`:
```sh om test/ config update --set claim#1.type=pool --set claim#1.name=tank --set claim#1.limit=250m om / config show ```
```sh curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=claim#1.type=pool' \ --data-urlencode 'set=claim#1.name=tank' \ --data-urlencode 'set=claim#1.limit=250m' \ "https://:1215/api/object/path/test/nscfg/namespace/config" curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/root/nscfg/namespace/config/file" ```
A namespace declaring no claim on a resource is not capped on it. A claim naming no limit says the namespace uses the resource, not that it is capped on it. ### Pool claims A pool claim counts the size each volume of the namespace was created or resized with, cluster-wide. An allocation or a resize taking the namespace over its limit is refused, and the pool is reported as one that does not match, with the reason: ``` $ om test/svc/app provision ... no pool matching criteria: [tank] the test namespace may claim 250mi of it and already claims 200mi, so it cannot claim 100mi more ``` It counts what was asked for, not what is written: a pool hands out what it promised, and that promise is what is being rationed. So a namespace can be at its limit on a pool that is nearly empty, and a pool can be full while every namespace is under its limit. ### Network claims A network claim counts the addresses the namespace holds in the network, cluster-wide. An allocation taking the namespace over its limit is refused: ``` $ om test/svc/app start ... ip#1: start: network backend2: the test namespace may hold 10 address(es) of it and already holds 10 ``` Addresses are counted, not reservations: a failover object holds the same address on every node it is configured on, and that is one address taken from the network, while the instances of a flex each hold one of their own. The limit is checked when an address is allocated, not when one already held is asked for again, so a namespace that reached its limit still restarts what it already runs. Lowering a limit below what a namespace already holds is therefore allowed, and takes effect on the next allocation rather than by taking addresses back. ### Compute claims A `cpu` or `memory` claim bounds what the objects of the namespace may be capped to, cluster-wide. It names no resource, because the cluster has only one compute: ```ini [claim#3] type = cpu limit = 800% default = 50% [claim#4] type = memory limit = 16g default = 512m ``` The cpu `limit` uses the `pg_cpu_quota` notation: `800%` or `100%@8` is eight cpus. `@all` is refused, because it depends on the size of a node. The memory `limit` is a size.
```sh om test/ config update --set claim#3.type=cpu --set claim#3.limit=800% --set claim#3.default=50% ```
```sh curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=claim#3.type=cpu' \ --data-urlencode 'set=claim#3.limit=800%' \ --data-urlencode 'set=claim#3.default=50%' \ "https://:1215/api/object/path/test/nscfg/namespace/config" ```
**What an object claims.** An object claims what its processes are capped to (`pg_cpu_quota`, `pg_mem_limit`), counted on every instance it can run at once: * A resource is bounded by its own cap. A subset or an object cap bounds the sum of the caps under it. * A failover object counts one started instance. A flex object counts `flex_target` of them. Any node might run a started instance, so each one is counted on the node where its caps are highest. * Every other node counts the object's standby resources, which run there too. So a flex object with a target of 2 and a 50% cpu container claims 1 cpu, however many nodes it lists. The daemon publishes each object's claim in its instance configuration, as `claims`. **Default caps.** A container, or a podman, docker or oci task, whose resource, subset and object all say no cap is given the claim's `default`. It runs with that cap, and is counted for it. The processes of `app` resources never get the default: capping them is up to the object. **Uncapped processes.** A process that nothing caps can take the whole node, so it cannot be counted. An object running one in a claimed namespace cannot grow its claim until it is capped: ``` $ om test/svc/web config update --set app#1.type=simple --set app#1.start="/bin/sleep 1000" Error: test/svc/web: [403] Forbidden: claim overrun: ... runs a process capped by nothing on it: cap it with pg_cpu_quota on its resource or its object (the default of a claim caps the containers only) ``` **When it is checked.** The claim is weighed when a configuration is written, before anything runs. A write that raises an object's claim past the limit is refused: ``` $ om test/svc/web config update --set container#2.pg_cpu_quota=60% Error: test/svc/web: [403] Forbidden: claim overrun: test/svc/web claims cpu 1.1 cpu, memory 384mi, and cpu: the test namespace may claim 1 cpu of it and already claims 0.75 cpu, so it cannot claim 0.35 cpu more ``` A write that leaves an object's claim unchanged, or lowers it, is always accepted. An object created before the namespace had a claim can therefore still be edited. **Starting more instances than counted.** Starting an instance on top of the ones its claim counts, such as a second instance of a failover object, is refused to users other than root: ``` [403] Forbidden: claim overrun: the test namespace claims the compute of 1 started instance(s) of test/svc/web, and 1 already run(s) ``` Root is let through, and the daemon logs a warning. Compute claims are how a namespace is bounded when its containers run [rootless](apps.design.rootless.md). Those run in their users' cgroup trees, which the namespace `pg_*` caps do not reach. ### Where the limit is read The limit is read from the namespace configuration on the node doing the allocation, never over the api. A namespace configuration is present on every cluster node, and most namespaces claim nothing, so the common case asks nothing of the daemon. Only a namespace that is actually capped needs the cluster-wide count of what it already holds, and failing to reach the daemon for that count leaves the claim unchecked rather than refused: a cap is something the cluster brokers, and where no daemon answers there is nothing brokering. ## References A `{namespace}` reference is available in service configuration. ## Cgroups Services in an explicit namespace have the namespace name inserted in their cgroup path: ``` opensvc.slice/.slice/.slice ``` ## The "root" pseudo namespace Services created before namespace support, and services created without an explicit namespace, are assigned to a pseudo namespace named `root`. Their configs are stored in `/etc/opensvc/`. The root pseudo namespace name is used in cluster DNS names (`.root.svc.`) and DNS search paths, but is not embedded in Docker container names or cgroup paths. The `{namespace}` reference evaluates to `root`. ### Select objects in the root namespace ```sh $ om 'root/**' ls $ OSVC_NAMESPACE=root om '**' ls ``` ### Create objects in the root namespace ```sh $ unset OSVC_NAMESPACE $ om svc1 create ``` ## Relative object paths An object referring to another one can name it with a path relative to its own namespace, by replacing the namespace with a `.`: ```ini [DEFAULT] parents = ./svc/db [fs#1] install = /etc/app.conf from ./cfg/app key app.conf ``` `./svc/db` reads as "the `db` service, in whatever namespace I am in". For `test/svc/app` it resolves to `test/svc/db`, and for `prod/svc/app` to `prod/svc/db`, with no change to the configuration. That is the point of the notation: a set of objects designed together can be cloned from a development namespace to a production one, and keeps referring to its own members rather than reaching back into the namespace it came from. A fully qualified `test/svc/db` would follow the clone and still point at `test`. Node configuration is not namespaced, so the same notation in `node.conf` resolves in the `system` namespace: `./sec/relays` is `system/sec/relays`. ## Relations Children and parents relations use object paths or object names. - If a name is declared in the parents, slaves, or children lists, the related objects are searched in the same namespace. `svc2` and `./svc/svc2` are the same relation, the second stating it explicitly. - A relation to an object in another namespace must be declared using its path instead of its name. - Relations to objects in the root pseudo namespace must be defined as `root//` to avoid ambiguity. # Configs and Secrets An application needs configuration files, certificates and credentials. Baking them into the image ties the image to one environment. Leaving them on the nodes means every node has to be prepared identically, by something that is not the agent. OpenSVC stores them in objects of their own, `cfg` for configuration and `sec` for secrets, and installs them as files into a service's storage when it starts. `sec` keys are encrypted at rest with the cluster secret. ## Put the data in a datastore
```bash om myapp/cfg/app create om myapp/cfg/app key add --name app.conf --from /tmp/app.conf om myapp/cfg/app key list ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary '[DEFAULT]' \ "https://:1215/api/object/path/myapp/cfg/app/config/file" curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d "[{\"action\": \"add\", \"name\": \"app.conf\", \"string\": $(jq -Rs . < /tmp/app.conf)}]" \ "https://:1215/api/object/path/myapp/cfg/app/data" curl -s -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/myapp/cfg/app/data/keys" ```
The api tab above uses the `$TOKEN` and the listener endpoint set up in [Cluster API](configure.api.md). `PATCH …/data` applies a batch of `add`, `change`, `remove` and `rename` operations as one transaction, and a binary value goes in the `bytes` field, base64 encoded, rather than in `string`. Secrets work the same way, with `om myapp/sec/creds`. ## Install it where the application will look The `install` keyword tells a resource to lay keys down inside the storage it manages, as files: ```ini [fs#1] type = directory path = /var/tmp/myapp install = /etc/app.conf from ./cfg/app key app.conf [app#1] type = simple start = /opt/myapp/bin/start ``` Starting the instance shows the order this happens in: INF myapp: fs#1: create directory /var/tmp/myapp INF myapp: fs#1: create directory /var/tmp/myapp/etc to host key app.conf INF myapp: fs#1: install key app.conf from myapp/cfg/app to /var/tmp/myapp/etc/app.conf with owner : perm -rw-r--r-- INF myapp: app#1: run: /opt/myapp/bin/start The file is in place before the application is launched. That is the point: the resource holding the storage prepares the dataset, and only then do the resources that consume it start. `./cfg/app` is a [relative object path](apps.design.namespaces.md#relative-object-paths), so the same configuration works in whatever namespace it is cloned to. ## Which resources can install Any resource that manages a directory the agent can write into, in other words any driver exposing a head, accepts `install`: * `volume` * `fs` (`fs.directory`, the host filesystem drivers, `fs.zfs`) Resources start in driver group order, and both of those groups start before `container` and `app`. So a volume or filesystem is always able to stage data for the containers and applications that follow it, and never the other way round. ## Sharing across namespaces A datastore serves its own namespace and no other. An object elsewhere naming it is refused, and the whole start is rolled back: ERR consumer: fs#1: start: unauthorized install /etc/pw from prod/sec/shared key pw INF consumer: rollback The `share` keyword on the datastore names the namespaces allowed to decode its keys: ```ini [DEFAULT] share = prod staging ``` The same install then succeeds: INF consumer: fs#1: install key pw from prod/sec/shared to .../etc/pw with owner : perm -rw------- This is what keeps one certificate, licence or database password in one place instead of copied into every namespace that needs it. The copies are the problem: they drift, and rotating the original leaves them behind. `*` shares with every namespace. The default is the datastore's own namespace, so sharing is always a deliberate act. > **Note:** sharing lets another namespace *decode* the keys. Who may read the > datastore object at all is a separate question, answered by > [Access Control](configure.rbac.md). ## Beyond plain files The `install` keyword takes more than a source and a destination: ```ini install = /etc/app.conf from ./cfg/app key app.conf mode 0600 user app group app signal HUP:container#1 ``` * `mode`, `user`, `group` set the installed file's permissions and ownership. Without them a `sec` key installs `0600` and a `cfg` key `0644`, so a secret is not left world-readable by omission. * `signal` sends a signal to another resource's processes when the content changes, so a running daemon reloads its configuration instead of being restarted. * `template` treats the key as a Go template, rendered on each node at each install. See [Templates](#templates) below. * `source` seeds the key from a URI or a local file during provisioning, if it does not exist yet. A template is stored as it is written, and rendered at each install. * `required` stops the install if that item fails, rather than continuing. A trailing `/` on the destination, or omitting `from`, makes it a directory rather than a file. A globbing pattern in `key` installs every matching key. The full syntax is in the `install` keyword reference. ## Templates A `template` line keeps the key as it is written, a Go template, and renders it into the installed file on each node, at each install. One key then serves every node and every object, each file saying what its node knows when it is installed: ```ini [fs#1] type = tmpfs mnt = /srv/{fqdn} install = /app.conf from ./cfg/app key app.conf mode 0600 template ``` with the `app.conf` key of `cfg/app`: ``` listen = {{ref "nodename"}}:{{.port}} peers = {{ref "clusternodes"}} password = {{key "./sec/app" "password"}} ``` The data is the `[env]` section of the object: `{{.port}}` is the evaluated value of `env.port`. The functions are: * `ref ""`: the value of a [reference](apps.design.references.md), written without its braces, evaluated for this node, as `{{ref "nodename"}}` or `{{ref "volume#1.mnt"}}`. * `split `: the words of a string, as a list: `{{range split (ref "clusternodes")}}{{.}} {{end}}`. * `key "" ""`: the content of a key of another datastore, which shares its keys with the namespace of the object. The file holds what the key holds, so a file rendering a sec key wants a `mode` as tight as the key's. * `network ""`, `networks`, `tunnels`: the networks of the node, and the peers a [routed bridge](configure.networks.md#routed-bridge) reaches, with their addresses and the tunnel to each. `tunnels` lists each tunnel once: `{{range tunnels}}{{.Nodename}} {{.Local}} {{.Remote}}{{end}}`. A template reading a key that does not exist, an `[env]` value not set or a reference not known fails to render, and its file is not written rather than written with a hole. ### When a file is rendered again * At each install: a start, a provision, or a task with `install`. * When the template key changes, on each node. * When a key the template read with `key` changes, on each node: a file rendered from a sec follows the sec. The keys read are recorded at each render, so this follows the first render on the node. A file rendered the same as the one installed is left as it is, so the `signal` of the line is sent when the content changes only. ### Telling what reads the files An install that wrote files publishes a `FileInstalled` event, naming the resource and the files, whatever ran it. An install finding its files up to date publishes nothing. A task can run on it, to have what reads the files read them again: ```ini [task#reload] type = host container = container#app command = app reload on = FileInstalled,rid=fs#1 ``` A file rendered from what the node knows, as the networks, is installed again by a task running on the event telling it changed. Both tasks are described in [Event-triggered Runs](internals.daemon.scheduler.md#event-triggered-runs), and the [strongSwan Mesh](../howtos/strongswan.mesh.md) howto puts them together. > ➡️ See Also > * [Namespaces](apps.design.namespaces.md) for the relative path notation. > * [Provisioning](apps.deploy.provisioning.md) for when `source` seeding runs. > * [Events](internals.daemon.events.md) for the `FileInstalled` event. # Certificates A `sec` holds a TLS certificate as it holds any secret: as keys, encrypted at rest, [installed](apps.design.datastores.md) into the storage of the services that serve it. The agent also makes the certificate, and renews it before it expires, from an ACME directory like Let's Encrypt or generated by the sec itself. ## The keys of a certificate The certificate is described by keywords of the sec: `cn` is its subject, and the first of its names, `alt_names` its other names. `bits`, `validity`, `email`, and the `c`, `st`, `l`, `o` and `ou` fields complete what a generated certificate says. ```ini [DEFAULT] cn = www.example.com alt_names = example.com ``` Wherever it comes from, the certificate is kept as four keys: | key | holds | | :--- | :--- | | `private_key` | the private key, PEM encoded | | `certificate` | the certificate | | `certificate_chain` | the certificate, followed by the certificates of its issuers | | `fullpem` | the private key followed by the chain, the one file haproxy reads | An application installs the ones it reads, as it would any key: ```ini [volume#1] install = /certs/www.pem from ./sec/web key fullpem signal HUP:container#1 ``` ## Generate a certificate `certificate create` generates a certificate, signed by the sec itself: ```bash om ns1/sec/web create --kw cn=www.example.com om ns1/sec/web certificate create ``` With `ca` naming another sec of the namespace that holds a certificate, the certificate is signed by that one instead. A self-signed certificate is what an internal endpoint or a first test needs. A browser warns about it. ## Obtain a certificate from an ACME directory A sec naming a directory in its `acme` section obtains its certificate there: ```ini [DEFAULT] cn = www.example.com alt_names = example.com email = admin@example.com [acme] directory = letsencrypt ``` `letsencrypt` and `letsencrypt-staging` name the two directories of Let's Encrypt. Any other value is the url of a directory. The staging directory is the one to try a setup out with: its rate limits are far higher, and its certificates are not trusted, so a mistake shows before it costs anything. Changing the directory afterwards renews the certificate from the new one. The directory proves that the requester controls the domains, `cn` and `alt_names`, with the **http-01 challenge**. It asks for a token, then fetches it from each domain at `http:///.well-known/acme-challenge/`. So the http server answering for the domains on port 80 must serve that path from the directory the token is written in, the **webroot**. The other challenges, through the dns or tls-alpn, are not supported. The account at the directory is registered on the first renewal, with the `email` of the sec as its contact when it has one, and kept in the sec, as the certificate is. Any node renewing the certificate next renews it with the same account. ## Renew A renewal does something only when it is due: * there is no certificate yet * the certificate names other domains than `cn` and `alt_names` * it expires within `acme.renew_before`, 30 days by default * an ACME directory is asked, and the certificate is self-signed or was issued by another directory Otherwise it says why it did nothing. So it is safe, and meant, to run on a schedule: no certificate issued: the certificate is valid until 2026-12-31T19:19:00+01:00 A sec naming no directory renews by generating its certificate again, as `certificate create` does. Its renewal is due on the same conditions, a self-signed certificate excepted. A renewed certificate does not wait for the next start of the services using it. Every volume installing a key of the sec, on every node running it, gets the new key, and the `signal` of its install line is sent, so the server reloads its certificates: INF web: volume#1: reinstall key fullpem from ns1/sec/web to /srv/web.ns1.vol.cluster1/certs/www.pem with owner 99:99 perm -rw------- INF web: container#1: send SIGHUP signal to container ns1..web.container.1 (pid 3375461) ### From the service publishing the domains: task.acme The natural place to renew from is the service whose http server answers for the domains, because the challenge token must be written where that server reads it. A `task.acme` resource of that service renews the secs it lists, and writes the token in one of its volumes: ```ini [task#acme] type = acme secs = web webroot = volume#1:/acme-challenges schedule = @1d run_requires = container#1(up) ``` * `secs` lists the secs to renew. `web` and `./sec/web` are secs of the namespace of the service. `ns2/sec/web` is a sec of another namespace, which must [share](apps.design.datastores.md#sharing-across-namespaces) it with this one. * `webroot` names a directory in a volume or a filesystem of the service, by the rid of its resource, as `volume#1:/acme-challenges`. The token is written under `.well-known/acme-challenge/` there, and never outside the volume, whatever links it holds. * `run_requires` keeps the task from proving the domains while the server is not there to answer the directory. The task runs where the service runs, so the server it writes the token for is the one the domains reach. Run it now rather than wait for its schedule:
```bash om ns1/svc/web instance run acme --node --follow ```
```bash curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/ns1/svc/web/action/run?rid=task%23acme" ```
INF ns1/svc/web: task#acme: ns1/sec/web: certificate issued for www.example.com example.com: no certificate A `task.acme` is one a namespace administrator may configure. Its action is the agent's own rather than a command its writer chooses, it renews only the secs its namespace may use, and the only thing it writes on the node is the token, inside a volume of its own service. The [Ingress Gateway](configure.igw.md) page deploys a haproxy renewing its certificates this way. ### From the command line `certificate renew` renews one sec where it runs, the token written in the host directory `acme.webroot` names, or `--webroot`: ```bash om ns1/sec/web certificate renew --webroot /var/www/html ``` It suits a web server running on the node itself. `acme.webroot` being a path of the node, setting it requires the root grant. `--force` renews a certificate that is not due. The command has no api counterpart, a renewal from a service being the `task.acme` above. ### Through the agent listener A sec whose certificate the agent listener presents, listed in the cluster `listener.tls_secs`, needs no webroot: the listener of every node answers its challenges on `listener.acme_port`, from the token the renewal stores in the sec and the daemon replicates. The public name of the cluster API is certified this way, see [Cluster API](configure.api.md#option-3-a-public-certificate-from-an-acme-directory): ```bash om system/sec/public certificate renew ``` > ➡️ See Also > * [Configs and Secrets](apps.design.datastores.md) for the install lines and > the sharing of a sec. > * [Ingress Gateway](configure.igw.md) for a haproxy renewing its > certificates. # Resource Capping A misbehaving application should not be able to take a node down with it. The agent puts every object it starts in its own cgroup, and the `pg_*` keywords turn that into a limit. ## The slice an object runs in Starting an instance builds a hierarchy under `opensvc.slice`, one level per scope: opensvc.slice └ opensvc-ns..slice └ opensvc-ns.-svc..slice └ opensvc-ns.-svc.-app.1.slice A resource's processes live in its own leaf, inside its object, inside its namespace. That nesting is what makes a limit set at any level apply to everything below it. ## Capping an object ```ini [DEFAULT] pg_cpus = 0 pg_mem_limit = 256m ``` Starting the instance logs what it applied: INF myapp: applied pg /opensvc.slice/...-svc.myapp.slice: cpus=0 mem_limit=256m INF myapp: app#1: run: om exec --pg /opensvc.slice/...-svc.myapp-app.1.slice -- ... And the kernel agrees: ```bash $ cat /sys/fs/cgroup/opensvc.slice/.../opensvc-svc.myapp.slice/memory.max 268435456 $ cat /sys/fs/cgroup/opensvc.slice/.../opensvc-svc.myapp.slice/cpuset.cpus 0 ``` The keywords are the usual cgroup controls: | Keyword | Caps | | :--- | :--- | | `pg_cpus` | the cpus the object may run on, as a list or range: `0,1,2` or `0-2` | | `pg_mems` | the memory nodes it may allocate from, same syntax | | `pg_cpu_shares` | its share of cpu **when the node is cpu-bound**, relative to other objects | | `pg_cpu_quota` | its cpu time whether or not the node is busy: `50%` is half of one cpu, `50%@all` half of every cpu the node has, `10%@2` a tenth of two | | `pg_cpu_burst` | cpu time banked while under the quota and spent above it in a short spike, in the `pg_cpu_quota` notation. It cannot exceed the quota. Unified hierarchy only | | `pg_mem_limit` | resident memory, in bytes. Exceeding it wakes the OOM killer | | `pg_mem_high` | resident memory above which the kernel reclaims and throttles, without killing. Set it below `pg_mem_limit`. Unified hierarchy only | | `pg_vmem_limit` | memory plus swap | | `pg_pids_max` | processes and threads running at once. A fork beyond it fails | | `pg_mem_oom_control` | `0` lets the OOM killer run, `1` freezes the group instead. v1 hierarchy only | | `pg_mem_swappiness` | how readily its pages are swapped. v1 hierarchy only | | `pg_blkio_weight` | its share of block io, between `10` and `1000` | `pg_cpu_shares` and `pg_cpu_quota` are the pair worth telling apart: shares only arbitrate a contended cpu, whereas a quota caps the group on an idle node too. `pg_mem_high` and `pg_mem_limit` are the other pair: past the first, the group runs slower while the kernel reclaims its memory, and past the second it is killed. A memory claim counts `pg_mem_limit`, the one that bounds the group. systemd has no property for the cpu burst, so it neither keeps it nor resets it. The agent writes it every time it applies the caps, which is before each container starts in the group. The two marked *v1 hierarchy only* cap nothing on a node running the unified hierarchy, which has neither `memory.swappiness` nor `memory.oom_control`. Setting one there is warned about, and ignored. ## Capping a resource The same keywords set on a resource cap that resource alone, which is how a sidecar is kept from starving the process it assists: ```ini [app#1] type = simple start = /opt/myapp/bin/server [app#2] type = simple start = /opt/myapp/bin/indexer pg_cpu_shares = 128 ``` ## Changing the caps of a running object Editing the keywords changes the configuration, not the running instances: the new caps apply at the next start. `cap` changes both at once. It sets the keywords, then applies them to every instance running, on every node, without a restart:
``` om test/svc/myapp cap --set pg_cpu_quota=60% --set app#2.pg_cpu_shares=128 ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/test/svc/myapp/action/cap?set=pg_cpu_quota%3D60%25&set=app%232.pg_cpu_shares%3D128" ```
A `--set` names a keyword of the object, as `pg_cpu_quota=60%`, or of a subset or a resource, as `app#2.pg_cpu_shares=128`, and takes a [scope](apps.design.scoping.md) as any keyword does: `pg_cpu_quota@node2=60%`. Only the `pg_*` keywords are accepted, and only with `=`. The command help lists them, and `om config doc --kw pg_cpu_quota` prints the full text of one. `cap` needs the admin grant on the namespace of the object. The caps are checked as any configuration change is: a value the keyword does not take, a keyword the rbac policy does not grant, or a cap beyond the [compute claims](apps.design.namespaces.md#compute-claims) of the namespace is refused, and nothing is written. `cap` is an orchestration, so it is refused while another one is in progress on the object, and nothing is written then either. Like the other orchestrations, it returns once the daemon queues it, and `--wait` holds it until every node has applied the caps. An instance not running has nothing to apply them to, and gets them when it starts. With no `--set`, `cap` applies the caps the configuration already holds, which puts back a cap lifted or changed by hand:
``` om test/svc/myapp cap ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/test/svc/myapp/action/cap" ```
The api answers the `orchestration_id` of the queued orchestration, which is waited for as the [other actions'](apps.operate.action.md) are. ## Capping a namespace An `nscfg` object holds the defaults of its namespace, the `pg_*` keywords among them: ```bash om test/nscfg/namespace create --kw pg_mem_limit=4g ``` Every object in `test` is then capped by it, no matter who created it, which is how a namespace is handed to a team without handing them the node. A [rootless container](apps.design.rootless.md) is the exception. It runs in the cgroup tree of its user, outside `opensvc.slice`, and the namespace and node caps are not copied there. To bound what a namespace takes wherever its containers run, use [compute claims](apps.design.namespaces.md#compute-claims), which count the caps of the objects. ## Caps that systemd keeps podman asks systemd to start the group of a container, and systemd writes a slice's cgroup files from the unit properties whenever it starts that slice. A cap written only to the files would be reset to "no cap" the next time a container started. So the agent also sets the caps as runtime properties of the slice, on the system manager, or on the user's manager for a rootless container: ```bash $ systemctl show -p CPUQuotaPerSecUSec,MemoryMax 'opensvc-svc.myapp.slice' CPUQuotaPerSecUSec=500ms MemoryMax=268435456 ``` They are runtime properties: they are lost at reboot, and the next start applies them again from the configuration. ## Rootless containers The `pg_*` keywords of a rootless container, and those of its object and subset, cap groups the agent makes in the tree systemd delegates to the user: user.slice/user-.slice/user@.service └ opensvc.slice └ opensvc-svc..slice └ opensvc-svc.-container.1.slice The cap files stay owned by root, so the user cannot lift them. A cap whose controller the user's tree is not delegated is reported as not applied. By default that is `pg_cpus`, `pg_mems` and `pg_blkio_weight`. See [Rootless Containers](apps.design.rootless.md#delegating-more-controllers) for how to delegate them. ## Lifting a capping Removing a `pg_*` keyword does not lift what it capped. What was written stays written, because the agent has no way of telling a capping it wrote from one systemd or an operator wrote, and silently undoing the second would be worse than leaving the first. So lifting a capping is asked for, and there are two ways of asking. The keyword set to `default` puts that one capping back where a node that never capped anything leaves it: ```ini [DEFAULT] pg_cpu_quota = default ``` This is the one to reach for. It is the configuration, so every node converges to it: a peer taking the object over, or a node provisioned tomorrow, lifts the capping too. `om test/svc/myapp cap --set pg_cpu_quota=default` writes it and lifts the capping on the running instances at once. The command lifts every capping of an instance at once, whatever the keywords say: ```bash om test/svc/myapp instance pg reset ``` That one is for a capping the configuration knows nothing about — left by an older agent, by systemd, or by hand. It is not a lasting decision: a capping the keywords do name comes back at the next `pg update`, and at the next start. ## Turning it off ```ini [DEFAULT] create_pg = false ``` Grouping is on by default. Turning it off leaves the processes ungrouped and uncapped, and is worth doing only where the cgroup itself causes the problem. > ➡️ See Also > * [Namespaces](apps.design.namespaces.md) > * [Rootless Containers](apps.design.rootless.md) # Data Replication A failover service whose data lives on local disks needs a copy of that data on the nodes it may fail over to. The `sync` resources make that copy. They send the data from the node running the service to its peers, on a schedule. The two drivers most setups use: | Driver | Copies | Unit of transfer | | :--- | :--- | :--- | | `sync.rsync` | a directory tree | the files that changed | | `sync.zfs` | a zfs dataset, and its children when `recursive` is set | the blocks changed since the last snapshot the peer received | ## Choosing the peers The `target` keyword names the peers a resource replicates to: `nodes`, the other nodes of the service, `drpnodes`, its disaster recovery nodes, or both. ```ini [sync#1] type = zfs src = tank/{fqdn} dst = tank/{fqdn} target = nodes drpnodes schedule = @60m ``` `schedule` runs an update every hour here. An update run by hand, or a full copy, can be narrowed to some of those peers with `--target`. It takes `nodes`, `drpnodes`, `local`, or a node selector expression such as a node name, a glob like `n*`, or a label like `site=paris`. Only the peers the resource's `target` keyword reaches are selected: a `--target` that selects none of them is refused.
``` om test/svc/db instance update --rid sync#1 --target drpnodes ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/test/svc/db/action/update?rid=sync%231&target=drpnodes" ```
A sync runs on the node the service runs on, which is the node the api request names. That node is told by the reference resources of the service, the ones it holds its data on or is reached by: every resource but the app, sync and task ones. They must be up there, and a sync asked on a node where they are not is refused, as it would overwrite the running copy with a stale one. A service with no reference resource, only apps and syncs, gives no way to tell, and is not synced: give it one, a `fs.flag` if nothing else fits. `--force` sends from a node whose reference resources are warn. ## Stopping or switching while a sync runs Stopped under a sync, a service would hand its peers a copy the sync was writing. A stop or a switch waits for the syncs running on the instance to end, in the `wait syncs` state: 09:37:17 n1 wait syncs/up run=sync#2 | n2 stopped/down 09:37:36 n1 idle/down | n2 starting/down 09:37:39 n1 idle/down | n2 idle/up It waits `wait_syncs_timeout` at most, 10 minutes by default, set to what the largest syncs of the service take. Past it, the stop fails, and the service keeps running where it was, its monitoring on: syncs sync#2 still running after 10m0s: the instance is not stopped To stop or switch without waiting, ask to end the syncs instead. The rsync and ssh processes they started end with them, and a zfs copy the peer was receiving is discarded whole:
``` om test/svc/db switch --to n2 --interrupt-syncs ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"destination": ["n2"], "live": false, "interrupt_syncs": true}' \ "https://:1215/api/object/path/test/svc/db/action/switch" ```
`om test/svc/db stop --interrupt-syncs` stops the same way, `?interrupt_syncs=true` on the api. A stop run on the node, `om test/svc/db instance stop`, waits the same way for the syncs holding the object lock, and says so: wait for the update of the syncs (pid 340343, running for 3s) to end, 10m0s at most: --interrupt-syncs ends them instead No scheduled sync starts while a stop, a switch or another orchestration is in progress: the next period syncs, from wherever the service runs then. A drain does not wait: the shutdown of the instances of a drained node ends the syncs running first, the rsync and ssh processes they started included, so none keeps sending to the peer taking over. No scheduled sync starts on a node being drained. ## Holding the updates `update_requires` holds the updates and the full copies of a resource until other resources are in the states it names. A snapshot of a dataset, for one, is taken only while the filesystem on it is mounted: ```ini [fs#2] type = zfs dev = tank/{fqdn} mnt = /srv/{fqdn} [sync#3] type = zfssnap dataset = tank/{fqdn} name = hourly keep = 3 schedule = @60m update_requires = fs#2(up) ``` The scheduler does not schedule the update while the requirement is not met, and schedules it again once it is. An update asked by hand is refused, and says why: ERR test/svc/zfs30: sync#3: sync requires: the resource action requirements are not met: action update on resource sync#3 requires fs#2 in states (up), but is down A condition is `(,...)`, and states left out mean `up,stdby up`. The v2 names `sync_update_requires`, `sync_nodes_requires` and `sync_drp_requires` are read as `update_requires`. ## Reading the status A sync resource reports whether the copy is fresh. On the node running the service, it is up when every peer was synced recently. On a peer, it is up when that peer received a copy recently. "Recently" is `max_delay` when set. Otherwise the copy is stale once the sync the `schedule` makes due after the last one is late by half a schedule period: 45 seconds after the last sync for `@30s`, 36 hours for a daily sync. Past it, the resource is warn. A sync resource tells whether the data is replicated, not whether the service runs, so it does not count in the availability of the instance. In its overall status, a sync up counts as nothing. A sync warn, or down as the drivers of array replication may report, counts as a warning: | Instance | Other resources | `sync#1` | avail | overall | | :--- | :--- | :--- | :--- | :--- | | running | up | up | up | up | | running, a peer's copy stale | up | warn | up | warn | | standing by, receiving the copy | down | up | down | down | | standing by, its copy stale | down | warn | down | warn | So the node standing by for a service reads down, as it should, and only a stale copy raises a warning. A copy goes stale with no event to tell, so each resource says when its status will change, and the daemon evaluates the status again then. A copy found stale turns fresh again when the next sync lands: the source writes the peer's last sync record, and the peer evaluates its status on it. The warnings come and go within seconds of the copies, not at the next scheduled status evaluation. ### The recovery point objective The `max_delay` of a sync resource, explicit or derived from its schedule, is its recovery point objective: past it, the node standing by holds data older than the resource allows to lose. Each resource keeps its own. The logs of a database may be replicated with a 2 hours delay, and its redo logs with a 10 minutes delay, in the same service. A node standing by breaches the recovery point objective as soon as one of its copies is older than its resource allows: were it to take over then, it would lose more data than that resource's contract. Each instance publishes when that happens, `rpo_breached_at`, the earliest time among its sync resources, and each resource publishes its own. The node the data is replicated from publishes none: its data is current. `om mon` marks with a red `L` the instances past that time: test/svc/db up! ha 1/1 | O!^ X X!L and the status tree tags them, the warning of the resource naming the contract breached: ├ n3 down warn rpo-breached idle │ warn: n3 last sync is too old, at 2026-09-29 09:17:54 +0200 CEST, more than 45s ago (schedule @30s, half a period after the sync due) The time is compared to the clock when the status is read, so the mark is right whenever the instance status was last evaluated. Choosing a node to fail over to, prefer one with no mark. ## ZFS replication ### Each peer has its own base Each update takes a snapshot of the source, named after the resource and the time it was taken, to the microsecond, as `sync.1.20260928T154211.315216Z` for `sync#1`. It then sends each peer the changes between that snapshot and the newest snapshot the peer already holds, its base. The snapshots are matched by their guid, so a peer's base is found whatever either side calls it, after a failover too. Once a peer has received the new snapshot, its older ones are destroyed. The source keeps the newest snapshot, and the base of any peer that fell behind: dev2n1: sync.1.20260928T154154.161026Z sync.1.20260928T154203.972669Z dev2n2: sync.1.20260928T154203.972669Z dev2n3: sync.1.20260928T154154.161026Z Here `dev2n3` missed the last update. The update failed, and said why: ERR test/svc/zfssync: sync#1: dev2n3: list snapshots: dial tcp 10.29.0.13:22: connect: connection refused The next update sends `dev2n3` everything it missed, from the base it holds, and the three nodes are in step again: INF test/svc/zfssync: sync#1: /usr/sbin/zfs send -R -I tank/zfssynctest@sync.1.20260928T154154.161026Z tank/zfssynctest@sync.1.20260928T154211.315216Z | ssh dev2n3 '/usr/sbin/zfs receive -dF tank' A short outage heals by itself, and a peer out of reach does not hold up the others. A peer that holds no snapshot of the resource, a new node or one whose dataset was destroyed, is sent a full copy automatically. ### Limits on a lagging peer The base the source keeps for a lagging peer holds on to the blocks changed since. The longer the outage, the more space it holds, and a pool that fills up takes the running service down with it. Two keywords bound what a lagging peer may cost: | Keyword | Default | Bounds | | :--- | :--- | :--- | | `max_lag_age` | `24h` | how long a peer may miss updates, counted from the first one it missed | | `max_lag_size` | `20%` | the space the snapshots kept for it may hold on the source: a size, or a percentage of the space the pool would have free without them | Past either limit, the source destroys the peer's base and stops sending to it. The update says so: ERR test/svc/zfssync: sync#1: dev2n3: stranded: lagging for 1m6s, more than max_lag_age 1m0s: run 'om test/svc/zfssync instance full --rid sync#1 --target dev2n3' The status warns until the peer is synced again: └ dev2n1 n/a warn idle └ resources └ sync#1 ...../.. warn zfs tank/zfssynctest to nodes warn: dev2n3: not synced since 2026-09-28T17:43:26+02:00: lagging for 1m6s, more than max_lag_age 1m0s: run 'om test/svc/zfssync instance full --rid sync#1 --target dev2n3' The same happens to a peer that holds snapshots of the resource, none of them in common with the source, as after both sides ran the service during a split. ### Syncing a peer again A peer with no base left needs a full copy. The full copy replaces the peer's dataset, whatever it holds, so the agent does not decide it alone. Once the peer is back and holds nothing worth keeping, run on the node running the service the command the warning gives:
``` om test/svc/zfssync instance full --rid sync#1 --target dev2n3 ```
``` curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/node/name//instance/path/test/svc/zfssync/action/full?rid=sync%231&target=dev2n3" ```
The peer receives the whole dataset as of a new snapshot, which becomes its base, and the next updates are incremental again. The other peers are not touched. ### Upgrading from an older agent Older agents kept two snapshots per resource, `.sent` and `.tosend`, shared by all the peers. The first update after the upgrade uses them as the base of the peers holding them, and they are destroyed once no peer needs them. A peer out of reach during that update keeps them as its base, and is not sent a full copy when it comes back. # Rootless Containers A podman container, or a podman task, can run as an unprivileged unix user instead of root. The agent still drives it, but every podman command runs as that user, so the container, its image and its store belong to the user, and the processes of the container run under the user's subordinate ids. Root inside the container is the user on the host, not root. This is the podman driver's rootless mode. The docker and oci drivers have no equivalent. ## What to expect A rootless container is managed like any other: it starts, stops, fails over and reports its status through the same commands. What changes is on the node: * **The processes are the user's.** Root in the container is the user on the host, and ids from 1 are the user's subordinate ids. A process escaping the container has the rights of that user, no more. * **The store is the user's.** Images are pulled into `~/.local/share/containers` of the user, on each node. An image root pulled is not seen, and the user pulls it again. * **The container lives in the user's cgroup tree.** podman places it under `user.slice/user-.slice/user@.service`, the subtree systemd delegates to the user, not under `opensvc.slice`. The agent makes the groups of the object there and caps them. The files holding the caps stay root's, so the user cannot lift them. * **It outlives the daemon.** The containers run under the user's systemd instance, not under the agent, so restarting the daemon or the `opensvc-server` unit leaves them running, with their caps. * **The resolver is written where the user can read it.** The `resolv.conf` of the container is written under `/run/user//opensvc/`, rewritten on every start. * **Networking works as for a rootful container.** An `ip.netns` resource can plumb an address in the network namespace of a rootless container, as in the example below. * **Some caps need the controller delegated.** A default `user@.service` is delegated the `cpu`, `memory` and `pids` controllers. `pg_cpus`, `pg_mems` and `pg_blkio_weight` need `cpuset` and `io`, and are reported as not applied until those are delegated too. See [Delegating more controllers](#delegating-more-controllers). * **Namespace and node caps do not reach it.** The groups of the namespace and of the node are not copied into the user's tree. Copying them would give each user tree its own full budget, which caps nothing. The object caps and the resource caps apply. What the namespace consumes is accounted for by its [compute claims](apps.design.namespaces.md#compute-claims). ## Setting up the nodes Do this once per user, on every node the object can run on. ### 1. The user and its subordinate ids ```sh useradd --create-home webapp grep webapp /etc/subuid /etc/subgid ``` `useradd` grants subordinate ids on most distributions. If the `grep` prints nothing, grant a range yourself: ```sh usermod --add-subuids 200000-265535 --add-subgids 200000-265535 webapp ``` Give the user the **same uid, the same subordinate ranges, and the same home** on every node: the host ids of the container's files depend on them, and a volume failing over to another node has to be read there the same way. ### 2. A systemd instance that runs without a login ```sh loginctl enable-linger webapp ls -d /run/user/$(id -u webapp) ``` Without linger, the user's systemd instance, and every container under it, stops when the user's last session ends. It also does not run at boot, so the agent cannot start the container there. ### 3. Delegating more controllers This step is optional. It is needed only for `pg_cpus`, `pg_mems` or `pg_blkio_weight` on a rootless container: ```sh mkdir -p /etc/systemd/system/user@.service.d cat >/etc/systemd/system/user@.service.d/delegate.conf <
```sh om test/ config update --set rootless_users=webapp ```
```sh curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -G \ --data-urlencode 'set=rootless_users=webapp' \ "https://:1215/api/object/path/test/nscfg/namespace/config" ```
* `rootless_users` lists accounts by name or uid. An allowed account's primary group is allowed too. * `rootless_groups` lists any other group a container may run as, through `rootless_group`. A group reaches every file it owns, so a group like `docker` or `disk` reaches the node itself. * The root account and the root group are never allowed. A namespace admin setting `rootless_user` or `rootless_group` to anything else is refused. The check reads the value as it evaluates, through references and on each node: ``` [403] denied: container#0 runs as daemon on n1: the test namespace does not allow rootless_user daemon (uid 1): it allows webapp ``` Root is not bound by the list. A container running as an account its namespace does not allow shows the reason as a status warning. That happens when root set it up, or when the squatter removed the account afterwards. Allowing one account in two namespaces lets each reach the other's containers. That is allowed but warned about, both in the daemon log when the namespace config is written and in the status of every container running as that account. Give each namespace an account of its own. ## Configuring a rootless container Set `rootless_user` on the container. `rootless_group` replaces the user's primary group when needed. Leaving `rootless_user` unset runs the container as root, which is the default. ```ini [DEFAULT] nodes = n1 n2 [container#0] type = podman image = ghcr.io/opensvc/pause rm = true netns = none rootless_user = webapp [container#1] type = podman image = docker.io/library/nginx:latest rm = true netns = container#0 rootless_user = webapp pg_mem_limit = 256m pg_cpu_quota = 50% volume_mounts = web-vol-1/html:/usr/share/nginx/html:ro [ip#0] type = netns netns = container#0 network = backend [volume#1] pool = shm size = 1m install = /html/index.html from ./cfg/web key index.html mode 0640 user {container#1.uid.101} group {container#1.gid.101} ``` All the containers of a pod run as the same user: a container joining another one's network namespace has to be able to enter it. A `task.podman` takes the same `rootless_user` and `rootless_group` keywords, with the same requirements. Create the object as usual:
```sh om test/svc/web create --config web.conf om test/svc/web start ```
```sh curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/octet-stream" --data-binary @web.conf \ "https://:1215/api/object/path/test/svc/web/config/file" curl -s -X POST -H "Authorization: Bearer $TOKEN" \ "https://:1215/api/object/path/test/svc/web/action/start" ```
## Files the container reads A process of the container reads files as its own ids, and those map to other ids on the host. nginx runs as uid 101 in its image, which is not uid 101 on the host but a subordinate id of `webapp`. A file installed for it must be owned by that host id, or nginx cannot read it. The `{.uid.}` and `{.gid.}` references answer the host id that a container id runs as. `{.uid}` alone is the container's root, which is the user itself: | Reference | Rootless, `webapp` = 1001, subuid `200000:65536` | Rootful | | :--- | :--- | :--- | | `{container#1.uid}` | `1001` | `0` | | `{container#1.uid.101}` | `200100` | `101` | The references answer the right ids in either mode. An `install` that names its owners by reference therefore keeps working when the container switches between rootful and rootless. A container given a `userns` mapping answers no id, because podman makes that mapping only when the container starts. The directories on the way to a mounted file must also let the user through. A `pool.shm` volume head is `700` by default, which only root can enter. Give the pool, or the volume's `fs` resource, a `mode` that lets the user in, such as `711`. ## Looking inside The containers are in the user's store, so podman shows them only when it runs as the user: ```sh sudo -u webapp XDG_RUNTIME_DIR=/run/user/$(id -u webapp) podman ps ``` The caps the agent applied are in the user's tree: ```sh cat /sys/fs/cgroup/user.slice/user-1001.slice/user@1001.service/opensvc.slice/*/*container.1.slice/cpu.max 50000 100000 ``` > ➡️ See Also > * [Resource Capping](apps.design.pg.md) > * [Namespaces](apps.design.namespaces.md) > * [References](apps.design.references.md) # References A reference is a marker in the value of a configuration keyword, replaced during evaluation. A reference is formatted as `{}`. References can be used to: * Abstract changing parts of a configuration, so this configuration can be used as a template Example: The object name, object id, the devices that hosts the data * Factorize information, so changing it is easier and safer Example: A project name used to format the name of different resources (volume group, filesystem path, ...) * Contextualize part of a configuration with information known to the agent Example: The cluster nodes, cluster dns ip addresses, ... ## Intra-Configuration References The reference format is `{[
.]