Section DEFAULT

Supported keywords:

• app
• children
• comment
• comp_schedule
• create_pg
• disable
• drpnodes
• encapnodes
• env
• flex_max
• flex_min
• flex_primary
• flex_target
• hard_affinity
• hard_anti_affinity
• id
• info_schedule
• monitor_action
• monitor_schedule
• nodes
• orchestrate
• parents
• pg_blkio_weight
• pg_cpu_burst
• pg_cpu_quota
• pg_cpu_shares
• pg_cpus
• pg_mem_high
• pg_mem_limit
• pg_mem_oom_control
• pg_mem_swappiness
• pg_mems
• pg_pids_max
• pg_vmem_limit
• placement
• pre_monitor_action
• priority
• provision
• provision_timeout
• restore_from
• rollback
• run_schedule
• shared
• soft_affinity
• soft_anti_affinity
• start_timeout
• stat_timeout
• status_schedule
• status_timeout
• stonith
• stop_timeout
• sync_schedule
• sync_timeout
• timeout
• topology
• type
• unprovision
• unprovision_timeout
• wait_syncs_timeout

Keyword app

required:    false
scopable:    false
default:     default

Description:

A user-defined code linking to:

• who is responsible for this service.
• who is billable.

This code thus provides a most useful object grouping and filtering key.

Short and simple codes, like ERP, are easier to work with.

Keyword children

required:    false
scopable:    false
convert:     listlowercase

Description:

The list of services or instances expressed as <path>[@<nodename>] that must be down or stdby up to allow this service to be stopped by the daemon.

The list is whitespace-separated.

Children are looked up in the same namespace first, so for a test/svc/svc1, the svc2 relation is interpreted as test/svc/svc2.

The ./svc/svc2 notation states that explicitly. It resolves in the namespace of the object declaring it, so a set of objects cloned into another namespace keeps referring to its own members.

To declare a relation to svc2 in the root namespace, use the fully qualified path notation root/svc/svc2.

Keyword comment

required:    false
scopable:    false

Description:

A free form text describing the role of the object, of the node, or of the section it is set in.

The keyword is accepted in any section, so the DEFAULT section can document a configuration as a whole, and a resource, pool, heartbeat, array or network section can document itself.

The agent does not interpret the value.

Keyword comp_schedule

required:    false
scopable:    true
default:     ~00:00-06:00

Description:

The instance compliance run schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword create_pg

required:    false
scopable:    true
default:     true
convert:     bool

Description:

Use process grouping when possible.

If turned on, the agent will create a container group for:

• the object
• each resource group (ie, the subset:drivergroup tuple)
• each resource

A container group allows capping the memory, swap and cpu usage. These cappings can be defined using the pg_* keywords in the DEFAULT, the subset or the resource section.

Keyword disable

required:    false
scopable:    true
convert:     bool

Description:

Disables the object instance:

• Instance and all resource statuses become n/a.
• Instance actions (e.g. start, stop, …) are silently ignored.
• Resources stay disabled regardless of DEFAULT.disable=false.
• disable=false on a resource does not override DEFAULT.disable=true.

Keyword drpnodes

required:    false
scopable:    true
convert:     peers

Example:

drpnodes=n1 n2

Description:

A node selector expression specifying the list of cluster nodes hosting object instances when all primary nodes are unavailable, like in a DRP situation.

If not specified or left empty, the node evaluating the keyword is assumed to be the only instance hosting node.

Labels can be used to define a list of nodes by an arbitrary property. For example cn=fr cn=kr would be evaluated as n1 n2 n3 if n1 and n2 have the cn=fr label and n3 has the cn=kr label.

The glob syntax can be used in the node selector expression. For example n1 n[23] n4* would be expanded to n1 n2 n3 n4 in a n1 n2 n3 n4 n5 cluster.

The drpnodes can be data synchronization targets for sync resources.

Keyword encapnodes

required:    false
scopable:    false
convert:     list

Example:

encapnodes=n1 n2

Description:

A node selector expression specifying the list of cluster nodes hosting object encapsulated instances.

An object with container resources can have resources managed by OpensSVC agents deployed in these containers. These encapsulated agents form an encapsulated cluster, usually a single node cluster for a failover service.

For example a test/svc/s1 failover service, with a container#0 resource managing a e1 lxc host, can define encapnodes = e1. A app#1 resource with encap = true is then managed by the OpenSVC agent in e1.

Keyword env

required:    false
scopable:    false
aliases:     service_type

Default:

The same as the node env.

Description:

A code like PRD, DEV, etc… the agent can use to enforce data protection policies:

• A non-PRD object instance can not be started on a PRD node, unless the node allow_envs keyword allows its env
• A PRD object instance can be started on a non-PRD node (typically in a DRP situation)
• A PRD object instance running on a non-PRD node does not sync its data to its peers
• A non-PRD node does not sync data to a PRD node

The default value is read from the node env keyword.

Keyword flex_max

required:    false
scopable:    false
aliases:     flex_max_nodes
depends:     topology=flex
default:     {#nodes}
convert:     int

Default:

The number of elements in nodes.

Description:

The maximum number of up instances of this object in the cluster. Above this number the aggregated object status is degraded to warn.

The 0 value is interpreted as unlimited.

Keyword flex_min

required:    false
scopable:    false
aliases:     flex_min_nodes
depends:     topology=flex
default:     1
convert:     int

Description:

The minimum number of up instances of this object in the cluster. Below this number the aggregated object status is degraded to warn.

Keyword flex_primary

required:    false
scopable:    true
depends:     topology=flex
convert:     listlowercase

Default:

The first node of nodes.

Description:

The node in charge of syncing the other nodes in a flex object.

Keyword flex_target

required:    false
scopable:    false
depends:     topology=flex
default:     {flex_min}
convert:     int

Default:

The value of flex_min.

Description:

The optimal number of up instances of the object in the cluster. The value must be between flex_min and flex_max.

If orchestrate=ha, the daemon is free to take action to reach the flex_target.

Keyword hard_affinity

required:    false
scopable:    false
aliases:     affinity
convert:     listlowercase

Example:

hard_affinity=svc1 svc2

Description:

A whitespace separated list of object paths, a name alone being a svc of the object namespace.

An instance of the object starts only on a node where an instance of each of these objects is up: the daemon starts it, switches it or gives it back to no other node, and the start command refuses to start it elsewhere, unless --force-affinity. An instance running is not moved when one of these objects leaves its node.

Objects whose hard affinities name each other, none of them up anywhere, would each wait for the others: the daemon starts the one of the smallest priority, then of the first path in alphanumeric order, as if it had no hard affinity with the others, and the others follow it. A start asked for one of the others waits for it.

Keyword hard_anti_affinity

required:    false
scopable:    false
aliases:     anti_affinity
convert:     listlowercase

Example:

hard_anti_affinity=svc1 svc2

Description:

A whitespace separated list of object paths, a name alone being a svc of the object namespace.

An instance of the object starts only on a node where no instance of these objects is up, standby resources aside: the daemon starts it, switches it or gives it back to no other node, and the start command refuses to start it elsewhere, unless --force-affinity. An instance running is not moved when one of these objects comes to its node.

Keyword id

required:    false
scopable:    false
recorded:    written when what it names is made, and reset when the object is cloned

Default:

A random generated UUID.

Description:

A rfc4122 random uuid generated by the agent.

Keyword info_schedule

required:    false
scopable:    true
aliases:     resinfo_schedule
default:     @60m

Description:

The instance resource info refresh schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword monitor_action

required:    false
scopable:    true
candidates:  crash, freezestop, none, reboot, switch
default:     none
convert:     list
rbac:        Requires the root grant, except for the values switch, freezestop, none.

Example:

monitor_action=reboot

Description:

The action to trigger when a monitored resource is no longer in the “up” or “standby up” state, and all restart attempts for the resource have failed.

The reboot and crash monitor actions do not attempt to cleanly stop any processes. On Linux, they utilize system-level sysrq triggers.

This behavior is designed to ensure that the host stops writing to shared disks as quickly as possible, minimizing the risk of data corruption. This is critical because a failover node is likely preparing to write to the same shared disks.

You can append a fallback monitor action to this keyword. A common example is freezestop reboot. In this case, the reboot action will be executed if the stop fails or times out.

Other monitor_actions values:

• none: Is the No Operation monitor action (the default value).
• freezestop: freeze and subsequently stop the monitored instance.
• switch: try monitored instance stop to allow any other cluster nodes to takeover the instance.

Keyword monitor_schedule

required:    false
scopable:    true
default:     @1m

Description:

The instance monitored resources status evaluation schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword nodes

required:    false
scopable:    true
convert:     nodes

Default:

The lowercased hostname of the evaluating node.

Example:

nodes=n1 n*

Description:

A node selector expression specifying the list of cluster nodes hosting object instances.

If not specified or left empty, the node evaluating the keyword is assumed to be the only instance hosting node.

Labels can be used to define a list of nodes by an arbitrary property. For example cn=fr cn=kr would be evaluated as n1 n2 n3 if n1 and n2 have the cn=fr label and n3 has the cn=kr label.

The glob syntax can be used in the node selector expression. For example n1 n[23] n4* would be expanded to n1 n2 n3 n4 in a n1 n2 n3 n4 n5 cluster.

Keyword orchestrate

required:    false
scopable:    false
candidates:  no, ha, start
default:     no

Description:

Orchestrate defines how the daemon will manage the service.

• no The daemon does not try to keep the service up. On boot, the service won’t be started.

  The daemon does not try to reach the flex_target number of up instances for flex services.

• start The daemon starts the service on the first daemon start following a node boot, and moves it nowhere afterwards.

  A topology=failover service is started if it does not run elsewhere already and the local node is the natural placement leader. A topology=flex service is started if fewer than flex_target instances are up and the local node is one of the flex_target first natural placement leaders.

  Outside of that boot, the daemon does nothing to keep the service up: it does not failover a service whose instance goes down, and does not try to reach the flex_target number of up instances.

• ha Services with topology=failover failover automatically.

  The daemon tries to reach the flex_target number of up instances for flex services.

The resource restart policy is not affected by the orchestrate value.

Keyword parents

required:    false
scopable:    false
convert:     listlowercase

Description:

The list of services or instances expressed as <path>[@<nodename>] that must be up to allow this service to be started by the daemon.

The list is whitespace-separated.

Parents are looked up in the same namespace first, so for a test/svc/svc1, the svc2 relation is interpreted as test/svc/svc2.

The ./svc/svc2 notation states that explicitly. It resolves in the namespace of the object declaring it, so a set of objects cloned into another namespace keeps referring to its own members.

To declare a relation to svc2 in the root namespace, use the fully qualified path notation root/svc/svc2.

Keyword pg_blkio_weight

required:    false
scopable:    true

Example:

pg_blkio_weight=50

Description:

Block IO relative weight. Value: between 10 and 1000.

The kernel default is 100.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_cpu_burst

required:    false
scopable:    true

Example:

pg_cpu_burst=20%

Description:

The cpu time the group banks under its quota, and spends above it.

It is the time the processes of the group may bank while under their pg_cpu_quota, and spend above it in a short spike, in the notation of pg_cpu_quota: 20% lets the group run a fifth of a cpu above its quota during a period, out of what it left unused before.

It serves a latency-sensitive service capped tightly: a request arriving after an idle spell is served at the speed of the burst, rather than throttled for the rest of the period. Over a longer time the group still uses no more than its quota.

It needs pg_cpu_quota, and cannot exceed it: the kernel refuses a burst larger than the quota. Only the unified cgroup hierarchy has it.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_cpu_quota

required:    false
scopable:    true

Example:

pg_cpu_quota=50%@all

Description:

The percentage of cpu time the processes of the group may use.

Accepted expressions:

• 50% Half of one cpu.

• 50%@all Half of every cpu the node has.

• 10%@2 A tenth of two cpus.

Unlike pg_cpu_shares, which only arbitrates when the node is cpu-bound, a quota caps the group whether or not the node is busy.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_cpu_shares

required:    false
scopable:    true
convert:     size

Example:

pg_cpu_shares=512

Description:

The share of cpu the group gets when the node is cpu-bound.

The share is relative to the other groups.

The actual percentile depends on the shares allowed to the other services. Unset, the kernel default is used, which usually is 1024 shares.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_cpus

required:    false
scopable:    true
depends:     create_pg=true

Example:

pg_cpus=0-2

Description:

The cpus the processes of the group may run on.

Cpus are specified as list or range : 0,1,2 or 0-2.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_mem_high

required:    false
scopable:    true
convert:     size

Example:

pg_mem_high=384m

Description:

The memory past which the kernel throttles the group, in bytes.

Past it, the kernel slows the processes of the group down, reclaiming their memory and throttling their allocations.

Unlike pg_mem_limit, reaching it does not wake the Out-Of-Memory killer: a group above it keeps running, slower, while the kernel reclaims. Set below pg_mem_limit, it gives a process growing too much the time to be noticed, or to shrink, before it is killed.

It is a soft limit, so it is not what a memory claim counts: the claim counts pg_mem_limit. Only the unified cgroup hierarchy has it.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_mem_limit

required:    false
scopable:    true
convert:     size

Example:

pg_mem_limit=512m

Description:

The memory the processes of the group may use, in bytes.

The Out-Of-Memory killer is triggered in case of tresspassing.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_mem_oom_control

required:    false
scopable:    true

Example:

pg_mem_oom_control=1

Description:

Whether the Out of Memory killer runs for the group: 0 enables it.

It is a flag (0 or 1) enabling or disabling the Out of Memory killer for the processes of the group.

• If enabled (0), tasks that attempt to consume more memory than they are allowed are immediately killed by the OOM killer.
• If disabled (1), tasks are allowed to continue to try allocating memory, stressing the system.

The OOM killer is enabled by default in every cgroup using the memory controller.

This keyword caps nothing on a node holding the unified cgroup hierarchy, which has no memory.oom_control: that file is of the v1 hierarchy. Setting it there is warned about, and ignored.

Keyword pg_mem_swappiness

required:    false
scopable:    true

Example:

pg_mem_swappiness=40

Description:

Set a swappiness percentile value for the process group.

This keyword caps nothing on a node holding the unified cgroup hierarchy, which has no memory.swappiness: that file is of the v1 hierarchy. Setting it there is warned about, and ignored.

Keyword pg_mems

required:    false
scopable:    true

Example:

pg_mems=0-2

Description:

The memory nodes the processes of the group may allocate from.

Memory nodes are specified as list or range : 0,1,2 or 0-2.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_pids_max

required:    false
scopable:    true

Example:

pg_pids_max=512

Description:

The most processes and threads the group may run at once.

A fork beyond it fails in the process that asked for it, and nothing else of the node is touched: it is what keeps a runaway fork loop from exhausting the process table of the node.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword pg_vmem_limit

required:    false
scopable:    true
convert:     size

Example:

pg_vmem_limit=1g

Description:

The memory plus swap the processes of the group may use, in bytes.

The Out-Of-Memory killer is triggered in case of tresspassing. The specified value must be greater than pg_mem_limit.

Setting this keyword to default puts the capping back where a node that never capped anything leaves it. Removing the keyword does not: what was written stays written, whether om wrote it or something else did.

Keyword placement

required:    false
scopable:    false
candidates:  , last start, load avg, nodes order, none, score, shift, spread
default:     nodes order

Description:

Set a service instances placement policy:

• none

  No placement policy. a policy for dummy, observe-only, services.

• nodes order

  The left-most available node is allowed to start a service instance when necessary.

• last start

  The preferred instances is the one started last.

• load avg

  The least loaded node takes precedences.

• shift

  Shift the nodes order ranking by the service prefix converter to an integer.

• spread

  A spread policy tends to perfect leveling with many services.

• score

  The highest scoring node takes precedence (the score is a composite indice of load, mem and swap).

Keyword pre_monitor_action

required:    false
scopable:    true
rbac:        Requires the root grant.

Example:

pre_monitor_action=/bin/true

Description:

A callout to execute before the monitor_action.

For example, if monitor_action = freezestop, a pre_monitor_action script may decide to crash the server if it detects a situation were freezestop can not succeed (for example, a fs can not be umounted due to an unresponsive storage array).

Keyword priority

required:    false
scopable:    false
default:     50
convert:     int
rbac:        Requires the prioritizer grant.

Description:

When the daemon has so many actions to submit in parallel that the node.max_parallel limit is reached, this priority is used to determine which service are served first.

The priority is an just an number used as a sort key. The smaller the number, the higher the priority.

The priority setting is dropped from a service configuration injected via the api by a user not having the prioritizer grant.

Keyword provision

required:    false
scopable:    true
default:     true
convert:     bool

Description:

Set in the default section, provision=false cancels the instance provision action before looping over the resources. In this case, the action returns an error.

This acts like a glass the user has to break to access the provision button.

Keyword provision_timeout

required:    false
scopable:    true
convert:     duration

Example:

provision_timeout=1m30s

Description:

Wait for <duration> before declaring the action a failure.

Takes precedence over timeout.

Keyword restore_from

required:    false
scopable:    false
rbac:        Requires the root grant.

Example:

restore_from=store1 ns1/svc/svc1 7c0e41d2-9a1b-4c3d-8e5f-0a1b2c3d4e5f

Description:

The snapshot the heads of the object are restored from when they are provisioned: <store> <object path> <snapshot id>, the store by its name, the object the snapshot is of, and the id of the snapshot.

A provision restores the member of the snapshot named after a resource into the head of that resource when it moved the resource from unprovisioned to provisioned, after the heads are up and before the resources using their data start. A head provisioned already is left as it is, so a provision run again, or on a node added later, restores into the new heads only.

deploy --store <store> --snapshot <id> sets it, the configuration of the object being the one of the snapshot.

Keyword rollback

required:    false
scopable:    true
default:     true
convert:     bool

Description:

If set to false, the default rollback on start action error behaviour is disabled, leaving the instance in its half-started state (avail warn).

The daemon then refuses to failover a service if any instance is in warn availabity state. It is highly recommended to not use rollback=false if orchestrate=ha.

Keyword run_schedule

required:    false
scopable:    true

Description:

The instance tasks run action default schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword shared

required:    false
scopable:    true
default:     true
convert:     bool

Description:

If true, the resource will be considered shared during provision and unprovision actions.

A shared resource driver can implement a different behaviour depending on weither it is run from the leader instance, or not:

• When --leader is set, the driver creates and configures the system objects. For example the disk.disk driver allocates a SAN disk and discover its block devices.

• When --leader is not set, the driver does not redo the actions already done by the leader, but may do some. For example, the disk.disk driver skips the SAN disk allocation, but discovers the block devices.

The daemon takes care of setting the --leader flags on the commands it submits during deploy, purge, provision and unprovision orchestrations.

Warning: If admins want to submit --local provision or unprovision commands themselves, they have to set the --leader flag correctly.

Flex objects usually don’t use shared resources. But if they do, only the flex primary gets --leader commands.

Keyword soft_affinity

required:    false
scopable:    false
convert:     listlowercase

Example:

soft_affinity=svc1 svc2

Description:

A whitespace separated list of object paths, a name alone being a svc of the object namespace.

The daemon prefers the nodes where instances of these objects are up when it starts or switches the object, the more the better, the placement policy ordering the nodes it prefers the same. No node is excluded.

Keyword soft_anti_affinity

required:    false
scopable:    false
convert:     listlowercase

Example:

soft_anti_affinity=svc1 svc2

Description:

A whitespace separated list of object paths, a name alone being a svc of the object namespace.

The daemon prefers the nodes where no instance of these objects is up when it starts or switches the object, the placement policy ordering the nodes it prefers the same. No node is excluded.

Keyword start_timeout

required:    false
scopable:    true
convert:     duration

Example:

start_timeout=1m30s

Description:

Wait for <duration> before declaring the action a failure.

Takes precedence over timeout.

Keyword stat_timeout

required:    false
scopable:    true
convert:     duration

Description:

The fs resources status evaluation includes a stat syscall test. This keyword defines the maximum wait time for those stat calls to respond.

When expired, the resource status is degraded is to warn, which can trigger a monitor action (reboot or crash the node) if the resource is monitored.

Keyword status_schedule

required:    false
scopable:    true
default:     @10m

Description:

The instance status evaluation schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword status_timeout

required:    false
scopable:    true
default:     1m
convert:     duration

Example:

status_timeout=10s

Description:

The maximum duration of the instance status evaluation.

For example, the total start action duration is constrained by different timeouts:

• the start_timeout Limiting the start action duration.

• the stop_timeout Limiting the start rollback duration triggered by start errors.

• the status_timeout Limiting the post-start instance status evaluation duration.

Keyword stonith

required:    false
scopable:    false
depends:     topology=failover
default:     false
convert:     bool

Description:

Enable the stonith callout for this object.

Shoot The Other Node In The Head, aka fence, using a callout.

The callout is triggered after a quorum vote won, when the surviving node is about to start a local instance of a service that was known to be started on a unreachable peer node.

The callout is meant to prevent the peer from writing to shared disks, remote databases, and from responding to clients.

The callout itself is the command keyword of the peer stonith#<nodename> section of the node configuration.

Keyword stop_timeout

required:    false
scopable:    true
convert:     duration

Example:

stop_timeout=1m30s

Description:

Wait for <duration> before declaring the action a failure.

Takes precedence over timeout.

Keyword sync_schedule

required:    false
scopable:    true
default:     04:00-06:00

Description:

The instance sync default schedule.

See usr/share/doc/schedule for the schedule syntax.

Keyword sync_timeout

required:    false
scopable:    true
convert:     duration

Example:

sync_timeout=1m30s

Description:

Wait for <duration> before declaring the action a failure.

Takes precedence over timeout.

Keyword timeout

required:    false
scopable:    true
default:     1h
convert:     duration

Example:

timeout=2h

Description:

Wait for <duration> before declaring a state-changing action a failure.

A per-action <action>_timeout can override this value.

Keyword topology

required:    false
scopable:    false
aliases:     cluster_type
candidates:  failover, flex
default:     failover

Description:

• failover

  The service is allowed to be up on one node at a time.

• flex

  The service can be up on flex_target nodes, where flex_target must be in the [flex_min, flex_max] range.

Keyword type

required:    false
scopable:    false

Description:

The resource driver name.

Keyword unprovision

required:    false
scopable:    true
default:     true
convert:     bool

Description:

Set in the default section, unprovision=false cancels the instance provision action before looping over the resources. In this case, the action returns an error.

This acts like a glass the user has to break to access the unprovision button.

Keyword unprovision_timeout

required:    false
scopable:    true
convert:     duration

Example:

unprovision_timeout=1m30s

Description:

Wait for <duration> before declaring the action a failure.

Takes precedence over timeout.

Keyword wait_syncs_timeout

required:    false
scopable:    true
default:     10m
convert:     duration

Example:

wait_syncs_timeout=1h

Description:

How long a stop or a switch of the instance waits for the syncs running on it to end.

Stopped under a sync, the instance would hand the peers a copy the sync was writing, so the stop waits for the syncs, in the wait syncs state. Past this duration, the stop fails, and the instance keeps running.