Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Ingress Gateway

Services obtaining their IP address from a backend network are not reachable from outside the cluster. An ingress gateway is a service with a public address that routes the requests of the outside world to them.

HAProxy is our recommended program for it, routing layer 4 and layer 7 traffic. This page deploys one, with certificates the agent renews.

How the gateway finds the backends

The gateway names the backend servers by their cluster dns names, <name>.<namespace>.svc.<clustername>, and HAProxy resolves them again as they move, with its resolvers mechanism:

resolvers clusterdns
    parse-resolv-conf
    accepted_payload_size 8192

HAProxy runs in a container started by the agent, whose /etc/resolv.conf names the cluster nameservers, and parse-resolv-conf reads them from there. A backend then refers to these resolvers:

backend svc1
    server-template svc1_ 1 svc1.ns1.svc.${CLUSTERNAME}:80 resolvers clusterdns check init-addr none

init-addr none lets HAProxy start while a backend is not resolvable yet, its service being down or not deployed.

Topologies

  • One gateway for the cluster: a single HAProxy, in a failover service. The public address moves with it. This is the deployment below.
  • One gateway per node: every node runs a HAProxy exposing the same servers, in a topology = flex service, and an upstream load balancer picks one.

Deploy

The gateway is three objects of a namespace, here igw:

  • igw/sec/haproxy, the certificate of the https frontend
  • igw/cfg/haproxy, the haproxy configuration
  • igw/svc/haproxy, the service running haproxy, which installs the other two in a volume

The certificate

A self-signed certificate is enough to start with. The Certificates section below replaces it with one the browsers trust.

om igw/sec/haproxy create --kw cn=svc1.example.com
om igw/sec/haproxy certificate create

The haproxy configuration

global
    tune.lua.bool-sample-conversion normal
    lua-load /usr/local/etc/haproxy/acme-http01-webroot.lua
    # a reload waits no longer than this for the open connections to end
    hard-stop-after 30s

resolvers clusterdns
    parse-resolv-conf
    accepted_payload_size 8192

defaults
    mode http
    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend http
    bind :80
    acl acme path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET acme
    redirect scheme https code 301 if !acme

frontend https
    bind :443 ssl crt /usr/local/etc/haproxy/certs/
    use_backend svc1 if { hdr(host) -i svc1.example.com }

backend svc1
    server-template svc1_ 1 svc1.${NAMESPACE}.svc.${CLUSTERNAME}:80 resolvers clusterdns check init-addr none
  • The http frontend redirects to https, except the ACME challenge path, which a small lua responder serves from the webroot the certificate renewal writes in.
  • The https frontend loads every certificate of the certs/ directory, and presents the one whose names match the server name the client asks for. Serving another domain is installing one more certificate there.
  • hard-stop-after bounds a reload, as the one following a renewed certificate: the old process waits for the open connections to end, and a websocket can stay open for hours.

Save it as haproxy.cfg, and store it in the cfg, with the lua responder:

om igw/cfg/haproxy create
om igw/cfg/haproxy key add --name haproxy.cfg --from haproxy.cfg
om igw/cfg/haproxy key add --name acme-http01-webroot.lua \
  --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/igw_haproxy/common/acme-http01-webroot.lua

The service

[DEFAULT]
nodes = *
orchestrate = ha

[volume#1]
type = shm
name = {name}
size = 1m
user = 99
group = 99
install = `
	/haproxy/haproxy.cfg from ./cfg/{name} key haproxy.cfg signal HUP:container#1
	/haproxy/acme-http01-webroot.lua from ./cfg/{name} key acme-http01-webroot.lua signal HUP:container#1
	/haproxy/certs/{name}.pem from ./sec/{name} key fullpem signal HUP:container#1
	/haproxy/acme-challenges/
`

[ip#1]
type = netns
network = default
netns = container#0
shared = true

[container#0]
image = ghcr.io/opensvc/pause
rm = true
shared = true

[container#1]
image = docker.io/haproxy:3.2-alpine
rm = true
environment = CLUSTERNAME={clustername} NAMESPACE={namespace}
volume_mounts = `
	volume#1:/haproxy:/usr/local/etc/haproxy
	volume#1:/haproxy/acme-challenges:/acme-challenges
`
netns = container#0
restart = 3
shared = true

[task#acme]
type = acme
secs = {name}
webroot = volume#1:/haproxy/acme-challenges
schedule = @1d
run_requires = container#1(up)
  • volume#1 is a small memory-backed volume holding the files haproxy reads, installed from the cfg and the sec. Nothing in it needs keeping or replicating: it is installed anew on every start, so the service fails over without any storage.
  • The signal of the install lines makes haproxy reload when a key changes, on every node running it: a configuration edited, or a certificate renewed, takes effect without a restart.
  • user and group are the ones haproxy runs as in its image. A sec key installs readable by its owner only.
  • ip#1 takes an address from a cluster network, in the network namespace of the container#0 pause container, which the haproxy container joins.
  • volume_mounts names the paths of the volume by the rid of its resource. The lua responder reads the challenge tokens from /acme-challenges.
  • task#acme renews the certificates, see Certificates.

Save it as haproxy.conf, and deploy:

# Use a network spanning the cluster nodes, if you have one, and make sure
# the cluster nameservers answer requests from it.
om igw/svc/haproxy deploy --config haproxy.conf --kw ip#1.network=default --wait

Test

Deploy a web server as the svc1 backend, saved as svc1.conf:

[DEFAULT]
nodes = *
orchestrate = ha

[ip#1]
type = netns
network = default
netns = container#0

[container#0]
image = ghcr.io/opensvc/pause
rm = true

[container#1]
image = docker.io/nginx:alpine
rm = true
netns = container#0

om igw/svc/svc1 deploy --config svc1.conf --kw ip#1.network=default --wait

The gateway answers on its backend address already, from a cluster node:

IP=$(om igw/svc/haproxy resource ls ip#1 --node $HOSTNAME -o json | jq -r '.items[].data.status.info.ipaddr')

# fake the dns resolution of svc1.example.com to the gateway address
curl -k --resolve svc1.example.com:443:$IP https://svc1.example.com

Public address

For the clients outside the cluster, add a public address to the service, in the network namespace of container#0, so it moves with the gateway:

[ip#2]
type = netns
name = 203.0.113.10
netmask = 24
gateway = 203.0.113.1
dev = br-prd
mode = bridge
netns = container#0
shared = true

dev is the bridge of the nodes on the public network. The dns records of the domains served point to name. Cluster Networks describes the other modes.

Certificates

The https frontend presents the certificate of igw/sec/haproxy, self-signed so far. Once the domain resolves to the public address, obtain it from Let’s Encrypt instead, trying the setup out against its staging directory first:

om igw/sec/haproxy set --kw acme.directory=letsencrypt-staging
om igw/svc/haproxy instance run acme --node <node running the gateway> --follow

The run proves the domain through the http frontend, stores the certificate issued in the sec, and installs it, haproxy reloading. Once the staging certificate is in place, switch to the directory browsers trust, and run again:

om igw/sec/haproxy set --kw acme.directory=letsencrypt
om igw/svc/haproxy instance run acme --node <node running the gateway> --follow

From then on, the daily task#acme renews the certificate within 30 days of its expiry.

To serve another domain, create a sec for it, install its fullpem in certs/, add it to the task, and run the task:

om igw/sec/www create --kw cn=www.example.com --kw acme.directory=letsencrypt
om igw/svc/haproxy set \
  --kw volume#1.install+="/haproxy/certs/www.pem from ./sec/www key fullpem signal HUP:container#1" \
  --kw task#acme.secs+=www
om igw/svc/haproxy instance run acme --node <node running the gateway> --follow

INF igw/sec/www: set key fullpem
INF igw/svc/haproxy: volume#1: install key fullpem from igw/sec/www to /srv/haproxy.igw.vol.cluster1/haproxy/certs/www.pem with owner 99:99 perm -rw-------
INF igw/svc/haproxy: container#1: send SIGHUP signal to container igw..haproxy.container.1 (pid 3869733)
INF igw/svc/haproxy: task#acme: igw/sec/www: certificate issued for www.example.com: no certificate

A namespace administrator may configure every piece of this: the task writes only in the volume of its service, and renews only the secs the namespace may use. Certificates has the details: when a renewal is due, the secs of other namespaces, and the renewal from the command line.