One of my dedicated server hypervisors unexpectedly reset this morning, taking down all the VMs with it. I’ve been preparing for this moment since the beginning of 2020, when I decided to completely redesign my infrastructure according to my latest thinking about high-availability. I didn’t actually notice the reset immediately, because it didn’t cause an outage. Most services rerouted immediately onto other VMs on redundant hypervisors, database clusters stabilised and auto-promoted a new master as necessary, and dependent programs such as the Isoxya web crawler detected the dead database connections and relaunched themselves. For production stacks, this happened within 1 minute.
During this time, I was connected to a private VPN allowing access into restricted areas of the infrastructure. All I had to do was restart the connection and connectivity was restored, now routed through an entirely separate datacentre. In the past, I’d been using the commercial OpenVPN Access Server for the VPN. But its support for HA is limited; previously, this relied on LAN model UCARP-based failover, and now, with active-active clustering requiring multiple licences. Whilst I’d recommend it to anyone not wanting to go too deeply into running their own VPN server (I’ve used it for both client and personal projects in the past), I also found it rather expensive for my use-case. So, I decided to install an HA VPN of my own, using the community variant of OpenVPN 2.4.9, running on the latest CentOS 8.2.2004, via Ansible 2.9.10.
Installation
Packages
If you don’t already have it, install EPEL. This should come as a separate step, so the package cache can be refreshed prior to installing the main packages.
-
name: PACKAGE pre-install
package:
name:
- epel-release
Install the packages. easy-rsa is optional, but very helpful for running a certificate authority and for generating CSRs and signing them. It’s wise to keep the CA itself off-server, or, depending on your security requirements, potentially offline altogether.
-
name: PACKAGE install
package:
name:
- easy-rsa
- openvpn
SELinux
If you are running SELinux in enforcing mode on a security-hardened system, you might need to take additional steps, because for me, the packages didn’t work out-the-box when using PAM for clients without certificates. I’m frankly tired of seeing so many technical blogs say to simply disable or uninstall SELinux; instead, I compile a custom SELinux module, based on a minimal extension policy I developed by auditing the SELinux security logs. Your policy might look slightly different, especially on different OS versions (it might not be necessary for CentOS 7), and almost certainly on different OS distributions. I tend to put custom SELinux modules into a directory, and run a handler to compile and install them. Of course, if you’re not running SELinux, you don’t need this at all.
-
name: SELINUX modules
template:
src: "{{ item }}"
dest: /etc/selinux/{{ item | basename }}
mode: 0644
with_fileglob: "templates/selinux/*"
notify:
- 0 SELINUX compile modules
The custom SELinux policy, which I’ve called ` x_openvpn. I tend to namespace under x_` so my overrides are easier to remove later.
module x_openvpn 1.0;
require {
type chkpwd_t;
class capability dac_override;
}
#============= chkpwd_t ==============
allow chkpwd_t self:capability dac_override;
Next is the handler. I use a 0 prefix, because handlers are run in alphabetical order when there are multiple ones pending for that stage, and this ensures the module is compiled and installed prior to starting any services.
-
name: 0 SELINUX compile modules
command: /usr/local/sbin/selinux-compile-modules
The selinux-compile-modules script is installed everywhere in my infrastructure as part of my security module defined elsewhere.
#!/bin/bash -eu
set -o pipefail
modules=/etc/selinux
#-------------------------------------------------------------------------------
function selinux_compile() {
f=$1
f_=${f%.*}
echo "$f"
checkmodule -M -m "$f" -o "$f_.mod"
semodule_package -o "$f_.pp" -m "$f_.mod"
semodule -i "$f_.pp"
}
export -f selinux_compile
#-------------------------------------------------------------------------------
find "$modules" \
-name '*.te' \
-type f \
-exec bash -c 'selinux_compile "$1"' _ {} \;
Scripts
I install a number of optional scripts to assist with the operation of the VPN:
-
name: BIN copy
template:
src: "{{ item }}"
dest: /usr/local/sbin/{{ item | basename }}
mode: 0755
with_fileglob: "templates/bin/*"
The openvpn-init script copies the private key into the right location, fixes ownership, and relabels to apply the correct SELinux contexts. The security labelling of OpenVPN directories and files is strict in this regard, so I found this a straightforward compromise, since it only needs to be run once when setting up each server.
#!/bin/bash -eu
pki=/etc/openvpn/pki
server=/etc/openvpn/server
owner=root
group=openvpn
#-------------------------------------------------------------------------------
cp "$pki/private/{{ inventory_hostname_short }}.key" "$server/"
chown -R "$owner":"$group" "$server"
restorecon -Rv "$server"
The easyrsa_ script is a wrapper around easyrsa, which isn’t placed into the PATH by default anyway. This also creates the store, and sets the right paths, because I’m bound to forget this.
#!/bin/bash -eu
set -o pipefail
easyrsa=/usr/share/easy-rsa/3
pki=/etc/openvpn/pki
#-------------------------------------------------------------------------------
export EASYRSA_PKI=$pki
[[ -d "$pki" ]] || $easyrsa/easyrsa init-pki
$easyrsa/easyrsa "$@"
Lastly, the easyrsa-gen-req script assists my memory in generating certificate requests using the short hostname, with no password, since it will use the server authentication certificate type to identify as an authorised VPN server within the HA cluster.
#!/bin/sh -eu
#-------------------------------------------------------------------------------
easyrsa_ gen-req "$(hostname -s)" nopass
Part 2
That’s it for Part 1. In Part 2, we’ll configure the OpenVPN servers, and create an OVPN config file which can be used for the OpenVPN clients.