[tech] IPSec VPN working again and hostname change
bob at ucc.gu.uwa.edu.au
Thu Apr 22 12:52:13 AWST 2021
This email is long and detailed because there have been lots of changes to
our processes over the last year and we have a bunch of new members. Read
on if you want lots of detail, otherwise know that the ipsec vpn has now
moved to ipsec.ucc.asn.au and see https://wiki.ucc.asn.au/VPN for config
Last project night [NTU], [TPG] and I spent some time fixing the IPSec
VPN, which basically hasn't been working since UWA CloudFu^?lared
everything. The new cloudflarification broke LetsEncrypt DNS validation
from working and I think also broke secure.ucc.asn.au in general.
For those who don't know, we used to overload the secure.ucc.asn.au
hostname with pretty much every secure service that the club wanted to
use, even with those services across a bunch of machines. This meant that
we could pay for a single signed certificate and use it everywhere - keep
in mind this was set up in the days before letsencrypt. So the IPSec VPN
was still using secure.ucc even after we moved to letsencypt. Since this
is no longer necessary and we have a bunch of mixed legacy stuff floating
around in config files, I elected to move it to ipsec.ucc.asn.au. Down the
track I expect we'll end up with wireguard.ucc and
openvpn.ucc so I didn't want to use vpn.ucc.
First thing we did was to make ipsec.ucc.asn.au point to murasoi (the
router) by creating a new entry in /etc/bind/domains/primary/ucc.machines
on mooneye (the DNS server). Normally this would be a CNAME, but we ended
up having to do it as an A record so we could explicitly ensure that
ipsec.ucc doesn't have an IPv6 record. With a CNAME it would have just
pointed to murasois record which has IPv6.
For those new to this, UCC has an in-house script that automatically
generates DNS zone files for us and eliminates a bunch of sources of
errors. So after editing ucc.machines we cd to /etc/bind/domains/primary
and execute the zonemake.py script. This script outputs some instructions
at the end to reload the DNS server, reload the web server (to account for
new user accounts), and notably SYNC THE DNS CHANGES TO CLOUDFLARE using
octodns. Having run those three commands, we could then ping
ipsec.ucc.asn.au from external and get a response from 22.214.171.124 -
Next thing we had to do was to reconfigure strongswan (that's the VPN
server) on murasoi to make sure it was listening to ipsec.ucc.asn.au and
responding with the correct certificates. We luckily knew that the VPN was
working properly before the certificates broke, so we didn't have to do
most of the usual VPN setup this time around. Strongswan has lots of
moving parts and a bunch of config files in assorted directories so it
took us a while to find the right config file. For future reference, the
one with the main config stuff for ucc is in /etc/ipsec.conf. The only
things we had to do here is change leftid from secure.ucc to ipsec.ucc,
and leftcert to reflect the (not yet created) ipsec.ucc letsencrypt
certificate. After changing the config file we ran `service ipsec restart'
to make it pick up the new config, but of course that failed because we
hadn't created the certificate yet.
Creating the certificate is *almost* as easy as using standard letsencrypt
certbot. The difference is that we have an alternate external dns service
that allows certbot on any machine to use dns verification. Step one was
to add a dns entry for the acme challenge that points to the external dns
service, so we added the following lines to ucc.machines along the same
lines as existing entries. Note the "root-" part of the CNAME is the
username of the API key we're using.
zones: ucc asn
Again we ran zonemake and the subsequent commands to sync everything. Note
that the above will not respond to pings for testing.
For certbot to work we have an internal machine called quovadis, which has
an API that certbot needs to update each time certbot runs. This was a
simple matter of getting the hook and config files from gitlab as noted on
https://quovadis.ucc.asn.au/quovadis/ and following the instructions on
that same page. We placed the script and config file in
/etc/letsencrypt/ucc-hooks/ , and grabbed the API credentials off another
machine (since this is a UCC machine we use root, but other people can
create an API key using their UCC username for their own use). Then we ran
the certbot cert creation command with the `--manual-auth-hook
/etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh' option (again
following the instructions on the quovadis.ucc page). I note that the
first time we ran this command it failed, but it subsequently ran fine;
perhaps we hit some sort of timeout on the very first run?
Anyway, having successfully run certbot we now had our certificate set in
/etc/letsencrypt/live/ipsec.ucc.asn.au/ and all the permissions were
correct. We started strongswan again, but monitoring the log data in
/var/log/syslog was showing that it still wasn't finding the certificates.
Some googling later, we found that apparmor was preventing charon (a
component of strongswan) from reading the cert. With apparmor disabled for
charon and stroke per
we were finally able to start the daemon without error.
Alas, we still couldn't connect. The final piece to the puzzle was
identified by [TPG], who noted that the full certificate chain wasn't
being dished up to clients. This was fixed with a symlink from
/etc/letsencrypt/live/ipsec.ucc.asn.au/chain.pem and suddenly we were off
Finally we cleaned up the old certbot jobs for secure.ucc.asn.au with
`certbot delete' and any other references to secure.ucc that we could
Poke my via email or irc if you have any questions. Enjoy!
bob at ucc.asn.au
|"If you can't beat them, join them, and then beat them." |
| ---Peter's Laws |
More information about the tech