[tech] IPSec VPN working again and hostname change

[Andrew Adamson]

Hi All,

This email is long and detailed because there have been lots of changes to 
our processes over the last year and we have a bunch of new members. Read 
on if you want lots of detail, otherwise know that the ipsec vpn has now 
moved to ipsec.ucc.asn.au and see https://wiki.ucc.asn.au/VPN for config 
details.

Last project night [NTU], [TPG] and I spent some time fixing the IPSec 
VPN, which basically hasn't been working since UWA CloudFu^?lared 
everything. The new cloudflarification broke LetsEncrypt DNS validation 
from working and I think also broke secure.ucc.asn.au in general.

For those who don't know, we used to overload the secure.ucc.asn.au 
hostname with pretty much every secure service that the club wanted to 
use, even with those services across a bunch of machines. This meant that 
we could pay for a single signed certificate and use it everywhere - keep 
in mind this was set up in the days before letsencrypt. So the IPSec VPN 
was still using secure.ucc even after we moved to letsencypt. Since this 
is no longer necessary and we have a bunch of mixed legacy stuff floating 
around in config files, I elected to move it to ipsec.ucc.asn.au. Down the 
track I expect we'll end up with wireguard.ucc and 
openvpn.ucc so I didn't want to use vpn.ucc. 

First thing we did was to make ipsec.ucc.asn.au point to murasoi (the 
router) by creating a new entry in /etc/bind/domains/primary/ucc.machines 
on mooneye (the DNS server). Normally this would be a CNAME, but we ended 
up having to do it as an A record so we could explicitly ensure that 
ipsec.ucc doesn't have an IPv6 record. With a CNAME it would have just 
pointed to murasois record which has IPv6. 

For those new to this, UCC has an in-house script that automatically 
generates DNS zone files for us and eliminates a bunch of sources of 
errors. So after editing ucc.machines we cd to /etc/bind/domains/primary 
and execute the zonemake.py script. This script outputs some instructions 
at the end to reload the DNS server, reload the web server (to account for 
new user accounts), and notably SYNC THE DNS CHANGES TO CLOUDFLARE using 
octodns. Having run those three commands, we could then ping 
ipsec.ucc.asn.au from external and get a response from 130.95.13.1 - 
hurrah!

Next thing we had to do was to reconfigure strongswan (that's the VPN 
server) on murasoi to make sure it was listening to ipsec.ucc.asn.au and 
responding with the correct certificates. We luckily knew that the VPN was 
working properly before the certificates broke, so we didn't have to do 
most of the usual VPN setup this time around. Strongswan has lots of 
moving parts and a bunch of config files in assorted directories so it 
took us a while to find the right config file. For future reference, the 
one with the main config stuff for ucc is in /etc/ipsec.conf. The only 
things we had to do here is change leftid from secure.ucc to ipsec.ucc, 
and leftcert to reflect the (not yet created) ipsec.ucc letsencrypt  
certificate. After changing the config file we ran `service ipsec restart' 
to make it pick up the new config, but of course that failed because we 
hadn't created the certificate yet. 

Creating the certificate is *almost* as easy as using standard letsencrypt 
certbot. The difference is that we have an alternate external dns service 
that allows certbot on any machine to use dns verification. Step one was 
to add a dns entry for the acme challenge that points to the external dns 
service, so we added the following lines to ucc.machines along the same 
lines as existing entries. Note the "root-" part of the CNAME is the 
username of the API key we're using.

[_acme-challenge.ipsec]
CNAME: root-ipsec.quovadis-challenges.ucc.asn.au.
zones: ucc asn

Again we ran zonemake and the subsequent commands to sync everything. Note 
that the above will not respond to pings for testing. 

For certbot to work we have an internal machine called quovadis, which has 
an API that certbot needs to update each time certbot runs. This was a 
simple matter of getting the hook and config files from gitlab as noted on 
https://quovadis.ucc.asn.au/quovadis/ and following the instructions on 
that same page. We placed the script and config file in 
/etc/letsencrypt/ucc-hooks/ , and grabbed the API credentials off another 
machine (since this is a UCC machine we use root, but other people can 
create an API key using their UCC username for their own use). Then we ran 
the certbot cert creation command with the `--manual-auth-hook 
/etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh' option (again 
following the instructions on the quovadis.ucc page). I note that the 
first time we ran this command it failed, but it subsequently ran fine; 
perhaps we hit some sort of timeout on the very first run?

Anyway, having successfully run certbot we now had our certificate set in 
/etc/letsencrypt/live/ipsec.ucc.asn.au/ and all the permissions were 
correct. We started strongswan again, but monitoring the log data in 
/var/log/syslog was showing that it still wasn't finding the certificates. 
Some googling later, we found that apparmor was preventing charon (a 
component of strongswan) from reading the cert. With apparmor disabled for 
charon and stroke per 
https://molchanoff.me/administration/ikev2-vpn-server-with-strongswan-and-lets-encrypt/
 we were finally able to start the daemon without error. 

Alas, we still couldn't connect. The final piece to the puzzle was 
identified by [TPG], who noted that the full certificate chain wasn't 
being dished up to clients. This was fixed with a symlink from 
/etc/ipsec.d/cacerts/ipsec.ucc.asn.au_chain.pem to 
/etc/letsencrypt/live/ipsec.ucc.asn.au/chain.pem and suddenly we were off 
and racing. 

Finally we cleaned up the old certbot jobs for secure.ucc.asn.au with 
`certbot delete' and any other references to secure.ucc that we could 
find. 

Poke my via email or irc if you have any questions. Enjoy!

Andrew Adamson
bob at ucc.asn.au

|"If you can't beat them, join them, and then beat them."                |
| ---Peter's Laws                                                        |