[tech] UCC WiFi Fixed

[dylanh333]

Hi All,

I have now successfully fixed the WiFi issues that UCC was having today (and potentially longer)!

Basically, it looks like our Unifi AP - smallwing - was in an intermediate state of trying to install a firmware update, giving up, rebooting, but not being fully provisioned again by the controller afterwards, presumably due to it pending the completion of the update process.
The update process itself was failing due to smallwing trying to directly download the updated by running `curl -s --retry 3 --retry-delay 3 -L -o /tmp/fwupdate.bin -w %{http_code} https://fw-download.ubnt.com/data/unifi-firmware/a120-U7NHD-4.0.66-5826dea6be734c72962d7f30c85ab4ea.bin`, but not being able to resolve "fw-download.ubnt.com" due to its DNS server being set to 192.168.2.1. Identifying both of these issues was done by actually SSHing into admin at smallwing using the credentials stored under Settings > Site > Device Authentication on the controller (https://salmon.ucc.asn.au:8443/), and not those stored in UCCPass (I'll update those accordingly). To see what curl was doing, I literally just ran `ps`, and it was at the bottom of the list.
The initial fix was to get the Unifi controller to cache the update binary (Settings > Maintenance > Firmware > Actions > Cache), and after the AP had finished chucking a tantrum and rebooted: re-deploying the firmware. This time around, when I SSHed into the AP to see what it was up to, it was runnig curl with the IP address of the controller instead of fw-download.ubnt.com (which it still could not resolve).
Once the firmware update completed successfully, the AP was properly provisioned by the controller this time and had all its config (this config gets downloaded from the controller on every boot instead of being stored in flash - to avoid wearing out the flash memory), and a few minutes later, I started seeing clients connect, so I knew that it was fixed!

With WiFi now working again, I decided to dig deeper into why the AP was getting the wrong DNS settings. Pretty much after going on a wild goose chase, I found it right there in the GUI under Devices > smallwing > Settings > Network > Preferred DNS, so I changed it from 192.168.2.1 to 130.95.13.9, and sure enough it started resolving DNS names. The bad news is: it appears to be firewalled off from the Internet, so the next time it needs a firmware update, it'll pretty much run into the same issues _unless_ we cache the firmware on the controller first.

TL;DR: Going forward, DON'T run a firmware update on smallwing (and any future APs we may acquire), UNLESS you cache that firmware version on the controller first by going to Settings > Maintenance > Firmware > Actions > Cache.

Cheers,
Dylan Hicks [333]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20191121/3942f25c/attachment.htm