[tech] Done! Re: Planned Upgrades to `molmol`

[Nick Bannon]

On Mon, Apr 24, 2023 at 08:40:12PM +0800, Lorenzo Iannuzzi wrote:
> As you may be aware with the ucc-announce email that just went out, Gary and
> I have planned to upgrade `molmol` with the new 4TB SSDs that we have
> purchased. To ensure that we are ahead of any foreseeable issues, I have
> added a new file to `/home/wheel/docs/` entitled
> `Plan2UpgradeMolmolMay6.org` detailing our plan. Could the people with zfs
> know-how look at the file just to make sure we aren't forgetting anything? I
> have also attached the file in this email so you can save a copy just in
> case things go wrong.

Nicely done! I tweaked a couple of items, seems like that's where we're at now.

We've always tried to use CNAMEs for major services, e.g. (`home`,
`away`, `services` instead of `motsugo`, `molmol`, `molmol`) respectively.
[MPT] and I noticed in the process that the Proxmox cluster was using the
explicit hostname for the NAS storage pool, fixed. There's a bit more
work we'd have to do for real high-availability-style server cutovers,
but just that is almost enough for low-effort "change the name and reboot"
cutovers, so that's what we did. As result, the actual outage times were
pretty short.

1. `zfs send` the entire space pool from `molmol` to `wobbegong`
  - this was regular in the leadup to the maintenance work:
    ```
    root at molmol:/sbin# crontab -l
    [...]
    00 2 * * * /root/zfs-send-script.sh
    
    root at molmol:~# cat /root/zfs-send-script.sh
    #!/bin/sh
    # Create a new snapshot
    zfs snapshot -r space@$(date +%Y%m%d)
    [...]
    # Send the snapshot to wobbegong
    zfs send -RvLci space@$(date -v -1d +%Y%m%d) space@$(date +%Y%m%d) | ssh -i /root/.ssh/zsendrecv root at wobbegong zfs receive -F -v space
    ```
2. change CNAMEs and set up NFS exports from `wobbegong`
  - https://docs.freebsd.org/en/books/handbook/network-servers/#network-nfs
3. move the important clients over to the new `/services` etc. for the duration
4. rebuild/reshape `molmol`
5. `zfs send` the entire space pool back to `molmol`
  - there is also a new scratch2 pool
6. change CNAMEs back to `molmol`
7. move all clients back to the current `services` `away`
   - e.g. on motsugo just now:
    ```
    motsugo# umount -f /services
    motsugo# umount -f /away
    motsugo# find /away/ucc/ -ls
      1334877      4 drwxr-xr-x   3 root     root         4096 Nov  4  2016 /away/ucc/
      1334879      4 drwxr-xr-x   3 root     root         4096 Nov  4  2016 /away/ucc/elliotnunn
      1334881      4 drwxr-xr-x   2 root     root         4096 Nov  4  2016 /away/ucc/elliotnunn/cooee
    motsugo# rm -r /away/ucc
    motsugo# mount -av
    /                        : ignored
    /boot                    : already mounted
    /var                     : already mounted
    /media/cdrom0            : ignored
    /home                    : already mounted
    /tmp                     : already mounted
    /scratch                 : already mounted 
    /webcamrestore           : already mounted 
    mount.nfs: timeout set for Mon May  8 12:10:24 2023
    mount.nfs: trying text-based options 'nfsvers=3,tcp,rsize=8192,wsize=8192,soft,addr=130.95.13.6'
    mount.nfs: prog 100003, trying vers=3, prot=6
    mount.nfs: trying 130.95.13.6 prog 100003 vers 3 prot TCP port 2049
    mount.nfs: prog 100005, trying vers=3, prot=6
    mount.nfs: trying 130.95.13.6 prog 100005 vers 3 prot TCP port 1009
    /away                    : successfully mounted
    [...]
    /services                : successfully mounted
    /mp3s                    : successfully mounted
    none                     : ignored
    ```

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Plan2UpgradeMolmolMay6.20230501.org
Type: application/vnd.lotus-organizer
Size: 966 bytes
Desc: not available
URL: <https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20230508/414e3736/attachment.org>