[tech] UCC backup status; mollitz, molmol, motsugo

[Nick Bannon]

On Mon, Aug 10, 2020 at 08:11:00PM +0800, Nick Bannon wrote:
> On Mon, May 11, 2020 at 10:25:34PM +0800, Nick Bannon wrote:
> >   * mollitz, an 8GiB DELL PowerEdge 2950, boots off a 60GB OCZ-VERTEX2 SSD
> ...or maybe it doesn't anymore.
> Looks like it had an outage on 2020-07-20 [...]

It's making progress, but mollitz (legacy backups) is still on the
workbench and not back in its usual comfy location.

It now has 24GiB of ECC RAM and a fresh install on a 120GB GIGABYTE
GP-GSTF QLC SSD bootdrive. It runs its cronjobs but it's missing:

* Prometheus metrics for uccmonitor
  * hopefully just a https://gitlab.ucc.asn.au/ucc-systems/ansiblemonitoring away
* a proper packaged install of its old tools like megaclisas-status .
(assistance welcome)

rdiff-backup limps on: I think it doesn't handle huge files very well
(and there are a few multi-gigabyte files stashed away in motsugo:/home
and molmol:/away that probably need cleaning up) and it certainly doesn't
handle files that change while the backup is running, so would imply
that the ideal config would be to run it from a filesystem snapshot,
not live systems... which we're not currently doing.

That might be why molmol OOM's sometimes and kills random processes. It
needs monitoring too, but if rdiff-backup is the trigger, maybe all it
needs is some extra swap space to fix temporary OOMs.

There's some weekly stats being collected by stats-log.sh each Sunday in:
* mollitz:/backups/log
* motsugo:/home/du
* molmol:/space/away/du
...which we can use to target cleanups. (assistance welcome)

I had hoped ( <20201126120002.4B5DC200CE at motsugo.ucc.gu.uwa.edu.au> ) to
look at neo-mollitz backups drives and a replacement for molmol's dead
SLOG SSD for Black Friday/Cyber Monday. It's nearly over and I haven't
pulled the trigger on an order yet, but with the exchange rate still up
close to $1.00AUD = $0.74USD we can get some excellent value in any case.

molmol probably had a pair of Samsung SSD 860 EVO 250GB's. I think
it's best to get a new drive for the Ceph cluster and hand molmol down
an old EVO 500GB.

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal