[tech] Tech/Wheel Meeting 2021-01-10 14:00 - One hour reminder

[root]

Wheel Meeting Agenda - Sunday 2021-01-10 14:00
==============================================
	- VENUE: UCC Clubroom
	  - and online at https://meetings.ucc.asn.au/b/bob-yrk-uy6 ?

*Meeting opened xx:xx*

## Attendance
- Present
- Apologies
- Absent

## Next meeting
- Schedule next meeting
  - what's happening before O-Day 2021-02-19 ?
  - ACTION: xxx, who hasn't tried it recently?:
    - Update the agenda, update the crontab, check at T-7days that the notice really went out
    - Set and verify reminders of next meeting: `motsugo# crontab -e`
      - skip the `4day` , unless there's issues at `1week` ?
- Curate agenda.next

## Standing items (brief)

### Visibly reinduct members new (and old?) with the "Wheel Group Ethical Guidelines"
  - examining an Ethical Guideline, e.g. asking:
    - What's an example situation in which it could be encountered?
    - What other guidelines or rules could it conflict with? How would
      one resolve it?

### Status check: Regular updates, monitoring
  - atlantic cluster
    - should we `apt upgrade` ? do some reboots?
  - e.g. Debian oldstable 9 "stretch" -> Debian stable 10 "buster"
    - find candidates on ocsinventory
      - has it stopped reporting versions in Debian 10?
        https://ocsinventory.ucc.asn.au/ocsreports/index.php?function=visu_search&fields=HARDWARE-LASTCOME&comp=tall&values=07/07/2020%2007:19&values2=&type_field=
  - molmol
    - Dead SSD? at Mon  9 Nov 08:00:10 AWST 2020
      ```
      molmol: /space/scratch/nick>zpool status|grep -C4 DEGRADED
      logs
        mirror-4                       DEGRADED     0     0     0
          ada0p3                       ONLINE       0     0     0
          3087349144323640050          UNAVAIL      0     0     0  was /dev/gpt/molmol-slog0
      ```
    - Upgrade and performance analysis:
      1. monitoring: add prometheus metric export
      2. iozone performance-and-latency-under-load benchmark
      3. enable metaslab debugging mode: https://serverfault.com/questions/511154/zfs-performance-do-i-need-to-keep-free-space-in-a-pool-or-a-file-system
      4. iozone performance-and-latency-under-load benchmark
      5. OS upgrade
      6. iozone performance-and-latency-under-load benchmark

### Status check: Backups

### Status check: Password/Key rotations
  - https://en.wikipedia.org/wiki/Pro_re_nata

## ..._then_ New wheel members, additions, nominations
- Welcome to wheel!
  - Read /home/wheel/docs/WelcomeToWheel
- winadmin, sprocket
  - [BRD]@2020-08-13 `uid=12426(bird) gid=10021(gumby) groups=10021(gumby),10069(committee),12203(door),666(winadmin),777(sprocket)`
  - `uid=12469(hilmi) gid=10021(gumby) groups=10021(gumby)`

## New Matters
- [NTU] `Dec  8 05:41:46 motsugo kernel: [10920663.319201] Oops: 0000 [#1] SMP PTI`
  - thanks for the remote power cycle, [TPG]
  - `murasoi:/var/log/ucc/messages` is pretty noisy!
    - can we fix some of the root causes of the noise?
    - kerosene's timezone is UTC, probably should be local?
- [TRS]@2020-11-03: SOGo ( https://sogo.nu/ ) has been down for a while too
  - [NTU] molmol had stopped responding - out of memory and the wrong thing got killed?
    - remote power cycle of molmol
    - OOM possibly triggered by rdiff-backup on huge files?
      - ACTION: [???] clean up the huge files, see `mollitz:/backups/log`
    - is SOGo working again?
      - ACTION: [???] can we add a prometheus+grafana health check for SOGo?
- [NTU] Live demo: ceph magikarp:osd.3 is smaller than it should be, let's fix it

## Matters arising previously

- ACTION: xxx, who hasn't tried it recently?: Set and verify reminders of next meeting: `motsugo# crontab -e`
- ACTION: DONE? [MTL]+[MPT] poking zonemake.py and its API-driven replacements and children
  - ACTION: DONE? [MPT] cf_tools / zonemake.py / octodns: generate API tokens for uccpass
  - https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-December/005411.html
  - [NTU] How can internal-only proxmox cluster VM hosts automate new letsencrypt certs?
- ACTION: [MTL] to look at UCC web reverse proxies
- ACTION: [MPT] UWA IT liason: matrix test domain
- ACTION: [MPT] update https://wiki.ucc.asn.au/Network with latest traffic paths
- ACTION: [TEC] to look at dashboards for murasoi network traffic
  - [NTU] can it capture when bulk TCP resets are sent by upstream connection-tracking routers?
    - graph the age of existing non-LAN connections, look for spikes to zero?
    - graph new connections, look for spikes?

*Meeting closed xx:xx*

----

```
# https://demo.hedgedoc.org/Hlsapf47RsqpgIjqLVfMUw
cd /home/wheel/docs/meetings
CODIMD_SERVER=https://demo.hedgedoc.org codimd export --md Hlsapf47RsqpgIjqLVfMUw ./$(date +%Y-%m-%d).txt
git commit -a "minutes"
```

# vim: tabstop=4 shiftwidth=4 expandtab