[tech] Tech/Wheel Meeting 2022-02-12 14:00 - 24 hour reminder

[root]

Tech/Wheel Meeting Agenda - Saturday 2022-02-12 14:00
=====================================================
- VENUE: UCC Clubroom
  - and online at https://meetings.ucc.asn.au/b/tech

*Meeting opened 14:MM*

## Attendance
- Present
- Apologies
- Absent


## Next meeting
- Schedule next meeting
  - *day 2022-MM-ddTHH:MM
- ACTION: [???] shall be this meeting's secretary! This entails:
  - Copying the following checklist into a new issue under [[https://gitlab.ucc.asn.au/UCC/tech-todo-list/-/issues]] (or clone #28) , and assigning it to yourself.
    - This is to keep track of any async secretarial duties detailed ahead. See our new Action Items section below.
  - [ ] ACTION: Save and commit the minutes of today's meeting, during the meeting; and at the end
  - [ ] Set and verify reminders of next meeting: `motsugo# crontab -e`
    - [ ] Promptly update agenda.next with the TIME/DATE/VENUE
    - [ ] Perform initial curation of agenda.next, and move any longstanding action items out of it and into GitLab (see Action Items section below).
    - [ ] Check at T-7days that the notice really went out, fix for T-4days if needed
- [ ] Everyone, before next meeting: Curate agenda.next, and move any items you think should be tracked as GitLab issues into GitLab issues, as above.


## Optional items - choose at the start of the meeting
- Ethical guidelines
- Monitoring
- Backups
  - New, locally staged, gigabit
  - to overlap with rdiff-backup/oldstable v1.2.x
    - completion of upgrades to stable
- Password rotations
- New members
- Quick check of ChangeLog
- Lessons learnt


## Current Action Items
### Boilerplate
- Now maintained in GitLab at [[https://gitlab.ucc.asn.au/UCC/tech-todo-list/-/issues/]]
- Briefly discuss anything in here that's worth discussing, but don't spend too long rehashing unresolved issues that have already been discussed ;)
- Going forward:
  - New actions: when new ACTION items arise, put them in the minutes once, but add to GitLab
  - Ongoing actions: don't keep them in the agenda, unless they definitely need to be discussed in the next meeting
  - Completed actions: mention in the agend that it's been completed, and and briefly discuss if need be

### Action items to discuss
- Issue #30: "Investigate ACPI syslog spam on Mudkip and Magikarp"
  - Investigate ACPI syslog spam on Mudkip and Magikarp, which are repeatedly logging errors like the below, which may be causing extra wear on their OS disks (SD cards) than would normally be expected.
  - We could also sidestep this issue (or complement the resolution thereof) by making /var/log a tmpfs mount, and adjusting our central logging so that nothing important is lost upon reboot (suggested by [BOB]).
  - Example errors:
    - "Feb 01 16:34:37 magikarp kernel: ACPI Error: No handler for Region [POWR] (000000002f01d0b2) [IPMI] (20190816/evregion-134)"
    - "Feb 01 16:34:37 magikarp kernel: ACPI Error: Region IPMI (ID=7) has no handler (20190816/exfldio-265)"
- Issue #31: "Resolve Ceph HEALTH_WARN \"mons are allowing insecure global_id reclaim\""
  - ceph status on all our VM hosts is reporting the warning that "mons are allowing insecure global_id reclaim".
  - [MPT] reports that this relates to [[https://docs.ceph.com/en/latest/security/CVE-2021-20288]], and that "all our servers are running 14.2.22 (with the fixed global_id behaviour) so it should be safe to roll out the config fix".

## Known Broken Stuff


## Matters arising previously
- Issue #29: "Mudkip is down"
  - Fan 1 and 7 "failed", with 7 being a consistent offender, and 1 being intermittent
  - Basically the HP servers will force a shutdown if more than 1 fan is degraded/failed, even if the temperatures and system health is otherwise okay
  - A bit heavy handed, but ¯\_(ツ)_/¯
  - Anyway, we had 4 spare fans in the server room, and on 10/01/2022,[MTL] replaced the faulty ones with two of these
  - Failty fans 1 and 7 have been labelled "degraded" and "FITH", respectively
  - Thanks Mark!

## Extra items (rename/refile as appropriate)

*Meeting closed HH:MM*

----

```
# https://demo.hedgedoc.org/Hlsapf47RsqpgIjqLVfMUw
cd /home/wheel/docs/meetings
HEDGEDOC_SERVER=https://demo.hedgedoc.org /home/wheel/bin/hedgedoc export --md Hlsapf47RsqpgIjqLVfMUw ./$(date +%Y-%m-%d).txt
git commit -am "Tech meeting minutes $(date +%Y-%m-%d)"
```

<!-- vim: tabstop=2 softtabstop=2 shiftwidth=2 expandtab
-->
<!-- Local Variables: -->
<!-- tab-width: 2 -->
<!-- End: -->