[tech] Target dates, was Re: molmol SSD storage upgrades

[Nick Bannon]

On Tue, Mar 07, 2023 at 12:24:10AM +0800, Nick Bannon wrote:
> [molmol is full]
> So! That budget is in the right ballpark for upgrading ourselves towards
> all-SSD active storage, but there are some hiccups that we discussed at
> the last tech meeting: /home/wheel/docs/meetings/2023-02-21.txt
> 
> - The first two 4TB drives have arrived: that will let us fix the urgent
>   `molmol` space problem with even less disruption than we were planning,
>   but see also below

We need to keep this moving. [GPO], [MPT] and I had a bit of a huddle
about it last night and as [GPO] pointed out, we have to think about the how
long it's going to take to be done, i.e. dates.

- Urgent `molmol` space problem
  - Target date: one week?
  - we've got drives that can do it, but we're just deciding now if we
    want to hang onto them given the megabuy confusion:
  - The megabuy.com.au non-Pro Ironwolf 125 link, with the Pro picture,
    says "Est. Delivery: 1-3 days", implying some stock...
  - The megabuy.com.au Ironwolf Pro 125 link, with the non-Pro picture,
    says "Est. Delivery: Contact Us", implying no stock?
  - megabuy has been contacted but we still need to decide what we want
    - based on what they say or later plans - not much later
  - maybe non-Pro drives are fine, given a good ZIL/SLOG/WAL

- next: extra SSD space for `molmol`
  - Target date: solidly underway, making orders, within one week of
    the urgent item? but willing to wait for stock/shipping
  - new, Australian, with warranty, so we can do returns if we like?
    - pricey, no spares?, low stock?
    - if the Ironwolf 125 is OK? maybe mirror with Crucial MX500, 4TB/$365?
  - or secondhand/self-warranted, with spares, locally or online (see last email)
    - enough space to be comfortable in the current budget?
    - more waiting for more packages
    - enterprise 3.84TB/7.64TB with PLDP? server pulls/grey market
    - look for some bargain 4TB/8TB without PLDP?

- good ZIL/SLOG/WAL for `molmol`?
  - Target date: months is OK
    - or never, put into the next server rebuild
    - but... has same backup/restore interruption of service which could
      stop it from happening
  - I think we can more directly measure the difference between garbage
    tier drives (like WD Green) and good ones, for synchronous write IOPS:
    - since FreeBSD 12.x, `diskinfo(8)` has a `diskinfo -S`/slogbench
    - like `time fio --direct=1 --sync=1 --rw=write --bs=4k`
    - or `dd if=XX of=/dev/sdYY bs=4k count=10000 oflag=direct,dsync`
    - Re: https://ceph.io/en/news/blog/2014/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
  - find good boot/cache drives
    - maybe SATA is OK?
    - maybe USD$50-$100 m.2 Optane H20's are good? fast cache, on some bulk QLC
    - maybe cheap USD$5-$10 little m.2 Optane M10's?
      - https://www.ixsystems.com/blog/why-zil-size-matters-or-doesnt/
        - via https://libreddit.librenesia.com/r/zfs/comments/bmahlj/optane_as_zilslog_only_used_less_than_1_gb_is

> - USD$78.11 - https://www.ebay.com/itm/204226168575
>   - Dual NVMe PCIe Adapter,M.2 NVMe SSD to PCI-E 3.1 X8/X16 Card Support
>     M.2 (ASMedia ASMedia ASM 2812 chipset, which does not depend on the
>     PCIe fork/bifurcation supported by motherboard)
>   - i.e. with a PCIe switch. 4 ports would be nicer?
> - 2+ units of USD$6.64 + free shipping?
>   - Intel Optane Memory M10 MEMPEK1J016GAH 16GB NVMe PCIe M.2 2280 HP P/N L08717-001

- good ZIL/SLOG/WAL for other legacy servers, even `wobbegong`?
  - could speed up backup/restore interruption of service? or make poor performance adequate
  - network as well? Synology's https://staticice.com.au/cgi-bin/search.cgi?q=E10M20-T1
    in `wobbegong`? AUD$449 new or a little cheaper on ebay
    - 10GBASE-T and a couple of NVMe M.2 slots
    - other servers' networks deserve a SFP+
      - AUD$123.25 https://www.amazon.com.au/E10G41BTDAG1P5-Ethernet-Converged-X520-DA1-X520-SR1/dp/B01LZRSQM9
      - plus AUD$34.15 https://www.amazon.com.au/EDIMAX-Backward-Compatible-Oxygen-Free-EA1-010D/dp/B0B9JB884J
      - copper 10G 802.3an and multigig 802.3bz could be useful in places,
        but has latency and some power/heat?
        - Wirenest single port multi-gig cards for USD$99 https://www.ebay.com/itm/364167700081
        - maybe 802.3an-only for USD$109? https://www.amazon.com/Wirenest-Intel-x540-T2-Copper-Server/dp/B0BW2QHFWG
  - new motherboards get cheap m.2 onboard cards
    - can anyone find a good 4*m.2 adaptor with PCIe switch?

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal