[tech] Filling up /var on murasoi

[James Arcus]

P.S. Based on the contents of the auth logs, I believe the filesystem 
had
been filling up since ~20:30 last night.

Cheers,

James [MPT]

On 2020-01-27 23:17, James Arcus wrote:
> Hi all,
> 
>  [TEC]'s new Grafana alerting has already paid dividends, reporting
> murasoi's var as 100% immediately after coming online.
> 
>  While there was ~10G of space in the volume group to grow into, which
> I immediately did (`lvresize -L +10G murasoi/var` and `resize2fs
> /dev/mapper/murasoi-var`), I noticed that there was a large amount of
> space on the physical partitions that was not being used by the raid1
> array which sits directly atop it:
> 
> root at murasoi:/# lsblk
> NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> ...
> sdc                     8:32   0 465.8G  0 disk
> └─sdc1                  8:33   0   110G  0 part
>   └─md0                 9:0    0  68.4G  0 raid1
> sdd                     8:48   0 465.8G  0 disk
> └─sdd1                  8:49   0   110G  0 part
>   └─md0                 9:0    0  68.4G  0 raid1
> ...
> 
> What the above is saying is that the data stored across the different
> partitions of murasoi could occupy a maximum of 68.4 GB, but the
> underlying storage has space for 110 GB with the current partition
> layout, or 465.8 GB if the physical partition was expanded to fill the
> whole disk. This likely happened because the RAID array `/dev/md0` had
> been physically copied from a smaller disk in the past and hadn't yet
> been made aware of the extra space available.
> 
> The steps are fairly simple and can be run without bringing the system
> down:
> 
> # mdadm --grow --size max /dev/md0
> # pvresize /dev/md0
> 
> First, the RAID array is grown to fill its full allocation on disk,
> then LVM (the next layer up, which deals with allocating space to
> different areas of the filesystem) is informed that it has more raw
> blocks to play with. Now the extra space is available to allocate to
> one or more partitions in the volume group. I have left /var as is,
> but there's plenty of space to expand into, even within the 110 GB.
> 
> Cheers,
> 
> James [MPT]
> _______________________________________________
> List Archives: http://lists.ucc.asn.au/pipermail/tech
> 
> Unsubscribe here:
> https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/jimbo%40ucc.asn.au