[tech] Filling up /var on murasoi

[James Arcus]

Hi all,

 [TEC]'s new Grafana alerting has already paid dividends, reporting
murasoi's var as 100% immediately after coming online.

 While there was ~10G of space in the volume group to grow into, which I
immediately did (`lvresize -L +10G murasoi/var` and `resize2fs
/dev/mapper/murasoi-var`), I noticed that there was a large amount of
space on the physical partitions that was not being used by the raid1
array which sits directly atop it:

root at murasoi:/# lsblk 
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT 
...
sdc                     8:32   0 465.8G  0 disk  
└─sdc1                  8:33   0   110G  0 part  
  └─md0                 9:0    0  68.4G  0 raid1 
sdd                     8:48   0 465.8G  0 disk  
└─sdd1                  8:49   0   110G  0 part  
  └─md0                 9:0    0  68.4G  0 raid1  
... 

What the above is saying is that the data stored across the different
partitions of murasoi could occupy a maximum of 68.4 GB, but the
underlying storage has space for 110 GB with the current partition
layout, or 465.8 GB if the physical partition was expanded to fill the
whole disk. This likely happened because the RAID array `/dev/md0` had
been physically copied from a smaller disk in the past and hadn't yet
been made aware of the extra space available. 

The steps are fairly simple and can be run without bringing the system
down: 

# mdadm --grow --size max /dev/md0 
# pvresize /dev/md0 

First, the RAID array is grown to fill its full allocation on disk, then
LVM (the next layer up, which deals with allocating space to different
areas of the filesystem) is informed that it has more raw blocks to play
with. Now the extra space is available to allocate to one or more
partitions in the volume group. I have left /var as is, but there's
plenty of space to expand into, even within the 110 GB. 

Cheers, 

James [MPT]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20200127/81753a90/attachment-0001.htm