[ucc-announce] Downtime Saturday 12/2/11

Bob Adamson bob at ucc.gu.uwa.edu.au
Sun Feb 13 20:49:51 WST 2011


Hi,

First of all, apologies for the extended downtime, especially on the irc 
server. The home directory move took more than twice as long as expected 
due to a couple of large Mail dirs full of small files, and then mylah 
(which hosts mussel as a vm) decided not to boot. It took several wheel 
members most of today to diagnose and fix the problem, hence ucc services 
have only come back online in the last couple of hours. If you discover 
anything still not working (and it should be working), please email 
wheel at ucc or mention it on irc

On a happier note, motsugo is now open to normal user logins -
ssh.ucc.asn.au is still pointing at martello for the moment, but I 
encourage everyone to switch over to using motsugo.ucc.asn.au for the long 
term.

Bob


On Fri, 11 Feb 2011, Bob Adamson wrote:
> Hi again,
> 
> There has been a small change to how we're going to do the move, which 
> will hopefully preserve what little sanity Wheel have left. There have 
> been concerns raised that having a read-only home dir will cause a number 
> of shells/processes to hang, which is generally pretty shit. So 
> unfortunately for users of martello and mussel, you are going to get 
> kicked off while we do the move. This will also cause most of ucc's 
> services to go down, including user logins and irc. Please make everyone's 
> lives easier by being disconnected from martello and mussel by 3pm on 
> Saturday.
> 
> Judging from a test of resync time for /home that I ran today, the new 
> estimate for downtime is around 3 hours. As previously mentioned, keep an 
> eye on http://twitter.com/ucc_status for announcements.
> 
> Bob
> 
> PS: I misquoted the CPU model in the previous email, it's dual Xeon E5520
> 
> On Fri, 11 Feb 2011, Bob Adamson wrote:
> > Hi All,
> > 
> > You may be aware that the club recently purchased a new server to upgrade 
> > and replace our ageing main user server, martello. The new server is 
> > called motsugo and is a dual Intel E6550 Xeon with 12GB RAM and around 
> > 5TB of redundant storage.
> > 
> > This Saturday the 12th at 3pm, Wheel will be moving user home directories 
> > from martello to motsugo and (hopefully) opening motsugo to member logins. 
> > Martello will still be available with home directories after the move, and 
> > will be running concurrently with motsugo for at least a month.
> > 
> > The move should be able to be completed without having to kick all users 
> > off martello, however home directories will be read-only for at least an 
> > hour. Anyone with open handles to /home and /scratch will have their 
> > processes killed; that means you should:
> > 
> > Before the move:
> > -close or stop all ucc email clients
> > -not have any terminals open in the /home and /scratch directories
> > -not have any files open that are in your home directory
> > -stop any scheduled jobs that run on /home or write to your home directory (cron or otherwise) 
> > -stop all processes that log to your home directory (eg irc logging) 
> > -stop any version control systems running in your home dir (eg monotone) 
> > -THE ABOVE APPLIES TO ALL SERVERS THAT USE /HOME (mussel, mermaid, martello, manduba, mylah, mooneye, dagon) 
> > -users of martello's /scratch should redirect their exploits to 
> > motsugo.ucc.gu.uwa.edu.au in the long term. In the short term, 
> > /scratch will be nfs exported from motsugo back to martello.
> > 
> > During the move:
> > -mail delivery will be paused and will queue until /home is restored on motsugo.
> > -dispense and door services will be offline
> > -ucc git and cvs repo's will be read only
> > -member websites will be unavailable for a short time
> > -you will not be able to write to your home directory!
> > 
> > After move:
> > Keep an eye on http://twitter.com/ucc_status for announcements
> > 
> > Please note that motsugo has manbo's old IP address, so you may get a 
> > warning if you have ssh-ed to manbo before.
> 



More information about the ucc-announce mailing list