[tech] Mussel

[David Adam]

On Thu, 6 Jun 2013, Sam Moore wrote:
> Mussel isn't running very well lately. Unfortunately we haven't caught it yet.
> 
> We think the problem is due to OOM caused by apache spawning too many
> processes. Currently apache uses about 40% of memory with no load, and has
> about 90 threads running. When something actually does request a lot of stuff,
> mussel runs out of memory.

It died again on Friday night; this time, it was responding on the Xen 
console but not on the network. Root login was possible but very, very 
slow and a shell never actually started. In the end, all I could get out 
was SysRq-related data (running jobs, memory usage, etc.) and bounced the 
system. Logs are on Mylah in /backups/mussel-console{,1}

It didn't actually appear to be out of memory this time; possibly a CPU 
thrashing thing, although it's hard to tell from the Xen tools.

There were a few saslauthd processes running, and I wonder if we might 
have hit a bug which makes it use lots and lots of CPU; see 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708552 / 
https://bugs.launchpad.net/ubuntu/+source/cyrus-sasl2/+bug/997217

I've upgraded the Cyrus SASL packages and I guess we'll see what happens.

David Adam
UCC Wheel Member
zanchey at ucc.gu.uwa.edu.au