[tech] An eventful evening on the servers

[Daniel Axtens]

Hi all!

Tonight, I tried to set sigma, up my colo-esque machine (I don't actually own the hardware, thanks [ASH] and UCC). Lots of stuff happened:

 (1) when bigmouth was set up in DNS, zonemake.py wasn't run. So when I added sigma and ran the script, I found out that there was some problem with that stanza which (I don't understand, but) I fixed by commenting out the number of comments line and changing Comment0 to Comment. The error message, fwiw, is at http://pastebin.com/RHj8gRps. Thanks [JCF].

 (2) then I tried to set up DHCP. The server failed to restart, leading me to discover that the /var partition had filled up with syslog-ng repeatedly saying this:

Dec  2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='57'
Dec  2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='56'
Dec  2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='57'
Dec  2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='56'

/var/log/error was 19GB; the messages had been recurring at many times a second since ~Nov 26. The disk filled up a few hours ago.

 (3) the cause of that problem lies in syslog-ng. 
Per [1] and [2] I believe this problem was caused by the program( ) directives in the syslog-ng configuration, in what I believe is best described as utterly brain-dead behaviour. program() directives are currently only being used to play sounds on certain dispenses. The lines I have commented out are as follows, if any of the people affected want to have a crack at fixing the problem:

## [TRS] evil for ubuntu soundtrack
#destination ubuntu_install { program("/root/playubuntu.pl >/dev/null"); };
...
# ubuntu crack
#log { source(s_all); filter(f_daemon); destination(ubuntu_install); };
...
#Retired 20110417, this logging stuff is for the old dispense. [BOB]
# [MSH] coke
filter f_coke { facility(local4); };
#destination d_cokewatch { program("/root/cokempdmon rickroll /root/playrick.sh '(door:0)' /root/playdoor.sh '(door:0) for bob' /root/playbob.sh '(door:0) for stuartp' /root/playstuartp.sh '(door:0) for beas$
...
#log {
#        source(s_remote);
#        filter(f_coke);
#        destination(d_cokewatch);
#};

It may be worth configuring more frequent rotations than the current (AIUI) weekly rotations. A rotation based on size would be particularly worthwhile. A project for a new wheel member, perhaps?

Everything seems to work now.
Some other things worth noting:

 (a) I believe we use RCS to control revisions for some config files but I don't know how or for which ones. If you do, feel free to educate me.

 (b) My colo is, after all this, now living on .104 and has the DNS name sigma. Wheel members: If anything goes wrong with it, feel free to turn it off. I have installed fail2ban, and haven't opened it up on the firewall at all.

That is all for now. Hope you're enjoying holidays, work or whatever your present station in life is.
[DJA]

[1] http://engardelinux.org/modules/index/list_archives.cgi?list=syslog-ng-users&page=0024.html&month=2010-07
[2] http://notes.benv.junerules.com/syslog-ng-pollerr-occurred-while-idle-fdnn/