From danielax at gmail.com Fri Dec 2 22:51:08 2011 From: danielax at gmail.com (Daniel Axtens) Date: Fri, 2 Dec 2011 22:51:08 +0800 Subject: [tech] An eventful evening on the servers Message-ID: <0773F689-F1CA-4A53-B27D-C891AF1D58BC@gmail.com> Hi all! Tonight, I tried to set sigma, up my colo-esque machine (I don't actually own the hardware, thanks [ASH] and UCC). Lots of stuff happened: (1) when bigmouth was set up in DNS, zonemake.py wasn't run. So when I added sigma and ran the script, I found out that there was some problem with that stanza which (I don't understand, but) I fixed by commenting out the number of comments line and changing Comment0 to Comment. The error message, fwiw, is at http://pastebin.com/RHj8gRps. Thanks [JCF]. (2) then I tried to set up DHCP. The server failed to restart, leading me to discover that the /var partition had filled up with syslog-ng repeatedly saying this: Dec 2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='57' Dec 2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='56' Dec 2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='57' Dec 2 06:51:54 murasoi syslog-ng[5005]: POLLERR occurred while idle; fd='56' /var/log/error was 19GB; the messages had been recurring at many times a second since ~Nov 26. The disk filled up a few hours ago. (3) the cause of that problem lies in syslog-ng. Per [1] and [2] I believe this problem was caused by the program( ) directives in the syslog-ng configuration, in what I believe is best described as utterly brain-dead behaviour. program() directives are currently only being used to play sounds on certain dispenses. The lines I have commented out are as follows, if any of the people affected want to have a crack at fixing the problem: ## [TRS] evil for ubuntu soundtrack #destination ubuntu_install { program("/root/playubuntu.pl >/dev/null"); }; ... # ubuntu crack #log { source(s_all); filter(f_daemon); destination(ubuntu_install); }; ... #Retired 20110417, this logging stuff is for the old dispense. [BOB] # [MSH] coke filter f_coke { facility(local4); }; #destination d_cokewatch { program("/root/cokempdmon rickroll /root/playrick.sh '(door:0)' /root/playdoor.sh '(door:0) for bob' /root/playbob.sh '(door:0) for stuartp' /root/playstuartp.sh '(door:0) for beas$ ... #log { # source(s_remote); # filter(f_coke); # destination(d_cokewatch); #}; It may be worth configuring more frequent rotations than the current (AIUI) weekly rotations. A rotation based on size would be particularly worthwhile. A project for a new wheel member, perhaps? Everything seems to work now. Some other things worth noting: (a) I believe we use RCS to control revisions for some config files but I don't know how or for which ones. If you do, feel free to educate me. (b) My colo is, after all this, now living on .104 and has the DNS name sigma. Wheel members: If anything goes wrong with it, feel free to turn it off. I have installed fail2ban, and haven't opened it up on the firewall at all. That is all for now. Hope you're enjoying holidays, work or whatever your present station in life is. [DJA] [1] http://engardelinux.org/modules/index/list_archives.cgi?list=syslog-ng-users&page=0024.html&month=2010-07 [2] http://notes.benv.junerules.com/syslog-ng-pollerr-occurred-while-idle-fdnn/ From maset at ucc.asn.au Sat Dec 3 10:03:47 2011 From: maset at ucc.asn.au (Anil Sharma) Date: Sat, 3 Dec 2011 10:03:47 +0800 Subject: [tech] Fwd: arpwatch on murasoi In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Anil Sharma Date: 2 December 2011 23:16 Subject: arpwatch on murasoi To: wheel arpwatch on murasoi was attached to eth0, and this was spitting out errors because the arp packets were still tagged with vlan information. /var/log blew up. I've edited /etc/arpwatch.conf to only monitor eth0.2, eth0.3, eth0.5, eth0.6, eth0.8, and eth1. If other networks should be monitored, add them to the list in that file. DO NOT add eth0. Cheers, Anil. From maset at ucc.asn.au Mon Dec 19 12:50:18 2011 From: maset at ucc.asn.au (Anil Sharma) Date: Mon, 19 Dec 2011 12:50:18 +0800 Subject: [tech] IPv6 connectivity Message-ID: I put in a ticket in Service Desk detailing our IPv6 connectivity problems. Toivo replied: "We know about the problem. It's waiting on change control before we can fix it." Cheers, Anil. From maset at ucc.asn.au Wed Dec 21 09:33:31 2011 From: maset at ucc.asn.au (Anil Sharma) Date: Wed, 21 Dec 2011 09:33:31 +0800 Subject: [tech] UCC has IPv6 connectivity again [nt] Message-ID: From bob at ucc.gu.uwa.edu.au Wed Dec 21 19:17:43 2011 From: bob at ucc.gu.uwa.edu.au (Bob Adamson) Date: Wed, 21 Dec 2011 19:17:43 +0800 (WST) Subject: [tech] Spare Ultra320 disks Message-ID: Hi all, I've donated four 300GB Ultra320 SCSI disks to the club which came from where I work. One has already gone into mylah to replace the disk that died a month ago, and the other three are spare. I'm afraid I don't know their history so I make no assertions as to their quality, but the one in mylah appears to be in good health. Maybe one could go into meersau if nobody ever gets around to making it into a vm. Andrew Adamson UCC President bob at ucc.asn.au |"The faster you move, the slower time passes, the longer you live." | | ---Peter's Laws | From maset at ucc.asn.au Thu Dec 29 13:13:38 2011 From: maset at ucc.asn.au (Anil Sharma) Date: Thu, 29 Dec 2011 13:13:38 +0800 Subject: [tech] uccrouter.ucc.asn.au/traffic-tester Message-ID: Hi all I found another bit missing from the murasoi install - traffic-tester. I've set it up as best as I can, but there are a few things that need fixing: - the equivalent of /home/grahame/svn/ucs.uwa.edu.au/aarnet/iptables/freenets-aggr.txt needs to be setup - cronjob (or however it is autogenerated) for the above file setup - SSL certificate was copied from madako, doesn't seem to be working (I don't know about such things) Cheers, Anil.