From frekk at ucc.asn.au Sat Mar 2 20:57:16 2019 From: frekk at ucc.asn.au (Felix von Perger) Date: Sat, 2 Mar 2019 20:57:16 +0800 Subject: [tech] Fwd: Prices for Stuff from Wholesale & Ebay Links In-Reply-To: References: Message-ID: <2b0eb662-9f71-86ff-0ef6-6342a80bef69@ucc.asn.au> Here are some sample prices for equipment? from Melissa - note that the wholesaler names have been redacted. Regarding monitors, I'd be tempted to stick with the 24" variety, since those are approximately the same size as the others we already have, and they are only 1080p resolution anyway. - [FVP] -------- Forwarded Message -------- Subject: Prices for Stuff from Wholesale Date: Fri, 1 Mar 2019 14:07:35 +0800 From: Melissa Star To: president at ucc.asn.au Hi Felix and Everyone, Just reconfirming prices and stock availability for various things: Some things are in stock, some are not, check notes. * * * * *Wholesaler 1Wholesaler 2* * * * * *SAMSUNG 860 QVO - 1TB160158Note: Wholesaler 1 has Perth Availability from 6th March only. Wholesaler 2 have plenty of Stock in Melbourne but can courier over.* *(35K IOPS) * *SAMSUNG 860 EVO - 1TBX189??? ??? ?? Wholesaler 2 currently has Perth stock, Wholesaler 1 does not.* *(98/99K IOPS)* * * *SAMSUNG 860 QVO - ?2TBX 302 (!!!)Exactly 8 in stock in Perth, more in Melbourne* * * *SAMSUG EVO M2 - 1TBX 189ETA LATE NEXT WEEK* * * *INTEL 660P M2 - 1TB2162 in Perth, about 8 others around the place. Rare. More stock at Wholesaler 2 but $10 more expensive.* *150K + IOPS* * * *????* * * AOC 27" 1ms 144Hz Full HD FreeSync Frameless Gaming Monitor - G2790PX $365 + GST. Stock in WA NOW (7 in stock) AOC 24.5" 1ms 144Hz Full HD FreeSync Frameless Gaming Monitor - G2590PX $300 + GST. Stock in WA on 15th March, we have to reserve. ------------------------------------------------------------------------ Here are some relevant Ebay Links: *Dell R710* https://www.ebay.com.au/itm/Dell-R710-2-x-E5520-2-27Ghz-48GB-RAM-2-x-PSU/173751309985?_trkparms=aid%3D111001%26algo%3DREC.SEED%26ao%3D1%26asc%3D20160908131621%26meid%3D6ceb75c482b9414fa39c8c113dc37e3f%26pid%3D100678%26rk%3D1%26rkt%3D15%26sd%3D173751309985%26itm%3D173751309985&_trksid=p2481888.c100678.m3607&_trkparms=pageci%3A221f2b2e-3be8-11e9-9945-74dbd18015f3%7Cparentrq%3A37dc542d1690ab67025673bdfffd48fa%7Ciid%3A1 *Compatibility with XEON 56XX Processors:* * * *https://store.flagshiptech.com/dell-poweredge-r710-intel-xeon-5600-series-cpus-processors/?sort=featured&page=2 * * * Rack Shelves of Various Types: https://www.ebay.com.au/itm/1U-SLIDE-SHELF-19-Adjustable-Depth-300mm-Suit-19-600mm-Deep-Server-Rack/122436251015?hash=item1c81c4ed87:g:u80AAOSwInxXOAKQ:rk:1:pf:1&frcectupt=true https://www.ebay.com.au/itm/1U-FIXED-SHELF-700mm-Suit-19-Inch-1000mm-Deep-Free-Standing-Server-Cabinet/122436231008?_trkparms=aid%3D222007%26algo%3DSIM.MBE%26ao%3D2%26asc%3D20140106155344%26meid%3D84fd2bcf86f049a297719d11ab4f1f5f%26pid%3D100005%26rk%3D4%26rkt%3D12%26sd%3D122436251015%26itm%3D122436231008&_trksid=p2047675.c100005.m1851 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190302/67ecfb06/attachment.htm From zanchey at ucc.gu.uwa.edu.au Sat Mar 2 21:27:24 2019 From: zanchey at ucc.gu.uwa.edu.au (David Adam) Date: Sat, 2 Mar 2019 21:27:24 +0800 (AWST) Subject: [tech] Dead disk in Molmol Message-ID: Hi all, Molmol has dropped one of its SSDs: Feb 26 14:15:10 molmol kernel: ahcich1: Timeout on slot 25 port 0 Feb 26 14:15:10 molmol kernel: ahcich1: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd c0 serr 00000000 cmd 0004d917 Feb 26 14:15:10 molmol kernel: (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 Feb 26 14:30:34 molmol kernel: (ada1:ahcich1:0:0:0): CAM status: Command timeout Feb 26 14:30:34 molmol kernel: (ada1:ahcich1:0:0:0): Retrying command Feb 26 14:30:34 molmol kernel: ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080) (etc.) It's detached from the bus and won't reattach. The device is a Samsung SSD 840 PRO Series DXM05B0Q (s/n S1ATNSAD864731A) - note that there are two of these in the machine! I'm not sure whether it is hotpluggable or not. This SSD was providing one half of the SLOG mirror [1] and a RAID partition for the root filesystem. The other half is provided by the other Samsung 840 PRO: zfs pool status: NAME STATE READ WRITE CKSUM logs mirror-4 DEGRADED 0 0 0 5535644740799039914 REMOVED 0 0 0 was /dev/gpt/molmol-slog gpt/molmol-slog0 ONLINE 0 0 0 Checking status of gmirror(8) devices: Name Status Components mirror/gmirror0 DEGRADED ada0p2 (ACTIVE) If one has gone, I suspect the other is not far behind (SLOG devices do a lot of writing), so it is probably worth replacing at least one and possibly both. This may be part of why performance has tanked recently (although I have no evidence to support this statement). They don't need to be big - we're currently using 80 GB of the 256 GB disk - but they do need to be reliable and fast. I have zero idea what the best part to pick is; any thoughts? David Adam zanchey@ UCC Wheel Member [1]: https://pthree.org/2012/12/06/zfs-administration-part-iii-the-zfs-intent-log/ From bob at ucc.asn.au Sun Mar 3 15:19:14 2019 From: bob at ucc.asn.au (Bob Adamson) Date: Sun, 3 Mar 2019 15:19:14 +0800 Subject: [tech] [committee] Fwd: Prices for Stuff from Wholesale & Ebay Links In-Reply-To: <2b0eb662-9f71-86ff-0ef6-6342a80bef69@ucc.asn.au> References: <2b0eb662-9f71-86ff-0ef6-6342a80bef69@ucc.asn.au> Message-ID: <003f01d4d191$6d0bf3f0$4723dbd0$@ucc.asn.au> Sweet, can we please get two SAMSUNG 860 QVO (2TB) disks to replace the currently failing pair in motsugo @ $302 each. We would normally replace these with WD Black 2TB disks (WD2003FZEX) @ $189 each, however motsugo is due for replacement soon and I envisage those two SSD's becoming bulk data disks in clubroom desktops when motsugo is upgraded. Cheers, Bob From: committee-bounces+bob=ucc.gu.uwa.edu.au at ucc.asn.au On Behalf Of Felix von Perger Sent: Saturday, 2 March 2019 8:57 PM To: tech at ucc.asn.au Cc: committee at ucc.asn.au Subject: [committee] Fwd: Prices for Stuff from Wholesale & Ebay Links Here are some sample prices for equipment from Melissa - note that the wholesaler names have been redacted. Regarding monitors, I'd be tempted to stick with the 24" variety, since those are approximately the same size as the others we already have, and they are only 1080p resolution anyway. - [FVP] -------- Forwarded Message -------- Subject: Prices for Stuff from Wholesale Date: Fri, 1 Mar 2019 14:07:35 +0800 From: Melissa Star To: president at ucc.asn.au Hi Felix and Everyone, Just reconfirming prices and stock availability for various things: Some things are in stock, some are not, check notes. Wholesaler 1 Wholesaler 2 SAMSUNG 860 QVO - 1TB 160 158 Note: Wholesaler 1 has Perth Availability from 6th March only. Wholesaler 2 have plenty of Stock in Melbourne but can courier over. (35K IOPS) SAMSUNG 860 EVO - 1TB X 189 Wholesaler 2 currently has Perth stock, Wholesaler 1 does not. (98/99K IOPS) SAMSUNG 860 QVO - 2TB X 302 (!!!) Exactly 8 in stock in Perth, more in Melbourne SAMSUG EVO M2 - 1TB X 189 ETA LATE NEXT WEEK INTEL 660P M2 - 1TB 216 2 in Perth, about 8 others around the place. Rare. More stock at Wholesaler 2 but $10 more expensive. 150K + IOPS ???? AOC 27" 1ms 144Hz Full HD FreeSync Frameless Gaming Monitor - G2790PX $365 + GST. Stock in WA NOW (7 in stock) AOC 24.5" 1ms 144Hz Full HD FreeSync Frameless Gaming Monitor - G2590PX $300 + GST. Stock in WA on 15th March, we have to reserve. _____ Here are some relevant Ebay Links: Dell R710 https://www.ebay.com.au/itm/Dell-R710-2-x-E5520-2-27Ghz-48GB-RAM-2-x-PSU/173751309985?_trkparms=aid%3D111001%26algo%3DREC.SEED%26ao%3D1%26asc%3D20160908131621%26meid%3D6ceb75c482b9414fa39c8c113dc37e3f%26pid%3D100678%26rk%3D1%26rkt%3D15%26sd%3D173751309985%26itm%3D173751309985 &_trksid=p2481888.c100678.m3607&_trkparms=pageci%3A221f2b2e-3be8-11e9-9945-74dbd18015f3%7Cparentrq%3A37dc542d1690ab67025673bdfffd48fa%7Ciid%3A1 Compatibility with XEON 56XX Processors: https://store.flagshiptech.com/dell-poweredge-r710-intel-xeon-5600-series-cpus-processors/?sort=featured &page=2 Rack Shelves of Various Types: https://www.ebay.com.au/itm/1U-SLIDE-SHELF-19-Adjustable-Depth-300mm-Suit-19-600mm-Deep-Server-Rack/122436251015?hash=item1c81c4ed87:g:u80AAOSwInxXOAKQ:rk:1:pf:1 &frcectupt=true https://www.ebay.com.au/itm/1U-FIXED-SHELF-700mm-Suit-19-Inch-1000mm-Deep-Free-Standing-Server-Cabinet/122436231008?_trkparms=aid%3D222007%26algo%3DSIM.MBE%26ao%3D2%26asc%3D20140106155344%26meid%3D84fd2bcf86f049a297719d11ab4f1f5f%26pid%3D100005%26rk%3D4%26rkt%3D12%26sd%3D122436251015%26itm%3D122436231008 &_trksid=p2047675.c100005.m1851 -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190303/f50fcb6b/attachment-0001.htm From bob at ucc.asn.au Sun Mar 3 15:21:02 2019 From: bob at ucc.asn.au (Bob Adamson) Date: Sun, 3 Mar 2019 15:21:02 +0800 Subject: [tech] Dead disk in Molmol In-Reply-To: References: Message-ID: <004c01d4d191$ada21250$08e636f0$@ucc.asn.au> Hi All, I think we should look for something a bit more enterprisey for this task, since it is such a critical component - this machine hosts a lot of club and member VM storage, as well as clubroom desktop home directories. Given the issues we seem to be having with speeds, I don't think we should skimp on disks this time around. This page, though from 2014, details how we might check the performance of SSD's as a Ceph journaling device, which (aiui) uses synchronous writes similar to the requirements of NFS on ZFS: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is -suitable-as-a-journal-device/ . The results are somewhat scant, but what is apparent is the order of magnitude in speed difference between consumer and enterprise SSDs for this use. Despite having many bays on the front, molmol only supports 8 SAS disks, 2 SATA3 disks, and 4 SATA2 disks. The 8 SAS ports are taken up by spinning disks at the moment, and the 2 SATA3 ports are used by the system/SLOG disks. There's really no point in using the SATA2 ports due to their speed. The mobo is a supermicro x9srh-7tf , so it has 1xPCIe 3.0 x16 slot and 1xPCIe 3.0 x8 slot. Given that the mobo has 10G ethernet onboard, I think both of those slots should be free. The case itself is 2RU, so we could support 2 low profile PCIe SSD cards. Anyway, what I'm thinking is to replace the failing system disk with another similar SSD, then chuck a single, fast, PCIe SSD in it for the SLOG and L2ARC only. If it fails, aiui we don't have a corrupt file system, we just lose the last 5 seconds of data (correct me if I'm wrong here?). This is based on a few google results, like https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSSLOGLossEffects $409 plus delivery for an Intel Optane 900P: https://www.scorptec.com.au/product/Hard-Drives-&-SSDs/SSD-2.5-&-PCI-Express /70481-SSDPED1D280GASX Plus $90 to replace the failed system disk with a 250GB 860EVO: https://www.scorptec.com.au/product/Hard-Drives-&-SSDs/SSD-2.5-&-PCI-Express /71382-MZ-76E250BW $15 delivery $514 total Thoughts? I'm happy to order it, just approve it at a committee meeting (or outside of one via circular) and let me know. Thanks, Bob -----Original Message----- From: tech-bounces+bob=ucc.gu.uwa.edu.au at ucc.asn.au On Behalf Of David Adam Sent: Saturday, 2 March 2019 9:27 PM To: tech at ucc.gu.uwa.edu.au Subject: [tech] Dead disk in Molmol Hi all, Molmol has dropped one of its SSDs: Feb 26 14:15:10 molmol kernel: ahcich1: Timeout on slot 25 port 0 Feb 26 14:15:10 molmol kernel: ahcich1: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd c0 serr 00000000 cmd 0004d917 Feb 26 14:15:10 molmol kernel: (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 Feb 26 14:30:34 molmol kernel: (ada1:ahcich1:0:0:0): CAM status: Command timeout Feb 26 14:30:34 molmol kernel: (ada1:ahcich1:0:0:0): Retrying command Feb 26 14:30:34 molmol kernel: ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080) (etc.) It's detached from the bus and won't reattach. The device is a Samsung SSD 840 PRO Series DXM05B0Q (s/n S1ATNSAD864731A) - note that there are two of these in the machine! I'm not sure whether it is hotpluggable or not. This SSD was providing one half of the SLOG mirror [1] and a RAID partition for the root filesystem. The other half is provided by the other Samsung 840 PRO: zfs pool status: NAME STATE READ WRITE CKSUM logs mirror-4 DEGRADED 0 0 0 5535644740799039914 REMOVED 0 0 0 was /dev/gpt/molmol-slog gpt/molmol-slog0 ONLINE 0 0 0 Checking status of gmirror(8) devices: Name Status Components mirror/gmirror0 DEGRADED ada0p2 (ACTIVE) If one has gone, I suspect the other is not far behind (SLOG devices do a lot of writing), so it is probably worth replacing at least one and possibly both. This may be part of why performance has tanked recently (although I have no evidence to support this statement). They don't need to be big - we're currently using 80 GB of the 256 GB disk - but they do need to be reliable and fast. I have zero idea what the best part to pick is; any thoughts? David Adam zanchey@ UCC Wheel Member [1]: https://pthree.org/2012/12/06/zfs-administration-part-iii-the-zfs-intent-log / _______________________________________________ List Archives: http://lists.ucc.asn.au/pipermail/tech Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/bob%40ucc.gu.uwa.edu.au From melissa at netexperts.com.au Tue Mar 5 15:36:14 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 5 Mar 2019 15:36:14 +0800 Subject: [tech] Responding to Server Room Heat, Ashera Shutdown and Remote Control Message-ID: Hi Everyone, After discussion with Felix last night, I am proposing the following: Remote Power-Off for Ashera (and optionally other servers) There was some discussion last night regarding the need to be able to remotely power down servers. For Ashera, I have created a shutdown at ashera.ucc.asn.au account. Logging in to this account via ssh will cause the machine to immediately power down. Note that there is no "confirmation prompt" and once the machine is powered down it must be physically restarted in the server room. This was achieved by: 1. Creating the shutdown user and assigning group "operator". 2. Change Shell (chsh) to /usr/local/bin/bash 3. Placing the command "shutdown -P now" into the .bashrc file. This can also be done using the init file of any other shell of your choice. I haven't set up SSH keys for the account yet, but could either use the standard SSH authorized_keys file for wheel, or preferably I could implement a command accessible to wheel that would cause ashera (and if desired, other servers) to shut down as needed. As Ashera is my own sever co-located at UCC, I've already implemented this feature. Making Remote Power-Off possible for UCC servers on command (apocalypse command) If you are interested, I could also write a bash script that would respond to "apocalypse " by shutting down the appropriate machines. Access would be wheel only. E.g. apocalypse ashera now would shut down ashera inmmediately. apocalypse now would shut down any defined group of servers. (E.g. apocalypse VMs now, or apocalypse left-rack now). apocalypse * now would power down every server that is part of the system. I could then write a script that could read a heat sensor in the server room and automatically begin to shut down machines in order of priority as heat increases past a critical point. Let me know what you think. Regards, Melissa -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190305/dadc9898/attachment.htm From charcoaldragon at hotmail.com Fri Mar 8 21:01:10 2019 From: charcoaldragon at hotmail.com (Thomas Miller) Date: Fri, 8 Mar 2019 13:01:10 +0000 Subject: [tech] Broken monitor Message-ID: Hi tech. After discussing with [BOB] i am sending an email to inform that a monitor was accidentally broken. The monitor had been precariously placed on the the machines on the floor beside the cluttered coffee table in the centre of the room, as seen in the picture attached. Perhaps this is an untimely reminder that cleanliness in the clubroom serves many purposes. Tom [TFM] -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190308/45508085/attachment-0001.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: 16.jpg Type: image/jpeg Size: 44990 bytes Desc: 16.jpg Url : https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190308/45508085/attachment-0001.jpg From bob at ucc.gu.uwa.edu.au Tue Mar 12 12:21:32 2019 From: bob at ucc.gu.uwa.edu.au (Andrew Adamson) Date: Tue, 12 Mar 2019 12:21:32 +0800 (AWST) Subject: [tech] Gothamburg upgrade Message-ID: This chain got started in the wheel@ list as part of another conversation, and doesn't need to be a closed one. Edited for relevancy. Andrew Adamson bob at ucc.asn.au |"If you can't beat them, join them, and then beat them." | | ---Peter's Laws | ---------- Forwarded message ---------- Date: Tue, 12 Mar 2019 12:13:32 +0800 From: Melissa Star To: James Cox Cc: Wheel , committee-only at ucc.asn.au Subject: Re: [wheel] Wheel application process Hi James, Reading up about it now on MacRumors?? To run Mojave without ?graphics issues? we would basically need a ?small metal video card? and then Mojave patcher :) Small, so it fits, and doesn?t overheat the machine, and supporting the metal API.? Possibly, something like this: https://www.ebay.com.au/itm/HP-Nvidia-GeForce-GT630-2GB-DDR3-128bit-PCIe-x16-Graphics-Card-DP-DVI-702084-001/202508328689?hash=item2f266fdef1:g:OOsAAOSw0qdceHRo? There are also mixed reports with the GT710, and Nvidia CPUs. I could buy one to try it out. Picture On 12 Mar 2019, at 8:10 am, James Cox wrote: Gothamburg is an Xserve 3,1 and on its max supported OS. (I've just updated the wiki) Since it's not a production system I don't see anything wrong with upgrading it past supported territory... But to get to Mojave we would need a new GPU. I'm not sure there's any point upgrading it only as far as say High Sierra? [RME]~Coxy On Tue, 12 Mar 2019 at 01:56, Melissa Star wrote: That issue aside, I would love to bring Gothunberg up to date and have an X-Serve I could actually use on campus. If I could have decent user privileges there, and you agreed, I could get it up to a more recent OS/X, and put irc and other relevant apps on it, and would actually end up using it a lot in addition to Ashera. I?m not sure if it otherwise gets used very much. From uwaunigames at gmail.com Wed Mar 13 15:29:58 2019 From: uwaunigames at gmail.com (Unigames UWA) Date: Wed, 13 Mar 2019 15:29:58 +0800 Subject: [tech] Mailing lists + Markdown parser Message-ID: Hey there tech@, Unigames (Donald) here. Two inquiries: 1) Unigames would like to have a few more mailing lists. Ideally we would like: - unigames (our current one) - unigames-magic (new) - unigames-rpgs (new) - unigames-wargames (new) - unigames-freshers (new) Ideally, we would also like unigames-freshers to not allow user subscriptions, but make it so that only an mailing list admind can subscribe people. Is this doable? Also, sub-inquiry: If I sent mail to unigames at ucc.asn.au, would it go to the mailing list, or whatever email we have it set to forward to, or both? (None at the moment I don't believe, but good to check.) 2) Currently, Unigames' website is hosted on UCC's servers, and I'm making slow upgrades to it as part of my secretary job this year. One of those things involves posting minutes in markdown. Currently though, I have to upload a .md file and a rendered .html file to get it to look pretty. Is there a way to allow .md files to be rendered automatically when viewed, just on our domain? Thanks everyone in advance <3 Donald Sutherland (donsuth) Unigames Secretary -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190313/1d477d63/attachment.htm From trs80 at ucc.gu.uwa.edu.au Wed Mar 13 16:49:19 2019 From: trs80 at ucc.gu.uwa.edu.au (James Andrewartha) Date: Wed, 13 Mar 2019 16:49:19 +0800 (AWST) Subject: [tech] Mailing lists + Markdown parser In-Reply-To: References: Message-ID: On Wed, 13 Mar 2019, Unigames UWA wrote: > Hey there tech@, Unigames (Donald) here.? > Two inquiries: > 1) Unigames would like to have a few more mailing lists. Ideally we would like: > - unigames (our current one) > - unigames-magic (new) > - unigames-rpgs (new) > - unigames-wargames (new) > - unigames-freshers (new) This is feasible, who should we put as the mailing list admins? > Ideally, we would also like unigames-freshers to not allow user subscriptions, but make it so that only an mailing list admind > can subscribe people. Is this doable? Yeah, it's easy to do that. > Also, sub-inquiry: If I sent mail to unigames at ucc.asn.au, would it go to the mailing list, or whatever email we have it set to > forward to, or both? (None at the moment I don't believe, but good to check.) Mailing list. > 2) Currently, Unigames' website is hosted on UCC's servers, and I'm making slow upgrades to it as part of my secretary job this > year. > > One of those things involves posting minutes in markdown. Currently though, I have to upload a .md file and a rendered > .html?file to get it to look pretty. > > Is there a way to allow .md files to be rendered automatically when viewed, just on our domain?? Hmm, .ucc files can contain markdown but then they'll be rendered with the UCC theme. Also the process flow is fucking nasty (and I use that term advisedly, first it parses the markdown into an older format called "DOCTYPE grahame" which is then processed with XSLT into HTML). Some quick googling turns up several better options, https://github.com/hamano/apache-mod-markdown https://github.com/sminnee/markdown-handler http://projects.skurfer.com/Example.mdown Which one to pick is an open question, the second and third ones in theory can be installed by you so perhaps try them first. -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From melissa at netexperts.com.au Wed Mar 13 19:33:16 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Wed, 13 Mar 2019 19:33:16 +0800 Subject: [tech] My notes today. Message-ID: Hi everyone, These are the notes I took into today's meeting for what to discuss, amended in light of what we spoke about. My purpose here was to gauge committee interest before taking further time to further research something that would not be likely to be supported. 1. A New Linux Server to be administereded by students. Following on from the idea of having a set of "development" or "sandbox" machines that are not as strictly controlled for root access. * Students able to learn to administer Linux on real hardware without having to be wheel members * Potential mail server for "illuminati.org.au" and other domains we may get. * Potential Docker host. I can talk more about Docker and what it is if anyone doesn't know (or google Docker, they have a great website). I estimate a suitable server for this purpose could be purchased second hand in the $500 range, or a more powerful and ideal solution for about $1000. 2. A Microsoft Terminal Services and Exchange Server The benefits of such a machine would be: * Be able to learn Windows Server, Microsoft Exchange, Active Directory and Terminal Services administration. * Offering Windows desktops and Microsoft Office for people who are logged on to Linux machines (repeated request by various people in the club, including while I was writing this e-mail!) * Offering an emergency Windows remote desktop to anyone whose laptop starts misbehaving in class, or for people who can't afford a machine powerful enough to do what they would otherwise want to. Hopefully either the uni or Microsoft would be able to license the relevant software to us (if we have in principle approval I'm willing to do the work). To work for many concurrent users this would require quite a lot of RAM and multiple fast CPU cores, but there is a very big ask for the ability to run Windows apps over the top of non-Windows machines, including making it easier for people to use Linux on their laptops at uni and still access e.g. Microsoft Word. NOT DISCUSSED BECAUSE WE REAN OUT OF TIME. 4. Upgrade of Gothenburg and installation of Application Programs * Upgrade to Mojave may or may not be possible with purchase of a cheap GT710 graphics card. If not then ugprading as far as High Sierra will give us a large additional set of standard Mac applications. To test the viability of doing this, I have purchased a functionally identical Mac Pro 3,1 for myself (which I then need for another project, unrelated to UCC) which will allow me to pre-test the concept, I will receive it next week and report on the outcome. I'm quite interested in trying this out both because of the ability to easily do a multi-user Mac environment off any club or other machine (similar to the Terminal Server discussed above) and for the technical challenge in of itself. Regards, Melissa From frekk at ucc.asn.au Wed Mar 13 23:12:31 2019 From: frekk at ucc.asn.au (Felix von Perger) Date: Wed, 13 Mar 2019 23:12:31 +0800 Subject: [tech] Renaming accounts, was "Wheel being the Illuminati..." In-Reply-To: References: Message-ID: <60ed8211-2568-5df7-726c-2067c516320f@ucc.asn.au> Hi all, Moving this to tech@ since it's not really committee-related. Renaming accounts is a fairly nightmarish process that is probably easiest done by creating an entirely new account, then moving files and dispense credit over to the new one. It might be possible to rename the account in AD, although that would require further testing, and would likely cause more problems than it would solve. In recent history I was able to "rename" /dragonxdoom/ to /donsuth/ by creating a new account under the correct name, then copying the files across to a directory under the new account's homedir, then chowning everything so that the permissions were correct. Note that simply moving the old homedir to the new location doesn't work since it breaks Windows roaming profiles completely (even if the permissions are correct, it will say "access denied" when you try to login under the new username). As for dispense, simply transferring the old account balance to the new account should be sufficient, and it doesn't seem like deleting dispense accounts is implemented anywhere. Card logins for the snack machine will probably remain associated to the old account unless you can somehow (re)associate them with the new account, which may be a problem. On another note, the /etc/bind/domains/primary/zonemake.py script on mooneye currently seems broken, so it's not possible currently to refresh member domains and virtualhosts. It might be related to possible (now fixed?) duplicate account UIDs but I haven't had time to debug. - [FVP] On 13/3/19 10:48 pm, James Andrewartha wrote: > On Wed, 13 Mar 2019, David Adam wrote: > >> If you would like to rename your account, you can ask a Wheel member or >> email wheel at ucc.gu.uwa.edu.au. (I don't know that we've ever actually done >> this, but https://serverfault.com/a/929192/10056 makes it sound like it's >> possible.) > We did it with Alwyn's account, but that was 18 years ago in the age of NIS. > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190313/a8536527/attachment.htm From trs80 at ucc.gu.uwa.edu.au Thu Mar 14 00:27:10 2019 From: trs80 at ucc.gu.uwa.edu.au (James Andrewartha) Date: Thu, 14 Mar 2019 00:27:10 +0800 (AWST) Subject: [tech] Renaming accounts, was "Wheel being the Illuminati..." In-Reply-To: <60ed8211-2568-5df7-726c-2067c516320f@ucc.asn.au> References: <60ed8211-2568-5df7-726c-2067c516320f@ucc.asn.au> Message-ID: On Wed, 13 Mar 2019, Felix von Perger wrote: > On another note, the /etc/bind/domains/primary/zonemake.py script on mooneye currently seems broken, so it's not possible > currently to refresh member domains and virtualhosts. It might be related to possible (now fixed?) duplicate account UIDs but I > haven't had time to debug. User covfefe (is it even a real account) has a bogus GID 20021 (as well as the regular gumby GID 10021). Also all the users are still duplicated in mooneye's /etc/passwd which means a lot of stuff is generated twice. -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From james at cox.cx Fri Mar 15 09:19:46 2019 From: james at cox.cx (James Cox) Date: Fri, 15 Mar 2019 09:19:46 +0800 Subject: [tech] Systems Change Log (was wheel generational issues) In-Reply-To: References:

Message-ID: I have created a new page https://wiki.ucc.asn.au/ChangeLog It is linked from Mission Control. Please feel free to edit, add, whatever. [RME]~Coxy ---------- Forwarded message --------- From: James Andrewartha Date: Fri, 15 Mar 2019 at 09:04 Subject: Re: [committee] [wheel] Wheel generational issues To: James Cox Cc: committee , , Lyndon White On Fri, 15 Mar 2019, James Cox wrote: > Sure, but tech@ posts aren't a commit log either. > And notification is no good to the hypothetical future person who wasn't subbed at the time (or deletes their email.) Yes but that's why there's a mailing list archive. A commit log is fine as an addition https://i.kym-cdn.com/photos/images/newsfeed/000/538/731/0fc.gif -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190315/2aa6b347/attachment.htm From frekk at ucc.asn.au Sun Mar 17 16:52:40 2019 From: frekk at ucc.asn.au (Felix von Perger) Date: Sun, 17 Mar 2019 16:52:40 +0800 Subject: [tech] [wheel] Wheel meeting - Monday 2019-04-08 19:00 Message-ID: Hi all, Following up from discussions over the last week or so, and to welcome one (or more) new members, I'd like to propose a gathering of the members of the Wheel Group on Monday 2019-04-08 at 19:00 AWST. For anyone who wishes to get some food beforehand, we'll probably head over to Chelsea Pizza around 17:30 or so. To add anything to the agenda, simply edit /home/wheel/docs/meetings/2019-04-08.txtor reply to this email. The agenda will also be visible online at https://www.ucc.asn.au/infobase/minutes/2019/2019-04-08_wheel.txt. Best regards, Felix von Perger [FVP] UCC President & Wheel member -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190317/8ea4fb7d/attachment.htm From nick at ucc.gu.uwa.edu.au Sun Mar 17 19:24:42 2019 From: nick at ucc.gu.uwa.edu.au (Nick Bannon) Date: Sun, 17 Mar 2019 19:24:42 +0800 Subject: [tech] Serial terminals in the hallway - PS/2 keyboard needed In-Reply-To: <20180312154343.zro2xbw7b42hwf3w@ucc.gu.uwa.edu.au> References: <20110929104349.GA28262@ucc.gu.uwa.edu.au> <20180312154343.zro2xbw7b42hwf3w@ucc.gu.uwa.edu.au> Message-ID: <20190317112442.thtopj6fg6me5gz3@ucc.gu.uwa.edu.au> On Mon, Mar 12, 2018 at 11:43:43PM +0800, Nick Bannon wrote: [...] > https://terminals-wiki.org/wiki/index.php/DEC_VT510 The good news: the terminal is now mounted on a fit-for-purpose? shelf as part of a 19" rack which needed a home, as of yesterday's cleanup. The bad news - although the VT510 takes nice easy-to-find PS/2 keyboards, we couldn't find one to fit the available space. One of our USB ones does fit, but it may not support PS/2 mode (via an adaptor) - the correct adaptor or any other 445mm-wide (or narrower) PS/2-capable keyboard would be very handy. So, at least temporarily, it's an amber https://terminals-wiki.org/wiki/index.php/DEC_VT320 ...with our precious narrow keyboard. It has been reported that it's not working as well as the VT510, particularly with the fish shell? but if a software fix can't be foudn easily, a suitable keyboard for the VT510 would be the easy hardware fix. Having said that, having multiple working terminal ports and/or terminals would be a great project. The Cyclades TS-1000 PPC terminal server is in the top of the rack, and it could probably use an ethernet switch, too... > Software flow control, so 9600bps it is for now. ...which might be one way get a properly wired cable with hardware flow control. > There's also a HP small-form-factor PC, it's not booting (or netbooting) > anything interesting right now. Possible PSU issue? Nick. -- Nick Bannon | "I made this letter longer than usual because nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal From zanchey at ucc.gu.uwa.edu.au Sun Mar 17 20:33:18 2019 From: zanchey at ucc.gu.uwa.edu.au (David Adam) Date: Sun, 17 Mar 2019 20:33:18 +0800 (AWST) Subject: [tech] Serial terminals in the hallway - PS/2 keyboard needed In-Reply-To: <20190317112442.thtopj6fg6me5gz3@ucc.gu.uwa.edu.au> References: <20110929104349.GA28262@ucc.gu.uwa.edu.au> <20180312154343.zro2xbw7b42hwf3w@ucc.gu.uwa.edu.au> <20190317112442.thtopj6fg6me5gz3@ucc.gu.uwa.edu.au> Message-ID: On Sun, 17 Mar 2019, Nick Bannon wrote: > So, at least temporarily, it's an amber > https://terminals-wiki.org/wiki/index.php/DEC_VT320 > ...with our precious narrow keyboard. > > It has been reported that it's not working as well as the VT510, > particularly with the fish shell? but if a software fix can't be foudn > easily, a suitable keyboard for the VT510 would be the easy hardware fix. If someone can get a coredump of a hung fish process (`gcore PID` from another terminal), I can take a look. David Adam zanchey at ucc.gu.uwa.edu.au From mtearle at tearle.com Mon Mar 18 06:18:38 2019 From: mtearle at tearle.com (Mark Tearle) Date: Sun, 17 Mar 2019 18:18:38 -0400 Subject: [tech] [committee] [wheel] Wheel meeting - Monday 2019-04-08 19:00 In-Reply-To: References: Message-ID: Hi folks I'll need to send my apologies as geographically challenged ( :( ) Will try and make some more further progress on mailman before then... Mark -- Mark Tearle On Sun, 17 Mar 2019, at 8:53 AM, Felix von Perger wrote: > Hi all, > Following up from discussions over the last week or so, and to welcome one (or more) new members, I'd like to propose a gathering of the members of the Wheel Group on Monday 2019-04-08 at 19:00 AWST. For anyone who wishes to get some food beforehand, we'll probably head over to Chelsea Pizza around 17:30 or so. > To add anything to the agenda, simply edit /home/wheel/docs/meetings/2019-04-08.txt or reply to this email. > The agenda will also be visible online at https://www.ucc.asn.au/infobase/minutes/2019/2019-04-08_wheel.txt. > Best regards, > Felix von Perger [FVP] > UCC President & Wheel member > _______________________________________________ > List Archives: http://lists.ucc.asn.au/pipermail/committee -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190317/c99f6251/attachment.htm From elliotnunn at fastmail.com Mon Mar 18 15:10:08 2019 From: elliotnunn at fastmail.com (Elliot Nunn) Date: Mon, 18 Mar 2019 15:10:08 +0800 Subject: [tech] [wheel] Wheel meeting - Monday 2019-04-08 19:00 In-Reply-To: References: Message-ID: <9DA54B3A-3DAE-4953-9F81-01E845CFCD19@fastmail.com> Dear all, Please accept my apologies for this meeting. I will be down south for uni until mid April. [LE@] > On 17 Mar 2019, at 4:52 pm, Felix von Perger wrote: > > Following up from discussions over the last week or so, and to welcome one (or more) new members, I'd like to propose a gathering of the members of the Wheel Group on Monday 2019-04-08 at 19:00 AWST. For anyone who wishes to get some food beforehand, we'll probably head over to Chelsea Pizza around 17:30 or so. > > From melissa at netexperts.com.au Mon Mar 18 15:13:50 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Mon, 18 Mar 2019 15:13:50 +0800 Subject: [tech] Temperature Monitoring in Server Room [repost] Message-ID: Hi Everyone, I just realised - if you have smartmontools installed on linux machines, each hard drive or SSD will provide its ?Airflow Temperature?, which I can extract via script. I'm thinking of centralising this for all the servers I run, and collecting the data to chart, having a display at home that gives me live info for all machines under my control. I could make a similar display for UCC, which could be on the website and/or a monitor in the club room (although this would likely be in the winter holidays due to increasing workload). Note the reallocated sector count for SSDs, once this starts to happen, the drive should be replaced. For SSDs (and also HDDs) mounted at the front of servers, because they are getting airflow to the sensor sucked in directly from ambient air, and are thermally insulated from the rest of the machine, this will be equal to the temperature of the room. For example, right now, the UCC server room temperature is 29 degrees, according to 3 of the four installed drives, and 30 degrees according to the 4th one. For PCs, the same test will provide the temperature in the case. Some drives also have a count of total hours run outside of their acceptable temperature range and G/shocks or drops, as well as all types of other interesting data. If there is an interest, I could parse this data, and the page with Ashera-related information could provide it and could also e-mail (and/or SMS) warnings to anyone on the list if the temperature passes a key threshold. Here is what the data actually looks like (I've highlighted the airflow temperature): smartctl -d sat -a /dev/pass1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-STABLE amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: Samsung SSD 860 QVO 1TB Serial Number: S4CZNG0M138175F LU WWN Device Id: 5 002538 e701b1df5 Firmware Version: RVQ01B6Q User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Mar 18 15:03:46 2019 AWST SMART support is: Available - device has SMART capability. SMART support is: Enabled ... (cut to prevent this email becoming ridiculous) ... SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 648 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 15 177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0032 071 058 000 Old_age Always - 29 195 Hardware_ECC_Recovered 0x001a 200 200 000 Old_age Always - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 235 Unknown_Attribute 0x0012 099 099 000 Old_age Always - 13 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 336661820 SMART Error Log Version: 1 No Errors Logged Regards, Melissa -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190318/a7a2006b/attachment.htm From dylanh333 at ucc.asn.au Mon Mar 18 16:59:21 2019 From: dylanh333 at ucc.asn.au (Dylan H) Date: Mon, 18 Mar 2019 16:59:21 +0800 Subject: [tech] Temperature Monitoring in Server Room [repost] In-Reply-To: Message-ID: <39711d45-ce79-4cb8-a209-f3585d958d52@email.android.com> An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190318/f2698145/attachment-0001.htm From zanchey at ucc.gu.uwa.edu.au Mon Mar 18 21:58:32 2019 From: zanchey at ucc.gu.uwa.edu.au (David Adam) Date: Mon, 18 Mar 2019 21:58:32 +0800 (AWST) Subject: [tech] Temperature Monitoring in Server Room [repost] In-Reply-To: References: Message-ID: On Mon, 18 Mar 2019, Melissa Star wrote: > I just realised - if you have smartmontools installed on linux machines, > each hard drive or SSD will provide its ?Airflow Temperature?, which I > can extract via script. > > I'm thinking of centralising this for all the servers I run, and > collecting the data to chart, having a display at home that gives me > live info for all machines under my control. We used to do this on all the servers, but I think evil is the only one still running: https://ucc.asn.au/stats/ It reads the fan and temperature data from lm-sensors (run `sensors` on a bare metal machine to get an idea of what's available), plus various system statistics, and writes them into some custom RRDs. It is approximately zero fun to maintain. Collectd (https://github.com/collectd/collectd) has both SMART and lm-sensors plugins, and was the most sensible tool for our use last time I checked, so if we were going to set anything up I'd start with that. It hasn't been updated for a couple of years but is fairly mature. You don't need root access to start playing around with it. The fanciest option would be to write a Cockpit plugin (https://cockpit-project.org/ currently available at https://secure.ucc.asn.au/missioncontrol/), but we don't have the timeseries store stuff set up in that and it sounds like a lot more work. David Adam zanchey From andrew at ucc.gu.uwa.edu.au Mon Mar 18 22:54:00 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Mon, 18 Mar 2019 22:54:00 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: Message-ID: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> On 2019-03-18 9:58 PM, David Adam wrote: > On Mon, 18 Mar 2019, Melissa Star wrote: >> I just realised - if you have smartmontools installed on linux machines, >> each hard drive or SSD will provide its ?Airflow Temperature?, which I >> can extract via script. >> >> I'm thinking of centralising this for all the servers I run, and >> collecting the data to chart, having a display at home that gives me >> live info for all machines under my control. > > We used to do this on all the servers, but I think evil is the only one > still running: Rather than rolling your own temperature monitoring scripts and code to display them, I highly recommend installing Nagios/Icinga or equivalent. That will monitor network services (web, database, NTP, SSH, etc), host state, disk space, rack and internal temperatures, voltages, fan speeds, etc, on tens or hundreds of machines. Here's the Icinga2 setup for the MWA telescope - it's using a mix of built-in and third-party plugins for the sort of things you'd see in a normal server room, plus custom plugins to monitor the actual telescope hardware and software health. http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups (username 'guest', password 'mwa-guest') The performance data (raw values from every sensor or measurement) is automatically piped from icinga to a Whisper/Carbon backend, and we use Graphite to view the time series plots: http://graphite.mwa128t.org/dashboard You can either go to Dashboard/Finder and choose one of our pre-saved plot layouts (please don't change them, or save new ones), or drill down through the monitoring point tree using the top half of the page (starting with icinga2. then going down through a hostname, then a service on that host, until you reach a ....value leaf node, and add a graph showing that value to the dashboard). I usually prefer to use the Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree (left nav)'. Andrew From trs80 at ucc.gu.uwa.edu.au Tue Mar 19 11:05:29 2019 From: trs80 at ucc.gu.uwa.edu.au (James Andrewartha) Date: Tue, 19 Mar 2019 11:05:29 +0800 (AWST) Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> Message-ID: Apologies for top-posting, but per #ucc my recommendation for low-effort monitoring is LibreNMS. It is very SNMP based but there are extensions for things like SMART, and it can also get environmental data from IPMI controllers. It can export to Graphite and be viewed with Grafana if someone can be bothered, but the default webui is fine to start with. https://docs.librenms.org/Extensions/Applications/#smart https://docs.librenms.org/Extensions/Metric-Storage/ I am explicitly going to leave this for someone else to set up, my recommendation is a new VM, (although you could do a container) since it requires PHP 7.1.3 at a minimum. On Mon, 18 Mar 2019, Andrew Williams wrote: > On 2019-03-18 9:58 PM, David Adam wrote: > > On Mon, 18 Mar 2019, Melissa Star wrote: > >> I just realised - if you have smartmontools installed on linux machines, > >> each hard drive or SSD will provide its ?Airflow Temperature?, which I > >> can extract via script. > >> > >> I'm thinking of centralising this for all the servers I run, and > >> collecting the data to chart, having a display at home that gives me > >> live info for all machines under my control. > > > > We used to do this on all the servers, but I think evil is the only one > > still running: > > Rather than rolling your own temperature monitoring scripts and code to > display them, I highly recommend installing Nagios/Icinga or equivalent. > That will monitor network services (web, database, NTP, SSH, etc), host > state, disk space, rack and internal temperatures, voltages, fan speeds, > etc, on tens or hundreds of machines. > > Here's the Icinga2 setup for the MWA telescope - it's using a mix of > built-in and third-party plugins for the sort of things you'd see in a > normal server room, plus custom plugins to monitor the actual telescope > hardware and software health. > > http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups > > (username 'guest', password 'mwa-guest') > > The performance data (raw values from every sensor or measurement) is > automatically piped from icinga to a Whisper/Carbon backend, and we use > Graphite to view the time series plots: > > http://graphite.mwa128t.org/dashboard > > You can either go to Dashboard/Finder and choose one of our pre-saved > plot layouts (please don't change them, or save new ones), or drill down > through the monitoring point tree using the top half of the page > (starting with icinga2. then going down through a hostname, then a > service on that host, until you reach a ....value leaf node, and add a > graph showing that value to the dashboard). I usually prefer to use the > Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree > (left nav)'. > > Andrew > _______________________________________________ > List Archives: http://lists.ucc.asn.au/pipermail/tech > > Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/trs80%40ucc.gu.uwa.edu.au -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From melissa at netexperts.com.au Tue Mar 19 12:14:56 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 12:14:56 +0800 Subject: [tech] Temperature Monitoring in Server Room [repost] In-Reply-To: <39711d45-ce79-4cb8-a209-f3585d958d52@email.android.com> References: <39711d45-ce79-4cb8-a209-f3585d958d52@email.android.com> Message-ID: <79807713-FCF4-40F4-A120-7E72476A748F@netexperts.com.au> Hi Dylan, It depends on: * Whether the sector failure rate is increasing, particularly if the increase is marked (that indicates a drive near end of life). * How critical the data on the drive is. Generally, once sectors start to fail, all of the memory on the drive will have aged to the point that more and more will fail over time. The drive is certainly still usable for non-critical applications on a desktop machine on data that isn't financial, health related, or otherwise mission critical (meaning changes to the data will either get you sued for millions of dollars or endanger human life). The expectation is a server SSD, that may be storing financial or critical data, should look like this: 5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 15708 5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 15708 That data is from two mirrored SSDs on Nyx, one of my production servers. Drives have been running for a total of 15,700 hours or two year at 24/7/365 If that number changes from zero, I will require OVH to replace the drive in question, and have a 3AM scheduled outage while the array rebuilds. And from my 7 year old, 2012 Macbook Pro 5 Reallocated_Sector_Ct 0x0033 100 100 000 Pre-fail Always - 0 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 18196 In spite of 173 Wear_Leveling_Count 0x0032 191 191 100 Old_age Always - 304961552547 If you have a gaming PC, and you keep daily backups of your uni work, you definitely can keep using an SSD once it starts showing wear, although I'd begin to budget for its replacement, and there may be an argument for selling it on Ebay (with a warning about its reallocated sectors, if you are honest). If it's as UCC, then... I wouldn't allow it on Ashera, but the rest of the machines are up to Wheel discretion. If anyone's life, or livelihood depends on data not being corrupted, please Goddess no. Where the data on the drive is work 100x or more what the drive is worth, you don't want such scenarios. Regards, Melissa > On 18 Mar 2019, at 4:59 pm, Dylan H wrote: > > Hi Melissa, > > Can you please clarify whether it really is an issue to have any reallocated sectors at all on an SSD, even if only very few? > > My understanding is that the Raw Value is the actual number of sectors reallocated, and the Value is the normalised value, which should not reach the threshold (lower is worse, so a Value of 98 is better than 20). > Assuming this, I don't imagine a small handful of reallocated sectors is much of an issue, unless the normalised value is getting close to the threshold (0 in this case), as most SSDs come with a sizeable chunk of spare, hidden sectors to swap in, as far as I am aware. > > My main concern is that replacing an SSD that only has a small handful of reallocated sectors but a normalised value of 90+ (for example) would be a bit wasteful, when it still has 90% to go before reaching the vendor-defined threshold. > > Thanks! > > Kind regards, > Dylan Hicks [333] > > > On 18 Mar 2019 3:13 pm, Melissa Star wrote: > Hi Everyone, > > I just realised - if you have smartmontools installed on linux machines, each hard drive or SSD will provide its ?Airflow Temperature?, which I can extract via script. > > I'm thinking of centralising this for all the servers I run, and collecting the data to chart, having a display at home that gives me live info for all machines under my control. > > I could make a similar display for UCC, which could be on the website and/or a monitor in the club room (although this would likely be in the winter holidays due to increasing workload). > > Note the reallocated sector count for SSDs, once this starts to happen, the drive should be replaced. > > For SSDs (and also HDDs) mounted at the front of servers, because they are getting airflow to the sensor sucked in directly from ambient air, and are thermally insulated from the rest of the machine, this will be equal to the temperature of the room. > > For example, right now, the UCC server room temperature is 29 degrees, according to 3 of the four installed drives, and 30 degrees according to the 4th one. > > For PCs, the same test will provide the temperature in the case. Some drives also have a count of total hours run outside of their acceptable temperature range and G/shocks or drops, as well as all types of other interesting data. > > If there is an interest, I could parse this data, and the page with Ashera-related information could provide it and could also e-mail (and/or SMS) warnings to anyone on the list if the temperature passes a key threshold. > > Here is what the data actually looks like (I've highlighted the airflow temperature): > > smartctl -d sat -a /dev/pass1 > smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-STABLE amd64] (local build) > Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org > > === START OF INFORMATION SECTION === > Device Model: Samsung SSD 860 QVO 1TB > Serial Number: S4CZNG0M138175F > LU WWN Device Id: 5 002538 e701b1df5 > Firmware Version: RVQ01B6Q > User Capacity: 1,000,204,886,016 bytes [1.00 TB] > Sector Size: 512 bytes logical/physical > Rotation Rate: Solid State Device > Form Factor: 2.5 inches > Device is: Not in smartctl database [for details use: -P showall] > ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 > SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) > Local Time is: Mon Mar 18 15:03:46 2019 AWST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > ... (cut to prevent this email becoming ridiculous) ... > > SMART Attributes Data Structure revision number: 1 > Vendor Specific SMART Attributes with Thresholds: > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE > 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 > 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 648 > 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 15 > 177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0 > 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 > 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 > 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 > 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 > 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 > 190 Airflow_Temperature_Cel 0x0032 071 058 000 Old_age Always - 29 > 195 Hardware_ECC_Recovered 0x001a 200 200 000 Old_age Always - 0 > 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 > 235 Unknown_Attribute 0x0012 099 099 000 Old_age Always - 13 > 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 336661820 > > SMART Error Log Version: 1 > No Errors Logged > > > > > > > Regards, > > Melissa > -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190319/39728ea0/attachment-0001.htm From melissa at netexperts.com.au Tue Mar 19 12:20:21 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 12:20:21 +0800 Subject: [tech] Temperature Monitoring in Server Room [repost] In-Reply-To: References: Message-ID: Hi David, Smartmontools get the info in a way that is easy to parse, and by having a chron job write its output to a file that can be read by a non-privileged user, PHP doesn't need to be able to execute anything or otherwise make a hole in the security model. Writing a server status web-page, which I've said I will do for Ashera-security over the winter break, makes sense to me, and I also have a compelling commercial reason to do it since I run multiple servers that would get me in trouble if they lose data even between periodic backups, and previously relied on data centres to look after this stuff for me. If I want to move from a "dedicated server" to a "colocation" model and be responsible for my own hardware, I need to implement such tools in any case, and they need to either aggregate multiple servers, sending me email or SMS warnings, or both. What has occured to me (and thanks to UCC I am learning and growing professionally!) is that SSDs wear at fairly predictable amounts and that an SSD RAID is therefore likely to fail catastrophically once its drives have exceeded total data written limits, and if they are intel drives (mine are at OVH), they apparently go into read only mode and then self destruct at very close to to the TDW limit Regards, Melissa > On 18 Mar 2019, at 9:58 pm, David Adam wrote: > > On Mon, 18 Mar 2019, Melissa Star wrote: >> I just realised - if you have smartmontools installed on linux machines, >> each hard drive or SSD will provide its ?Airflow Temperature?, which I >> can extract via script. >> >> I'm thinking of centralising this for all the servers I run, and >> collecting the data to chart, having a display at home that gives me >> live info for all machines under my control. > > We used to do this on all the servers, but I think evil is the only one > still running: > https://ucc.asn.au/stats/ > It reads the fan and temperature data from lm-sensors (run `sensors` on a > bare metal machine to get an idea of what's available), plus various > system statistics, and writes them into some custom RRDs. It is > approximately zero fun to maintain. > > Collectd (https://github.com/collectd/collectd) has both SMART and > lm-sensors plugins, and was the most sensible tool for our use last time I > checked, so if we were going to set anything up I'd start with that. It > hasn't been updated for a couple of years but is fairly mature. You don't > need root access to start playing around with it. > > The fanciest option would be to write a Cockpit plugin > (https://cockpit-project.org/ currently available at > https://secure.ucc.asn.au/missioncontrol/), but we don't have the > timeseries store stuff set up in that and it sounds like a lot more work. > > David Adam > zanchey From melissa at netexperts.com.au Tue Mar 19 12:21:50 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 12:21:50 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> Message-ID: <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> Hi Andrew, Thanks, this looks interesting. I may try this, rather than writing my own. Or I may write my own anyway from the exercise, but use it as a point of comparison. It's important for me that the tool can both send me SMS warnings and auto-shutdown a server in extreme conditions. Regards, Melissa > On 18 Mar 2019, at 10:54 pm, Andrew Williams wrote: > > On 2019-03-18 9:58 PM, David Adam wrote: >> On Mon, 18 Mar 2019, Melissa Star wrote: >>> I just realised - if you have smartmontools installed on linux machines, >>> each hard drive or SSD will provide its ?Airflow Temperature?, which I >>> can extract via script. >>> >>> I'm thinking of centralising this for all the servers I run, and >>> collecting the data to chart, having a display at home that gives me >>> live info for all machines under my control. >> We used to do this on all the servers, but I think evil is the only one >> still running: > > Rather than rolling your own temperature monitoring scripts and code to display them, I highly recommend installing Nagios/Icinga or equivalent. That will monitor network services (web, database, NTP, SSH, etc), host state, disk space, rack and internal temperatures, voltages, fan speeds, etc, on tens or hundreds of machines. > > Here's the Icinga2 setup for the MWA telescope - it's using a mix of built-in and third-party plugins for the sort of things you'd see in a normal server room, plus custom plugins to monitor the actual telescope hardware and software health. > > http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups > > (username 'guest', password 'mwa-guest') > > The performance data (raw values from every sensor or measurement) is automatically piped from icinga to a Whisper/Carbon backend, and we use Graphite to view the time series plots: > > http://graphite.mwa128t.org/dashboard > > You can either go to Dashboard/Finder and choose one of our pre-saved plot layouts (please don't change them, or save new ones), or drill down through the monitoring point tree using the top half of the page (starting with icinga2. then going down through a hostname, then a service on that host, until you reach a ....value leaf node, and add a graph showing that value to the dashboard). I usually prefer to use the Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree (left nav)'. > > Andrew From andrew at ucc.gu.uwa.edu.au Tue Mar 19 14:22:45 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Tue, 19 Mar 2019 14:22:45 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> Message-ID: <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au> On 2019-03-19 12:21 PM, Melissa Star wrote: > Hi Andrew, > > Thanks, this looks interesting. > > I may try this, rather than writing my own. Or I may write my own anyway from the exercise, but use it as a point of comparison. > > It's important for me that the tool can both send me SMS warnings and auto-shutdown a server in extreme conditions. Yep, Icinga can send SMS and email alerts. Each contact or contact group can define what hours/days they are on-call to receive alerts, and it also handles escalation from lower-level contact groups up to higher levels if the alert isn't acknowledged in a defined time frame. We don't use any of the fancy features because there's only a few of us running things. It can also perform arbitrary actions using event scripts - restarting apache, shutting down a server, or powering down machines or a whole rack using a UPS or smart PDU's (we've got APC PDU's in each rack, and if the air temperature in any rack gets too high, icinga powers down the servers in that rack). The event scripts an do whatever you want - I wrote one that SSH's into a Raspberry Pi in our office to turn on a big orange strobe light if the correlator software crashes on site... All the monitoring is done using plugins, using the Nagios plugin API which has been around for a long time - there are hundreds of available plugins. The icinga host runs many of the plugins directly (for example, to check whether a postgres server on a remote host is alive). To measure something directly on a remote machine (like disk space) it can ask another icinga instance in the same cluster to run it (if you want to run an icinga instance on each machine), or the icinga server on one machine can SSH to each remote machine to run plugins there to test things like disk space, load average, motherboard temperatures, etc. Nagios was the original monitoring tool, but the open source version is ancient and horrible to use now - I've not used the commercial version. It was forked long ago into 'Icinga', keeping the same code and config file format, and is almost as ugly as Nagios. I'm actually using 'Icinga2', which is a complete rewrite using an entirely new configuration file format. It has many more features, and is much nicer to use. Andrew From andrew at ucc.gu.uwa.edu.au Tue Mar 19 15:55:24 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Tue, 19 Mar 2019 15:55:24 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au> Message-ID: On 2019-03-19 3:36 PM, Melissa Star wrote: > As for what UCC does, are you saying that the club should choose externally written code over the code written by a club member? > > I've had a look at Icigna2, and it seems really good, to be fair. But it would not give me the experience of interacting more directly with the hardware and writing some interesting code. > > And it would not encourage others to make the type of home-grown solutions that encourage learning. It depends what people want to learn - having experience with a widely-used server monitoring tool would be good experience for anyone wanting to get a sysadmin job in a large organisation. Working with hardware and rolling your own code is good experience for anyone wanting to be a developer. > One thought - running such a thing on a VM is a bad idea given it should run on every single machine to report its own internal state, not all machines can run VMs, and that will waste a hell of a lot of CPU and resources. You _can_ run an instance of icinga on every machine, and set them up as a cluster, but you wouldn't run each instance in a VM - that way it could only report the state of its own VM, not the host machine. In practice, I found it easier to run one instance of the icinga process and web server (which we have a VM for, but you don't need one). That process then uses SSH to run the plugins on each machine - essentially, to monitor disk space, for example, it's doing the equivalent of 'ssh foo at bar "df"' once every hour, or day, or whatever interval you specify. You need to set up a monitoring account with the icinga public SSH key on each monitored machine, to accept the SSH connections without a password. Andrew From michael at deegan.id.au Tue Mar 19 16:13:36 2019 From: michael at deegan.id.au (Michael Deegan) Date: Tue, 19 Mar 2019 16:13:36 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

Message-ID: <20190319081336.vhmm5j3ks6tgn4w2@deegan.id.au> Afternoon, On Tue, Mar 19, 2019 at 03:55:24PM +0800, Andrew Williams wrote: > > As for what UCC does, are you saying that the club should choose externally written code over the code written by a club member? > > > > I've had a look at Icigna2, and it seems really good, to be fair. But it would not give me the experience of interacting more directly with the hardware and writing some interesting code. > > > > And it would not encourage others to make the type of home-grown solutions that encourage learning. > > It depends what people want to learn - having experience with a > widely-used server monitoring tool would be good experience for anyone > wanting to get a sysadmin job in a large organisation. Working with > hardware and rolling your own code is good experience for anyone wanting > to be a developer. Agreed; if the aim is learning (or you simply have time to burn) then definitely try the DIY path. Though most people eventually reach the point where they prioritise functioning systems over the satisfaction of having done everything themselves, and if someone else has already written a tool that does the job adequately, then why not? :P Also, do try to avoid "Not Invented Here" Syndrome, which is a Thing. > > One thought - running such a thing on a VM is a bad idea given it should run on every single machine to report its own internal state, not all machines can run VMs, and that will waste a hell of a lot of CPU and resources. > > You _can_ run an instance of icinga on every machine, and set them up as > a cluster, but you wouldn't run each instance in a VM - that way it > could only report the state of its own VM, not the host machine. > > In practice, I found it easier to run one instance of the icinga process > and web server (which we have a VM for, but you don't need one). That > process then uses SSH to run the plugins on each machine - essentially, > to monitor disk space, for example, it's doing the equivalent of 'ssh > foo at bar "df"' once every hour, or day, or whatever interval you specify. > You need to set up a monitoring account with the icinga public SSH key > on each monitored machine, to accept the SSH connections without a > password. If you're looking for performance graphing, in addition to alerting, then I recommend taking a look at Munin (which is quite easy to set up, and requires a small agent installed on each monitored host), Cacti (which generally collects its data via SNMP), or MRTG. HTH, -MD -- ----------------------------------------------------------------------------- Michael Deegan Hugaholic http://www.deegan.id.au/ ------------------------ Jung, zr jbeel? ---------------------------------- From andrew at ucc.gu.uwa.edu.au Tue Mar 19 16:29:28 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Tue, 19 Mar 2019 16:29:28 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <20190319081336.vhmm5j3ks6tgn4w2@deegan.id.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

<20190319081336.vhmm5j3ks6tgn4w2@deegan.id.au> Message-ID: <436af147-dd46-c357-cf6e-430b00802d47@ucc.gu.uwa.edu.au> On 2019-03-19 4:13 PM, Michael Deegan wrote: > If you're looking for performance graphing, in addition to alerting, then I > recommend taking a look at Munin (which is quite easy to set up, and > requires a small agent installed on each monitored host), Cacti (which > generally collects its data via SNMP), or MRTG. Thanks, I hadn't heard of Munin or Cacti. I did have MRTG installed, but I swapped to 'Observium' and it is _fantastic_ for monitoring network performance on a whole bunch of switches. Much easier to set up (especially the autodiscovery), monitors a whole lot more things, and the web interface is wonderful. Andrew From dylanh333 at ucc.asn.au Tue Mar 19 16:54:52 2019 From: dylanh333 at ucc.asn.au (Dylan H) Date: Tue, 19 Mar 2019 16:54:52 +0800 Subject: [tech] Temperature Monitoring in Server Room [repost] In-Reply-To: <79807713-FCF4-40F4-A120-7E72476A748F@netexperts.com.au> Message-ID: <58182032-1bf8-43e2-beff-1f7e80605045@email.android.com> An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190319/9b6e2cf7/attachment-0001.htm From andrew at ucc.gu.uwa.edu.au Tue Mar 19 18:16:35 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Tue, 19 Mar 2019 18:16:35 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

Message-ID: <6f31ca67-1696-e3b3-14ea-8af16231ac7b@ucc.gu.uwa.edu.au> On 2019-03-19 5:08 PM, Melissa Star wrote: > So how difficult is it for icinga to call external code in response to events? As an example, here's some of our icinga config for a single host (apc54), which is the smart PDU in rack 54: # The host definition: object Host "apc54" { import "mwa-apc" address = "10.128.0.154" vars.parent_switch = "dell-mc-access04" } # Service definitions that apply to that host (and the other PDU's): apply Service "pdu_temp_monitored" { import "mwa-service" display_name = "Rack Temperature" check_command = "check_pdu_temp" check_interval = 60s retry_interval = 30s max_check_attempts = 3 event_command = "event_pdutemp" assign where "apcees" in host.groups } apply Service "pdu_currents" { import "mwa-service" display_name = "PDU plug currents" check_command = "check_pdu_currents" check_interval = 600s retry_interval = 120s max_check_attempts = 3 assign where "apcees" in host.groups } # Command definitions, as called in the above service definitions. # These commands return a couple of lines of data in the Nagios API # format, with a return code of 0 (OK), 1 (WARNING), 2 (CRITICAL) # or 3 (UNKNOWN/UNREACHABLE), plus a human readable message and the # measurement value/s. You can use existing plugins, or write your own. # This command is one that I wrote, to extract values from the PDU using # SNMP, and write Nagios API format check results to stdout. object CheckCommand "check_pdu_temp" { import "plugin-check-command" command = [ "/usr/local/bin/zeus", "-n" ] arguments = { "-H" = "$host.name$" } } object CheckCommand "check_pdu_currents" { import "plugin-check-command" command = [ "/usr/local/bin/zeus", "-p" ] arguments = { "-H" = "$host.name$" } } # The event command, defined in the check_pdutemp service. This is an # arbitrary executable, and this command definition passes in the # state (eg CRITICAL or OK), the state_type (SOFT or HARD) and # the number of times the plugin has been called and returned that state # since the last time it was 'OK'. It is called every time the service # changes state, so it's up to the command to interpret the values # passed and act accordingly. # For example, an event command might do nothing if the state changes # from something else to 'OK, restart the web server if it's # the first check that returned CRITICAL, or reboot the server if it's # the third. (I don't recommend actually doing that, BTW). object EventCommand "event_pdutemp" { import "plugin-event-command" command = "/usr/local/bin/event_pdutemp.py $host.name$ $service.state$ $service.state_type$ $service.check_attempt$" } Andrew From melissa at netexperts.com.au Tue Mar 19 15:36:44 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 15:36:44 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au> Message-ID: Hi Andrew, > Nagios was the original monitoring tool, but the open source version is ancient and horrible to use now - I've not used the commercial version. It was forked long ago into 'Icinga', keeping the same code and config file format, and is almost as ugly as Nagios. I'm actually using 'Icinga2', which is a complete rewrite using an entirely new configuration file format. It has many more features, and is much nicer to use. Thanks for letting me know about this. As it happens, though, it is probably easier for me to write code to do this one specific task - watching ambient temperature, notifying me, and shutting my server down before it gets damaged - then to learn a new package. As for what UCC does, are you saying that the club should choose externally written code over the code written by a club member? I've had a look at Icigna2, and it seems really good, to be fair. But it would not give me the experience of interacting more directly with the hardware and writing some interesting code. And it would not encourage others to make the type of home-grown solutions that encourage learning. One thought - running such a thing on a VM is a bad idea given it should run on every single machine to report its own internal state, not all machines can run VMs, and that will waste a hell of a lot of CPU and resources. One more thought, if we're running PHP56 *anywhere* we should have a plan to migrate out. Regards, Melissa From melissa at netexperts.com.au Tue Mar 19 15:52:48 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 15:52:48 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> Message-ID: <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> Hi James, > I am explicitly going to leave this for someone else to set up, my > recommendation is a new VM, (although you could do a container) since it > requires PHP 7.1.3 at a minimum. Isn't UCC a do-ocracy? I'm sure there are more standard ways, for example, to dispense drinks or maintain club membership records. And we could be using Microsoft Word or even OpenOffice to take minutes and SharePoint or perhaps OwnCloud to maintain records. Except we don't. From what I've seen so far, where possible we use solutions made by UCC members, even if they are not technically superior, because they are ours. Leaving aside the fact that: * You turned away from me and left me out of the decision making process that came out of my idea. * You suggested running system management tools in a VM because that would involve install PHP 7.x on the machine, and we've got stuff running on 5.x (note not all machines at UCC are VM hosts). * You don't mind us running PHP 5.x with all the resulting security risks Regards, Melissa > > On Mon, 18 Mar 2019, Andrew Williams wrote: > >> On 2019-03-18 9:58 PM, David Adam wrote: >>> On Mon, 18 Mar 2019, Melissa Star wrote: >>>> I just realised - if you have smartmontools installed on linux machines, >>>> each hard drive or SSD will provide its ?Airflow Temperature?, which I >>>> can extract via script. >>>> >>>> I'm thinking of centralising this for all the servers I run, and >>>> collecting the data to chart, having a display at home that gives me >>>> live info for all machines under my control. >>> >>> We used to do this on all the servers, but I think evil is the only one >>> still running: >> >> Rather than rolling your own temperature monitoring scripts and code to >> display them, I highly recommend installing Nagios/Icinga or equivalent. >> That will monitor network services (web, database, NTP, SSH, etc), host >> state, disk space, rack and internal temperatures, voltages, fan speeds, >> etc, on tens or hundreds of machines. >> >> Here's the Icinga2 setup for the MWA telescope - it's using a mix of >> built-in and third-party plugins for the sort of things you'd see in a >> normal server room, plus custom plugins to monitor the actual telescope >> hardware and software health. >> >> http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups >> >> (username 'guest', password 'mwa-guest') >> >> The performance data (raw values from every sensor or measurement) is >> automatically piped from icinga to a Whisper/Carbon backend, and we use >> Graphite to view the time series plots: >> >> http://graphite.mwa128t.org/dashboard >> >> You can either go to Dashboard/Finder and choose one of our pre-saved >> plot layouts (please don't change them, or save new ones), or drill down >> through the monitoring point tree using the top half of the page >> (starting with icinga2. then going down through a hostname, then a >> service on that host, until you reach a ....value leaf node, and add a >> graph showing that value to the dashboard). I usually prefer to use the >> Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree >> (left nav)'. >> >> Andrew >> _______________________________________________ >> List Archives: http://lists.ucc.asn.au/pipermail/tech >> >> Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/trs80%40ucc.gu.uwa.edu.au > > > -- > # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ > # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | > [ "There's nobody getting rich writing ]| -- Collect and hide your | > [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From melissa at netexperts.com.au Tue Mar 19 17:08:12 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 17:08:12 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

Message-ID: Hi, Admittedly being a developer it?s often easier for me to code my way out of a problem than to work out if a third party tool is reliable and secure. Of course, like every developer I ?assume? my code is always reliable and secure, until the horrible real world annoys my by proving otherwise. I don?t just ?want to be? a developer or a system admin, I am both and have been for years. But I do want to be a better developer and sysadmin, and having tools warn me that my servers are overheating or an SSD is starting to reallocate bad blocks at increasing speed is a good idea. In my case I have an application development framework that manages servers already but doesn?t handle this functionality. It would be a useful addition to the framework. So how difficult is it for icinga to call external code in response to events? Sent from my iPhone > On 19 Mar 2019, at 3:55 pm, Andrew Williams wrote: > >> On 2019-03-19 3:36 PM, Melissa Star wrote: >> >> As for what UCC does, are you saying that the club should choose externally written code over the code written by a club member? >> I've had a look at Icigna2, and it seems really good, to be fair. But it would not give me the experience of interacting more directly with the hardware and writing some interesting code. >> And it would not encourage others to make the type of home-grown solutions that encourage learning. > > It depends what people want to learn - having experience with a widely-used server monitoring tool would be good experience for anyone wanting to get a sysadmin job in a large organisation. Working with hardware and rolling your own code is good experience for anyone wanting to be a developer. > >> One thought - running such a thing on a VM is a bad idea given it should run on every single machine to report its own internal state, not all machines can run VMs, and that will waste a hell of a lot of CPU and resources. > > You _can_ run an instance of icinga on every machine, and set them up as a cluster, but you wouldn't run each instance in a VM - that way it could only report the state of its own VM, not the host machine. > > In practice, I found it easier to run one instance of the icinga process and web server (which we have a VM for, but you don't need one). That process then uses SSH to run the plugins on each machine - essentially, to monitor disk space, for example, it's doing the equivalent of 'ssh foo at bar "df"' once every hour, or day, or whatever interval you specify. You need to set up a monitoring account with the icinga public SSH key on each monitored machine, to accept the SSH connections without a password. > > Andrew From melissa at netexperts.com.au Tue Mar 19 17:14:35 2019 From: melissa at netexperts.com.au (Melissa Star) Date: Tue, 19 Mar 2019 17:14:35 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

Message-ID: <338961A9-0623-4908-B8DA-89F1BF5358E1@netexperts.com.au> One other thought... Passwordless ssh everywhere is very egalitarian in the sense of ?equal pwnage for all? :) I suspect many of the utility that are required for this need root access too. My idea would be to have a chron job run the utils on each machine and export raw data to json every e.g. 30 seconds so monitored machines don?t need to have ssh root access in to monitor anything. This would stop various types of exploits where security stats could reveal important info about the machine. Also the task in question would have zero external access. Sent from my iPhone > On 19 Mar 2019, at 3:55 pm, Andrew Williams wrote: > >> On 2019-03-19 3:36 PM, Melissa Star wrote: >> >> As for what UCC does, are you saying that the club should choose externally written code over the code written by a club member? >> I've had a look at Icigna2, and it seems really good, to be fair. But it would not give me the experience of interacting more directly with the hardware and writing some interesting code. >> And it would not encourage others to make the type of home-grown solutions that encourage learning. > > It depends what people want to learn - having experience with a widely-used server monitoring tool would be good experience for anyone wanting to get a sysadmin job in a large organisation. Working with hardware and rolling your own code is good experience for anyone wanting to be a developer. > >> One thought - running such a thing on a VM is a bad idea given it should run on every single machine to report its own internal state, not all machines can run VMs, and that will waste a hell of a lot of CPU and resources. > > You _can_ run an instance of icinga on every machine, and set them up as a cluster, but you wouldn't run each instance in a VM - that way it could only report the state of its own VM, not the host machine. > > In practice, I found it easier to run one instance of the icinga process and web server (which we have a VM for, but you don't need one). That process then uses SSH to run the plugins on each machine - essentially, to monitor disk space, for example, it's doing the equivalent of 'ssh foo at bar "df"' once every hour, or day, or whatever interval you specify. You need to set up a monitoring account with the icinga public SSH key on each monitored machine, to accept the SSH connections without a password. > > Andrew From susie at ucc.asn.au Tue Mar 19 20:48:12 2019 From: susie at ucc.asn.au (Susie Johnston) Date: Tue, 19 Mar 2019 20:48:12 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> Message-ID: <17A45D93-713C-4A82-8CD6-56A3F8E4B533@ucc.asn.au> Php 5 was on evil which actually isn?t an UCC machine. I?ll poke Matt or Alwyn about upgrading it (it is their machine). I saw motsugo was on an earlier version of php 7. I haven?t done investigations past that. Susie > On 19 Mar 2019, at 3:52 pm, Melissa Star wrote: > > Hi James, > >> I am explicitly going to leave this for someone else to set up, my >> recommendation is a new VM, (although you could do a container) since it >> requires PHP 7.1.3 at a minimum. > > Isn't UCC a do-ocracy? > > I'm sure there are more standard ways, for example, to dispense drinks or maintain club membership records. > > And we could be using Microsoft Word or even OpenOffice to take minutes and SharePoint or perhaps OwnCloud to maintain records. > > Except we don't. From what I've seen so far, where possible we use solutions made by UCC members, even if they are not technically superior, because they are ours. > > Leaving aside the fact that: > > * You turned away from me and left me out of the decision making process that came out of my idea. > > * You suggested running system management tools in a VM because that would involve install PHP 7.x on the machine, and we've got stuff running on 5.x (note not all machines at UCC are VM hosts). > > * You don't mind us running PHP 5.x with all the resulting security risks > > > Regards, > > Melissa > > > > >> >>> On Mon, 18 Mar 2019, Andrew Williams wrote: >>> >>>> On 2019-03-18 9:58 PM, David Adam wrote: >>>>> On Mon, 18 Mar 2019, Melissa Star wrote: >>>>> I just realised - if you have smartmontools installed on linux machines, >>>>> each hard drive or SSD will provide its ?Airflow Temperature?, which I >>>>> can extract via script. >>>>> >>>>> I'm thinking of centralising this for all the servers I run, and >>>>> collecting the data to chart, having a display at home that gives me >>>>> live info for all machines under my control. >>>> >>>> We used to do this on all the servers, but I think evil is the only one >>>> still running: >>> >>> Rather than rolling your own temperature monitoring scripts and code to >>> display them, I highly recommend installing Nagios/Icinga or equivalent. >>> That will monitor network services (web, database, NTP, SSH, etc), host >>> state, disk space, rack and internal temperatures, voltages, fan speeds, >>> etc, on tens or hundreds of machines. >>> >>> Here's the Icinga2 setup for the MWA telescope - it's using a mix of >>> built-in and third-party plugins for the sort of things you'd see in a >>> normal server room, plus custom plugins to monitor the actual telescope >>> hardware and software health. >>> >>> http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups >>> >>> (username 'guest', password 'mwa-guest') >>> >>> The performance data (raw values from every sensor or measurement) is >>> automatically piped from icinga to a Whisper/Carbon backend, and we use >>> Graphite to view the time series plots: >>> >>> http://graphite.mwa128t.org/dashboard >>> >>> You can either go to Dashboard/Finder and choose one of our pre-saved >>> plot layouts (please don't change them, or save new ones), or drill down >>> through the monitoring point tree using the top half of the page >>> (starting with icinga2. then going down through a hostname, then a >>> service on that host, until you reach a ....value leaf node, and add a >>> graph showing that value to the dashboard). I usually prefer to use the >>> Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree >>> (left nav)'. >>> >>> Andrew >>> _______________________________________________ >>> List Archives: http://lists.ucc.asn.au/pipermail/tech >>> >>> Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/trs80%40ucc.gu.uwa.edu.au >> >> >> -- >> # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ >> # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | >> [ "There's nobody getting rich writing ]| -- Collect and hide your | >> [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / > > _______________________________________________ > List Archives: http://lists.ucc.asn.au/pipermail/tech > > Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/susie%40ucc.gu.uwa.edu.au From susie at ucc.asn.au Tue Mar 19 20:57:52 2019 From: susie at ucc.asn.au (Susie Johnston) Date: Tue, 19 Mar 2019 20:57:52 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> Message-ID: <05A4B516-4006-4DE1-8060-84F5FD07FDF1@ucc.asn.au> Also, there is no reason we can?t have both member-made and industry standard solutions running at the same time. That?s how new solutions come about :) E.g. dropbear (Or on a tiny scale my weather chart I wrote many years ago that i still find useful http://susie.ucc.asn.au/weather/ ) > On 19 Mar 2019, at 3:52 pm, Melissa Star wrote: > > Hi James, > >> I am explicitly going to leave this for someone else to set up, my >> recommendation is a new VM, (although you could do a container) since it >> requires PHP 7.1.3 at a minimum. > > Isn't UCC a do-ocracy? > > I'm sure there are more standard ways, for example, to dispense drinks or maintain club membership records. > > And we could be using Microsoft Word or even OpenOffice to take minutes and SharePoint or perhaps OwnCloud to maintain records. > > Except we don't. From what I've seen so far, where possible we use solutions made by UCC members, even if they are not technically superior, because they are ours. > > Leaving aside the fact that: > > * You turned away from me and left me out of the decision making process that came out of my idea. > > * You suggested running system management tools in a VM because that would involve install PHP 7.x on the machine, and we've got stuff running on 5.x (note not all machines at UCC are VM hosts). > > * You don't mind us running PHP 5.x with all the resulting security risks > > > Regards, > > Melissa > > > > >> >>> On Mon, 18 Mar 2019, Andrew Williams wrote: >>> >>>> On 2019-03-18 9:58 PM, David Adam wrote: >>>>> On Mon, 18 Mar 2019, Melissa Star wrote: >>>>> I just realised - if you have smartmontools installed on linux machines, >>>>> each hard drive or SSD will provide its ?Airflow Temperature?, which I >>>>> can extract via script. >>>>> >>>>> I'm thinking of centralising this for all the servers I run, and >>>>> collecting the data to chart, having a display at home that gives me >>>>> live info for all machines under my control. >>>> >>>> We used to do this on all the servers, but I think evil is the only one >>>> still running: >>> >>> Rather than rolling your own temperature monitoring scripts and code to >>> display them, I highly recommend installing Nagios/Icinga or equivalent. >>> That will monitor network services (web, database, NTP, SSH, etc), host >>> state, disk space, rack and internal temperatures, voltages, fan speeds, >>> etc, on tens or hundreds of machines. >>> >>> Here's the Icinga2 setup for the MWA telescope - it's using a mix of >>> built-in and third-party plugins for the sort of things you'd see in a >>> normal server room, plus custom plugins to monitor the actual telescope >>> hardware and software health. >>> >>> http://icinga.mwa128t.org/icingaweb2/monitoring/list/hostgroups >>> >>> (username 'guest', password 'mwa-guest') >>> >>> The performance data (raw values from every sensor or measurement) is >>> automatically piped from icinga to a Whisper/Carbon backend, and we use >>> Graphite to view the time series plots: >>> >>> http://graphite.mwa128t.org/dashboard >>> >>> You can either go to Dashboard/Finder and choose one of our pre-saved >>> plot layouts (please don't change them, or save new ones), or drill down >>> through the monitoring point tree using the top half of the page >>> (starting with icinga2. then going down through a hostname, then a >>> service on that host, until you reach a ....value leaf node, and add a >>> graph showing that value to the dashboard). I usually prefer to use the >>> Tree interface instead - go to Dashboard/Configure UI, then choose 'Tree >>> (left nav)'. >>> >>> Andrew >>> _______________________________________________ >>> List Archives: http://lists.ucc.asn.au/pipermail/tech >>> >>> Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/trs80%40ucc.gu.uwa.edu.au >> >> >> -- >> # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ >> # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | >> [ "There's nobody getting rich writing ]| -- Collect and hide your | >> [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / > > _______________________________________________ > List Archives: http://lists.ucc.asn.au/pipermail/tech > > Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/susie%40ucc.gu.uwa.edu.au -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190319/d072dc10/attachment.htm From frekk at ucc.asn.au Tue Mar 19 22:46:18 2019 From: frekk at ucc.asn.au (Felix von Perger) Date: Tue, 19 Mar 2019 22:46:18 +0800 Subject: [tech] Parts to purchase Message-ID: Hi all, Regarding the various budgets from the 2018 committee relating to various computer-related components, I've compiled a list of all the various things we want to buy and a suggestion of where they should be purchased. Many of these quotes were provided by [MSX] who is able to purchase these at wholesale prices. * 2 x 144hz monitors o Budget: $500 AUD each ($1000 total), see minutes here o Suggested purchase: *AOC 24.5" 1ms 144Hz Full HD FreeSync Frameless Gaming Monitor - G2590PX* [MSX: $300 + GST] * 2 x 1TB M.2 SSDs (for Christmas and Catfish) o Budget: $300 per SSD, see minutes here o Suggested purchase: *SAMSUG EVO M.2 - 1TB*[MSX: $189 from /Wholesaler 2/] o Already purchased 2019-03-12 * 3 x 1TB SATA SSDs (for Corvo, Cichlid & Corydoras) o Budget: $300 per SSD, see minutes here o Suggested purchase: *SAMSUNG 860 EVO SATA - 1TB *[MSX: $189 from /Wholesaler 2/] * 6 x 1TB SATA SSDs (for upgrading ceph cluster in proxmox hosts, 2 SSDs per host) o Budget: $250 per SSD, see minutes here o Suggested purchase: *SAMSUNG 860 EVO SATA - 1TB (?)* o maybe an idea to get something more enterprise-grade? Replacement parts for various failing hardware, see minutes here * dead disks in molmol: o Budget: $600 for some super-fancy PCIe SSD thing to be used as a journal drive, such as this o Budget: $100 for a ~250GB SSD to replace the failed system disk (ie. Samsung 860 PRO/EVO) * failing / dead disks in motsugo: o note, we already have one spare WD Black 2TB on the rack, 2 disks are failing o Budget: $190 for an additional WD Black 2TB Please let me know what bits people are able to buy and when so we don't accidentally end up buying multiple lots of things. Best regards, [FVP] -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190319/3aae8c86/attachment-0001.htm From trs80 at ucc.gu.uwa.edu.au Tue Mar 19 22:47:50 2019 From: trs80 at ucc.gu.uwa.edu.au (James Andrewartha) Date: Tue, 19 Mar 2019 22:47:50 +0800 (AWST) Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <436af147-dd46-c357-cf6e-430b00802d47@ucc.gu.uwa.edu.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

<20190319081336.vhmm5j3ks6tgn4w2@deegan.id.au> <436af147-dd46-c357-cf6e-430b00802d47@ucc.gu.uwa.edu.au> Message-ID: On Tue, 19 Mar 2019, Andrew Williams wrote: > On 2019-03-19 4:13 PM, Michael Deegan wrote: > > > If you're looking for performance graphing, in addition to alerting, then I > > recommend taking a look at Munin (which is quite easy to set up, and > > requires a small agent installed on each monitored host), Cacti (which > > generally collects its data via SNMP), or MRTG. > > Thanks, I hadn't heard of Munin or Cacti. > > I did have MRTG installed, but I swapped to 'Observium' and it is > _fantastic_ for monitoring network performance on a whole bunch of > switches. Much easier to set up (especially the autodiscovery), monitors > a whole lot more things, and the web interface is wonderful. LibreNMS is a fork of the last GPL version of Observium with a much less toxic maintenance team. Nobody should be using Cacti in this day and age. -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From trs80 at ucc.gu.uwa.edu.au Tue Mar 19 23:06:01 2019 From: trs80 at ucc.gu.uwa.edu.au (James Andrewartha) Date: Tue, 19 Mar 2019 23:06:01 +0800 (AWST) Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <565B1BFA-507A-4FF3-BB6E-579834AD4AA2@netexperts.com.au> Message-ID: Hi Melissa, On Tue, 19 Mar 2019, Melissa Star wrote: > Hi James, > > > I am explicitly going to leave this for someone else to set up, my > > recommendation is a new VM, (although you could do a container) since it > > requires PHP 7.1.3 at a minimum. > > Isn't UCC a do-ocracy? Sure. Which is why I said it was my recommendation, and not something I was actually going to do myself. It is far better for a non-old-guard member to do this and take ownership of it. What form it takes is up to whoever actually does it, I was bringing my recommendation from Discord/IRC to the lists for posterity, since not everyone reads both. > I'm sure there are more standard ways, for example, to dispense drinks or maintain club membership records. > > And we could be using Microsoft Word or even OpenOffice to take minutes and SharePoint or perhaps OwnCloud to maintain records. > > Except we don't. From what I've seen so far, where possible we use solutions made by UCC members, even if they are not technically superior, because they are ours. We do whatever is done. Please do not let my suggestions act as stop energy. > Leaving aside the fact that: > > * You turned away from me and left me out of the decision making process that came out of my idea. I will grant this, although I was responding more to Andrew's recommendation of Icinga than your suggestion of DIYing. > * You suggested running system management tools in a VM because that would involve install PHP 7.x on the machine, and we've got stuff running on 5.x (note not all machines at UCC are VM hosts). The LibreNMS server only runs on one machine (well in an environment the size of UCC). All the communication with hosts is done via SNMP or IPMI - we just need to install snmpd on the various servers we want to monitor. Switches already have SNMP, and most of our physical servers have IPMI. I suggest a VM for the server alone because UCC runs Debian in general, and there is no stable version of Debian with PHP >= 7.1.3, and so requires intalling from a third-party repository, which is best done in the isolation of a VM. > * You don't mind us running PHP 5.x with all the resulting security risks Debian stretch (the current stable) has 7.0, but not 7.1. Thanks, -- # TRS-80 trs80(a)ucc.gu.uwa.edu.au #/ "Otherwise Bub here will do \ # UCC Wheel Member http://trs80.ucc.asn.au/ #| what squirrels do best | [ "There's nobody getting rich writing ]| -- Collect and hide your | [ software that I know of" -- Bill Gates, 1980 ]\ nuts." -- Acid Reflux #231 / From andrew at ucc.gu.uwa.edu.au Tue Mar 19 23:12:11 2019 From: andrew at ucc.gu.uwa.edu.au (Andrew Williams) Date: Tue, 19 Mar 2019 23:12:11 +0800 Subject: [tech] [committee] Temperature Monitoring in Server Room [repost] In-Reply-To: References: <3497ba42-1d66-4ca8-df64-350e9d79d493@ucc.gu.uwa.edu.au> <051431EC-EE16-4504-8BF9-15E7AF5DB51B@netexperts.com.au> <3da7cd60-85b7-17d2-105f-bc903459cb29@ucc.gu.uwa.edu.au>

<20190319081336.vhmm5j3ks6tgn4w2@deegan.id.au> <436af147-dd46-c357-cf6e-430b00802d47@ucc.gu.uwa.edu.au> Message-ID: On 2019-03-19 10:47 PM, James Andrewartha wrote: > LibreNMS is a fork of the last GPL version of Observium with a much less > toxic maintenance team. Nobody should be using Cacti in this day and age. Interesting - looks like a cross between Observium CE (which, as far as I can see, only monitors network switches?), and Icinga. I can't see it's worth the effort in swapping over from a working setup, but I'll keep it in mind. The mapping elements of Observium and LibreNMS are cool, but for us, it'd be a really tight cluster of points in Tech Park, an 800km line on the map, and a really tight cluster in the Murchison... Andrew From alexjthat at gmail.com Fri Mar 22 18:18:14 2019 From: alexjthat at gmail.com (Alex Thatcher) Date: Fri, 22 Mar 2019 18:18:14 +0800 Subject: [tech] Christmas/Catfish: Installation of new SSDs, reinstall of windows Message-ID: Hello tech@, Just letting you know that I installed the two new 1TB M.2 SSDs committee purchased into Christmas and Catfish as system disks, re-installing windows as a result. I've set up partitions as follows for both, feel free to suggest adjustments: SSD: ~500GB system partition, ~500GB games/other partition HDD: ~1TB games partition The old 500GB SSD for Christmas has been placed in the machine room, not formatted. I've followed https://wiki.ucc.asn.au/StandardOperatingEnvironment , specifically everything under the windows 7 section and the chocolatey/regedit in the windows 10 section. A few things in chocolately failed to install, but they should be able to be installed manually as required. Please let me know if any issues occur. Regards, Alex [AJT] -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190322/9837b7d0/attachment.htm From frenchie at ucc.gu.uwa.edu.au Sun Mar 24 18:26:37 2019 From: frenchie at ucc.gu.uwa.edu.au (James French) Date: Sun, 24 Mar 2019 18:26:37 +0800 Subject: [tech] SSSD Kerberos Failures on Mooneye Message-ID: Evening All, While looking into a mail forwarding issue, I've picked up that SSSD/kerberos seems to be fairly unhappy on mooneye at the moment. We're seeing the following error in daemon.log every 5-10 minutes: mooneye [sssd[krb5_child[9901]]]: Preauthentication failed Everything seems to be working ok otherwise, but it's not making me feel warm and fuzzy seeing regular kerberos issues on our main mail server. I'm not currently across the AD auth well enough to want to go poking too deeply into what's going on lest I break things so does anyone more familiar with the setup have any ideas? A quick survey around out other machines, we're seeing the same error logged infrequently (<10 times a day) on a number of our other Linux machines. None as frequently as mooneye however. It may just be noise, but worth looking at. Regards, F. -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190324/2804fba0/attachment.htm From jimbo at ucc.gu.uwa.edu.au Tue Mar 26 15:00:15 2019 From: jimbo at ucc.gu.uwa.edu.au (jimbo) Date: Tue, 26 Mar 2019 15:00:15 +0800 Subject: [tech] Possible dead clubroom network port Message-ID: <781-5c99ce00-d-51134c00@80823771> Hi tech, Earlier today, [DAS] noted that Christmas wouldn't allow sign-ins to network accounts and didn't show an internet connection. We investigated and determined that it worked while plugged into a different clubroom network port, and other computers wouldn't work when plugged into Christmas's port. I have not investigated much further but it appears port 11 is not functioning (at least for IPv4). I have left Christmas plugged into port 10. Port 10's previous occupier, the SW clubroom webcam which was currently non-functional anyway, has been left unplugged. Cheers, James [MPT] ? -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190326/5a60a718/attachment.htm From bob at ucc.gu.uwa.edu.au Tue Mar 26 18:57:14 2019 From: bob at ucc.gu.uwa.edu.au (Andrew Adamson) Date: Tue, 26 Mar 2019 18:57:14 +0800 (AWST) Subject: [tech] New-Undead-Maltair Message-ID: Hi All, Maltair is back up and running again on an HP DL380 Gen 8 which the club bought at auction. Specs are: Dual E5-2690 @ 2.90GHz 196GB RAM Dual PSU disks from old maltair...two 120GB SSD's mirrored as system disks, plus a 500G ceph disk We got bitten by the onboard HP P420i raid controller, which supports JBOD/HBA mode after some hacking (that's Just a Boring Old Disk passthrough for those who don't know), but doesn't let you boot from a non-raided disk. Eugh, how enterprisey. The solution was to grab the MegaRAID SAS 2008 card out of the old IBM x3550 new-dead-maltair and put it in to new-undead-maltair. The card is motherboard mounted in the IBM, but still uses a standard PCIe slot - I happened to find a PCI mounting bracket that fit it on the machine room shelf, so it all sits nicely in the HP case. One notable problem I had is that the SAS backplane cables in the HP could not reach the new card location, so I had to steal a cable out of old maltair. New-undead-maltair needs two - new-dead-maltair only had one that was long enough, so only disks 1-4 (the left bays) are actually connected to anything. The RIGHT FOUR BAYS ARE CURRENTLY DISCONNECTED. Perhaps someone with a little time could get another long cable out of the other dead x3550 and install it. The machine booted straight away with the new SAS card, but network didn't work because the interface names were different. Back in my day I would have edited /etc/udev/rules.d/70-persistent-net.rules to assign an interface name (like eth0) to a particular port...this doesn't seem to be the case anymore, I tried it and basically nothing happened *shakes fist at systemd*. Anyway, eno1 and eno2 seem to be the new hotness and I'm disinclined to fight that, so I just updated /etc/network/interfaces with the new name and off it went. Andrew Adamson bob at ucc.asn.au |"If you can't beat them, join them, and then beat them." | | ---Peter's Laws | From jimbo at ucc.asn.au Wed Mar 27 12:50:32 2019 From: jimbo at ucc.asn.au (James Arcus) Date: Wed, 27 Mar 2019 12:50:32 +0800 Subject: [tech] merlo losing /home Message-ID: <38d861b1-2a77-1f75-03c2-6cac08479242@ucc.asn.au> Hi tech, wheel, At some point last night (2019-03-26/27), it appears that merlo lost its mount of /home. I discovered the issue at 12:35 today when I was unable to log in via pin to the vending machine but card login still worked. As such the latest transactions have been written to a local copy of /home/other/cokelog rather than the network share. The last transaction logged to the network is: ??? Mar 26 21:04:09 merlo odispense2: refund 'nullc0ke' (coke:0) to frekk by frekk [cost 94, balance 1144] and the first logged locally to merlo is: ??? Mar 27 08:53:31 merlo odispense2: dispense 'door' (door:0) for coffee by root [cost 0, balance 100] The local merlo log also reveals there are a number of other transactions from previous instances of this happening. I have remounted /home on merlo but have yet to investigate further. Cheers, James [MPT] From oxinabox at ucc.asn.au Sat Mar 30 20:43:55 2019 From: oxinabox at ucc.asn.au (Lyndon White) Date: Sat, 30 Mar 2019 12:43:55 +0000 Subject: [tech] [wheel] Wheel meeting - Monday 2019-04-08 19:00 apologies In-Reply-To: <9DA54B3A-3DAE-4953-9F81-01E845CFCD19@fastmail.com> Message-ID: An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190330/ba9fbf5c/attachment.htm From dylanh333 at ucc.asn.au Sun Mar 31 08:56:50 2019 From: dylanh333 at ucc.asn.au (Dylan H) Date: Sun, 31 Mar 2019 08:56:50 +0800 Subject: [tech] [committee] [wheel] Wheel meeting - Monday 2019-04-08 19:00 In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190331/ea887751/attachment.htm From jimbo at ucc.gu.uwa.edu.au Sun Mar 31 12:39:14 2019 From: jimbo at ucc.gu.uwa.edu.au (jimbo) Date: Sun, 31 Mar 2019 12:39:14 +0800 Subject: [tech] New software installed on Motsugo Message-ID: <782-5ca04480-7-21986c80@198980464> Hi tech, Just letting everyone know I installed Emacs 25 on Motsugo on Friday. This is the latest version available in the current Debian stable repositories. As such, version 25 is the new default. If you wish to still use version 24, it remains installed and can be invoked by the name emacs24. Cheers, James [MPT] -------------- next part -------------- An HTML attachment was scrubbed... URL: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20190331/57b2e6f7/attachment.htm