[tech] webcam archives, molmol

[Nick Bannon]

While poking mollitz, it's become clear that the webcam archives (mostly
on molmol) have become a bit unmanageable. Could we have a hand finishing
the cleanup?

  * cleaning up /home/other/webcam and /home/other/webcam/public-html
    and uploading it to https://gitlab.ucc.asn.au/ with an appropriate
    license

  * the webcam archives used to be stored on mylah and had to
    be recovered from a broken filesystem: there's a terabyte of
    motsugo:/webcamrestore and a few handfuls of "bad" files in
    molmol:/space/services/webcam/archive

  * it takes molmol tens of hours to "find /space", let alone do daily
    backups. There's tens of millions of camera/month/day/hour/minute.jpg
    files in the archive , so the metadata doesn't fit in RAM anymore.
    [TPG]'s added an option to improve that, see below.

  * molmol has a drive failure, one of the SSD log mirrors:
    molmol$ zpool status|grep -C4 DEGRADED
  pool: space
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 28h16m with 0 errors on Fri Nov 13 08:12:17 2020
config:

        NAME                             STATE     READ WRITE CKSUM
        space                            DEGRADED     0     0     0
[...]
        logs
          mirror-4                       DEGRADED     0     0     0
            ada0p3                       ONLINE       0     0     0
            3087349144323640050          UNAVAIL      0     0     0  was /dev/gpt/molmol-slog0

  * we tried AVI/MPEG-encoding daily webcam files but the compression
    rates were rather disappointing. Seems better to just zip them
    up so they taken fewer inodes, and it's easy to undo if we need
    to change/rearrange something. Quick curl(1) benchmarks show that
    access is still pretty fast, though I'd like to compare accessing
    files at the start and end of a zip file.

  * and so... further cleanup might interact with a neat update [TPG]
    just did to /home/other/webcam/public-html/archive.php

diff --git a/archive.php b/archive.php
index bbdd5e1..e646d09 100644
--- a/archive.php
+++ b/archive.php
@@ -53,6 +53,36 @@
                        passthru($cmd);
                        exit();
                }
+
+               $zipfile_day = $month_folder.$timestamp->format("Ymd").".zip";
+               if( is_readable($zipfile_day) ) {
+                       $zip = new ZipArchive;
+                       $zip->open($zipfile_day);
+                       $iname = $timestamp->format("d/H/i").".jpg";
+                       $c = $zip->getFromName($iname);
+                       if($c !== false) {
+                               header("Content-Type: image/jpeg");
+                               header("Content-Length: ".strlen($c));
+                               header("Last-Modified: ".gmdate("D, d M Y H:i:s", filectime($zipfile_day))." GMT");
+                               echo $c;
+                               exit();
+                       }
+               }
+
+               $zipfile_month = "archive/" . $camera . "/" .  $timestamp->format("Ym").".zip";
+               if( is_readable($zipfile_month) ) {
+                       $zip = new ZipArchive;
+                       $zip->open($zipfile_month);
+                       $c = $zip->getFromName($timestamp->format("Ym/d/H/i").".jpg");
+                       if($c !== false) {

  * so we could use a "check-and-zipit" script that could cd to
    "/space/services/webcam/archive/ipcamera*" and:
    $ zip -mr0 20xxyy.zip 20xxyy
    ...after checking that the only contents of 20xxyy/ are valid .jpg files

    * in some cases, the old recovered "files" have been directories or
      even multi-gigabyte files of nulls. Most valid JPEG files have
      been <100K, though there's some poorly-encoded files of static
      that are larger. Attached is a shortlist to check:
      molmol$ find /space/services/webcam/archive -size +190k -print -ls |gzip -9v > /space/scratch/nick/webcam-190KB.files.ls.gz

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal
-------------- next part --------------
A non-text attachment was scrubbed...
Name: webcam-190KB.files.ls.gz
Type: application/gzip
Size: 22361 bytes
Desc: not available
URL: <https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20201114/f51c647a/attachment-0001.gz>