[tech] Referrer Checking for Hotlinking

[Davyd Madeley]

As an idea to try and cut down our traffic bill some more (or web
backtraffic is ~1/11th of our outbound traffic) I knocked up a script to
analyse our referrers to look for hotlinking.

This script is ~davyd/src/ucc/log-parse.py

The output of this analysis for the last month or so is at:
    http://davyd.ucc.asn.au/ucc-test/ucc-referrers.txt (warning 2MB)

The first half is ordered by referrer (to look for chronic hotlinkers),
the second half is ordered by request (to look for popular items). From
what I've seen of the log, things aren't incredibly bad.

Obviously there is no blanket solution for dealing with hotlinking, the
best we can hope for is to identify chronic problems (eg. randoms
linking to images from the UCC on myspace) and do image substitution.
Perhaps offer something standard people can add to their .htaccess files
that references a "why hotlinking is bad" page, and an explanation to
help them fix it (I've always assumed that most hotlinking is out of
naivity rather than being malicious).

--d