[tech] Mussel/Apache changes as of 2025-04-15

[Roy Xu]

Hi all,

This is a brief update about the changes on Mussel’s Apache server. Due to intensive crawlers and botnet attackers (running dir scan and exploit, etc.), our Apache server has stopped working quite a few times recently. I’ve made some changes as part of the countermeasures, also as an improvement to the web server.

Firstly, I changed the mpm mode from mpm_prefork to mpm_event. Event is now the default mpm option for the current Apache version (https://httpd.apache.org/docs/2.4/mpm.html), and it should be more tolerable to the high-frequency crawling issue. The connection parameters specified by mods-available/mpm_event.conf are not tuned for the hardware capability and workload yet.

As a result of switching to threaded working mode, I had to move from mod_php to php_fpm (php7.3_fpm), which doesn't allow PHP directives in Apache config files used in some of our sites’ conf files. [MSH] fixed hg.conf, I think secure.conf is still broken but it should be retired? Please let me know if there’s any other website that behaves incorrectly.

I also changed the ServerTokens to Prod and ServerSignatures to Off, for hiding the server software version info. We shouldn’t rely on obfuscation for security, but I think it’s best not to broadcast our software’s version to the internet.

Lastly, briefly mentioning here, some of us are also investigating other mitigations for the AI bots. We’ve tweaked the robots.txt for those that behave well, and currently looking into using Cloudflare’s anti-AI features for the rest, such as AI Labyrinth.

Mussel needs a rebuild, we might as well explore other modern options for security and performance when we do it.

Cheers,
[ROY]