DaveDevelopment::Dave Marshall

Well then, a few months down the road and I’ve hopefully made a few more improvements to DaveProxy.

First off is some modifications to the CGIProxy script, thanks to the guys over at Proxy Society. It basically allows me to control which type of files can be accessed through the proxy, based on the file extension.

I did have Squid running as a httpd accelerator for a while, but in the end I decided that the benefits were not really worth the hassle. On the other hand, I can see where this type of setup could prove very useful, particularly with other CGI scripts, just not proxy one’s.

I also started rotating the cgi-bin folder by adding a random number to the end, changing all the configuration scripts and the index page of the site every two hours. This certainly stopped a lot of hotlinking, but stopped people from adding favourites through the site and also any incoming traffic from Proxy.org. I also tried using apache to deny any requests without referers, this worked well for a short time…

The site had some major downtime recently and although I’m not entirely sure I think I figured out what happened. About three days ago, I had a massive influx of traffic, which ended with me serving thousands of 403 denied requests, which culminated in the proxy not being able to handle any real requests because apache and the proxy itself were firing that many 403’s back. I personally think these requests were mainly coming from China. Pages like this providing links to FreeBSD and not to mention all the forum pages hotlinking porn, are just abusing the site and it’s not fair on me or the genuine users. First I thought about geotargetting and deny requests, but that wouldn’t help anyway. A 403 might cost me 1kB, which doesn’t sound like much, but times that by thousands and my server would be crippled again. So why not drop it before it gets to Apache? A quick google search not only led me to sites listing chinese and korean ‘A’ class network ranges, but also to iptables scripts set to drop all traffic for them. Changed the port from SMTP to HTTP and I was in business. Even implemented this at work for our SMTP server, cut the spam in half!

Add to that some more links, a news page, a links page and a little bit of promotion here and there and that’s DaveProxy as of now.

Just spent an hour or so tweaking the server running DaveProxy, mainly the introduction of mod_perl and a couple of little tweaks to Apache. Both together it should help keep it up during the worst periods of traffic.

I decided I’d take this course of action rather than moving over to PHProxy, although CGIProxy is very heavy on resources, using mod_perl should cut out the perl interpreter instantiation for every request and also allows easier monitoring using Monit. While PHProxy is better on the resources side of things, CGIProxy handles more features.

Depending on how well it goes over the next week, I may introduce squid as a cache between Apache and clients.