You are currently browsing the Maeyanie.com weblog archives for August, 2009.

22 Aug 2009

Blocklist Merging on Linux


A little while back, I wrote a post about using iptables PeerGuardian blocklists efficiently. However, that program only uses a single list; it expects it to be pre-processed by another program.

Originally I used Bluetack’s Blocklist Manager. It’s quite good, but it’s Windows-only, slow, and eats gobs of memory. (Seriously, 100+ MB, and 24/7 if you want it to auto-update.)

Today, I got bored, and wrote my own as a tiny command-line program dubbed BLM.

It only merges the blocklists, though I included a couple of scripts to show downloading them with wget automagically too. Also, it’s designed to output the merged list in PeerGuardian format to stdout, which works very nicely with my pg2ipset utility from that post I linked above.

My suggestion is to make a file with a bunch of URLs of blocklists in .gz format (the .tar.bz2 includes a list of the Bluetack ones) then add a script something like this to your crontab or /etc/cron.daily:

cd /opt/blocklist
wget --timestamping `grep -v ^# urls.txt`
zcat *.gz | ./blm | ./pg2ipset | ipset -R

Modifying this to your personal paths and needs, as always. :)