Efficient IPTables PeerGuardian Blocklist

Mae's Days

I like being watched as little as the next girl. But so far, the only Linux-based automatic PeerGuardian blocklist updater I’ve seen has been horribly inefficient, using over 200,000 individual iptables rules. It takes forever to parse, and can seriously slow down a low-end system.

I thought to myself, “here has to be a better way.” And, of course… there is. Here’s how.

The trick to speeding up lookups in a huge list is to not just search the whole list. Hashes and trees are commonly used, but iptables doesn’t really have either by default. It has a few small hashes, but they can only store single IPs. The collected Bluetack blocklist is some 2.2 billion IPs; if they’re entered individually this might cause some memory issues.

But there is a handy extension which works very nicely for this, IPSet. You can download it from http://ipset.netfilter.org/install.html. It installed and loaded with no problems or kernel recompiles or anything ugly like that on my system.

It has a handy save/restore feature, similar to iptables. So all we need to do is convert a PG list into one of its files.
I wrote a quick program to do just this, dubbed pg2ipset (link fixed now, oops!)
It’s very simple, uses no libraries, so you can compile it with something like gcc -O3 -o pg2ipset pg2ipset.c

One of the things I love about Linux (or UNIX in general) is pipes. So you can do something like this:
curl -L http://www.bluetack.co.uk/config/level1.gz | gunzip -c | pg2ipset - - LEVEL1 | ipset -R
That line will load the level 1 PG blocklist into an IPSet called “LEVEL1”. You can then add iptables rules like this:
iptables -A INPUT -m set --set LEVEL1 src -j DROP
iptables -A FORWARD -m set --set LEVEL1 src -j DROP
iptables -A FORWARD -m set --set LEVEL1 dst -j REJECT
iptables -A OUTPUT -m set --set LEVEL1 dst -j REJECT

This will cause anything a blocked IP sends you to mysteriously vanish as if you weren’t even there, while any of your programs which try to connect to a blocked IP are informed it won’t work. Adjust to your liking, as always; if you have ipt_TARPIT, this would be a great place.

To update the list, you can have a cron job do something like this every day or so:
curl -L http://www.bluetack.co.uk/config/level1.gz | gunzip -c | pg2ipset - - LEVEL1-NEW | ipset -R
ipset -X LEVEL1-NEW

This loads the blocklist to “LEVEL1-NEW”, swaps it with “LEVEL1” once it’s done loading, then deletes the old list, leaving no unprotected gap in the middle.

It’s fairly quick, even on the ancient 800 MHz Pentium III which I use for a router. I haven’t tested it extensively, so if anyone has trouble, please leave a comment and I’ll try to help.

This entry was posted on Thursday, December 18th, 2008 at 9:08 AM and is filed under Mae's Days. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

17 Responses to “Efficient IPTables PeerGuardian Blocklist”

  1. I've been looking for a way to implement IPSet-based blocklists ever since i saw they were used in Shorewall (which i don't want to install) - thanks for the article, and the cool converter...

    Can i ask for some elaboration on why you chose the iptreemap set-type - choosing the ideal set-type for lists exceeding 64k of lines has been confusing the hell out of me!

    Thanks in advance.

  2. I used "iptreemap" since it can handle both IP ranges (in from-to or CIDR blocks) and single IPs, and can handle a huge number of them without being overly slow or memory-hogging.

    The hash types are probably faster, and nethash would certainly be more efficient for CIDR blocks (it can't handle single IPs though). But they're limited to 64k entries like you mentioned, where iptreemap isn't.

  3. I'm a linux nub, but I've been learning about networks and networking from the CCNA courses I've been taking, and getting rather bored with my network, wanted to implement a hardware firewall. I ended up choosing Shorewall, and the first use that I thought of was using a Bluetack filter list (somehow) on it. But after a quick couple minutes on google, I had almost given up.

    That is, until I found your site. Great article, it helped a lot.
    The rest of your site is pretty informative as well, I especially like the 'about' section.

  4. hi.. the link to pg2ipset.c does not work anymore 🙁

  5. Hi Maeyanie, a friend of mine and I were interesting in your script to do the conversions of peerguardian to iptables, but it appears that file is missing from the server. Could you reupload it or post a link to pastebin.ca with the source pasted? You sound like a very talented lady, keep up the good work.

  6. It looks like your source code link to pg2ipset is broken. Can you update this or point to a new location?

    Great entry, btw!

  7. This is a great article, but the pg2ipset.c source code link results in a 404 not found. I would love to implement your howto. Thanks!

  8. Oops... seems somewhere in migrating from one host to another, that file got lost. It's back up now.

  9. It is crystal clear that the size of your chosen lists matters. Well, my own list is as tiny as some 1000 entries (mostly IPs, some ranges) but give it a try and just keep an eye on some of your logs.



  10. [...] A little while back, I wrote a post about using iptables PeerGuardian blocklists efficiently. However, that program only uses a single list; it expects it to be pre-processed by [...]

  11. @ fran on August 20th, 2009 at 12:47 am

    Nice one thanks for the list i have added that also... Cheers!.

  12. Nice work, although I didn't get it working. got this error:

    ipset v4.1: host/network `Access' not found

    Seems it doesn't like it when the level1 file has this format:

    AITS Network: Access to Music:

  13. Good catch... yes, that's a bug. Fixed it, new version up. Thanks for pointing that out. 🙂

  14. Hi, I really want to use this script. I've gotten hung up on the libmnl installation. There's no directions for installation and I've tried the general things that I thought would work (./configure, etc.). Is it possible to get help with this since they don't have a readme on how to install this on http://www.netfilter.org/.

    Thanks a ton if you can help!

  15. Really not sure... last version of this I built I don't recall needing anything special. Just tried compiling the latest, and autoconf fails, so I can't get very far.

  16. Ok, that seems to be because I didn't have libtool installed. But of course it couldn't *say* that, it just gave me the somewhat cryptic message:
    configure.ac:101: error: possibly undefined macro: AC_PROG_LIBTOOL

    Anyhow. Installing libmnl wasn't anything unusual, I used the 1.0.1 package and just ran the standard ./configure, make, make install.

    If you grab it from git, you probably have to run ./autogen.sh first, but from there it should be the same.

    The whole thing has sure gotten more work than when I first wrote this though. It takes kernel patching and the whole works now. I guess that's what counts as "progress" these days.

  17. I've updated your script, added a Makefile, and also included an example bash script for cron updating. I posted it over at my github, and of course gave you credit: