Dissecting OpenBSD’s divert(4) Part 1: Introduction

For more than four years I have been using and tinkering with OpenBSD’s divert(4). At one point after OpenBSD 4.9 was released, I ran into an annoying bug in divert(4) that totally prevented me from using it. At the time I had no idea how to fix it, so I did the next best thing by filing a detailed bug report.

Eventually I realized that the bug isn’t going to fix itself, so I decided it was time to roll up my sleeves and wade into the code. So after 2.5 years of on-and-off tinkering and staring at the code and head-scratching and facedesking I finally fixed it, thanks to a ton of help from Bret Lambert (blambert@). The problem turned out to be due to checksums, which is another interesting topic but that’s a story for another day.

In that whole process I had a chance to study divert(4)’s internals in detail, and it was also my first successful attempt at kernel hacking (or rather, 33 unsuccessful attempts and one successful final attempt :)).

I have since moved on to fixing and improving divert(4) in other ways. But I thought it might be worthwhile to do a brain dump what I learned, both to refresh myself when I start hacking on divert(4) again and for others who may be interested. So today I’m starting (or attempting to start) a series of blog posts on divert(4).

What exactly is divert(4)?

OpenBSD’s divert(4) is a relatively unknown but powerful feature in PF that allows whole network packets to be diverted to userspace for Layer 7 inspection. The whole packet here means everything from the IP header to the payload. If you’ve ever wanted to intercept and inspect every single packet of a session in userspace, including the TCP handshake packets themselves in the case of a TCP session, you can use divert(4) to do so (there are other ways to do this, but divert(4) makes it much easier in my opinion).

Unlike a libpcap-based packet sniffer which works by listening passively to copies of packets, a divert(4) userspace program receives the whole packet itself and gets to actively decide whether the packet should be sent on its way.

divert(4) was first released as part of OpenBSD 4.7. It was originally committed by Michele Marchetto (michele@). As mentioned in the commit message, OpenBSD’s divert(4) is compatible with FreeBSD’s divert sockets:

OpenBSD divert(4) is meant to be compatible with software running on
top of FreeBSD's divert sockets even though they are pretty
different and will become even more with time.

One of the most popular programs that uses divert(4) is Snort when it is used in inline mode (thus running Snort as what the industry calls an “intrusion prevention system”). Since OpenBSD’s divert(4) is currently compatible with FreeBSD’s divert(4), Snort can work with OpenBSD’s divert sockets as well.

One key difference though is that OpenBSD’s divert(4) works with OpenBSD’s PF, while FreeBSD’s works with IPFW (but not FreeBSD’s PF). Note that I have never used IPFW, so I won’t be able to do a full comparison with IPFW’s divert.

Another program in the OpenBSD world that uses divert(4) is Florian Obser’s (florian@) dnsfilter. Florian wrote an interesting post about why he wrote dnsfilter. There’s also an OpenBSD port available for dnsfilter.

An example

I tend to learn best by example so let’s start with one. Packets are diverted to divert(4) sockets using the divert-packet PF parameter. For example, the following rule queues all outbound TCP port 80 traffic to a divert(4) socket:

pass out quick on egress inet proto tcp to port 80 divert-packet port 700

IMPORTANT: divert-packet should not be confused with the divert-to and divert-reply PF parameters. It is unfortunate that they share similar names, but they are different features.

For the above PF rule to be useful, you will need to run a userspace program that listens to divert(4) sockets. For a quick test you can use the example program on the divert(4) man page.

Here’s an abominable one-liner that you can use to extract that program out of the man page:

mandoc -Tascii /usr/share/man/man4/divert.4 | \
    awk '/#include/,/^     }/ { gsub(/^     /, ""); print}' \
    >divert-example.c

If you compile and run that program, and use a browser on the same host where the above PF rule is loaded, you should be able to see activity from that example program.

The example program simply reads every packet and reinjects it back into the kernel as-is. A more sophisticated program could perform further inspection of the packet and decide to discard the packet by not reinjecting it (or even modifying it).

Also, note how divert(4) operates its own port number space. This allows you to run multiple divert(4) userspace programs. For example, if you choose to, you could run one program on divert port 700 to handle HTTP, and another program on divert port 800 to handle a different protocol.

Parts of divert(4)

divert(4) has roughly three parts to it:

1. Diverting packets from the kernel to userspace
2. Processing in userspace
3. Processing reinjected packets

If the planets align, I will write the next few blog posts in this series in that order, where I will dissect each part of divert(4) in detail.

I also plan to discuss some pitfalls with divert(4) including some bugs that I’m still trying to fix.

If this stuff sounds like your cup of tea, stay tuned and don’t divert your attention away from this blog! (pun obviously intended) An easy way to do this is to subscribe to my RSS feed and/or follow me on Twitter where I will announce new blog posts.