Lawrence Teo's Pseudorandom Thoughts

A developer's journal on OpenBSD, security, and beyond

Dissecting OpenBSD’s divert(4) Part 1: Introduction

For more than four years I have been using and tinkering with OpenBSD’s divert(4). At one point after OpenBSD 4.9 was released, I ran into an annoying bug in divert(4) that totally prevented me from using it. At the time I had no idea how to fix it, so I did the next best thing by filing a detailed bug report.

Eventually I realized that the bug isn’t going to fix itself, so I decided it was time to roll up my sleeves and wade into the code. So after 2.5 years of on-and-off tinkering and staring at the code and head-scratching and facedesking I finally fixed it, thanks to a ton of help from Bret Lambert (blambert@). The problem turned out to be due to checksums, which is another interesting topic but that’s a story for another day.

In that whole process I had a chance to study divert(4)’s internals in detail, and it was also my first successful attempt at kernel hacking (or rather, 33 unsuccessful attempts and one successful final attempt :)).

I have since moved on to fixing and improving divert(4) in other ways. But I thought it might be worthwhile to do a brain dump what I learned, both to refresh myself when I start hacking on divert(4) again and for others who may be interested. So today I’m starting (or attempting to start) a series of blog posts on divert(4).

What exactly is divert(4)?

OpenBSD’s divert(4) is a relatively unknown but powerful feature in PF that allows whole network packets to be diverted to userspace for Layer 7 inspection. The whole packet here means everything from the IP header to the payload. If you’ve ever wanted to intercept and inspect every single packet of a session in userspace, including the TCP handshake packets themselves in the case of a TCP session, you can use divert(4) to do so (there are other ways to do this, but divert(4) makes it much easier in my opinion).

Unlike a libpcap-based packet sniffer which works by listening passively to copies of packets, a divert(4) userspace program receives the whole packet itself and gets to actively decide whether the packet should be sent on its way.

divert(4) was first released as part of OpenBSD 4.7. It was originally committed by Michele Marchetto (michele@). As mentioned in the commit message, OpenBSD’s divert(4) is compatible with FreeBSD’s divert sockets:

OpenBSD divert(4) is meant to be compatible with software running on
top of FreeBSD's divert sockets even though they are pretty
different and will become even more with time.

One of the most popular programs that uses divert(4) is Snort when it is used in inline mode (thus running Snort as what the industry calls an “intrusion prevention system”). Since OpenBSD’s divert(4) is currently compatible with FreeBSD’s divert(4), Snort can work with OpenBSD’s divert sockets as well.

One key difference though is that OpenBSD’s divert(4) works with OpenBSD’s PF, while FreeBSD’s works with IPFW (but not FreeBSD’s PF). Note that I have never used IPFW, so I won’t be able to do a full comparison with IPFW’s divert.

Another program in the OpenBSD world that uses divert(4) is Florian Obser’s (florian@) dnsfilter. Florian wrote an interesting post about why he wrote dnsfilter. There’s also an OpenBSD port available for dnsfilter.

An example

I tend to learn best by example so let’s start with one. Packets are diverted to divert(4) sockets using the divert-packet PF parameter. For example, the following rule queues all outbound TCP port 80 traffic to a divert(4) socket:

pass out quick on egress inet proto tcp to port 80 divert-packet port 700

IMPORTANT: divert-packet should not be confused with the divert-to and divert-reply PF parameters. It is unfortunate that they share similar names, but they are different features.

For the above PF rule to be useful, you will need to run a userspace program that listens to divert(4) sockets. For a quick test you can use the example program on the divert(4) man page.

Here’s an abominable one-liner that you can use to extract that program out of the man page:

1
2
3
mandoc -Tascii /usr/share/man/man4/divert.4 | \
    awk '/#include/,/^     }/ { gsub(/^     /, ""); print}' \
    >divert-example.c

If you compile and run that program, and use a browser on the same host where the above PF rule is loaded, you should be able to see activity from that example program.

The example program simply reads every packet and reinjects it back into the kernel as-is. A more sophisticated program could perform further inspection of the packet and decide to discard the packet by not reinjecting it (or even modifying it).

Also, note how divert(4) operates its own port number space. This allows you to run multiple divert(4) userspace programs. For example, if you choose to, you could run one program on divert port 700 to handle HTTP, and another program on divert port 800 to handle a different protocol.

Parts of divert(4)

divert(4) has roughly three parts to it:

  1. Diverting packets from the kernel to userspace
  2. Processing in userspace
  3. Processing reinjected packets

If the planets align, I will write the next few blog posts in this series in that order, where I will dissect each part of divert(4) in detail.

I also plan to discuss some pitfalls with divert(4) including some bugs that I’m still trying to fix.

If this stuff sounds like your cup of tea, stay tuned and don’t divert your attention away from this blog! (pun obviously intended) An easy way to do this is to subscribe to my RSS feed and/or follow me on Twitter where I will announce new blog posts.

reallocarray() in OpenBSD: Integer Overflow Detection for Free

The upcoming OpenBSD 5.6 release introduces a new libc function called reallocarray(3) that extends realloc(3) with built-in integer overflow detection. In this post, I’ll discuss why it’s useful and how it can be used to fix unsafe code.

If you’re not familiar with integer overflows or need a refresher, Ray Lai’s Undeadly article on integer overflows is a good place to start, especially if you would like to read about it in the context of OpenBSD.

In his article which was written in 2006 (way before reallocarray() existed), Ray wrote, “Unfortunately there is no safe replacement for realloc(3). To prevent multiplication overflow, a check must be added”, where the check is as follows:

1
2
3
4
5
6
7
8
9
10
11
size_t newsize, num, size;
char *newptr, *ptr;
...
if (num && size && SIZE_T_MAX / num < size) {
        errno = ENOMEM;
        return (NULL);
}
newsize = num * size;
if ((newptr = realloc(ptr, newsize)) == NULL)
        return (NULL);
ptr = newptr;

Adding that check before every realloc() call is clunky, not to mention error-prone. Ray recognized this and recommended the use of a function called xrealloc() — an early effort that attempts to solve what reallocarray() now solves. xrealloc() is implemented in parts of the OpenBSD source tree but is not part of OpenBSD’s libc.

reallocarray() solves this integer overflow problem more thoroughly by wrapping around realloc() like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/*
 * This is sqrt(SIZE_MAX+1), as s1*s2 <= SIZE_MAX
 * if both s1 < MUL_NO_OVERFLOW and s2 < MUL_NO_OVERFLOW
 */
#define MUL_NO_OVERFLOW (1UL << (sizeof(size_t) * 4))

void *
reallocarray(void *optr, size_t nmemb, size_t size)
{
    if ((nmemb >= MUL_NO_OVERFLOW || size >= MUL_NO_OVERFLOW) &&
        nmemb > 0 && SIZE_MAX / nmemb < size) {
            errno = ENOMEM;
            return NULL;
    }
    return realloc(optr, size * nmemb);
}

As you can see, reallocarray() checks to see if the attempted multiplication will result in an overflow before calling realloc(). This means you can avoid writing checks every time you need to call realloc() – thus gaining integer overflow detection virtually for free.

Another advantage over xrealloc() is that reallocarray() is available in OpenBSD’s libc, which means all C programs in OpenBSD can easily use it.

“But wait! There’s more!”™

At this point, you may be thinking, “Okay, it’s a wrapper around realloc(). What’s the big deal?” Well, it is a wrapper around realloc() with built-in integer overflow detection, but it’s also so much more than that!

In my opinion, reallocarray() is useful in three cases:

  1. Replacing unsafe malloc() calls
  2. Avoiding unneeded calloc() calls
  3. Replacing unsafe realloc() calls

Let’s look at each case in turn.

1. Replacing unsafe malloc() calls

malloc() as we all know is used to allocate memory for an object with a given size.

Its prototype is simple:

1
2
void *
malloc(size_t size);

When a programmer wants to allocate memory for multiple objects, the obvious but unsafe way is to call malloc() like this:

1
2
3
p = malloc(num * size);     /* unsafe! potential overflow */
if (p == NULL)
        err(1, "malloc");

As noted in the comment, the multiplication of num and size could result in a potential overflow.

With reallocarray(), the above malloc(num * size) call can be replaced with:

1
2
3
p = reallocarray(NULL, num, size);
if (p == NULL)
        err(1, "reallocarray");

Since reallocarray() does the overflow check for you, you’re done!

2. Avoiding unneeded calloc() calls

Before reallocarray() came along, the recommended way in OpenBSD to fix malloc(num * size) calls is to replace them with calloc() instead:

1
2
3
p = calloc(num, size);
if (p == NULL)
        err(1, "calloc");

calloc() comes with built-in integer overflow detection, at least on OpenBSD. But calloc() also zeroes out the allocated memory. There are times when you don’t need or want to zero out the memory, for example if you’re going to intialize the entire allocated memory immediately after the calloc() call anyway.

This is where reallocarray() shines again. Just like the above case, the call can be replaced with:

1
2
3
p = reallocarray(NULL, num, size);
if (p == NULL)
        err(1, "reallocarray");

This allows the memory to be allocated without the cost of zeroing the memory, while at the same time detecting integer overflows. Neat, huh?

3. Replacing unsafe realloc() calls

As noted by Ray, there is no safe replacement for realloc(). realloc() is frequently called like this:

1
2
3
4
newp = realloc(p, num * size); /* unsafe! potential overflow */
if (newp == NULL)
        err(1, "realloc");
p = newp;

Like the malloc(num * size) case, the multiplication occurs without a check, which could result in an integer overflow.

reallocarray() can replace the unsafe realloc() call like this:

1
2
3
4
newp = reallocarray(p, num, size);
if (newp == NULL)
        err(1, "reallocarray");
p = newp;

You have to be careful here though, because if the code is part of a function which returns on error instead of exiting completely, the original p would not be freed and you will lose it. For example:

1
2
3
4
newp = reallocarray(p, num, size);
if (newp == NULL)
        return (NULL); /* original p is lost -> memory leak! */
p = newp;

A better way is to borrow the recommended idiom for realloc() from OpenBSD’s realloc(3) man page and apply it to reallocarray():

1
2
3
4
5
6
7
newp = reallocarray(p, num, size);
if (newp == NULL) {
        free(p);
        p = NULL;
        return (NULL);
}
p = newp;

What’s next?

reallocarray() is a non-portable OpenBSD extension. But thanks to its liberal ISC license, you are free to use it with your own code as long as you adhere to the terms of the license. So not only do you get integer overflow for free, you get the code for free as well!

Due to its usefulness, at the time of writing, more than 130 commits have been made to the OpenBSD tree that involve reallocarray(). Theo himself has been on a merciless reallocarray() rampage this past month. :–) Some of the fixes have been easy and obvious; others have been very tricky. The audit continues…

Happy hacking!

UPDATE (2014/10/29): reallocarray() has been added to libbsd 0.7.0. Note that I have never used libbsd so this is not an endorsement of libbsd. :–)

UPDATE (2014/10/30): Revised the “Replacing unsafe realloc() calls” section to discuss potential memory leak issues. Thank you to Andrew Dalgleish for pointing out this issue in the original version of the blog post.

A Sneak Peek at the Upcoming OpenBSD 5.6 Release

OpenBSD 5.6 CD cover In exactly one month from now, OpenBSD 5.6 will be released on November 1, 2014 — like clockwork as always. But, you don’t have to wait until November 1 if you want it earlier, because pre-orders are now up at the brand new OpenBSD store!

OpenBSD 5.6 is of course the first OpenBSD release with LibreSSL, the now-famous fork of the OpenSSL library. But while LibreSSL is an important milestone for OpenBSD, there are many other things in the OpenBSD 5.6 release that warrant attention as well.

The developers are still busy preparing the OpenBSD 5.6 release notes. Meanwhile, if you’re curious about what’s new in OpenBSD 5.6, you can get a sneak peek from various places on the Internet if you know where to look! I would like to highlight three places in particular.

plus56.html

The first place to look at is the big list of changes between OpenBSD 5.5 and 5.6 that has been painstakingly compiled by Brett Mahar (brett@) throughout the development cycle. I think Brett does an amazing job of collating all this information together for every release! I have no idea how he manages to go through every source-changes@ post to produce this list so consistently. :)

Hackathon reports

The second place that you can look at are the hackathon reports written by the developers. Two hackathons were held between OpenBSD 5.5 and 5.6: m2k14 in Marrakech, Morocco (April 17-22, 2014) and g2k14 in Ljubljana, Slovenia (July 8-14, 2014).

The hackathon reports are published on the OpenBSD Journal website. I have listed them here:

m2k14 hackathon reports

g2k14 hackathon reports

As for the g2k14 hackathon reports, thanks to Undeadly editor tbert, they are all listed in a single article on the OpenBSD Journal! Here it is:

g2k14: Reports Roundup

Slides and papers

The third place to look at are the slides and papers written by OpenBSD developers. Two notable events that took place between OpenBSD 5.5 and 5.6 are BSDCan 2014 and EuroBSDCon 2014. You can check out OpenBSD-related slides and papers for these events and more at:

As you can see there’s a wealth of information out there on what’s new in OpenBSD 5.6 that really shows the amazing amount of work that is done for every release.

I would like to end by highlighting this tweet/retweet from Ryan McBride (mcbride@), which I think summarizes the awesomeness of the OpenBSD release process like no other:

mcbride on OpenBSD releases

So remember to pre-order OpenBSD 5.6 or donate to OpenBSD to keep this awesome project alive and thriving!