OpenNSM (AFL-Fuzz and Bro)

Welcome to the meeting of the Open Network Security Monitoring Group at the University of Illinois at Urbana-Champaign. A weekly NSM DFIR group which has projects and presentations and all kinds of good stuff. We're going to drop right into the group updates to begin. So a few things, uh, Snort 2.9 images were created for our contain NSM project. So, uh, Mike Downey contributed these. He contributed about 20-something Snort images for the 2.9 series, and the Dockerfiles are on GitHub, and I built out the images and they're available on Docker Hub. So you can docker pull OpenNSM/snort:2.9.whatever version and pull them down and use them. Also, I released Suricata 2.0.8 and 2.0.9, and those are also available on the container NSM repository on Docker— or on GitHub and on the Docker repositories. And 2.9— or 2.0.9 was released last week. It was a bug fix release. And that comes into the networking news. Where you can see that they had— they put a blog post out for the release and they just fixed a number of bugs including what seemed to be a fairly severe one that crashes when certain rules are used. So that's not really good at all. Also they fixed a bug in the defrag invasion and they did some better detection of TCP retransmissions. Also I put up a team page on opennsm.net so if you've been contributing I have you listed here. Hopefully I remember everybody. But click on team. And eventually we'll have the new website up. That's still something that needs to be completed. But so far that's what I have. And it'll be— we'll have a list of what you— to give you guys recognition for your contributions, we'll have a list to the right side of your name and a link to your blog or whatever, the things that you've been helping out with. Going back to the notes, uh, CapLoader by the people that do, um, Network Miner, uh, NetResSec, they released CapLoader 1.3 and this particular one has a number of new features and basically it's a pcap parsing utility if you didn't know already for Windows and this one supports multiple filters, BPF and display filters, kind of like Wireshark has display filters. But if you read the thing, it shows they work a little bit differently. Also, they have, um, support for, um, domain whitelisting from the Alexa list as well as domain extraction via passive DNS parsing, among a few other things. Let's see if there's anything else that looks notable. Go down here. So if you're on Windows, I highly recommend trying out this tool. Oh, they got, uh, this one's kind of interesting— producer-consumer ratios. They have PCRs now. As well as flow transcript can be done by double-clicking on a flow and a few other things. So this tool has been coming along. Support for SCTP, which you don't see too often, but that's kind of cool, among a number of other things in the PCAP carving code. Next week is— actually not next week, this week is the Ohio Linux Fest. Now this isn't really NSM related, but we bring this one up because a number of your NSM members will be attending. I'll be I'll be attending, Shane will be attending, Clem will be attending, Mike will be attending, and Waylon will be attending. So that's this week. We're leaving Thursday night and we'll be there until— oh, and Simon, I'm sorry Simon, I forgot about you. We'll be there until what, Saturday or Sunday? So if you are attending that, come talk to us or give us a shout out online, we'll meet up. Moving along, signature selection. This is the meeting section where we talk about new signatures or detections. And here's one that came out on type squatting with Bro. I didn't know if Justin wants to make a few mentions on it or not. But essentially they're trying to find type squatted domain names in the SMTP streams. And they're using what is called Levenshtein, or Levenshtein distance, which seems to be an algorithm for calculating that. It looks like they use from 0 to 4 characters to determine the type squatting. Of course, they got whitelist support at the top. You can see that you can set a— add some values to a set for whitelisting various domain names, etc. So this is kind of a cool tool if you want to detect attacks that are used that are SMTP-based using type squatted domains. So fat fingering it essentially. Where users would do that and the attacker would luckily get results, or they see the URL and they think that it's actually legitimate, but they missed that, hey, a character is off. So that's really cool if you want to check that out. Now we're just gonna go ahead right on into our talk today. So we have Justin Azoff with us, who's a security engineer at the NCSA and a developer for the Bro team on the Bro project. He does a lot of administration and development, including releasing a number of tools like PassiveDNS, Dumno, the BHR site, as well as HTTPFlood, NetFlow Indexer, ninfo, etc. So I'm gonna pass it off to him. He's gonna be talking about fuzzing Bro-cut today.

Yeah, so I have been working on fuzzing Bro with AFL. Part of the problem is that Bro is one of the worst things you could want to fuzz because it's this giant C++ program that also has its embedded scripting language, and generally with AFL you kind of want to fuzz a smaller component that doesn't utilize, you know, every single C++ feature and library under the sun. I did have some success doing it, but mostly it found bugs so far in dealing with invalid short truncated packets. It's actually almost impossible to fuzz a pcap easily using AFL because AFL wants to generate a random input, and a random input is never going to be a valid pcap. So my latest thing I've been working on, I had to put it to the side for now, is the post-processor support in AFL to fix a random pcap to have valid packet headers, which is a lot of low-level TCP stuff. But one of the first things I used AFL for, which is actually a great example because A, it finds bugs very quickly, and B, it's a very small standalone C program that's fairly easy to understand. So one of the components in Bro is called Bro-cut, and it works like cut. So I have a log file here, and you can kind of use cut like if you wanted the second field or the third field, you know, cut will do that. Bro-cut's a little nicer because you can say I want the UID field and I want the query field, and it knows the different indexes, and you could tell it, you know, I don't want those two fields, and it'll give you everything else. It also handles the decoding of the timestamp to a human-readable time. And that's really all it does. It's just very, very, very high performance. It does zero memory allocations. It does almost the bare minimum of work necessary, so it's some pretty tight C++ code. It's 400 lines of code, most of that is options and dealing with errors. So I wrote it, and one of the other Bro developers, Daniel Thayer, fixed it up and actually made it some pretty nice C code and fixed a lot of the error handling. But I wanted to know, you know, does it have any bugs that we're missing? You know, it's a fairly small subset of inputs that it should accept, and since you're generally running this over untrusted Bro logs, you don't want to have any buffer overflows owning your system when you're trying to analyze logs. So that's how bro-cut works, is you can pipe logs to it and it will give you the fields. Fairly straightforward. So the way AFL works is you need to give it— it's very complicated— an input directory and an output directory. So let's take this log we have and stick it in input. So that's my very complicated input corpus. You could give it lots of files. Generally they recommend a handful of very small files that do enough to exercise the input. Like, BroCut tends to work the same way if you had 3 columns versus 30 columns, and it works the same way if you had 1 line as opposed to 1,000 lines. So adding more data isn't going to help. If anything, it just slows down the process of it actually finding the bugs. So to jump right in to use AFL, it's basically a wrapper for Clang. So in this case, we want to use afl-clang instead of gcc, and we can just recompile it in this directory, and you'll see it instrumented a couple hundred locations. And the way AFL works is it doesn't just do random inputs. It kinda starts out doing random inputs and then it tries to see what that change in input affected how the code ran. So much like you can step through code with a debugger, AFL will realize, you know, if I change these bytes, the code does the same code path. But if I change this byte over here, that did something in the code and now it's running this code path. And I think I have it open here somewhere. One of— yes, so one of their examples for testing is this piece of code where if you send it the 3-letter string "pwn" the binary will crash. And AFL will really quickly figure this out because it'll see that, "Hey, when the first character is 'p'" like, this code runs and it doesn't run if the first character is not "p". So a first character of "p" is interesting. And then it'll start figuring out, "Hey, when I make the second character 'w' this code runs, and hey, it didn't run before, so that's cool." And it'll very quickly figure out that 'pwn' crashes the program. Where if you were doing strictly random, that would take much longer because it just doesn't know that a 'p' affects the program if it was strictly randomizing things. So back to BroCut. So now that it's compiled, I should have— it should work the same way, so if I pipe it to brocut, it works the same way as it did before. So now we're ready to go. You just need to use the AFL fuzz wrapper tool and follow the options it tells you to use. You tell it I want to use I as the input directory, O as the output directory, and my binary is brocut. And I believe that's all you need since it's very fast. It'll calibrate it pretty quickly and start fuzzing in a couple of seconds. Oh, and I have one problem due to my colors in my terminal. AFL disagrees with them. Fix that. So now you can see right away— so sometimes hangs are nothing to worry about. It, once it benchmarks it initially and figures out that it's a fairly fast program, it will set a very aggressive timeout. But the very interesting things are AFL's already found 27 different code paths, so it's changed the input enough, and as you see, it's finding many code paths. And what I can do— oh, ah, geez, I forget what directory I was in— auxbro, aux, brocut, and in that o directory there's a cur Thought there was. Hold on a second.

Cur—

ah, oh geez. Okay, so I accidentally made a -o directory as my output directory because I was typing too fast. And there is a cur input file. So if I echo— so this is the file it's running at any given second through BroCut. So it is just random code characters and stuff. Yeah, well, it's, it's just flipping bits and flipping bytes. And trying to find problems, and at this rate it's currently telling me it's doing 1,000 executions a second, which seems okay. If I use the persistent mode support, I could probably get that up to 10k, because what it's actually doing is, I think it forks the binary once and then just clones it. It doesn't actually run it, so it's able to do some speedups. Um, so yeah, and it's already found 69 code paths and trying very, very hard to find which byte it needs to change to crash it. And there's actually support. You can run this distributed and use multiple cores.

So right now we're just waiting for a crash.

Yeah, I was fortunate the first couple times I did this that it found a crash within just a couple of minutes. Very, very, very quickly. It's not the most exciting thing in the world to watch, I know.

Um, this seems to be a pretty popular fuzzer anymore. Like on their website they show people using it to like SSH vulnerabilities, all kinds of services. Seems to be the real deal as far as finding vulnerabilities goes.

Yeah, it's very, very, very popular. They have a kind of testimonial thing. Yeah, it's found bugs in a ton of different libraries. Internet Explorer, Firefox. Yeah, it's great at finding bugs. Anything that handles data, it's really easy to work with. You know, anything that kind of just wants a blob and deals with it. And that's the challenge with Bro is that it doesn't want a blob, it wants a valid pcap file. Right. So it's actually somewhat difficult It's a lot harder than BroCut, where all BroCut wants is an input file. It doesn't really care. So yeah, it's very, very, very— And if we're lucky— Nope, no crashes yet. 72 code paths. And what's interesting is since BroCut is so small, you can actually think about it. If I grepped for—

Did you—

you can't see it. Clear the screen. If I grepped for if statements in BroCut, Oh, some of these are not. We have input we don't want. Ifs. There are 60 if statements in Bro, or Brocut rather. So you could imagine, oh, 60 if statements. So if you actually kind of went through the code and tried to think, you know, in a function, right? Find one of these smaller functions. You know, a function like this, how many different code paths would there be? Well, this is a code path, this is a code path, each one of these fields, that's doing something. So AFL is very quickly figuring out how it needs to change the input to get into all these different if statements and try to find different behaviors. It's found 73, no crashes yet. That's irritating. I had to run—

this isn't the patched one, is it?

Oh, that's a good question. I, um, no, bro. 2.4 had the bugs. It's fixed in 2.4.1. Um, because I know where the bugs are.

So yeah, Justin used AFL fuzz to find all these bugs and then he fixed them all. Or at least all the ones he found.

Yeah, it was. I believe so. I could diff it. Like if we diff brocut.c here to source/bro, aux/bro, aux/brocut, brocut.c. Yeah, there's a number. Yeah, it was, yeah, this, that was basically one of the fixes, checking for a null input field separator. Really? It's not going to crash? I wonder, you know, last time I used a much smaller input file with only like 2 columns. I wonder if that's why this is taking longer.

So what kind of bugs, just to give an example, a few examples, did it find with like in your program? There's certain characters it didn't like?

Yeah, it does. Let's, uh, so as I talked about, the smaller input files are better, and I think this one I grabbed— so we can actually use BroCut. To— I think it's the include the header. Yes, we want to include the header. Let's just grab the timestamp and say the query. So we just want two fields, a much smaller input file, and let's replace that right over that file. So now input one is just this much, much, much smaller And let me get rid of that öde— Let's try this again. Much smaller input file, it should be able to figure out things much faster. And let's take a look at what it's been doing. Yeah, so now you can see there's much less fields in the file, there's much less opportunity for it to change things that don't matter. Because remember in the original file, I had this really long query and it's going to, of course, try changing the bytes in that and it's never going to do anything. And like I said, Bro-cut works exactly the same way as if it has 2 fields as it has 1,000 fields. So we don't need very many lines. Yeah, some of the things AFL tests are very out there. It has a number of approaches. Some of the things it does If it sees a number, it tries changing that to different magic numbers. Like, let's try changing that to, you know, 2^16, or 2^16 1, or let's try just changing it to -1, or 0. So right now it's— it actually tells you what it's doing, the different fuzzing strategies. The bit flips, byte flips, arithmetic, the known ints, that's what I just mentioned about using 2^32.

Ah, yes.

I see. So right now, right now it's doing Havok, which I forget what Havok is.

What are they representing there when they're saying 2 slash some number?

I think the different— well, that's a good point. I think maybe if it found a different code path by changing that. I usually don't pay too much attention to that. Mostly the big thing you want is you want it to find paths, but not too many paths. One of the problems I had trying to fuzz Bro is it just finds a new path every second. Just everything it does changes the code paths and it is just completely confused by that because it has no idea what's going on. Where this finding a new path, it actually tells you last new path 30 seconds ago. So finding a new path every minute or so is a kind of good rate. It means it's finding new paths but it's not finding a new path every time it changes something. Which means you're never really going to find anything.

And does this log the details of every path that it found to a log file or anything?

Uh, not the paths it creates, uh, the queue. So in this queue directory, there's all these different inputs that it's going to be working through. So every time it finds something that affects it, it I think uses that as a starting point for making more changes. Oh, and I just realized— yeah, I'm using the wrong one. I think. One second here. Ctrl+C. I just realized if I'm fuzzing the one that's fixed— no, I'm not. Fuzzing the one in the current directory. I've done that before. That wastes a lot of time. Fuzzing the binary in your PATH instead of the one that you actually compiled with AFL. I know this is like watching paint dry at this point.

Talk about, um, what you've been trying to do to get Bro, like some of the obstacles to actually fuzz Bro's analyzers.

So I've tried a ton of different approaches to get Bro to work. So the first approach was worked exactly like this. Just give Bro an input file as a pcap. The problem was, as you can see, this is doing 1,000 inputs a second. When you just tell Bro run this pcap, you get about 1 execution a second. So this has already done 240,000 executions. With Bro, that would have taken 240,000 seconds to get to the same point that we're at. And I actually forgot I had it running on one of our test systems, and it ran for about a month because you asked me why is the CPU so high. And I was like, oh great, I wonder if it found something. I completely forgot, and it had barely gotten, I think, a million executions and found nothing. Nothing after a month. So AFL supports a number of persistent modes where you can tell it to execute the binary up into a point, knowing that anything before that's initialization and is very slow. And once you do that in Bro, you get it up to, I think, about 5 executions a second. And with the multiple, multiple processor mode, I think I got it to do about 20. So basically get to the point where we had after a month, we can get to that point after about a day and a half. Uh, the problem I then ran into is I actually did find a couple of bugs that are now fixed in Bro dealing with truncated packets that are just too small, you know, ARP packets that are too small. So generally things that I think are largely impossible to see in the wild but still the kind of things that you should fix. And probably the reason why I found them are because they never happen in the wild, because you— I don't think most networking devices won't forward, you know, a truncated packet that's smaller than the minimum size it should be for an ARP packet and things like that. Um, and the reason why those were the only bugs I found, because otherwise the bit flips and the byte swaps that it does don't create valid T— or valid PCAPs. So the checksum is invalid, or the length in the TCP header doesn't match the length in the packet. So libpcap itself ignores it or complains. So actually, I wonder if I still have it. I could show you just the nightmare, the absolute nightmare. Um, was it post— post live? Yeah, this It's an absolute nightmare, it's absolutely terrible code, but it kind of worked. It worked enough to get— oh, it doesn't want to tab complete it. So I took their example, so I had to do this. It was an absolute nightmare. I basically worked out how to figure out what the new data length would have been in the fuzzed packet. And update the appropriate bytes in the, first the libpcap header, and then the IP length header to get Bro to accept the packet. And that actually worked, but it was still kind of slow and never really found any bugs. So it kind of turned out to be Not very useful, but this is just the kind of things you have to deal with because while BroCut is perfectly happy accepting a 100% random file, libpcap doesn't. So you have to make a file that's acceptable before you can even get Bro reading it.

So the bigger issue is that you can't just fuzz the protocol, the higher level.

Exactly. So that's why I have this branch. I could find it. It's on GitHub on the Bro, uh, Repository. There's two fuzzing branches I think I uploaded. Oop. Fuzzing branches, not fuzzing repositories. And it was not— it's not fun. Okay, so the defer_to_init, I could show off this. This uses one of the approaches of AFL where you do this thing. Or not that thing. It works here, doesn't work here. Yeah, I had to figure out where to do it. This. So there's an AFL-provided function that you could insert into your code and basically say, when fuzzing, you can resume from here. You don't have to do anything above it. And the tricky thing is figuring out where you can put that in BREL. I actually had to kind of use binary search to figure out how low down could I stick this. And I had to reorder some code to get it to work. And then I commented out some code to have Bro not write out persistent files as kind of an optimization. So this is the kind of thing that AFL supports. It actually supports a whole bunch of different modes to do things like that. And just doing this makes it run a couple of times faster. That's how you get it from 1 execution a second to about 5. And it's very not invasive. You don't really have to change much. There's just a couple of tweaks that I changed to work around some other bugs in Bro when you're trying to fuzz it. And the other branch, this was the one where I really started hacking and slashing at Bro to get it to— because the end result often is you want to test the analyzer, you want to test the HTTP analyzer. You don't care about packets, you don't care about pcaps, all that you want to do is give it, like, the GET request and see if it crashes when it sees that GET request. And I have a branch, it's— oh, it's main. Where is main? Oh, main. Alphabet is hard. So I could find my function, it's in here somewhere. I'll know it when I see it. Where I basically ripped out a lot of the stuff that Bro normally does to try to get it to just run an analyzer, and it's actually extremely difficult to do that because it was never really designed to do that. Ah, and here we are. So, yeah, so using one of AFL's modes where you give it a loop, in order to just test an analyzer, you have to figure out how to make an analyzer. But an analyzer requires a connection, and a connection requires a session, and a session— so you basically first need to make a connection and give it the right data to initialize the connection. Then you need a hash key. Then you need a session. Now you can make a connection using the session, the key— I forget what the ID was for, but it doesn't really use it— my connection ID I made above, some unique ID and some other field that I don't need. Then you need to set it to be TCP, insert it into my session table, and only then can you make an analyzer. However, an HTTP analyzer needs a TCP analyzer, so you need to do these lines. So after all of this, which took me a couple hours to work out, because all I wanted to do was this— this is all I wanted to do— but you can't just do that. Then, once you have it at that point, you can call deliver_stream and give it some data. And it actually does run the analyzer. The problem I had, and it— and I only— now I know why it didn't work, is this actually does get the analyzer to analyze the data. The problem is a lot of stuff in Bro kind of ping-pongs between C++ code and Bro script code from events, and I wasn't handling events. So when I was delivering the stream into the analyzer, it was queuing up all these events in the script land that were never getting ran. So it wasn't really comparing apples to apples like when you give Bro a pcap. Right. It was not actually running any of the script code that would be ran when you have an analyzer. So I need to kind of go back to square one and use this, but figure out how to also run the events. I think you just need to call event drain or something like that. Yeah, I think I need to call this a certain number of times until there's no more events. And then make sure— I'm not actually sure where the script language gets called. So yeah, it turns out to be much more complicated than you'd think. So I haven't—

Yeah, this is pretty crazy.

Yeah, it's— Bro is really the worst thing you'd ever want to fuzz. Like, I was reading, they were talking, there were some good write-ups, because they fuzzed Node.js. And it's a similar problem, you're trying to fuzz an interpreted language and it's not really easy. So now that my laptop is very hot, let's see if we found any crashes. No crashes. Wonder what I'm doing wrong. How long has this been running for? Tells me somewhere. 12 minutes. Actually, yeah, I think when I fuzzed it I found a whole bunch of crashes within 20 minutes, which was actually great after trying to fuzz Bro for hours and days and never finding any crashes. Crashes.

So you can run AFL Fuzz directly on OS X?

Oh yes, you can run directly on OS X.

In Homebrew?

Yes, it's in Homebrew. To run it on Linux, it's actually a little better because some of the stuff that I had in Linux isn't in Homebrew to do like the persistent mode, but I haven't upgraded Homebrew in a while, so it might have it. So you can see it's just really out there trying crazy, crazy things in order to find— Yes, we can use watch. It is just trying, yeah, a file with just a comment, a file with— it tries repeating things, um, one line with a bunch of different things. One neat feature that AFL has that I've not yet used is you can actually give— I think they call it a dictionary, and they've used that for fuzzing SQLite where they give it— so if AFL fuzz was just doing random data it would never in a million years come up with a valid SQL statement. So instead, what you can do is tell it, okay, SELECT is a word, WHERE is a word, NOT is a word. Put these words together, don't care so much about fuzzing individual characters. And they had a blog post where they said they found a number of bugs in SQLite. Yeah, I think you've seen that one. Yeah, with minimal effort. So I'm actually surprised it hasn't found the crashing bugs yet. Especially because I know what causes it to crash. It's not very complicated. You know, why don't I demo since I have it here? Um, persistent mode. I could show you how quick it works when you give it a binary. I'll have to modify it slightly.

Yeah, that's a good idea.

Yeah, let's do that. Uh, afl-test, go into afl-test. Um, so one interesting thing, so Bro was having some crashes. I think I have it here in, um, yeah, UTF. There was a function that actually came with Clang, this, uh, UTF-8 conversion function. Is it in here? Oh, I don't actually have the header that it uses, but it was this function. Bro was crashing in this function. And we couldn't figure out why at first. So I wrote this test harness to fuzz it, to send it all these different kinds of inputs and outputs based on how Bro was calling it. And I fuzzed it with hundreds of millions of inputs and it never crashed. And it turned out the bug was in the code, similar code in Bro, that did these two lines that was creating the endpointer. But it was interesting to have the confidence that, geez, I've ran this function a couple hundred million times. It has never crashed. Pretty sure the bug is not in this function. And it was only crashing in the function is because if you give it an invalid end pointer, it will crash every time. It's just kind of the contract of that function. So let's— we don't want to use the persistent mode because I don't think I have it on this. Machine. So we're just going to get rid of that, get rid of that, and now afl— or clang -o test test.c. Uh-oh. Oh, complaining about headers? Their example was not a real-world example, that's mean. stdio.h. Let's see. Really? Did they? They left out a couple of lines. That's interesting. char buf. Alright, now it compiled. So now we know if we— if we test it and send pwn— Error 3. Oh, pwn, we get abort. Any other input, we just get an exit. So let's make an input file.

Oh, just gotta clear your screen.

Oh. So I made an i directory and an o directory. Let's make a file in the i directory. Um, say yes. So now we'll run afl-fuzz. --ii --oo ./test. Now my laptop will hate me even more right now. And we need to— I think it already found it.

So if it crashes once, it'll keep running? It'll start the program back up?

Is that right? Yeah, so it already found it. It only had to execute it a couple thousand times.

Now over on the left-hand side where it says stage progress and it says now trying Havok, are those different algorithms or modes?

—that tries to— Yeah, remember, so the bit flips, the known ints, the dictionary, trimming. There's all the different strategies. But it already found it in a couple of seconds. So it found 4 crashes, 1 unique, 5 crashes, 1 unique. And I'll show you why that is. So now if we look in this O directory, we have some crashes. ls instead of cat. OK, so it only actually output 1 of them. And we can see— come on, tab complete. Ah, so it actually wasn't perfect. It realized PWN with a couple of bytes at the end crashed.

Oh, so it actually gives you the input too.

Yeah. So yeah, so this way if you were testing a program, you could send the input to the bug tracker. And so it's repeatable. That's the big thing is you get a crash and now you can repeat it for someone else. So there's a tool that I've never used called AFL-CMin. Which is the Corpus Minimization Tool. So let's see if we could run it and see if we can get it to figure out— so now we can take— I believe what you do is you tell it the input directory is crashes. Actually, we need to make, let's say, a min. So we want to figure out— so we know this file crashed it, but maybe you don't know, because we know that the code is the PWN that crashed it. Let's say you didn't know that. And you want to write, come up with the simplest input possible to exhibit this bug. That's what afl-semin is for. I realize that sounds like something else now. You're all like, uh, so we give it the input directory as the crashes, the -o is min, and test. Oh, wow, what did that actually do? Testing target. I expected that to actually do something and it didn't. Why does— huh. I guess that tool does not necessarily do what I thought it does. Narrow says it's doing what we want to do. I guess it's not terribly important. Oh, I think that actually does something different. I think it's actually for narrowing down the input files.

But have you looked at any other tools here? Because I see like plot, maybe that generates a plot.

Oh yeah, so one thing AFL does while it's running, it's pretty cool, is in the output directory There's plot data, which is this giant CSV file of the timestamp, how many different paths it found, you know, how long it took to find. So basically, you know, if you came back after being away for a while and you weren't quite sure if it was really still making progress, or it found a bunch of crashes and you just wanted a little more information, you can dig in and understand the rate. And yeah, that's what— never actually ran AFLPlot. Let's see if it's easy to use. Oh wow, I guess it is really easy to use. afl-plot, the state dir is o, and graph output dir is graphs. Wow. So that one command output— oh look, an index.html. So let's open graphs/index.html. Nice. So yeah, apparently it is very easy to use. So we actually want, you know, is it finding paths? So you can see it's found no crashes. Hangs are likely just false positives.

Executions a second is kind of cool. Yeah, that's useful.

Yeah, that's pretty cool. So you can see it slowed down quite a bit when I was fuzzing the other binary and then sped back up. Really annoyed that it has not found the crashes yet. I wonder if I was just extremely fortunate in the input file I had selected when I tested it the first time that it very, very, very quickly found a couple of bugs, though it has only been exactly 21 minutes, 22 minutes.

Yeah, well, this is very— this is really neat. I think it'd be really cool. I know this would be a pretty intensive project, but— or demanding project, but it'd be really cool If you got a lot better with this and we did something like, once we have the AWS infrastructure, which I need to talk about too, to be able to provide this as like a service for NSM tools, if you get it figured out how the, a good way you could do like a reproducible pcap fuzzing. Yeah. You can just pass into a bunch of tools.

Yeah. That POST, that POST library wasn't Bro specific. So yeah, you basically what you need to do is write a C function that given a buffer fixes it. And their example was, you know, if you're fuzzing a PNG library, any PNG library is going to ignore a PNG file if it doesn't start with the PNG header. So their example that I modified was basically do all the random data, flip all the bits, but then put the PNG header back. Because if you don't have the PNG header, nothing is going to work. The problem is pcap files are a little more complicated. So you can see it's still kind of off. Doing some crazy things that are— per packet header. Exactly. Yeah. The per packet header was the biggest problem because if, if the, there's the TCP length, but there's also the pcap length. And if the pcap length is too long and AFL shorten the packet, the pcap will just say that there's a truncated packet and it won't even pass it to Bro. So yeah, I think coming up with a good library For keeping the lights on.

That's what I was trying to do earlier, guys. The lights just finally went off.

But yeah, so yeah, that library would be useful for Suricata and for any other tool. Yeah. Um, though Suricata might be easier to test in the more traditional way with AFL fuzz, where you just give it a blob and tell it run it through your pattern matchers. You might not need to use a pcap with Suricata. I don't know how much Suricata cares about sessions. Like a lot of Suricata does, I think would be fine testing a single blob where you don't, it doesn't necessarily work like Bro does where Bro really wants to reassemble the streams. So I could look at it. I don't know if anyone's fuzzed Suricata before. Uh, yeah. That's how I first— Oh, cool. Yeah, there's— I think there's definitely a lot of low-hanging fruit out there with tools that you could really just compile with AFL and run it and find some interesting bugs or even hangs. There's an open bug still in Bro that one of my earlier experiments found that since AFL doesn't know what it's doing, the first thing it tries modifying in the files ends up being the pcap timestamps, where it just makes them like 30 years into the future in the second packet. And when you try running it against Bro, Bro just hangs. And I have a feeling if you waited a couple years it might finish. I'm not 100% sure what it does, but so yeah, there's somewhat of a denial of service bug in Bro where if you give it a pcap with an invalid timestamp, it'll just hang. Yeah, there's— it's actually really well documented. Uh, AFL status, I believe. Description of the status screen. Yeah, it tells you what everything is doing. Um, so yeah, the big thing, so when you first start out, you want to make sure it's finding new paths. Last new path was only 26 seconds ago, so it's up to 105 code paths. So it's actually doing pretty good. I think when I left it running forever, I can't remember where it found a steady state. I could have sworn it was only in the 30s, but maybe I'm thinking of another tool I was trying. So yeah, last new path, that's good. So yeah, so I know the confusing stuff is so the map coverage and the density I forget what that is. I know the problem I had with Bro was unless you compile it with this option that they recommend, Bro gets a map density of 99.9% immediately, and I think AFL is pretty useless. So it's telling us percentages over 70 are very rarely, and absolute numbers below 200 are— suggest the program is very simple. Which it very is. So let's see where we are. We are map density 100. So that's actually, I think, fair for BroCut. It is a very simple tool and I believe it is instrumented properly.

So is it trying to find out what the map density is by itself before running it?

Um, no. What is the map density? Hold on a second. Uh, oh yeah. So there's, I think, an internal bitmap in AFL. For, like, when it flips a bit or when it tries something, did this affect the output? And I think what a map density says— I might be wrong— is that 0.1% of the things that have tried has affected the output, possibly. And the problem is when you have a map density of 100%, I think that's basically telling you everything that has ever tried affected the runtime. And it's never gonna be able to make heads or tails of that. Because it's just finding a new code path everything it changes. And I know the other confusing part is the strategies, what the different ones mean. Ah, so Havoc. So this explains what Havoc is. Uh, stacked random tweaks, okay. So it does multiple things on top of each other. So I guess it's a— Oh, that would explain all the crazy characters.

Exactly.

So it's doing a whole bunch of the different flips and things like that. And yeah, we are— still have not found the crashing bugs that I know are there, which is very frustrating. I even have a screenshot of it. Oh, wow. Yeah, it's kind of off in the weeds right now testing giant inputs. But it's finding new code paths, so it keeps doing it.

Do we have any questions for Justin?

I didn't quite catch that. Be so nice if this would crash.

We think we heard some of it. We could not make it out. Could you post it in chat or try to repeat it? Let me try to turn up the audio.

Oh, so I found my directory from what I fuzzed, broke out the last time, and apparently I did use one giant DNS record. So actually a harder test case. One thing I should actually be able to see in my—

So I turned up the audio. If anybody has any questions that are remote, please unmute yourself and try to ask them. Or you can post them in chat.

I don't want to cheat. I mean, I could make the input be the input that crashes it so you could see, but that's no fun. Or slightly different. But yeah, so when I ran it the last time, I had one input and it crashed. Very quick. Actually, let me see, since now I know how to graph. AFL— what was the tool that did the graphing? Plot. Plot. We can do output graph. Start graph. Or not start. What is start? So how long did it take unique crashes? So I started it, it looks like around 3:40, and it found a unique crash about 18 minutes later. Cool. So it was very successful when I tried it the first time. I think part of the issue is that it is non-deterministic. It's extremely random how it's doing what it's doing. So I don't think there's any guarantee that just because it found it in 20 minutes one time, it will find it in 20 minutes the second time. But if you want to just pretend that it found it again, I can show you. So the output that made it crash, there were two different ones, and it keeps track. It knows, uh, it knows which line of code caused it to crash, so it can actually differentiate between the same bug that crashed it, you know, with two slightly different variations. So one of them was a fairly large file, but you can see what it did. The timestamp, it has this giant number. So normally a timestamp is a, you know, Unix timestamp. It sent us this completely giant number, and when you ran the input This might be the fixed binary. Or sorry, when you run the output crashes, that first one with the giant timestamp. Oh, this is the fixed binary. Um, so if I run— it's in a terrible, terrible directory. Oxbro, oxbro cut, cut. Oh, this might be why it hasn't found any crashing bugs yet. Oh. Ah, I'm an idiot. I didn't actually give it any options when I fuzzed it, and you need to give it the timestamp option. So that was kind of useless because I did not give it the option to say, pull out the two fields. Oh, and we can— so that's why it didn't work. Because you need the option that actually deals with the timestamp. So I'll let that run, but we'll look at this a little. So yeah, so Bro-cut crashes with a segfault when you give it an invalid time, and the reason why that was— Bro 2.4.1 source— oops, not source— aux Bro cut aux— isn't organization great? And that was because there was a call to strftime, or not str— really? strftime? Oh. Yeah, because that's the thing. It's interesting. Oh wait, not strftime. What is the function that converts? Is it strftime? That's what it—

whenever we did— whenever I looked at the code in OCaml, I— It's strptime. In the OCaml script I know it was strftime.

I don't know if it's different. Now, F is format, P is parse. Where is the— oh, it was actually this. Sorry. Yeah, it's call A to L to convert it to a long and then call localtime on it. And the problem is this was returning null. And we're like, well, that'll never fail. So, and then this calls strftime with time_ptr, time_ptr is null, and it crashes. And I wonder— unfortunately now you potentially have to wait another 20 minutes, but it should really find it now. Now that you actually give it the option that was making it crash. And the other bug that it found was very interesting. Fortunately, it was actually good that it found it, but it wasn't the kind of bug that could ever happen in a log file. It was a— oh no, that was— there was another bug. We're going to wrap up after this one. All right. I think I have it in here. There was one bug that we fixed and then I found another bug. But I think I might have lost it. Oh, I think here they are. No, that's the time. Yeah, there were two bugs, one with the timestamp. Oh, I know how to find it because it was great. Once, once we found the inputs that crashed, we turned around and went into the, um, Uh, it's actually interesting. I wasn't sure how to fix them. So the first thing I did is I added them as test cases and I told Daniel, I'm like, hey, like, I'm not sure even if I added them as test cases right, but oh, it's in test. We have a test suite for BroCut. So the first thing I did was I added them as, um, yeah, bad logs. Where are they? Testing logs. So there are two files in here. I think— where is it? Uh, hold on. I lost the file. Bad logs tests the file called— separator is missing. Did we add this? I haven't looked at this in a while. If I look at the data— weird. Oh, that's interesting. Don't actually know what happened to those commits. 2 minutes, all right. Ah, it's frustrating. Well, you know what, I know what the bug is. I can just make a file that crashes. And that's still not it. Source, /proc/aux/procs. Yeah, one thing I definitely learned is fuzzing is not an exact science, and there really is a lot to learn about making good input files and input files that are good at testing the code. So actually, turned out crashed was if you just made the field separator null. And once you try bro-cut, this bro-cut, it will segfault. Because while the code tried to parse the separator and ensure that the separator existed, it would still very happily set the separator to the null character. And then what we do with that separator is we call str strsep. And apparently when you call strsep with a null field separator, your program segfaults. Okay.

Good lesson here. So yeah. Well, thank you, uh, Justin, for, uh, graciously sharing your time and your experiences with this really cool adventure in fuzzing, uh, what is Bro.

Oh, and, uh, one thing I should add, so when I was trying to figure out all this stuff, I joined the AFL fuzz channel I think on Freenode, and they were extremely helpful explaining what some of the options did and explaining that I was doing things very wrong. And, you know, because I would see some things about the map density and I was like, "Is that bad?" And they're like, "Yes, that's bad." So they are extremely friendly and very helpful. And highly recommend if you have some problems getting this going on your tool. To reach out to them. All right, well, thank you.

Um, to wrap up real quick, uh, we hit 7 on the nose now. Uh, we— I did get— I did contact, uh, Amazon, our representative for the research grant, and he said there's 2 grants they have issues with right now, and we are one of them. So, um, he just said he's kind of working on it, and I suspect that it'll get fixed, but right now he— there's no funds to allocate us, so he's working on it. So that'll be all. Everybody take care. And until next week, which will probably— I don't know if, Shane, we're going to have a presentation, or are you going to do projects again next week? Projects, as far as I can tell. OK. Well, just expect—

And I swear, within 20 minutes, this is going to be fine.

Thanks for turning it on. Crushing bugs again. And thanks, Justin, again.

Yeah, thanks for the talk. Yeah.