OpenNSM (Flow Indexer)

Hello everyone, welcome to the Open Network Security Monitoring Group meeting at the University of Illinois at Urbana-Champaign. We are a weekly network security monitoring slash digital forensic incident response group where we do research, have presentations, and work on projects that are related to the field. So, um, we have a guest speaker this evening, Justin Azoff, which we'll get to shortly, but we'll go through the notes. Um, let's go on down to the group update section. So we have the state of affairs, and this is just an update since we haven't had one in a while. I know we've been— we had a couple project meetings, I think, and then we haven't had any talks in a while, so I just want to give you an update on me. I've been really busy lately doing some contracting and I started my own company, blah blah blah, and I've just been swamped with work so I haven't had a chance to get open NSM like it was a year ago, or I should say 2 semesters ago. So I just wanted to let everybody know if you haven't seen as many talks and you've been a follower for us for some time, that that's why things have kind of fell off a little bit, but hopefully we'll get it back going here after I complete some things. And now we have a new chair, Steve Gatzey, who also can help, and a few other people that have been donating their, their time. Okay, so let's just move along here. So in the meeting sections, the network news, we have a few releases. So Wireshark last week released 2.0.2, and they also updated 1. 12.0, I believe it was, or 1.11.0. They keep— they track the two major versions, the 1 and 2. And the 2 is the rewrite with a lot of the new features and the new GUI and libraries and stuff. So definitely check out that if you haven't yet. The .2 release that we have right here is mostly a bug fix release. There's nothing really new that caught my eye in the changelog, but there was some improved protocol support for a number of protocols that were out there. Moving along, Surakata, they're actually having a few training workshops coming up in March. We have one in Washington, D.C., Maryland, Baltimore area, and then there's another one, go ahead and open that up. I want to give a shout out to these guys too, because I'll be at the, most likely be at the Maryland one. So there's also one in the San Francisco Bay area. So if you're interested in Suricata, come check that out. If you happen to come to the Washington, D.C. one, please, or the Maryland one, same area, hit me up. Without further ado, we're just gonna jump into the talk since we didn't have a lot of notes today. So Justin Azoff, if you're not familiar with him, he's a security engineer at the NCSA, that is the National Center for Supercomputing Applications, and he's a developer for the Bro project. So the Bro Network Security Monitoring System, he's been doing a lot of development work on that and operational work. From, you know, designing integration tools with it to fuzzing it to finding bugs. He's also worked on a number of tools, many of them he's developed himself, such as PassiveDNS, Dummno, the Blackhole Router System, and many others. So today he's going to talk about his tool FlowIndexer. So I'd like to give it up for Justin here. I'm going to go ahead and switch screens to him. If you're gonna hit share screen.

If it asks me to share, you have to go to the window. Share screen. Share. I want to do the whole desktop. I gotta click that. Should I optimize for full screen video? No, you're not doing full screen video.

All right, here we go.

Alright, so I actually kind of have two projects for indexing flows. There's my older project called NetFlow Indexer, which is written in Python, and a newer tool that I'm writing in Go called Flow Indexer, which is kind of a next-gen rewrite of that. So what Flow Indexer is all about is a lot of people have logs. They might be NetFlow logs, they might be Bro logs, you might have you know, 2 years of these logs. And one thing that happens a lot is you learn that some IP address is very bad and you need to know, did anyone ever try connecting to that IP address? Or, you know, recently, over the past year. There's a lot of systems that in theory will index all your data like Splunk or Elasticsearch if you pay lots and lots of money. And even if you do get it to index all your logs, searching for something like, has anyone ever talked to this IP address in the past year and a half, you still might end up waiting a couple of minutes for it to tell you that someone did or didn't connect. Plus you may have your logs archived already in, you know, gzip text files or something like that, and another system might say, oh yeah, we can index your data if you basically copy all of your data into this other system, use double or three times the disk space, you know, just to have another copy that you can search faster. And sometimes you don't want to do that because you might have, you know, a terabyte of logs. You don't want to have to have a copy of a terabyte of logs necessarily just so you can search it a little faster. So what NetFlow Indexer and what FlowIndexer does is all they do is index what the unique IPs seen in those log files were. Much like a Bloom filter. So I know for, I believe, NetFlow Indexer, I think somewhere in here, sites using NetFlow Indexer, the index generally for this tool was about 5% of the data size. So there was a site indexing 5.6 terabytes of data and their index size was 56 gigabytes. So if you're the type of place that has 5.6 terabytes of data, saying that you only need another 56 gigabytes to index it is pretty reasonable. If you were to say try to put 5.6 terabytes into Splunk or Elasticsearch, you might not have a good time. Um, and that's my older NetFlow indexer project that used Zapian. My newer project that uses Go which uses LevelDB for storage and bitsets for packing in the data, is about a third the size to do the same thing. So—

Can you explain what Zapian is?

Sure, Zapian is kind of a more generic document indexing system that I kind of misuse for NetFlow Indexer, 'cause the documents are really the log files and the terms are just the unique IP addresses. So there's a relatively few number of documents, and a relatively large number of IP addresses, and it's never really designed for that sort of thing. It works surprisingly well, you know, like there's tools that index email or web pages that are built on top of Zapier, and it's able to do a much better job at a more kind of freeform thing like that because it can answer fuzzy searches. I think it can do various Boolean searches that I just never do. So yeah, so flow indexer leverages LevelDB, which is just a flat key-value store where the keys are the IP addresses and the values are kind of a list of the log files that saw those IPs. The tricky thing that it does is I use a bitset library, which I thought would actually compress really well, and it actually uses more disk space than I thought. And I actually tried replacing it with MessagePack, which is an extremely compact serialization framework, The problem is I wrote all the code to do that, and it was, I think, about half the performance, maybe slightly slower depending on what I was indexing. And searching was about half as fast. Because it turns out that while the Bitset library I'm using isn't exactly the most compact on disk, it is ridiculously fast because I think it's written partially in assembly. So no one seems to really mind that the indexes are pretty small. So I can kind of show off— I have it running on our— one of our systems here. So let me give you an example of the new tool on kind of how it manages to index. So we have some older data that we're indexing, which totals— that's going to take a minute because that's about a year and a half's worth of log files. So that's 1.3 terabytes of compressed data. I think if you— actually, gzip will tell us. If we gzip -l it, that might take too long. Yeah, that's gonna take too long. I think it works out if I do just a representative, just do February's logs. I think it works out to about 10 to 1 generally for our Bro logs, 'cause they compress very well. So figure, I think this is about 9 to 10 terabytes of raw logs. February, really don't want to run, just want to represent. Okay, so on a day we get about 73%. So figure, so figure the 1 point, what did I say it was? 1.3 terabytes is much smaller than the raw logs, so much more than say double or 3 times, probably 7 or 8 terabytes at least. So how big are the indexes once you index all the unique IPs for that? So the old Zappian indexes are 20 gigabytes. So that 1.3 terabytes of logs indexes down to 20 terabytes, or 20 gigabytes, which for storage space, if you can store 1.3 terabytes, you can store another 20 gigabytes. My new tool, the flow indexer that uses LevelDB, what is that directory? Bro index test is 6.6 gigabytes. So about a third the size. So, and that's, uh, that stores every unique IP address and what log file it was in since June of 2014. It is 6.6 gigabytes. Let me actually delete some of this data so we can— I'll delete 2016 database. So the way this tool works is you write a config file which says where to bind to, but then what indexers do you want to use. So I have two configured: all the conn logs in Bro and all the notice logs in Bro that go to unique indexes. So you give it a glob and you tell it where to store the databases, and since you generally want to partition your data you do this funny little regex thing here which extracts the year and the month out of the path so you get one database for connlogs for a year and a month and for the notice logs, since they're so much smaller, one database per year. And this lets you really just configure it once and then start up the software. So if I start it back up, it will reopen all the data files, expand the globs, So now since I deleted the 2016 data for the notice logs, you see it's picking up every single notice log, reading it and indexing it. And you see notice logs are pretty small, so it's already done about half the year that it has to so far. And if we wait a couple of seconds, it'll be done. So reading conn logs is a little slower but not much else. So it's done. So that indexed at least all the notice logs for 2016 in a couple minutes. And if I go in here and actually compare the two— so the conn logs are really 6.6 gigabytes and the notice logs are 57 megs. So now there's a command line interface, but really it's designed to use a simple REST interface. So we can start asking it some questions. We can say, "Were there any notices?" So we use the notice index for 1234. And it says no. And it said no in 0.006 seconds. So that effectively searched the indexes for the notice logs for the past year and a half in 0.006 seconds. Cool. If we ask it for, I think, an IP address that will probably have an index. Or some notices, our name server, yeah. So 3 notices, or at least 3 instances in the notice logs. And that's the smaller log. So that's the, what did I say, 57 megs of data. So let's search a little more. Let's search the conn logs. So can someone give me an IP address? Any IP address. A real IP address, yeah.

Google DNS, 8.8.8.8?

Sure, we can use 8.8.8.8. Let's find out, did anyone ever talk to 8.8.8.8 over the last year and a half, and if so, which log files has he appeared in? Any guesses on how long this is going to take to search the indexes for about 18 months worth of data?

1 minute.

0.4 seconds. So we were able to answer the question, did anyone talk to 8.8.8.8 over the last year and a half in 0.4 seconds. Yes. And I'll get to some more stuff in a minute, but what's very interesting, so that had some data. Let's say someone released some information that said this IP address is bad, and I'm just gonna type, you know, some gibberish to get an IP address that we probably didn't see. So, let's say this is an IP address that if anyone ever talked to at any point in the past year and a half, we have a serious compromise. You want to know as soon as possible, did anyone talk to that IP address? You don't want to have to wait an hour to grep a year and a half's worth of logs. So, let's see if anyone ever talked to that. No. 0.006 seconds. So, you just answered pretty much potentially critically important question.

Type in 159.122.159.122.254. Uh-huh. And then the last octet.

Oh, is it on the chat? I think I got it.

Somebody post it in the chat. Do you want to have the chat?

I got it. This isn't going to be like Pornhub or something, is it? It's totally— I bet it is. I'm going to see, I'd have to, one second. I just want to know what I'm looking up first so it's not potentially embarrassing.

He says the server kind of compromised us.

Oh sure, yeah. Oh, that's interesting. So some hits on that. So what exactly is that?

Ask him.

Ask him that.

Oh, it's an IP address and emerging threats compromised on rules.

Ah. Yeah, so what you can then do is dump the raw data. So I recently implemented at least the skeleton support So instead of search, I can do dump, which then, now that you know you potentially have some data and you don't mind waiting, it will dump the raw log records and you can see, oh, so those are inbound SSH scans. And I bet if I go back to where I was just looking, er, Yeah, we null routed that a number of times. So, yeah, so they're doing bad things to us. So we are not doing things to them, which is good.

Waylon's asking where the data set is from.

That I'm indexing?

Not clear.

So the data that I'm indexing is the, our Bro connection logs, which is very similar to NetFlow data. So what's interesting is we also support CIDRs, so I can search for /24 and do the same thing. Since it stores the IP addresses sequentially, it's really just a range search for storing, or for searching for CIDR blocks. It's just really just as efficient as searching for an individual address. And yeah, so if now I do, So now that I know what that is, so I could search— oops— notice— interesting. Hold on a second. That was the address. Todd, we should have some— oh, unless OSSEC was what was blocking it. We might not have—

That was one of the things I was wondering.

Yeah, Bro may never have gotten far enough Cause if OSSEC blocks it on the first attempt, it probably never pisses Burrow off enough to block it. One thing, I actually do have initial backend support for indexing raw syslogs. Okay. Which I just need to optimize a little bit more before throwing it at our like year's worth of syslog. Cause it's potentially the difference between taking a week to index versus 2 weeks. Okay.

So if I spend a little more time optimizing it, Is there a web interface for this?

Not yet. I mean, it's the HTTP API, but I don't have a web interface yet.

Could you demo using the command line tool? You said there's a command line tool.

Yeah, there's kind of a lower-level command line tool, which I implemented a bit first. So if you go into the actual directories with the logs, And I run flow indexer, I can do dbpath, give it one of these dates and say search for, and it works the same way. So one of the reasons why it's important to have a daemon is LevelDB doesn't allow you to have multiple readers. I think RocksDB does, but LevelDB doesn't. So you have to kind of go through a single process to manage the databases. One thing I did learn is it's better anyway. Because that one process can have the databases open and cached, and you don't want to have to start a program, open the database, read the indexes, figure out what's what, do your search, and then exit. You end up wasting all your time starting and stopping compared to just running a persistent daemon.

Is there a way to run on all of them without like folding through all those?

Well, that's— the daemon does that for you. That's one of the things it does is it'll open all the databases and do the search across them all.

—then you can do the search with the command line tool?

I didn't implement that, no. I could. It's really better to run the daemon because the daemon will also pick up new log files, index them automatically, present them to you as soon as they're available. So you just have an init script that runs the daemon at all times, essentially? I mean, I've been just running it from screen when I've been developing it, but yeah, it's the kind of thing you just start up once and it presents all the log files. Yeah, so it's— so the older NetFlow indexer project supports a lot more features that I haven't quite implemented yet. There's a web interface for it that's actually kind of user-friendly. It can do a little more outputting, filtering, but it's kind of slow. And one problem I've had, since it's written in Python and uses Zabbix, no one can get it installed. It's always a an ordeal. I get so many emails on how to get the bindings installed and how to configure it. So one of the benefits of writing it in Go is I can develop it locally and just publish static binaries that are much easier for people to run. Like, I actually develop it on the Mac. So my development process is pretty much this. I use— oh, I have to export my Go directory. I build it with Gox, which cross-compiles it for me. And I rsync it over to our servers. And that's it. And then I just run it. So our servers don't even have Go on them. They don't know what Go is. It doesn't matter. It takes a little bit to cross-compile it, but not too long. So that's how you cross-compile and deploy it to a server. And then you just run it. How many binaries does that generate? Just the one. Just one? Yeah, single binary. It just does everything. There was a couple more things I wanted to mention. Oh yeah, so it's very modular the way I've been writing it. So there's different backends. There is the Bro backend, which understands— actually have a Bro backend and one that uses Brocut, thinking it might be faster, but it's not. Terribly faster. And that just reads the— oh, here it is— it reads the different columns because the IP addresses are always the second and the fourth columns. There is a Bro JSON backend which uses a JSON decoder to look for one of these fields in the logs, and those are all very specific. What I've been working on optimizing is the syslog backend, which is absolutely terrible. I borrowed some regular expressions from Bro and tried to tweak them. If this makes sense to you, can you help? Because one issue is that the regular expressions in Go are kind of slow. They weren't anchored and I anchored them, which actually like I think quadrupled the performance. It's the kind of thing that doesn't really matter when you're trying to match this on a couple of messages. But when you're trying to match this on hundreds and hundreds of gigabytes of logs, having this be efficient kind of matters. And IPv6 is part of the problem. Matching IPv4 is pretty fast because there's really only the one way you can write an IPv4 address. IPv6, there's way too many. Um, so the nice thing is all the backends support the same API, and there's actually a nice, uh, test suite for testing, somewhat incomplete, but testing and also benchmarking. I actually spend most of the time benchmarking. So Go actually has— we actually don't want to test. I think none, something like that. Oh, not none. Oh, it's the V. Get rid of the V. There we go. Yeah, Go actually supports, instead of writing just a test function that you give it like kind of success or failure kind of things, you can write functions that are designed to be benchmarks. So, oh, it's probably better. So I actually wrote some code to generate random logs because otherwise I have to ship— I can't spell random— ship a ton of logs. So I don't want to pollute the Git repository with hundreds of gigs of logs. So I wrote some code to generate logs, and you can see some of the problems. The JSON extractor is a bit slow. You know, I could have sworn when I was running this on my Linux computer, it was about the same speed.

What's the number in the middle?

How many iterations it was able to run the test for. So the syslog extractor actually caught up quite a bit. It's actually not terribly slow anymore. So I think I just need to optimize it a little more. So the nice thing about the syslog backend, I can probably demo this. If I, let's say, go into /tmp, make, make some logs. So the way it's designed to work is we can just make, say, a 1 and just say, "This is my log," and the IP is 192.168.4.1, just appears anywhere in it, and let's say 192.168.3.4.5 is another IP. So that's 1.txt, and make 2.txt, say, 2.3.4.5 again. Okay, so I have two log files. One has two lines with two IPs, one just has one IP. And if we index it using the kind of, er, using the lower-level tools, you say backend syslog And I think star— there we go! So, indexed. So you can see the first log file had 2 lines, which is interesting because it only said 1 unique IP, but it clearly had 2. Oh, my regular expression is broken. I believe when I anchored it, I screwed something up. I think every log file that I ever tested, the IPs do not start start on the line. So we'll try that again. That's much better. So I need more test cases. Though I— do you know of any log file that starts first bytes on the line is an IP address? I don't think I've ever seen that. But I should fix it. So now that that's been indexed, all 2 log files and 2 IPs, I can say search for 1.2.3.4. And you can see that was only in 1.txt, and 2.3.4.5 is in both. So that's basically all the tool does. And if I wrote a, um, projects FlowIndexer— where is my example config? Example config. If we took config.json and say delete all of that. Logs, backend syslog, and that's /tmp/logs/*.txt. That doesn't really matter, so we will— I think I can cheat and just say that. Database root is /tmp/logs, and database path will be logs.db. I think if I did that right, I can say, "flowindexer daemon." Oh. And with that config file, it now found the two log files and indexed them. If I open a new terminal and I do curl localhost:7070/search/index=logs, I'm gonna need to quote this. And q equals 2.3.4.5. Oops, did I not say 70? I said 8080. It's amazing what happens. Oops, and I don't remember— oh, I think that's actually getting routed to like Docker or something because I have probably 3 virtual machines and multiple projects running. So we're gonna say 60, 60. I broke that. I'm going to get another popup. Oh, nope. And there we go.

You can run that in a web browser?

You could, yeah, if you went to the URL. I'll have a nice Google-like web page with a dropdown. I could probably show you— so many windows here. I must never connect to this directly. I don't remember what port. Oh, so it'll look like this. This is what the NetFlow indexer web interface looks like. Very similar, where you could type, you know, an IP address and do a search. And one thing I haven't implemented is also it does the time extraction. So instead of giving you a giant log file, It just tells you the timestamp, which is much more user-friendly. So that'll get implemented. Oh, another thing I didn't— I forgot to mention is there's also a stats endpoint. So instead of searching, you can call stats. And it gives the first and the last scene. So if I go back to here, we were searching for that Suricata, I think, IP address. Oops, con. Instead of doing a search, you do a stats, which is even faster. And you just get the first and last scene. And if I pipe this to— Is there anything that will return the count? Total, see, nothing knows the total count. This is the total log files. Yeah, that's what I meant. The count of the log files. Total log files, yeah. So actually, so now that this daemon locally is running on the temp directory, one thing that's interesting—

Stats return JSON there? Yeah. But the other stuff didn't?

It's just a flat list of log files. I might, once I return some more metadata, I'll probably make it use JSON. Just initially there was no point. Yeah. So if I go into this temp logs and simulate something new writing data. So if I write a new file, 3.txt, say another IP is 5.6.7.8. And let's have 1.2.3.4 again. So if I save this— I forget what the default interval is actually. It's a config option, so we might have to wait a minute. Let's see what the default is. Uh, flow— uh, oh, 60 seconds. So I'll let that run for a minute. So in theory, what'll happen is that'll realize, hey, there's a new log file in /tmp/logs and it should pick up 3. So to show you— oops, hold on a second, I want to get this command before it indexes it. So there's 5.6.7.8, which— oops, and I did it again. I need 6060. Oh, I took too long. So you can see it already has it. As soon as it indexed the log file, it made it available to the web API, so you don't have to wait a long time for things to be available. And if I query the 2.3.4.5 again, or what did I add in 3? 1, 2, 3, 4, yeah. So it pretty much makes it immediately available. So we have it running, you know, our Bro logs are shipped over on the hourly, so pretty much a minute or two after the hour it picks up the new log file I think it takes 10 seconds to index an hour's worth of data, and then it's immediately available to be queried. So it doesn't take very long to keep up to date. And I think when I did the initial index, I think it took about a day to do the backlog of a year and a half. And it could probably be a little more efficient. It's just, you know, how much time do you want to spend optimizing something that probably only happens once, or Most of my time has been spent currently trying to make searching faster because you'll search a lot more than you index. So that's kind of all it does so far. There's, there's a lot of things planned, like I said, doing the date extraction and the dumping support. So if I do— I don't say— I don't know if I've tested this. So that was a search. If I replace search with dump, I get the log lines that match, and that's not optimized at all, but it does work. The bigger problem is it works for individual IPs, but I need to implement CIDR searching. So while I can search for that, I can't filter for that yet because it's— oops— because it's not smart enough to know that it needs to expand the range 'Cause while it knows that 5670/24 is in the log file, it is dumb and tries to search for the literal string 5670/24, which it's not going to find. But I know how to work around that, I just need to implement it. But relatively speaking, CIDR searches aren't as common as individual IP addresses. Normally when you get intel, it's just a single IP address. Single IP address. One other thing I could show off is, so often what we'll end up using a lot is stats, just the outright, did we see this IP address? So you can see this one is a yes, this IP is a no. So it's nice to know how often we can answer that question. So I wrote a crappy little benchmark test program in Python, and if I run it and I tell it to use 8 threads, and have each thread send 2,000 searches each. How long does that take? Which is important because I think this will then search for a total of 16,000 stats searches. And 16,000, that's a decent size in TILFeed, so probably more than you'd ever need to search on, on the backlog of all the data. And I think it actually does get faster over time. So I think if I ran it again, so that did— the interesting thing is each thread did 2,000 IPs in 30 seconds. So the Go server was able to use lots of cores. So overall it did 16,000 IP addresses in that same 30 seconds at 500 a second. So I think if I actually bump this up to like 24 threads, probably. I think it scales linearly up to a point. I don't think I've maxed it out yet. And that stats endpoint really isn't optimized as much as it could be. So it probably will only get faster, which is good because people don't like waiting. So really what caused me to write this tool initially is, you know, we had, you know, couple months worth of NetFlow data, and you'd get, you know, intel data that certain IP addresses were bad, and it would take 20 minutes to do a dump across all that data. And no one likes waiting 20 minutes. If it takes you 20 minutes every time you want to see if you ever talk to an IP address, you're just not gonna bother. If it takes you 1 second, well, then maybe you're gonna do it, you know, every chance you get. And I actually do have support initial support for nf-dump. I think it's— is it in the master repository? I have a couple of different branches. Oh yeah, so there's an nf-dump backend, which is actually pretty straightforward. It just runs nf-dump with the CSV output mode, and then it's pretty much very similar to the Bro.

So with that, you wouldn't set up globbing in the config file, or would you?

You still go because it indexes each file independently. So you'd still do globbing. Basically, to implement the extraction port of the backend to do the indexing, all you need to do is take a log file and return the IP addresses.

And I— You use nfdump to read the logs and then— Read the logs.

And so one of the things I have is this IP set that I wrote. So you call ips_add_string, So you— every time you give it an IP address, it deduplicates it on the fly, so you don't need as much memory to store all the addresses. It's basically just a hash table. So that's where it gets the unique information from. And oh, and it's done. So that actually only still did about 500. I know the first time I tested this, I got it up to about 1,000, I think, because I ran it enough times. And basically, once the disk cache warms up LevelDB starts performing faster and faster and faster. Because after all, the indexes are only 6 gigabytes, and I think this system has 16 or 24 gigabytes of RAM. It's just a VM. So in theory, you could get that entire 6-gigabyte index cached in RAM, and then you're not even touching the disk to do these searches. So, which is one of the reasons why it can go so fast. And so that's a nice thing. You only ever need to go to the logs when you have hits. And that was really the key kind of observation for writing this tool initially, is no one likes waiting if they had a compromise. If you're waiting and waiting and waiting and it's actually going to give you useful logs, like, that's okay. What sucks is waiting an hour for like a grep to run to find zero records. So with this tool, you never have to wait for zero records. Zero records is always instant. It's only ever, you only have to wait if you do like the stats and then you realize like, oh, we actually did communicate with that host, you know, so now we need to further investigate, you know, who actually talked to it. But since you know you're going to find something at that point, you don't mind, you know, waiting a couple of minutes for the grep to run. And this at least narrows it down for that Suricata IP address, it narrowed it down to 75 log files. Files. And as I ran before, that's down from— probably do the math in my head almost, you know, because it's 24 log files a day times a year and a half. So that narrowed it down to 74 files that you need to grep compared to 15,000. So, and that's every one of these 75 files that you grep, you will find a match. So if you're really doing incident response, you won't mind waiting. Oh, Wayland.

Does this assume no SIEM?

He's asking like a comparison between— I've been talking to some people online. I sent some people the link to this and they were running it. And what they actually did so far was integrated it into their SIEM. So they can click on an IP address and it calls the stats endpoint to find out when they might have seen it in the past. So this could be something that you use with a SIEM.

I think he's curious about, if you have a SIEM, why do you need this when you can just search?

But I suspect this would be faster. Yeah, I mean, as you see, it's quite a bit fast. Yeah, I mean, like I said, this is 1.3 terabytes of logs, and I can do, you know, stats queries for an IP address, you know, any random IP address we get a notice for. Not 444, in 0.016 seconds. I mean, does your scene do that in 0.016 seconds? I doubt it. I'd be very interested in knowing how it works if it can do it that fast. So yeah, so it's a very limited tool. Like, after I released NetFlow Index years ago, someone was like, it's great, but we want to search by port. And I'm like, no, we can't do that. Because part of the reasons why the indexes can be so small is because I'm not indexing the ports. If you start indexing all the ports, that will blow up the amount of data that you have to index, and it won't be 6 gigabytes anymore because now you need to store every IP address and port that you saw versus just every IP address. So there's, there's definitely trade-offs. You always filter after too. Yeah, and so what I'm sorry, I don't know. Was that Waylon? Or who asked that? Yeah. Yeah, so searching for months of logs for 2 hours only to come up empty-handed. So that would never happen again with something like this. You might still have to search for logs, but only if you have hits. And what I realized is no one minds searching if you're going to find something. So if you're familiar with Bloom filters, it's a very similar concept. This you can kind of think of it as a Bloom filter on top of your logs. Explain that analogy. Yeah, so it's somewhat different from a Bloom filter in that it's exact. Where a Bloom filter might have false positives, this won't have false positives. So Bloom filter is something that you generally run in front of your data. Like I think the LevelDB backend that I'm using uses Bloom filters internally. It's a way for it to avoid doing searches if it knows that the thing that you're searching for isn't in the dataset, because it basically uses kind of a hash of the data where it might have a false positive, and then in that case it just ends up doing the search you were going to do in the first place. So, so yeah, you kind of treat it as kind of the first line of searches, and that's, it's what we do every day. If we hear about an IP address that's bad, we hit the system to see if anyone ever talked to it. And 9 times out of 10, it's 0 hits. We move on with our day. If there is a record, okay, then we dig in and find out what the actual records were. So that's basically what it does. It does one thing really, really fast.

Now is, um, what's all on GitHub? What's open source?

Uh, all of it so far, yeah. So there's, there's my older project that's very well documented called NetFlow Indexer. I tell you how to install it and how to configure it, but it's just not as fast. The whole reason why I wrote FlowIndexer initially, you know, I was learning Go and I looked over all my projects and I realized, you know, I don't really have anything that's performance sensitive except maybe NetFlow Indexer. So I wrote a really terrible initial kind of sketch of parsing the logs, dumping into a database, and it was so fast, and the files were so much smaller. So I kind of thought, okay, that's neat, maybe I'll spend some more time on this at some point. It kind of didn't really do anything for like 6 months or something, and then I picked it up again recently. So that's the FlowIndexer project, it's on GitHub.

I have— Does the NetFlow one only do NetFlow?

No, it actually has a similar It has a similar architecture. It'll do Bro too? Yeah. But we should all go towards the newer ones. Yeah, it supported NetFlow or NFDump, FlowTools, Bro. I never added a syslog support to it. But yeah, I would definitely say new users, especially if you know Go and can help, use FlowIndexer. Basically, if you have a working Go, you run these two commands and you have a binary. And then you copy the example config file and you start up the daemon and you're done. You come back in like an hour or so and you have a nice index of all your logs and then you can start hitting the API. And there are the lower level commands, but they just— it's not so much that they'll be phased out, it's the same code that's used internally, but probably you wouldn't use them, you would use the daemon unless you had a good reason. There's the one command that I haven't implemented yet is the expand cider. There's no API endpoint for it because it needs to do a little deduplication, but otherwise the API through the web interface can do everything.

And yeah, are you looking for contributors for more backends? I was thinking like Argus or—

yeah, more backends. Backend should be easy to write. Um, right now they have two endpoints. It's basically an interface in Go. There's the extract IPs, so given an open file, you know, be able to extract each unique IP address. And then there's the filter, which is given the log file, a query string, and an output, basically just act as a grep pipe. So I could show you kind of how the syslog—

So all of those, do they all include that? Backend.go file?

They don't include it, they implement it basically. So you can see the syslog backend implements extract IPs. So this is all just the BS you need to do to read files line by line and then extract the regex and then add each unique match that you find. And then the filter, this is really just kind of a terrible implementation of grep. You get it line by line, if the query is in the line, you write it to the to the output file. It's pretty terrible. Like I said, it doesn't handle CIDR searches. What I might end up doing is using grep behind the scenes, because grep is just ridiculously fast, and it's really hard to compete with grep. So I may just use that. Is the improved performance, is that due to the Go query language, or are you talking about BubbleDB using that to find— Yeah. The language or the database? Probably half and half. Go is in general— for a lot of the things that I'm doing, it's actually about the same as Python. Because if you take like a string in Python and call like string.split to get each column, that's just running C code behind the scenes. Crazy optimized, very highly efficient C code. In Go, you have a string, you call string.split, it or whatever, it's just calling Go code. It's really doing the same thing. So on some levels it's not really any faster. It does, I think, use a little less memory. Really, probably the big win is the LevelDB backend that I'm using. And I really need to get RocksDB installed, which is Facebook's fork of LevelDB, where they have a number of performance improvements. The one downside, the reason why I'm using LevelDB is in Go, someone ported LevelDB to Go, so you pretty much just import it. You don't have to compile anything. It just embeds it right into your project. RocksDB is still C++, so you need to install it separately and it's more work.

So the actual database file, is it just —text file with key colon?

Yeah, it's a— I think I can show— I think I have a Python tool that will dump it somewhere. Is it a binary? It's a binary file. Is that what— no, that's not what T is. Oh, it's probably my— basically, I could kind of sketch out. So with these text files, 1, 2, 3, that I indexed, effectively what it will write— I definitely— I know I have a tool somewhere that dumps it to the screen. I'm just trying to find it. Oh, is it this? No, it is not that. So basically how it works— so first it sees 1.txt, so it stores like doc I forget off the top of my head, something like this: 1.txt is the key and the value is 1. Just coincidence that it's 1 to 1. Normally your documents aren't 1, 2, 3, 4. Then it will see the IPs. So if I go into /tmp/logs, so first it saw 1, 2, 3, 4, so then it will store 1, 2, 3, 4 and the value using the bitset of just 1. And then— and it actually stores this as bytes, not as a string. So it will store something like that. Then you index document 2. Document 2 has 2, 3, 4, 5. So what it will then add to the file is doc_id 2.txt is document 2, and then it will add document 2 to that. But really, since I'm using a bitset, it's effectively storing it in binary 3, you know, in binary, because this is really kind of like 1 and this is like 11 effectively. And that's one of the reasons why for most use cases it's pretty compact. It doesn't use a lot of storage space. And then once we store 3, 3, so then it'll write out doc_id 3.txt is 3, and that has a new 5678, which goes to key document 3, and then 1234 will be in 3, which again is binary, will end up looking something like that. And this one will look like that. It's a little less efficient than I had wanted because I think I realized even though mstring is a bitset and I know exactly how many bits it should be, I think it puts a header telling you how many bits there are, which isn't really needed for LevelDB. But it hasn't really been a problem. Like, maybe I could get that 6GB database down to 6.5GB or 5.5GB, I don't know. It's kind of have to pick your battles. So basically that's how it stores it. And actually then it actually stores this reversed, so it'll actually store mapping of 1 to 1.txt and 2 to 2.txt. Because when you do a search, effectively what happens is you do a search for, say, 2345 that returns that it's in— oh, it's in documents 1 and 2. Then it finds documents 1 and 2 and says 1 is 1.txt and 2 is 2.txt, and then it outputs those 2 documents. So it's a very, very simple datastore. It's kind of— it's actually an interface as well, so I'm— I actually started using I think BoltDB, which turned out to not perform as well as I had hoped, and then I switched to LevelDB. So that's actually an interface as well. Yeah. Oh, we lost our lights. So there's actually only a couple of functions that a backend has to implement, which is, you know, addDocument, hasDocument, queryString, expandCiter, which is really, it's really the core of it. So when you're indexing, you add one.txt with the list of IPs, and then later you query it for an IP and it returns a list of documents. So there's really not that much to it. I tend to rewrite a lot of this kind of on the fly. Like I implemented this one interface and I realized that's too limited, and I'll make like a structure a little more complicated, add some more metadata. I'll probably definitely do that when I change it to add support because really what I want is instead of returning a document like this, like no one wants that. What I want it to do is just return 2016-03-08 at, you know, 1600 to the user, which is much more user-friendly because you don't care about the file name. So I need to write some functions to turn log files into timestamps. In kind of a generic way, put that in the config file, and then you could have a little more unified API. Instead of getting filenames back, you get times, because that's really what you want. What I really kind of want to do, which would be really neat, once I get it so I'm returning timestamps, what I want to do is be able to return the appropriate amount of data to render this, because that's a really compact representation of like hits per day for an entire year. So I could have some code that renders this based on— Oh, the graph. Yeah, the graph. So imagine you do a search for an IP address and you get this style graph for when you had hits, as opposed to a giant list of documents, which is kind of hard to read. Because one thing that's really easy to tell when you're looking at, you know, someone that actually contributes more, say like this guy, is you can tell does someone work on the weekends, you know, if it's If it's dense and dark in the middle, it's during the week. So if you're looking up an IP that gets a lot of hits during the week, you can kind of tell that it's based on, say, user traffic versus if it's only on the weekends, that might tell you something else. So, yeah, so as soon as I start returning timestamp data and I can use it as structured information, I could probably have a script or something that renders this type of table. Put it on the web interface. So one step at a time. So yeah, so that's, that's FlowIndexer. It's kind of a very limited tool, but I'm just trying to do one thing really well.

So yeah, it's definitely useful.

I may try to deploy it on some networks that I work with. Yeah, it's— and one easy way is to be able to And it's funny, I mean, we have a lot of logs. And we're probably in the middle, like, you know, if Wayland or someone tries to run it, you know, the main campus, I'm sure your logs blow ours out of the water. But I ran this at home and it does like the last 6 months of my logs in like 5 minutes, indexes everything. I think my index files are like 10 megabytes. So it really scales down pretty well too. So if you just have like a tiny, like, NSM sensor at home running Bro or Snort or something, you know, you can easily run this. You don't need a giant system. Like I said, I think the box that I was running all these tests on is a VM with I think maybe 1 CPU and 16 gigs of RAM. So it really doesn't take much. Like I said, yeah, good luck trying to index 1.2 terabytes of log data with really anything else and be able to do it on a server with 16 gigs of RAM and 6 gigs of extra disk space.

So, all right. Yep, we're almost at 7. Perfect, good timing. All right, go ahead and hit stop share if you want. Oh, what else you want to do?

One question. Okay. And like I said, this is pretty much the first thing I've tried writing in Go that actually actually does something. So the code is kind of terrible. I've kind of been making it up as I go. I need a lot more test cases, but what's surprising is it tends to just work if I write something and, you know, think about the interfaces on how it's supposed to work. I've ran into very few things like segfaults, or— well, Go doesn't really segfault, but it kind of just tends to work, which is really surprising. Like, I added the web interface. I told it to run the web interface in a separate goroutine, which is like a thread. I actually have it running the indexers in parallel. Like, I changed a couple lines to do that. It worked the first try. No deadlocks or corruption or anything weird like that, which is pretty amazing. So I've been pretty happy with Go. I just, I have no idea what I'm doing, which is surprising that I've been making so much progress. Like, I didn't read any books or anything. I pretty much just read the tutorial and and just dived in. So it's pretty easy to work with. That's a good way to do it. I mean, and that gets you moving and that's kind of part of it. I tried that with Rust the other day. Didn't really make much progress. Rust, you need to read the book. So I don't know as much as I would like about IPv6, but is there a standard way to write the addresses that would make them easy to search? No. Okay. But you can normalize them. You can always normalize them. That's what I meant. Yeah. Yeah, could they be normalized to something that was easy to search? Yeah, the software does it for you because I store it as the raw bytes. I think I have a blog post on my crappy blog that I recently updated that I think I talk about that. So negative. Yeah, it is crap. I've had this blog for a couple of years. These are all my posts for the most part. Normalizing IPv6. So you can actually get the operating system to do it for you. So yeah, you can take an IP address— so this is kind of how it stores it, just as the— or actually it's this. It stores it as the raw hex bytes. So even if you take an IP address that has missing sections or lots of zeros in it, it all gets normalized to the same thing. So you won't have a problem where if your log file stores an IPv6 address in one format and you're trying to search the index in another format, you don't find it, it won't have that problem because it normalizes it to bytes.

So, yeah. Okay. Alright, I want to give a thanks to Justin for coming out and giving this presentation. It's always good to have you. And check out his other presentations online. We have a few on YouTube on AFL fuzz and ninfo and blackhole router. And contact him if you have any questions. I'm sure he's freely available. JustinAzoff@gmail.com? Is that the one to use? All right.

Same name on GitHub and Twitter and just about everywhere else. Sounds good. I don't do a good job of hiding. And yeah, if you know Go or would like to see different backends implemented, I could— if you have a log file and it's publicly available and there's an open source tool that parses the log, I could easily add support backends for it. It's pretty trivial to support a new log file. I mean, the syslog one even goes a long way. The syslog one could index probably any plain text log file. Really only needs custom support at this point for things like nf-dump that have binary logs. Mm-hmm. Okay.

All right, we're going to wrap up. Thanks everybody for attending.

Chet, stop. Stop staring. Perfect. All right.