OpenNSM (Stenographer - Graeme Connell)

Hello, hello all. It's, uh, time for the meeting of the Open Network Security Monitoring Group at the University of Illinois at Urbana-Champaign, a weekly NSM student group meeting where we do a lot of presentations and we do research and the like. Let's go ahead and jump into the group updates. So we do have a Twitter account now. You can use @OpenNSM. We'll be using that to tweet anything that we're working on, GitHub repositories or the actual talks and guest speakers. Et cetera. You can also use a hashtag #OpenNSM. We do have a sponsorship document up, so we are looking for sponsors. If you're an organization and you think that you'd be interested in sponsoring us, do check that out. I added a ranking system, so we have tiers now based on the amount of monetary donation that you can— you get certain benefits such as advertising and access to our lab once we actually able to fulfill that goal. We also have a GitHub organization. We actually changed a few weeks ago from a GitHub account, which didn't make much sense, now to an organization. So if you're interested and would like to collaborate on projects with us, do send me a message and I'll actually add you as a collaborator. I applied for an Amazon grant a few weeks ago and a— OSU state grants for host requests for our website. So Amazon grants to allow us to do some research in the cloud. Got a few ideas there. You can actually go over them in the sponsorship document. So kind of waiting to hear back from both of those organizations. And we do have our Xen system up and running now. So using the Xen project on Linux. It's a hypervisor and we'll be able to use that to begin our development systems. And eventually we're going to have— our goal is to have a 4-node Ganeti cluster, which is one of the things I'm currently working on and trying to acquire hardware for. And finally, the last thing we have is we have email updates. So we now have a few different email addresses that you can use to contact us. talk for speaker requests, admin for server and website information. Information, notices, etc., sponsorship, and info. So do check those out or pass those along if you know anybody who wants to ask us any questions. Okay, so now we're gonna move right into the meeting section. So Wireshark 1.12.4 was released, and that's actually just a big bug fix release. There's quite a few of them in there. You can check it out, and there are some updates from a number of the protocol dissectors. So do check that out. SANS blog put out an article showing on Windows what you can do if you do not have Wireshark or TCPdump or related programs installed to actually do a packet capture. And this is kind of cool, I actually didn't know this, but mostly because I don't use Windows very much. But there's a tool called trace that you can use from, you can call from the netsh command and actually record packet captures and then of course it's in its own Microsoft ETL format right here. So you will actually have to convert it but it actually gives you a little script at the bottom to actually do so. And then you can read it in with the other tools that read the standard PCAP format. So do check that out if you're on Windows. Also, Cuckoo Sandbox 1.2 was released and it's actually a huge update from them. They added support for XenServer and running Cuckoo on bare metal to do the analysis, among a large number of bug fixes and improvements in their actual web user interface. So do check that out as well. Brotop by Dustin Webber, who we had on a few weeks ago, has now version 3.0, and they actually added a He actually has the ability to toggle, toggle, excuse me, your different row logs over here in this corner and it'll actually show up as they come in real time. So that's kind of nice. Now we're moving on to conference corner where we talk about events that are coming up. So Marcin of Malwarebytes will be at the Engineering Open House here at the University of Illinois and that is next week. Sadly, I won't be able to make it. I'll be out of town, but if you're a student, I do recommend checking that out. Also, Richard Stallman will be here next week as well, and because of this, we will not have OpenNSM because it's during our time period and everybody's going to want to go to that, including myself. So we're going to skip next week. Also, UIUC's SigPony group, which is a special interest group for offensive security that meets weekly. They're actually hosting their own CTF, an online CTF, so do check that out. That will be next month. And also the announcement for BroCon. BroCon 2015 will be in August, and that will be held at MIT. So do check that out.

Jon, can I give a quick addition to your comments about the CTF?

Yeah, sure.

I think we're still needing a couple of challenges in exploit and reversing categories. So if anybody feels confident with creating an exploitable piece of code that after the exploit releases the key string, the flag string, or something similar in reversing, should get a hold of me, I guess, at Shane@ShaneRogers.info. There's no D in Rogers.

Okay, thank you. You can post that in the chat too if you'd like. And now, Josh, let's see if Josh is on. Yes, Josh is on. Hey, Josh. Josh is gonna demo his— he released an SSD parser for Bro. A little Bro script that I have attached here in the signature selection here. So we can go ahead and pull that up. And actually, I'm just gonna pass it to him. It's in your repo? No, it's on theirs, CrowdStrike. Okay. It's in the meeting notes here. You guys can check that out. Josh, you want to switch screens here?

Sure. Can you hear me?

Yeah, let me go ahead. Yes, we can hear you. Let me stop my screen sharing and we'll pass it to you. Stop share. Okay, and then you should be able to click the green screen share screen button.

Okay, can you guys see my terminal window?

No, we cannot. There we go. This took a little bit.

Okay, great.

It's on full screen for us now.

Awesome. So yeah, Jon, you reached out to me just to give you guys a brief overview of what this thing is. So I was doing some analysis for some of our customers at CrowdStrike, and I was looking at seeing which services were, you know, occurring quite a bit on their networks that we weren't already catching with Bro. And one that I noticed was the standard ports for SSDP, which is the discovery portion of UPnP. And I didn't think that there would be anything too significant there, but I wanted to know what kind of data was being passed around. So I wrote this pretty simple parser for Bro that's written entirely in Scriptland, not in Bro Core, so it's not C++, which would be the ideal way to write this. But for like a toy parser, Scriptland works fairly well. So just to give you guys a quick idea of what is in this log, and also a short overview on SSDP if you haven't looked at it too in-depth, it's essentially HTTP over UDP. A lot of the strings inside the payload match HTTP headers, so they're pretty standard conventions there. So what you get in the log, it creates ssdp_log if you run the script that we have on GitHub. It gives you the connection5 tuple, which is your unique ID, your flow data, and then all the SSDP-related data is also parsed out. So if the, if the If the packet is a search request, you get the search target. And all this data in this log is just pcap collected off the internet that I merged together. If the request is like a notification or a response to a request, then you get the same data. So it's the same type of data. It's just from the server point of view. Essentially, so like a device comes online on your network and starts notifying, you know, the network just to make itself known, or, or someone queries something, it responds, etc. And then in the response, you get a lot more information as well, including server, USN, and location, which is potentially interesting. Server tells you a bit about the device itself, what operating system it's using, the version of UPnP. And the, I believe, service. You also get specific details about what it's hosting, sometimes in the USN. So you can see here, this one says TV Emulator. And what is potentially most interesting about this is that you get the location of the description for the device. So here's the HTTP URI for that. So theoretically, you could write a second script to see if any devices query that to pull the description back. And similar to the HTTP log, it also— this will also log all of the headers seen in the connection. So scrolling over here, this example on the bottom line was a server, essentially what I'd call like a response query. Where the headers were contained— contained a location, a USN, a server, the search target, cache control data, host data, and NTS data, which is not logged. So let me change my screen here. Okay, can you guys see my— it should be Sublime Text.

Okay, great.

Yeah, looks good, actually. Looks really good.

Cool.

So, so this is the actual script, and I won't belabor this at all, but I just want to give you guys an idea of what it looks like. It looks very similar to a main.bro script for any of the native protocol parsers in Bro, where you have an info record that contains all the data in the log. There is some— there are some details there about what each field is supposed to contain. They're not all— they don't all follow the RFC, obviously. So, for example, in the server field, you're not always gonna get an operating system name or the OS version. And then moving past that, there are multiple functions that are called to clean up the data. This is where the script is not as efficient as a native core C++ analyzer would be. Because we're splitting the data a few times and pulling, and just pulling out what we want and formatting it. Additionally, there's a set session, just so before the log's written, we set the data in Bro to, you know, put the log data in a specific place, right? This is pretty common if you've ever looked at a main.bro file. And then there are two functions that are called from a signature. So this uses the signature framework to detect SSDP on any port, and these, these two functions, ssdp_request and ssdp_response, are called when either a request or response are seen. And that information, the data from the payload, is then sent up to the functions earlier, parsed out, brought back to this function, and then built into the log. And once that is complete, so once the connection is removed from Bro, you know, for whatever reason, removed from memory, ends successfully, whatever, the log is written. And because these are UDP packets, they, they don't typically hang out in memory very long, so log is very quick. And that's pretty much it. Do you guys have any fast questions about that?

Seems pretty straightforward. Okay. I appreciate you sharing that. That looks— that's nice to have for sure.

Yeah, certainly. And if you guys have any questions, you can reach out to me. My email is on the GitHub page, and I'm on Twitter as well.

Excellent. Thanks for sharing that, Josh.

No problem.

Alright, and now we have our guest speaker up here. Um, Graham, you ready?

Yeah, I'll start sharing right now.

Okay, yeah, so Graham is from Google, works at Google, and he works on a tool called Stenographer, which is a fast packet-to-disk spooling program. So, uh, Graham's gonna talk about that today, and I do appreciate you coming out, Graham. I've been actually looking forward to hearing about this for quite some time.

Wonderful! Let's get started here. Uh, let's see. I can begin. Okay! So, uh, I'm talking today about Stenographer, which is a system that we developed at Google over the last couple months, uh, whose purpose is basically writing packets to disk. We're on GitHub. We'd love it if you guys would come in, take a look at the code, give us any suggestions. But let's get started. So what is Stenographer? It writes packets to disk, and then it gets them back again. It was written at Google, but it was a 20% project. If you're not familiar with the idea of 20% time, it's basically we are paid to work 80% for the company and 20% to do things that might be useful eventually, sometimes, maybe. And so this was one of those 20% things. As a 20% project, it is supported by us, the authors, and not anybody else. If you try to call the tech, you know, tech support for Google and ask about this, they'll laugh at you. And it's basically designed to be a simple tool to solve a very simple problem. This is the Unix model of do one thing, do it well. In particular, it is not a, you know, full NIDS solution. It is not a, you know, security event manager. It is nothing like that. It is just one tool in a toolchain. So, what does it do? Well, it writes packets to disk. If only it were that simple. Unfortunately, there isn't actually a way that you can tell your computer, hey, here's a NIC, here's a disk, a while. So the first thing we have to do is actually write from NIC into RAM and then to disk. And one of the things that's actually really cool about Snoggr is it just writes the packets into RAM once. And that really doesn't sound very impressive at all, but it is. And here's why. Normally when you read a packet from a NIC or using a typical syscall like receive or receive from, something like that, you're actually doing a couple of copies of the packet around. So the packet will hit the NIC and it'll be copied into memory in the kernel's buffer space. Then you'll issue your syscall, which will have to context switch back to the kernel. The kernel will copy the packet again into your local buffer and then return control to you and you can work with it. But we just have like 2 context switches and 2 packet copies. When you're writing to disk, actually the same thing happens. So you issue a write on a file descriptor. It copies the packet or whatever buffer you're trying to write back into kernel space into a kernel buffer. Context switches into the kernel. Things get written back to disk. And so we really see that there's actually 3 copies here. There's Nick to kernel, kernel to user space, user space to kernel, and then kernel out to disk. And we can do that by skipping 2 of those 3 copies, which is pretty cool. And the way we do this is we use the features that Linux provides us. So this little penguin down here helps us quite a bit. So the first thing Linux has a capability called afpacket, which allows the user space program to say, hey kernel, I'm gonna want packets. I'm gonna want a lot of them and I'm gonna want them quickly. Please give me a piece of memory and then just put packets in there. And so the user space might say, you know, here's a 10 gig or here's, I would like 10 gigs for packets. The kernel will create that 10-gig region and then as packets will come in, it'll immediately put packets into that region. So the packets never actually hit like a kernel buffer space. They go directly into this memory region. So we skip that first copy entirely. Also, that memory region is accessible directly through user space. So we also skip any of the context switches when it comes to reading packets. This is pretty cool. Very cool. Second thing is that when we're writing, we actually do almost exactly the opposite. When we're writing to disk, we use a kernel mode called O_DIRECT for writes. And that basically tells the kernel, write directly from user space, do not do an additional copy into a kernel buffer beforehand. And so by using these two things, we avoid the two copies, right? We go directly from NIC to memory. And then directly from memory to disk. That's pretty neat. Um, one of the things we do along the way is we actually index packets. So it turns out that if you just write, you know, 10 gigabits per second of packets to disk, you have a whole bunch of really big, really useless files. So we add an index along the way, which allows us to later on query interesting packets without having to read through the entire contents of the disk stuff again. So our indexing is relatively minimal. We, whenever we get a new set of packets in memory, we go through that set of packets, we pull out IPv4 and IPv6 addresses, ports, Layer 4 protocols, and MPLS labels and VLANs. We write these indexes out to disk. Now, this sounds very simple. Sorry, what's up?

Yeah, hold on just a second. Hey, someone joined the call and is— their microphone's really loud. Could you mute it? I would, but I just— my machine just froze up too, so on my host machine.

I believe it's possibly the person in a car.

Hello?

Okay, I guess we'll continue.

Yes, go ahead and continue. I apologize.

Oh, no problem at all. So we write these packets to disk, and one thing that's very important to notice here is that this is the one and only time the stenographer actually looks at the network packets itself. So this is the time where Stenographer will take the bytes that come in from the network and try to do something with them. And the savvy among you will realize very quickly that this is probably where, if you were trying to break Stenographer itself, you would focus all of your efforts. So we'll get into how we actually save that in the future. But the short story is, if you ever process bytes that are coming in from a possibly malicious user, You have to be very careful. Okay. So, the next thing we can do, and again Linux provides this as well, built in, in the AF packet mechanism, is load balancing. The idea here is that instead of saying to the Linux kernel, "Hey, please give me a memory space that you can write packets in," you can say, "Hey, please give me 8 or 10," and load balance the packets across them. Now, it turns out that this is really, really important for one reason, and that is that disks are a lot slower than NICs. So, if you have a typical disk, like a high-speed disk these days is about 15,000 RPMs, and that'll get you up to about 200 megabytes per second, but generally around like 160 to 180, and that's about 1.2 gigabits per second. The NIC can very easily handle 10 gigabits per second, And that is a big problem. So we know that we need to split this up across multiple disks, and Linux allows us to do that by using afpacketfanout. The other nice thing that this does is it gives us an easy way to split up the actual processing of the packets as well, right? We want to index the packets, so we want to process them. And if we're already splitting things up, we can now, you know, use different CPUs to do this as well. So that's pretty cool. And so everything we've talked about right now is about filling up disk. These were really good at it. It turns out you can also fill up disk by like piping /dev/zero into the disk. But once we have the packets on disk, now we actually need to do fun things with it. The first thing we can do is give analysts the packets that we've written to disk. You'll notice here that this analyst is very sad because we're giving him all the packets that we put on disk. And it turns out that if you give a security analyst, you know, the last 4 hours of traffic on a 10-gig NIC, they're not very happy. But luckily we have that indexing. So now we have a very happy analyst who can just say, you know, I would like all the packets for IP 1.2.3.4. On port 456 and blah blah blah and get back just the ones he cares about. So in a nutshell, what Stenographer is really giving us is a real-time database of packet history, right? Packets are sort of put into this database as they're pulled off the NIC and put onto disk, and then analysts can issue queries to this database using a typical BPF filter that then allows them to get back pcap data. So there's a little server that runs on this machine. You do a POST request and say, you know, here's the BPF that I care about. It goes through all the indexes, finds all the packets that you care about based on this BPF, and spits them back as a single pcap file. You can then do whatever the heck you want with this pcap file. You can Wireshark, you can run Bro scripts on it, you can do a whole bunch of other stuff. Our BPF is actually a very small subset of sort of typical BPF stuff. We support host, port, VLAN, ipproto, TCP, UDP, and we also have a quick extension to handle time because normal, normal BPF doesn't have any concept of time. We can say things like after 3 hours ago or before a specific timestamp. To focus in just on a particular period in time. You can also use parens and ands and ors to get things, you know, just the way you like them. So it does one thing and it does it well. Why would you ever want to do that one thing? Well, the first type of people that gets really excited about this are people that actually do incident response and forensics, because normally it's impossible to actually do forensics on a network, you know. Something happens and all of the data related to what just happened is immediately lost. And so actually being able to go back a couple hours, a couple days, depending on your traffic, the amount of disks you have, stuff like that, becomes really powerful. The second thing you can do is proactively pull out specific pieces of network traffic based on alerts or other events to provide context for security analysts. So the idea here would be you'd have Snort running or Suricata or something like that. It would generate an alert, and if you know it, like a typical Snort alert will say, okay, here's an alert that happened, and here's like the 3 packets before and after it so that you have some context. We could, for example, tail the Snort log and pull back all traffic related to those 2 hosts in the past 3 hours, and we would get other TCP connections that are unrelated to the Snort event. We would get, you know, data going back over a slow connection over a very long period of time. We get all this other stuff that, as a security analyst who is responding to that event, would provide a ton of context and allow them to actually really see what was going on and handle it accordingly. So that's super cool. More details. So I'm going to talk a little bit right now about, you know, some of how we actually build this. I can go into much more detail as, as necessary. I'll leave that to the questions at the end, but if folks are specifically interested in, you know, the processes we went through to build this, any type of tooling that we used, etc., etc., happy to answer all that. So first off, a very high level of the architecture that we have. When we say that we're running Stenographer, we're actually running two separate processes. The first, called Stenographer itself, is a process that we wrote in Go, and this process handles disk management, so cleaning up old files as the disk fills up. It actually responds to queries, so it's the little web server that you talk to that will take in a query, translate that query into a set of index lookups, and then do seeks based on those index lookups to pull packets out of the packet files and return them all to the user. And the third and most important thing it does is it runs a secondary process called stenotype. Now, stenotype is the process that actually writes these packets. Everything I talked about before with af_packet, nodirect, and all that kind of stuff is going on inside stenotype itself. It's written in C++, and basically all it does is writes the packets and computes and writes the indexes for those packets. Communication between these two processes happens basically just on disk. The stenotype will write out new files, stenographer will see that new files exist and start using those files to serve requests to analysts. So I mentioned earlier that the indexing is sort of our scary part here, right? There's one piece of code that actually tries to parse out packet data, which is provided by possibly untrusted sources. And do crazy things with it. And so we really want to focus on that and make it as, you know, absolutely safe as possible. So we do a couple of things with this, and if you're ever writing anything that, you know, actually hits packet data or something else provided by an attacker, I recommend, you know, finding anything you can possibly do, doing it, and then asking 6 other people for other things they can do and doing those as well. The first thing we do is flip the indexer. So, we pull out the indexing-specific code and we run a very powerful fuzzer called AFL on it. AFL stands for American Fuzzy Lop. I don't know why it's called that. But AFL instruments code and basically instruments the C++ compiler. So, it compiles the code and finds all of the possible jumps So, everywhere where a test is done, and the code branches, it then runs the code that you give it with output— with input that it changes, and it keeps track of which pieces of input change which paths, which code paths are being done by the tested code. And this allows it actually, you know, with very high confidence, basically traverse every code path and try to find all possible bad things that are going on. That's pretty cool. The second thing we do is we drop privileges. This is a pretty easy thing. You know, once we've opened up the sockets that actually pull in the network information stuff, we can switch to an unprivileged user. We do compile-time hardening, which is basically going online, finding all of the G++ flags that, you know, have protections against various attacks and implementing all of them. So we have stack guards. We do randomization of libraries to prevent return to libc, a whole bunch of other stuff like that. And the last thing we do is self-sandboxing. I'm actually gonna jump into code here just very quickly and show you what we do here. So, seccomp— by the way, can everyone read this?

Can you increase the size?

There we go. That's good.

There we go.

There's this thing called seccomp, which is built into the Linux kernel, and it basically allows a process to say, hey, these are the types of system calls I'm going to make from now on. If I make any other ones, kill me. And so what we do here is we actually say, hey, okay, after we've, you know, opened up our sockets and stuff like that, sandbox me. Say these are the only system calls that I'm going to make from this point on. If I make any other system calls, kill me. And this is an extremely powerful way of really making sure that even if, you know, our system did go and someone came in and they had full remote code execution, there's still a lot of stuff they can't do because they can only do the things on this list. That's pretty neat. So, live demo. I'm going to pray to the demo gods here. We're going to actually jump in.

Woo-hoo!

They are pretty good demo gods.

Pray to them.

Yes.

3 seconds to back it.

Yes.

Yeah. Excellent. Okay. So, I'm running on a live machine that we have here at Google that's actually processing real network traffic. And the first thing I can do here is look and see that I do, in fact, do have Snodrver running. So, I mentioned we have these two processes that are running next to each other. Here's one. Here's the other one. This one is the one that actually writes out stuff to disk, and this is the one that's going to service my requests, clean up disk, that kind of thing. One important thing to note here is threads=8. This basically says I want my data as it comes in to be split up 8 ways. So that I can use 8 cores and 8 disks to write it out and stuff like that. If I actually look a little more at how this is set up, you can see I've got a whole bunch of mount points. These ones are pretty standard, but then we have these down here, sdb or sdc, that are where I'm actually storing packets. So, the first 8 here JBOD 0 through JBOD 7. By the way, JBOD is just a bunch of disks. So JBOD 0 through JBOD 7 are just single disks, ext4, 550 gigabytes, 15k RPM disks. You're gonna notice that all of them are around 88 or 89% full. That's because Stenographer has been configured to leave 10% of the disk free at any time. If it ever goes above 90%, start deleting files until it gets down to 90% again. The last one here, J8, is actually a little special. You'll notice it has a lot more space and uses a lot less than all the other ones. So these 8 up here actually store the packet data. This 9th one here stores the index files for all the other ones. So you can tell that, you know, 8 disks can be 90% full and the entire index file for all of those 8 disks will take up 50% 15% of a 9th disk. So its indexes are way, way smaller than the actual packets themselves. Graham?

Yes?

I just had a question. So what you're doing here is you're rotating out the packets in each of these drives, so as new one comes in, then old ones are being overwritten. Is that right?

Yes, definitely.

To keep it at 90%? Yeah, and so you can actually So what's the kind of overtime on that? Like how long does a packet last in that scenario?

Good question. The answer is it depends very much. So by the way, this is just my configuration file. You know how I do this. If I wanted to, it's, it's not necessary here, but this basically says here's all the threads I want to write out to. I could say for each one, maybe if this disk was smaller or larger, I could say keep this much percent open or not. But actually, let's take a look at one of these directories. So, JBot0, let's say, LHTF. So, here's the actual files that exist on disk. So, files are rotated out if either they hit 4 gigs or after every minute, whichever is first. So, these ones are all rotating. They're not 4 gigs, which means they're rotating once a minute. And you can see here, you know, here's 1 minute, 2 minutes, 3 minutes, 4 minutes, etc. So this basically means— remember that this traffic is load balanced across 8 disks. So currently on this system, we're processing in about 16 to 20 gigs every single— every minute, and that's being load balanced across 8 machines. So we start up here. We go right now from 20, from March 9th at 20 to March 9th at 23:36. So on this particular machine with the amount of traffic that's coming in, we're able to hold about 3.5 hours of disk, 3.5 hours on disk. Now I looked at this Monday morning, like I looked at this this morning and it held 2 days worth of data, right? The big reason for that is, you know, the amount of information that's coming in and the size of the disks you have determines how much you can hold. So, over the weekend, there's very little traffic. We get, you know, on the order of hundreds of megs per minute per disk, and we can hold a heck of a lot more of it. Right now, because we just finished a busy workday, we're holding about 3.5 hours. This tends to range from about 3 hours on the low end for this particular machine up to a large number of days if it's a long weekend or a holiday week. And this will, of course, entirely depend on your network traffic. So this is far less than the 10 gig input that we can handle. This is probably on the order of 1 to 2 gigabit per second. With 10 gig per second, it's pretty easy to consider. You have 10 gigs per second is about 1.25 gigabytes per second. You have 550 gig disks and you have 8 of them. So, you have 4 terabytes of space divided by 1.2 and that gives you a number of seconds you can hold. So, it totally depends on the amount coming in at any given time. So, looking back at the—

I have a question.

Yeah.

Oh, sorry. How is it divided for each disk? Is it like— does it do it by like flows so that one goes to this disk and one goes to the other disk? Like, how does it divide them?

That's a great question. It's actually configurable. So again, we use Linux afpacketfanout to do this, and there are a number of possible algorithms that you can ask fanout to do. Right now we're using extremely simple load balancing, which is basically mod by the number of packets that are coming in. And remember here that we're actually not trying to do any type of stream reassembly or anything like that, right? If I query these packets, I will get back all packets across all disks, and I can do stream reassembly there if I feel like it. But using the simplest approach here actually provides a the easiest solution. There's a number of other ways that you can do it. You can actually do it by a hash of the network tuple if you feel like it. You can do it based on the CPU that handles the IRQ for the particular set of packets that comes in. So there's a number of possible options for good ways to do this. But yeah, right now we're using the simplest thing we can.

Okay, cool.

And now, are you, uh, Are you implementing any means of like more permanent storage as these things are getting rotated out, or just like overwriting them? So, wondering like how long the packet's in there if you want to look back at this traffic.

Yeah, at the moment, I mean, we consider that 99% of this traffic is totally useless crap, right? I mean, there is very little reason to look at most of the stuff that we are logging here. And so, Our current idea, our current model for this, we're just rolling this out. I mean, we were just code complete like a month ago or less. We're just rolling this out now. But our idea here is to basically selectively pull PCAPs out of this and store those longer term. So we would, for example, tail our Snort logs or our Suricata logs or our Bro logs or whatever logs, look for interesting sets of hosts, you know, 1.1.1.1 talks to 2.2.2.2. Something fishy goes on, so we run a query against Stenographer, pull out all pcaps related to that particular pair of machines, upload that to some service and store that basically ad infinitum.

But the actual—

So it's sort of like a surgical pcap full packet storage.

Yeah, exactly. So this basically allows us to really just augment the context of network intrusion detection alerts that we get. And because we're storing, you know, we're pulling out those types of things, and that's an extremely small amount of the total traffic. So the amount of disks that we would need for long-term storage of that type of stuff is minuscule. It also means that uploading it to an additional, to an extra server doesn't take that much. The savvy person will quickly note that if we try to take a 10-gig NIC and upload it into our production network, we very quickly run out of network. Yeah. So this provides a nice easy way of doing that. So looking at this, at these packets, you can see we're getting, you know, about 2 gigs per minute per disk. If we actually look at the indexes for this, we see a much different thing, right? These are, you know, 40 megs up to like 50 megs for 3 gigabyte packet files. So the indexes are really extremely minimal and they actually allow us to query them very quickly as well. So all of this allows us to really get, get the data that we need very quickly. We currently do NetFlow and a couple of other things. Which means that this data on disk here is relatively redundant. But if I were to pick something that I wanted to pull up and store, you know, forever, it would be the index files, right? Because this basically provides us with an entire map of every packet that's been going across our network, what it's been going to, and what it's been going from. Doing NetFlow is actually a little easier because it's much better indexed than this. But, you know, this would be one interesting way to pull large-scale metadata out of this type of information. Any other questions?

Andy, Andy has a question. He says, do you always want a different disk per thread?

That's a great question. I think the answer in general that we found so far has been yes. And it's really just because a CPU can handle far more traffic than a disk can. Right now, I don't actually have the numbers for with indexing, but without indexing, we can write to a disk using like maxing out the disk and using, I think, like 3% of the CPU of a single core. I think with indexing, it's somewhere around like 50% of a single CPU. So we're really— It's really easy to max out a disk using a single CPU. If you happen to have very slow CPUs and very fast disks, if you had, for example, like really hardcore SSDs or something like that, you could totally write multiple things to a single disk. And that's actually why we decided to choose using sort of directories as our method of configuring this stuff, because all we say here is, you know, write to this directory, write your indexes to this directory. You as an administrator can choose what you want mounted underneath that directory. I mean, these could all be on a single disk or they could all be on multiple disks. And for instance, right now all of these index files actually are backed on a single disk and all these packet directories on separate disks. So, it's totally up to you as the admin.

Are those disks connected in any special way in your current configuration?

No. Okay. I mean, I think right now, just in order to— if I'm remembering correctly, this is actually attached to a RAID 0 controller. But that's simply because we didn't have enough PCI slots or whatever the current thing they're attaching to disk is these days.

Right, right.

You know, you need to be able to actually plug in that many disks, and it turns out a RAID controller is a pretty good way of doing that. But for example, if you did actually use like, if you used, you know, any version of RAID that actually did processing across multiple disks, that might be a reason to have like a single logical disk that you would write to. Yeah, so let's actually do some stuff here. So I'm gonna time this just to see how it goes. But right now we just have a very simple command line tool called steno-read. And you give it a BPF like 6543, and you give it some tcpdump flags. And right now all it does is it curls to the local stenographer server. It gives it the filter, it takes back the pcap, and then just for ease of use, it actually throws that pcap through tcpdump so you can do other stuff with it like print it out, save it to disk, that kind of stuff.

So for example— Can you clear your screen? We're missing the bottom.

Oh, sorry. Yeah, totally. Um, one second. Let's see.

Hello?

Okay, there we are.

Yeah, perfect.

Okay, so this first thing up here, this is BPF. This is a flag passed to the easy to dump. It basically just says don't go and do DNS requests for all this stuff. And then just because I'm tricky and because this is actual Google network traffic, I'm going to throw it through something else that makes it all crazy so you can't read what's going on on the network. So basically what I've done here is—

oh yeah—

all the digits with digits and all the uppercase characters with uppercase characters and lowercase with lowercase. So you can't actually tell what's going on, but it should be relatively obvious that this looks like keycap that's been processed by tcpdump. Ooh, crazy. And you can see it's relatively fast, right? You know, we're doing on the order of half a second. One of the things we noticed relatively early on is that as well as being good at doing all of the, like, reading packets and writing packets and stuff like that, that Linux seems to excel at. It also is really good at disk caches. So if we run a similar query multiple times, you know, we get very, very fast results for a couple of times. But— oops, let's try that again. If I actually want to show you how this really works, I should probably be slightly better at that. So I'm going to drop all my caches from my disk. Do, do, do, do, do, do. As you can see, there's a lot of stuff that's cached. And try this again. We'll just do wc -l here so you can see how many packets there are. Well, we run it. It runs. It runs. And OK, this time it took me 3 seconds, right? I run it again. OK, now we're back to the 1 second. Tenth of a second, half a second range here. So you can really see that the disk caches help us a whole ton. This becomes especially important if, for example, you're using parts of your filter that are very common. So for example, here, inside stenographer, it's going to find all packets with port 80. It's going to find all packets with this port. It's going to collapse those down by doing a set intersection. Then it's gonna go find those packets, pull them from disk. It turns out there's a lot of these, right? So, this is gonna take me a relatively long time. In the past, this has probably taken about a minute and a half. The nice thing is the second time I run it, the third time I run it, if I run this type of stuff regularly, this ends up being very quick. I won't bother you with the details. Takes about a minute and a half. But using this type of very simple sort of request and response, an analyst can come in here, look at the stored alerts, pull in the data they require, and do some stuff with it. The other thing I could do, for example, here, instead of actually writing this out, I could just write it to, to, you know, to .pcap here, and I now actually have A pcap file that I can start to have fun with. I can issue crazy requests if I feel like it. For example, I can say, give me all port 80 traffic. This will take me a very long time. So I'm gonna say this right to standard out. TV here will just give me some stats.

Can you clear again?

Yep. PV here will just give me sort of real-time stats as to what data is coming out of this. And then I have piped into dev/null. And you can see here that, you know, we're pulling in like 25 megs of packets per second. Life is good. You know, we're moving along really quickly. And this will drop pretty quickly, I think. So, let's see. Yeah, maybe not. But there's—

Can you pass other options in etpdump there?

Yeah, yeah, you can pass as many options as you feel like. So for example, let's see, where's my Excel here? I can say, like, snap length or something like that, right? And that will give me very long things. Yeah, I can say snap length, I can do those kind of things. It's all pretty cool. Works out well. Yeah, and the other nice thing is that I mentioned that when you're actually querying, you know, the BPF that you pass in is relatively limited. You can just say like, you know, port, host, things like that. The nice thing here is using tcpdump, I can actually add additional filters, right? I can say like, you know, TCP flags, blah, blah, blah, all this kind of other stuff. It'll do the initial query to Sonographer and get back the set of packets, and then we'll run that through tcpdump, which will drop all the other things that don't match its much more complicated, much more interesting filter. And it sort of allows you to do a two-pass filter on the data. That's pretty neat. Nice. Yeah, and the other thing we can do is write it out, open it up, parse it. Sorry, was there a question?

Oh, no, sorry, we were talking about the bar at the screen. Never mind. Okay.

Excellent. Well, go back here. The demo gods have been kind. Question time. Oh, quick disclaimer. We don't actually use this for user data at Google. Okay, good.

Other questions? Have you done any comparisons with other PCAP tools that excel at speed like Gulp or NetSniffNG? I don't know if you've done any comparisons with how well your tool performs versus theirs. I know NetSniffNG, for example, uses npacket. However, they don't actually use the fanout yet. To my knowledge. So I imagine yours will outperform them just because you're actually load balancing it. But have you done any comparisons yourself research-wise?

We have done a couple comparisons on a few things. There was, I'm blanking on the name of it now, but it's like Bro History or Bro Timeliner or something like that was one possibility.

Time Machine.

Time Machine, yeah. And that one looked very promising except it doesn't support IPv6 and the code's dead, like no one had touched it in the last like 4 years or so. Right. We also, I believe, looked at ntops-ntgs, which had very similar performance characteristics, but it was closed source and we wouldn't have been able to modify it as we saw fit. I believe those are the two that we looked at initially. I don't believe we looked at Gulp or anything like that.

Do you, uh, have you heard of tpacket version 3?

No, no, I haven't.

Okay, there's a structure—

That's what this is. Yeah, so afpacket, tpacket version 3. Yeah, that's what we're using internally.

Okay, I just checked. I know I've read from the kernel mailing list like that one will actually, if you do like small packet sizes like 64-byte frames, that has like a 20% increase according to discussions on the NetDev kernel mailing list. I was wondering if you were using that option.

Yeah, so version 1 and version 2 both require you to create blocks and then frames, and so for create a block that's 1 meg and then a frame that's like 2K, and it'll put one packet into each frame. The problem is that you have to size frames to sort of the largest packet you want to process. And if you don't do that, you either, you know, hit a snap length or you just waste a lot of space on packets. QPacket version 3, the reason that we use it and the reason it's so great is that it does away with this idea of frames inside of a block. And just writes a linked list inside the block as closely together as it can. And so, yeah, if you're using afpacket these days and you're not using tpacket3, you should really reconsider. It's a very large increase in usage.

Cool. Thanks for this, Donnay. Yeah. By the way, if anybody that is remote has a question, do type it or unmute yourself. There's one. Or, um, and then mute yourself after you ask a question, or type it in the chat and we'll read it off.

Yeah, there is actually a question in the chat. Somebody's asking, uh, is— as memory gets cheaper, would you think of using something like Apache Spark to write it distributed to RAM filesystem, process information, and then send info to disk?

That's interesting. Uh, so right now RAM is cheap. And disk is still a lot cheaper. I mean, we use relatively large amounts of RAM in the processes that we're running because basically the size of the RAM you use allows you to handle traffic spikes. All right. So right now we have— we're writing to 8 disks. Each disk is either using— I think it's using— each disk is using 16 gigs of RAM internally. And that basically allows us to say if there's a huge spike, so long as that spike doesn't take up at least 16 gigs, it allows the traffic to spool up. So RAM is definitely cheap. Disk is also definitely cheap. Both are getting cheaper. I think one thing this project has taught us, if nothing else, is that simple solutions actually work really well here. I mean, we're using nothing the Linux kernel doesn't provide. Relatively stock disks and stock RAM. We're using a typical, I think, Intel 10-gig NIC with like, what's it called, IGXe or one of those things, IGVXe or something like that. You know, the typical Linux kernel driver. We're doing no magic and it's working really well. And so that's something that we've found really, really nice.

I have another question. Have you guys, uh, did a performance test of it and see how much you can write gigabits per second to disk across your 8 disks?

Yeah, so, um, we typically, um, the test machine that we use, uh, we pull from one of our offices and it gets about 2.5 gigabits per second input. Uh, we quadrupled that and got it up to about 10 gigs per second. And we're very easily able to get that all to disk with no issues. The NIC then started dropping packets on us. So, you know, that was good. The big issue there actually, or one of the big things we realized after not too long, is that disks only have one throughput and they have to use it for both reads and writes. When we had 8 disks and we were writing, you know, 10 gigs, 10 gigabit per second on each of them, we were probably utilizing about 70 or 80% of the write bandwidth, which meant that when we tried to do reads and we tried to do seeks into these disks, those seeks were actually taking a long time because they queued up behind all these very large writes. So we learned very quickly that actually having, you know, sort of over-provisioning your disks allows you to read data back a lot faster. If you have, you know, 90% utilization for writes, you'll get your data back. It'll just take you a really long time.

I see. Is there a way to like query stats from Sonographer? Like for like packet drops and stuff? So you get the stats from there? Do you have to use the general kernel facility to do that?

Yeah. Let's see if I can find that. So we have a couple of things. So One of the things we do, we have this thing called stenocurl. So, just quickly, our access control is based on SSL certificates. So, the only reason we have these stenoread and stenocurl things is they're wrappers around something that pulls the correct client certs when it's talking to the server and verifies the server certs and stuff like that. So, that's— besides that, basically anytime you see stenoread or stenocurl, think curl. Let me see if I remember this. Trying to remember how this actually works. I can't remember right now. Debug stats, maybe? Yeah, there we go. This provides us some statistics on— I mean, specifically these statistics are for stenographer more than stenotype. So, stenotype is the one doing the actual reading and stuff like that. But this will provide us very granular things on sort of like, you know, how long it took to update the— to read the indexes versus how long it took to read the packets from disk, how many packets we've read in and passed to users, the various HTTP requests that we've done, how long they've taken, whether they were successful or failed, all that kind of stuff. So yeah, we haven't done too much with actual like looking at the specific number of packets that we're writing to disk and things like that, but it's definitely a possibility.

Okay, cool.

It seems like you might be able to ramp up the amount of traffic you can handle by adding more disks and fanning out more, and I'm just wondering if you'd done any experiments on that and seeing how much more traffic you can handle, and if then you find that it bottlenecks somewhere else.

Yeah, definitely. We haven't done more than 9 disks because I think the first time I said, "Hey, sysadmins, can I have a 9-disk machine?" They were like, "Why the hell would you ever want something that big?" Then they went and found one and they were very nice and they put it together. But I think our RAID controller only supports like up to 12 disks and, you know, can't go much higher than that. We definitely would see issues on the NIC. Itself. I mean, obviously our NICs are 10 gigs. We could run multiple stenographers on different NICs and that would probably be fine. Again, our CPU usage right now, this machine is running 16 cores including hyper-threading and we're using 8 of those for processing to disk. At the max, we were seeing, you know, when we were actually doing 10 gigs onto 8 disks, you're seeing about 60 to 70% CPU usage. So if we were to jump up to like, you know, 40 gigs per second on this machine, and for example write to /dev/null or something like that, our next bottleneck would probably be indexing the packets themselves.

Okay, makes sense. As far as writing to disk, how is that done? Like, I know it's like, like NetSIF-ng, for example, uses Well, it has an mmap option. It also has like scatter-gather outside of the general read/write system calls. I was wondering what you guys were using there, if you've tried any other types.

Yeah, definitely. I/O. The writing to disk is actually one of the things that, that's really interesting about this, in that we do it very, very simply and it works very, very well. So we don't do mmaping or anything like that. We simply open up a file descriptor. The writes that we do are O_DIRECT writes, and they are writes of the actual memory space that we're being given by Tpacket. So on disk, we actually have the in-memory representation of packets, which includes that linked list to find new packets and stuff like that. But really, we're just doing sequential writes to a single file. We're doing them with Linux's asynchronous I/O, so io.submit and io.get, and all these other things, which basically allows— so when I say a single thread here, I actually do mean a single thread. So a thread that's processing packets will get a 1-megabyte region of memory, index it, issue an asynchronous write, and then wait for the next one. And it will check every once in a while and say, okay, how many of the writes are finished? Return those, return the underlying memory regions back to the kernel for more packets. But really all we're doing is sequential writes out, you know, in order to disk. And it turns out the disks are really good at sequential writes, right? I mean, they're, that's kind of what they were built for. They just spin and do their thing. By also doing asynchronous writes all the time, we're keeping the disk write buffer full almost, you know, 24/7. So especially when we see spikes in memory, we'll have up to, you know, a couple hundred megabytes asynchronously being written. But the async I/O actually allows us to handle that extremely easily.

Okay, thanks.

Yeah, this would probably be very different if we tried to do it on like SSDs. I feel like for SSDs something like, you know, more random access writing would probably be much better. But when it comes to spinning platters, you can't really go faster than just, you know, straight doing it. Another thing quickly here is that we are using ext4, so we get the benefit of extents. And there's also an option, we don't have it turned on by default, but you have the option of pre-allocating the file size in advance. So when that option's turned on, any new file is immediately pre-allocated out to 4 gigs. And then when we finish with that file, we truncate it down. And this basically allows us to front-load the overhead of inode changes as opposed to just, you know, we're starting from a file that's size 0 and appending to it. Requires a whole bunch more inode writes.

Okay, cool. Good to know. Does anybody else have any questions? Anybody remote, like I said, go ahead and type them in or unmute yourself and ask.

A Knuckles says, what Linux version are you currently using?

Good question. That one. So I believe, let's see, there's most of the stuff that's available has been available for a while. AF packet, T packet v3 has been out for a very long time. Some of the new fanout modes have come out more recently, like I think maybe like 3.5, 3.6. There's a couple other security features like the ability to So you can set up a BPF filter on the socket that you're reading from, and Linux has the ability to actually lock that filter, which means that if someone were to come in and, you know, get remote access to this machine, they have the— and they have the socket, they wouldn't actually be able to remove the filter from it and see more traffic than we had. That locking doesn't come in until 3.9. So there are a couple of things like that that provide us a little more security a little later on. But really, we're talking on the order of, you know, if you have any Linux in the last year or two, you're probably fine. If you're on NetBSD, you're out of luck though, because there's a lot of stuff here that's specific.

Excellent.

Thanks a lot, guys. Are there any additional questions?

Yeah, I got one more, sorry, if you have time for it. I was just gonna ask if, you know, the way you guys have it so that it's a single machine that's taking this and it's being load balanced across the disks, have you guys thought about maybe like a design where you have multiple machines and then you're talking to one sonographer tool that's actually contacting it like a distributed system? For this?

Yeah, that's a great question. So we actually are trying to run this on most of our aggregation points across our Corp network in the same way that, you know, where do you put your NIDS? You put it, you know, at aggregation points. So ingress and egress points, Corp offices, that kind of stuff. So we will actually have this distributed out to a couple of machines We already have a bunch of scripting for other stuff that are running on those machines to like go out, get a whole bunch of information, pull it back. But one thing we are considering is a more central management solution for this, something that would know where all of these sonographer processes are running and be able to issue queries to all of them, pull back and merge all the data and provide it in a single stream. We haven't done any work on it yet, and it's going to be a very low priority, but if that's something that anyone's interested in working on, That's definitely a possibility. Again, as I mentioned, when it comes to actually pulling the data out of this, the data is all coming in over HTTPS using client and server cert verification. So it's extremely possible that this could be distributed out to multiple systems with, you know, verified client certs on a centralized device that could go out, pull this information, and get it.

Oh, speaking of that, is maybe an API in the future? Or I mean, I guess you could do things already though.

An API would be fine, but /post/bpf/get-back-pcap seems like a pretty good API for the moment. So, you know, if you really want it to be Rusty, I could throw a bunch of slashes in there and be like /pcap/port/83/whatever. And if you did it, you know, put instead of a POST, I'd be really mad at you. But yeah, for now, POST /pcap seems like a great way to do it.

Did you say that this was available on a Git repo or something?

Yeah, let me actually go over to that.

Could you post it in the chat? I would, but like I said, my host machine's still frozen, unfortunately.

Let's see, I think I I may might need to stop share. Chat. There we go.

Excellent.

Oh, thank you.

And definitely pull requests welcome. So if you want to go in and look at it and play around with it, have a blast. Oh, also, in particular, I will mention, for those of you that are interested, there's a couple of specific docs in there that are, you know, if you're interested in building systems like this and stuff like that, look at the README and look at the design doc, and both of those should provide a lot more details to like the internals, some of the decisions we made and why we made them.

Okay, great. Good to know. Alright, Graham, I want to thank you for donating your time. This was an awesome talk and I'm actually looking forward to— we're going to try to use this for sure. Actually, one of our research projects that we have in our sponsorship documents, we're trying to acquire hardware to actually do a comparison of all the major packet recording tools and do a performance comparison between them. So if we're able to get the funds for that, I think sonographer to be a great addition to our research. So I do appreciate you coming on board.

Yeah, wonderful. It's been a pleasure, guys. I'm going to sign out, but if you need to access, or if you'd like to talk to me, I believe Jon has my contact info. Also, you can go through GitHub and you should be able to find me. Feel free to reach out for any questions or comments on, on steno or anything else. I'll very quickly plug Here's another one. So this is another open source project I do that does packet processing in Go. If you're interested in Go and packet processing, check it out. Super fun.

Cool. Great.

Yeah, thanks a lot, guys.

Alright, thank you. Take care. Alright, and that concludes OpenNSM for this week. We will not be meeting next week. This week, Richard Stallman will be on campus, and it'll be during our time, so we're all going to go check out that talk. If anybody else is interested in sponsoring us, do check out the sponsorship doc. It's listed right here. Go ahead and open it here just to show everybody real briefly. It is a tier-based system. It shows the 4 tiers that we currently have and what you'll get for them. And then in addition, it shows you— it has everything detailed outlined as far as all the projects that we're actually thinking about having or wanting to do and actually what we need are the requirements for them. So do check that out. Also join our mailing list. This talk will be recorded and available up on YouTube and Vimeo. So we'll have that done tonight and it'll be posted on Twitter and the mailing list. So I do recommend if you're not already subscribed to the OpenNSM mailing list, you do that now. And I think that concludes it all. Thank you guys. Take care and have a good break. Alright, so did we decide on dinner?

Well, I didn't order pizza, so I guess that sort of decided it.

Okay. Yeah, I just don't have any cash. Otherwise—

Yeah, I wouldn't have cared about that, but sushi sounds good too.

Okay. You guys want to head over there? Yes.

That was cool.

Yeah.

I want to play with that.

Is that similar to Time Machine with Pro? Yeah, but it's a lot fucking better.

Pretty, pretty small footprint. They're pretty light, lightweight, and fast. Oh man, well, I wonder if you could set up like a sort of a tiered system, right? So you got one Steno— before I kill this, I want to make sure that fans out to several other machines that are all running Steno too, right? You know what I'm saying?