OpenNSM (GRR Rapid Response - Greg Castle)

All right. So I begin for OpenNSM, the Open Network Security Monitoring Group at the University of Illinois Urbana-Champaign. We're going to go ahead and go to the group updates now. So we do have a Twitter account, so @OpenNSM. You can also, if you want to use the hashtag #OpenNSM, you can. We are looking for sponsors to build our open, our our NSM lab. We'd like to host it in the cloud or the other idea that we have is a Geneti cluster. So that's something I'm kind of working on now, but we still need some hardware. So if anybody's interested in supporting that and additional research, it's all detailed at the Google Doc document right here. So just follow that link and see and then take a look at it. And then if you have any— if you're able to actually support us in any way, do get in contact with us.. We now also have a— we're now— our GitHub account is a GitHub organization. I changed from a regular account to an organization so it's easier to collaborate with other people. So if you would like to contribute to any of our repositories, including our meeting notes, send me your GitHub account and then you will be able to add any news items, research papers, tools to the notes yourself. You don't actually have to contact any one of us to add it for you. So it kind of cuts out the middleman, helps to hopefully be more efficient. I applied for an Amazon grant for our OpenNSM lab. I also applied for Oklahoma State University's open source lab for web hosting because we have to move our system from NCSA to another, to another service provider. I also wrote an article called "What is OpenNSM?" that you can find on this website right here. It's a blog I've been running for a few years that gives you more details about the group if you're unfamiliar with it. We're going to head into the meeting section. So NSM in the news, few things. A few weeks ago, Bert released— I think it was Bert, was it Mark? I'll just pull up the name so I don't get it wrong on video. So let's see here. Yeah, Mark. Mark Burnett released a huge amount of passwords. A lot of them were public record already, but the difference was they were just the passwords and he has all of the usernames for them. So there's a torrent you can actually download available at the bottom of this. You can get the magnet link here. So 87 megabytes of username and password data. Some of the data is sanitized like company names and whatnot, but it's basically released for research, so do check that out. Also, you may have heard about the Superfish certificate. So Lenovo machines, there's a number of articles going around on this, but essentially Lenovo have this company called Superfish who also worked with a company called Comodia out of Israel that does SSL interception stuff. So for advertising, they developed this piece of adware that's on all the Lenovo machines by default. And what actually happens, it has a man-in-the-middle attack on your system between your browser so it can actually inject ads into your system while you're viewing the web. And a security researcher, well, there's a bunch of them, but I'm going to point to the guy at Radisec, Robert Graham, who actually showed a simple dissection of it. Actually, he found the actual private certificate, the private one, the key is actually available in the adware along with the password. And he just used strings to pull it out, strings and grep, so a real ghetto way to do it, but it works. And so at that point, the big problem with it is that that's It's the same on all machines. It's not different for every single one. So at this point you can actually decrypt all the people's data if you have that information, right? You need the public and the private key. So that is actually available and online now at this point. There's also a website out now for testing that. So you can go to this link here, filippo.io/badfish, and that will actually check if your laptop is vulnerable. Also, there is the announcement of HTTP/2 being finalized with IETF. That's going to— for NSM, a few things with that is that it's going to— we can't handle at this point. At least I'm not aware of any tools that do because it is a binary protocol. This is version 2, and it offers TLS over it at the application layer, I believe. So there's going to be a lot of issues moving on to do actual monitoring of HTTP version 2 traffic. We'll see what's to come on that. Also, the guy Champ Clark from Sagan put out an article on using Sagan, the multi-threaded log analysis engine, with Bro's intelligence feeds. So you can use that together with Bro, and I hope we're gonna play with that at work if I get a chance. That article is available right here. And we're gonna move into the next meeting section, Conference Corner. So Security Onion 4-day training is actually coming up in Houston, Texas, if anybody's interested. It's probably mostly for professionals and not students because it is, the price is quite high. But if you would like to have a 4-day training class, do check that out. Bam Fisher, the author of Squeal, actually added a job post in here. So he's looking for a, his company is looking for a Cyber Security Technology Lead at GM. So if anybody's interested in that, they do a lot of NSM stuff over there in this position. You check that out. And now we're gonna move into Tool Tray. This is the meeting section where we talk about new tools or old tools that maybe we may have not heard of. So the first one I want to talk about is Bro Intel Generator. This was actually posted, it's a little shell script that was posted to the Bro mailing list a few weeks ago that takes, it actually takes in, it HTML document or a PDF and allows you to increase my size here. It allows you to pull out indicators like domain names, file hashes, and IP addresses from the PDF and format it into the Bro intel framework format. So we can do a quick demo here. I had to patch it real quick to actually get it to run on OS X. It looks for the two packages in a particular path that is not the case on OS X. For example, if you use Bro or Brew or Mac ports. So anyway, so you can take a look. intel-intruder.sh, so we're gonna feed it a file. I downloaded this, um, Cylance Operation Cleveland report and I'm gonna say, hey, this is a PDF, right? Use -p. It's gonna read through this PDF report and pull out the indicators of compromise. And so it gives you some information. It actually wrote out 3 files called 3 files in .dat and there's text files with the Bro format 'cause they use that as a convention. You can take a look at them, so just cat all 3 of these. You can see that you actually have the IP addresses, the hashes, and the domain names, but there's not in this particular case for that. You can actually just have the Bro intel frame or intel frame read that in and use that to match against traffic. So that was a cool tool. I'll probably send that patch to the guy too so he can add it in. Also another one is MalDNSSearch, a tool that I wrote a few years ago that matches, it basically takes a number of different log files like BIND DNS logs or Bro DNS log or conn.log or SonicWall logs or whatever, just supports a number of different ones and matches them against a malware hash or a malware list that has IP addresses or domain names. So it's a fairly simple script and I wanted to point this out too because we had this big update where I did 20% improvement a few days ago and then Justin Azoff of the Bro team committed one that actually is 100% improvement using associative arrays. So it's very fast now compared to what it was anyway. Hey, by the way, could someone hit the lights in the back? Alright, awesome. So I'll just demo that real quick. So here we're gonna use the passive, we're gonna say we're reading a passive DNS log from the passive DNS tool, just a bunch of domain names. And then we're gonna specify that particular log file, pdns.log, and then by default it actually pulls in a MalHast file list, I'll show you at the bottom. And at this point you can see it actually downloaded the default list from MalHast, or from mechanical labs and then it just searches through and tries to find any matches for you and then it should complete in just a moment. But you can pass -M and then there's a list of ones built in that will actually pull down the latest list from like Snort, from emerging threats or you can pull down the Mandator report or whatever else. There's a number of them built in. So at this point you can see I found 6 of the nearly 2,000 entries from that ML host list and it's gonna report it like that. Nifty little tool if you want to look at historical log data to match it against things. So that's that. And then the last thing I want to talk about is this tool called Sonographer. This is built by a guy at Google that Greg happens to know. It is a packet capture recording tool that focuses on speed. So I'm going to try to get this guy on to talk about it. So they're doing, it looks to be here, they're doing a 10 gigabits per second, a multi-core, multi-disk machine. So that's of interest, definitely. So we'll have to check that out sometime. And then finally, the next section is paper period. And so the United States Marine Corps, along with— I can't think of the name of what it was off the top of my head— Army Cyber Institute came out with this cyber defense review where they talk about law, history, tactics, operations, and policy. Of cybersecurity and they have— they're going to publish a journal and they don't have an issue out yet but it's going to be freely available online so you can check that out if you're interested in any of that information. And now we're going to turn it over to Greg. So Greg here from Google. He works on GIR, the Rapid Response Framework at Google. He's given talks at Black Hat and other conferences and specifically on this tool. And he'll be talking about GRR. I'm going to go ahead and hand it over to you, Greg. All right. Stop the screen sharing. And then you can click yours now. OK. How's that? Looks good.

OK. So yeah, my name's Greg. I'm going to talk to you about GRR today. If people have seen my Black Hat presentation, there's a little bit of that presentation in this. But I go in— this is probably a better overview presentation than that one was since that was kind of purely focused on like one piece of the tool. So I'm going to talk a little bit of kind of about a little bit of background about why we built it and but mostly focus on what it can do and give you a live demo as well. So really quick bit about me. I'm a Go developer. I sit in the Google Incident Response team at Google. I also have a pretty strong interest in OS X security. I was the OS X security guy at Google for a while. And kind of before that, I was doing a bunch of other security-related jobs, so pen testing, incident response, audits, et cetera. So I'm going to start kind of where Gur started, which was basically with— a need for live response, live forensics. So here's a kind of a really common use case for anyone that's doing any sort of network security monitoring. So you'll see some sort of beaconing activity to a known bad domain. So you've got this domain that you know is a malware command and control domain, and Joe's machine is madly beaconing out to it, sending HTTP requests that look like command and control traffic. Typical responder scenario might be to kind of pick up your forensics go bag and walk over to Joe's cube and start doing some inspection of his machine. Maybe you can get some packet capture of the traffic. Maybe you can do some memory analysis. Maybe you can retrieve some files and get a copy of the malware, this kind of thing. That all gets really complicated when Joe's here in Hawaii. He's, you know, maybe not great at security, but he's definitely really good at vacations. And he's having a really nice time on a boat off the Napali Coast in Hawaii. And it's going to be hard for you to go there. Well, it's maybe not going to be that hard for you to go there. It'll be inconvenient, but it'll be fun. But it doesn't scale very well. So he does have 3G internet there. So if you had a capability like Go, where you had a tool that was already in place, installed on the machine, that just needs an internet connection, then you could do some sort of at least initial triage of this problem. And maybe even do a full response to it just completely over the internet. So another really common kind of use case is that a new APT report is released. So maybe it's advanced, maybe it's persistent, maybe it's even a threat, but basically guaranteed that at least it will have an interesting codename and maybe even a logo these days. So my now fictional kind of go-to logo for this sort of things is Bear Eagle Shark Laser, which is this code name I made up for kind of fictional malware. So, you know, Bear Eagle Shark Laser report is out. We want to check all the things, and checking all the things can be really complicated. So there might be more than 50 indicators of compromise across two different operating systems, and all the things is tens of tens of thousands of machines across a large organization. That are all mobile, that are all coming and going all the time. So without a capability like Go, it's really hard to do— so we call that a fleet check. It's really hard to do a fleet check like that without some sort of capability like Go. So what is Go? Go stands for Go Rapid Response. Despite what almost everyone thinks, the G isn't for Google. It's a recursive acronym and tradition of Gnu and Wine and YAML. It's an open-source live forensics tool, whereby live forensics, I mean, you know, we're not pulling disks out of machines and doing dead disk forensics. We're operating on the machine as it's running while the user is still using it. It's a Python agent that runs on the machine that talks over the internet to a Python server, and there's no, there's no kind of requirement for a corporate network or or university network or a VPN of any kind. It's anywhere there's an internet connection, GOWill work. Built into the tool is a disk forensics capability in the form of the Sleuth Kit and also a memory forensics capability in the form of the Recall memory forensics framework, which is also open. Both of those things are also open source. It's scalable. We run it at very large scale ourselves, and there's a number of pretty decent scale open source installations these days in the kind of tens of thousands range of clients. It's a stable and low-impact client. We've been running this on desktops, laptops, and servers of some very opinionated engineers who watch the resource consumption on their machines and have top running constantly, that kind of stuff. So if we were doing a bad job here, we'd definitely know about it. And we think we've been doing a pretty good job. In being efficient with the resources we use on the machine. And we've got full-time developers allocated to this project. So at the moment, we've got 5. So it's a— we're committed to this tool for the long term. We've put significant development into it. So OK, so we had these use cases. We figured we needed a tool to do something like this. Why not, you know, why make the build decision? Why not buy Mandy and Mia or an EnCase Enterprise or an F-Response or something like that? And so one of the main drivers there was we wanted a really deep level of customization that we weren't going to get from a commercial tool where we didn't have full access to the source and the ability to change whatever we liked. So we wanted to be able to, you know, make a big change, respond to whatever threat we what we're facing today and maybe deploy some custom detection or custom defensive measures that would be hard to imagine kind of doing with any commercial tool where we're relying on other people to do work for us. And we wanted a large scale of concurrent analysis. So we wanted to have, you know, 50+ analysts analyzing 50 machines in a really deep way and being able to look across, you know, easily tens of thousands of machines to check things. And get answers within a few minutes. We want to be able to move as fast or faster than the attacker. So, you know, be able to look at a machine in depth, get some information from there, and then pivot with that information across the whole fleet very quickly. And we wanted real support for Mac, Windows, and Linux. So, you know, back when we first open sourced this in 2011, I think, you know, definitely all the commercial tools were basically Windows only, and I think it's still pretty much true today that the large commercial players here are mostly Windows-focused still. And now, you know, we've been using this for quite a while and quite extensively inside Google that, you know, we couldn't imagine not having a capability like this and doing any form of effective incident response. So that's a little overview of why we built it and what the motivations were. And I've given you an executive summary of the tool. So now I'm going to dig into how it looks in a bit more detail. So this is a server-side architecture diagram. So clients poll over the internet to front-end servers. And in any reasonably large-scale deployment, we'd expect a load balancer to sit in front of those. We don't provide that, but we expect that organizations can lay their hands on some sort of load balancing. And so the polls come in from the clients to the front ends, and the front ends write those messages into the database. I'm going to talk about kind of Datastore in more detail in a minute. But the workers basically take those messages out. So there's separate processes called workers that do the real work of the processing. Take those messages out, decide if there's any work for that particular client to do, and then write requests for more information back into the database, which the front ends then send back to the clients. And it kind of— the, the kind of data flows like this. On the server side, it's basically a big asynchronous state machine that's triggered by polls that come from the client. And there's a, there's a web UI which you'll see in the demo, and also there's an IPython-based command line interface. So the clients, what do they look like? As I said, the stable clients, we've been running them for a long time now. We do heavy monitoring on the clients. So we measure how many CPU seconds we use, what our memory footprint is like, how many bytes we send on the wire for basically every action that we take. And so when we run these actions that are big across the fleet, then we aggregate all those statistics up so we can see exactly, you know, we burnt this many CPU seconds, we sent this much, this many bytes on the wire, so we can kind of measure the impact of the actions that we're taking on all the individual machines. And not only do we measure it, but we also limit it. So we have some guarantees of, you know, we're not going to spend more than X seconds, CPU seconds on this. We're not going to send more than this on the wire. And if our memory footprint goes over x, then we're going to kill ourselves and kill the process and restart it. So clients work on a 10-minute poll period by default. So it's an HTTP poll that sends encrypted and signed protocol buffers over HTTP to the server every 10 minutes saying, is there any work for me to do? Is there any work for me to do? And one of the key design decisions here was, how smart should this client be? So we basically went for a very smart server and a somewhat smart client. So it's not completely dumb, but it's reasonably minimal. And the reason why we kind of chose this approach was we wanted to be able to time travel backwards on the server. So we wanted to be able to, say, do something like collect persistence mechanisms regularly and then sometime in the future look back on all the different things that we collected and say, oh yeah, we can see this run key appeared here, and that'll help us kind of with a time frame for, for compromise or identifying patient zero, that kind of stuff. And we also wanted to basically hold most of the complexity on the server so that our build and, and fix and deploy times for bugs or new features is much faster. So we can We can deploy a new server very quickly and put a new feature into the server very quickly. But having to update many, many thousands of clients across the whole organization is a fairly slow process. So some clients will be in a desk drawer, and they will— a laptop will be in a desk drawer that will only get turned on every 3 months or something. So basically, whenever you deploy a client, you're going to have that client around for a long time. So it also means we're going to update less if the client is not— if all the smarts aren't on the client. If most of the smarts are on the server, then we have to update it less often just because there's less stuff to change. And it also means that backwards compatibility is simpler just because less complexity, less stuff that we need to refactor and do differently. And we also, during an active investigation, we're also leaking less of what we're doing to a potentially compromised machine if most of the interesting decisions and analysis and parsing of data and all that sort of stuff happens on the server, then we can— it's not as obvious exactly how we're doing the investigation. And then on the server side, as I said, the front ends pass messages, the workers do the real work, and everything is designed to be basically completely asynchronous. So, um, the whole system is designed around clients disappearing at any moment. So, you know, I'll sleep my laptop and jump on a plane and, you know, thousands of people doing this all the time on our corporate networks. So we need to be completely prepared for, you know, a machine to just completely disappear for hours, minutes, or days, or months, or whatever, and then be able to pick up where it was when it comes back on the network. So that also means that you should be able to queue network— queue actions on the server. So if I want to investigate this machine, Often it's not online right now because it's a laptop, you know, it comes up and down a lot and travels between networks a lot. Then I'll basically be able to schedule a bunch of stuff that I want to happen when, when that machine comes online. And so when it does come online, all those, all those actions will be taken by Go. And I can also schedule a notification that will email me that says, hey, this machine that you were interested in analyzing is now online. And the results of the flows you ran— so in Go speak, a flow is kind of a high-level unit of work, like collect the file or whatever. The flows that you ran are now complete. So Datastore is very abstracted in Go. So we have a number of different implementations. And it's actually quite easy to switch between them. So these are the current open source offerings. We also have a— proprietary one that we use internally. And we also know of some other open source users who've actually written their own proprietary internal data stores as well. So there's MySQL times 2, because we wrote an original one. And then one of the open source users basically rewrote the whole thing to the point where it was so different that it's now MySQL Advanced, a separate data store. There's also a MongoDB and a— and a sharded SQLite so that you can kind of break the SQLite out into kind of many different files. So the ones that are running kind of open source deployments of any significant scale are the MySQL Advanced and the SQLite are the two ones that we know people are running it at a pretty big scale. And kind of one of the key, as I said already, like the, we want to preserve this axis of time. So we want to hold a bunch of information on the server about things that we've collected from the client. And we want to be able to know, be able to look back in time. And a key point, a key way to do that is by basically, we hardly ever delete stuff out of the database. So we basically create new versions for everything. So if we stat a file or if we do a new process listing, then those things get written into the database as new versions of things that are already there. So, demo. How does this look? Is this text visible? Should I zoom in a bit more?

Nope, looks good.

Looks okay? Okay. On this demo Go server, I've got just 2 clients connected. So I have a Windows 2012 client connected and an Ubuntu Trusty connected. And so I'm going to run through a few different things you can do with Go that will give you a reasonable overview of the capabilities. So you can imagine a scenario. You've done this hunt for your Bear Eagle Shark Laser malware, and then you've, you know, imagine that, you know, this isn't 2 machines in here. This is, you know, 100,000 machines or something, and you want to distinguish the ones that are compromised by that malware. So what you can do is search for the machines using— so this is a search box here. I actually just searched for dot, which is a shortcut to show you all the machines. So you can actually apply labels to machines. So I'm going to apply the Bear Eagle label to these two machines, which, you know, which appears over here on the right. So you can now treat these machines separately to the rest of the fleet. So one thing you can do which is cool is hunt them separately. So a hunt in GRR speak is when we do something across the whole fleet or groups of machines or large groups of machines, or even just kind of smaller groups of machines, but doing stuff on multiple machines. So I'm going to start a new hunt, and I'm actually going to use the Artifact Collector. So I'm going to talk about this in more detail, but basically this gives us a simple way to collect a bunch of different forensically interesting things from machines. So I'm going to collect, collect event logs from Windows. So there's a— this is zoomed in, so it's kind of a little bit more kind of scrolly than it would normally be. But you can see over on the right here, this is going to grab application event logs, system event logs, security event logs, and those are all actually artifacts themselves. So this is a meta artifact that collects other artifacts. So I'm going to collect—

whoops.

Let's move that. Try again. And I'm also gonna— from Linux, I'm going to grab the auth logs. So you can see over here what it's going to collect. This is just auth.log*. So add that into the list. I give this hunt a name, so this is Demo Hunts. So here, these, these parameters, the client limit, expiry time, and the rate are quite interesting. So the way we run hunts here is we basically usually put like a limit of about 100 machines on any hunt that we run to start with. So you can run it on 100 machines and then see what the impact is. So you can see, oh, I use this many CPU seconds, I use I sent this much, this many bytes, and that way you basically get a random sampling of, you know, the impact of what this is on the, on the whole fleet. And then the expiry time is just how long this hunt will be relevant for. So if a machine pulls in, you know, a month from now, should it still run this hunt or not? And the client rate is how fast to schedule new clients. So we can, if the hunt is really intensive, we can make this go really slowly. So this is 20 clients a minute. So to kind of help our server out and not overload it by basically DoSing it, we can, we can kind of make this go slower and gradually ramp up on the hunt, or we can just take this limit off and go as fast as we can. If we go as fast as we can, we can basically get every machine, sort of no matter how many there are, reporting results within under an hour is basically the usual time. So any machine that's online would report in in that time and complete the hunt. Okay, so I'm gonna start that. So here, if I wanted to, I can add an output plugin. So this is a way to get data out of GRR. I'm going to talk about this in more detail as well too. So there's a couple of options kind of available by default. So you can, you can write all the results out as CSV, in which if you're collecting files, that'll be like a bunch of metadata. So basically what you get from a file system stat. So you know, like MAC times, file size, that kind of stuff that would get written out. Or if like, you know, this is a hunt for something you're not expecting to find, then you could get it to send you an email for everyone that it does find because you might really want to know about it if it's some bad class of malware or something. So I'm not going to do that for this hunt. And here we can set a rule type so we can hunt just Windows machines, we can hunt just Linux machines, just OS X machines, a bunch of other parameters we can kind of control exactly which machines this hunt will run on, but for this one I'm actually just going to hunt the Bear Eagle label, so I'm just going to hunt those two machines that I labeled before. So this is what it looks like. I'm going to get event logs and Linux auth logs, and I'm going to use the artifact collector flow to get them. So I've created the hunt now, so it shows up here in the paused state. And so that just means that it's not actually running yet. So I'm going to talk about the authorization model a bit further in another few slides' time. But basically, before this step would work— so I'm about to run this. But before that would work normally, I'd have to get an approval to run it if you're using the kind of two-party authorization system. So while that's going, I'm going to— just quickly show you a really sort of, I don't know, a really kind of basic memory hunting capability. So Recall gives us a lot of really powerful memory hunting capabilities. So I'm just going to show you like a really simple one, which is doing a process listing by looking at just raw memory and then working through the data structures in raw memory using Recall. So this is on the Windows 2012 machine. So I'm going to kick that off. And while I'm waiting for that to go— well, actually, that's probably— let's go back and look at the hunt. That's probably finished. Yeah, so just looking here, so you can see that I can download the result, all of the results of the hunt, so I can zip them up. So in this case, it's event logs and auth logs from Linux. So I can just download all the hunt results in a tarball. And it's actually— Go is actually really smart about what data it will deliver to you. So often with a hunt, you'll get 10,000 copies of the identical same thing, and there's only 5 that are actually different. So Go will just build a tarball here that has a bunch of symlinks that will kind of show you all of these machines had the same file, but we'll only have like, you know, 6 files actually in it. So it won't— you won't download gigabytes of exactly the same data when you could just— you're only interested in like the few that are different. So here are the kind of results. We've got the auth log from Linux and the event logs from Windows. And so we can click on these to see them. So here's the, the event logs we collected. Just move this. And so there's a bunch of information here. I could download just this file if I wanted to using this button here. I could also download it if it's a really big file. You might be better off downloading with the command line just because, you know, browser might timeout and you might lose it. And you can see here, you know, we can look at, you know, a text version or a hex editor kind of view of the file that we collected. So just while I'm here, I'll also show you, just, you can actually browse the remote file system directly. So you know, if you just want to kind of look through some directories and look for something in particular. So it's like, here's System32 on this Windows machine. It— you can filter this stuff. So if we filter for like Notepad, that'll show up here. And then you can— you don't have to go through the interface I used before. You can download files from here directly. So let me just kind of move this so you can see. So I've actually downloaded this file before, so it's already here, but I could download the version I've already downloaded from the client, or I could tell it to, with this button, get a completely new version from the client. Okay, so I showed you the hunt, and I also ran this, this recall this Recall flow that gives you a process listing. So let's go look at the results of that. So here's the results of the Recall process listing. So this has come from raw memory using a memory driver to get at the raw memory on the machine. And then if you click on kind of any of these, it will give you a bunch of interesting information about this particular process and also kind of in the same format as the, in the Cybox format, so using the Cybox schema for how to represent a process. Okay, so that's the demo. So let me switch back to slides. So, you may have noticed this tool has a rather large amount of power. So, it has access to raw memory and raw disk, and so with those capabilities, you can basically know anything there is to know from that machine. So, this is the obligatory kind of Spider-Man reference for "With great power comes great responsibility." So I mentioned the two-party authorization system. So you don't have to run it this way, but this is kind of the authorization system that we ship with. Basically, to take any of those actions, so to start a hunt or to get access to any particular machine, you'd have to get an approval from someone else on the team. So you basically get this pop-up that says, you know, you need an approval for this. And then you can enter a reason, which is an investigation ID or a bug ID or something like that. So as well as authorization, there's a bunch of auditing and logging that's available. So you can log at the database level, which is something we do. Go has this concept of audit events that are created whenever anything interesting happens. So they're created when you request authorization from a machine, when you give authorization, when you run a flow. So if you collect a file or if you collect an artifact from a machine, any of those things generate these audit events, and you can set up listeners to do basically whatever you want with them. So you can push them into syslog and then transport them, you know, to get off-host logging, or you could integrate them with whatever kind of auditing and logging system you'd like them to integrate with. And you can create more too. So you can create audit events for kind of whatever actions you'd like to audit. There's also kind of another layer of visibility and accountability that comes with the approval emails. So when you request these approvals, you can have it go to a CC list. So there's a team list that you can use kind of to basically— so everyone can see, partly so that people aren't blocked waiting on a single person for approval when someone else could approval. Prove it, but also as a measure of accountability for kind of how the tool is being used. So that's kind of a reasonable, reasonably detailed introduction to the tool, and hopefully with the demo you've got a bit of a picture for kind of like how that works. So I wanted to step back up to the high level again and just talk about how we think this fits into the kind of broader picture of defensive tools. And this is kind of what we're going for. We're going for being this this advanced live forensics that works at scale, that works remotely, really reliably and quickly. And so kind of at a slightly more zoomed-in level, what does that mean? It means mostly being really, really good at collecting stuff. This is kind of what we're aiming at. So we want to be good at collecting things that are hard to get at because the operating system doesn't normally let you get at them, like full registry hives on Windows or files that are locked for reading, that kind of stuff. Or maybe deleted files, and also stuff that's hard to get any other way. So some really interesting— a lot of the really interesting forensics research these days is basically memory-based forensics. So Rekall is really at the front of all that effort and is doing some really interesting stuff with memory analysis that, you know, is kind of impossible to get the results any other way. So we want to have like this operating system view kind of under the OS disk forensics view and the memory forensics view, kind of all three of those ideally. So, you know, ideally you could look at a binary from all three of those perspectives and, and, and check and see if any of them are different. And we also want to be able to kind of address these like really hard to specify locations to where variations across the fleet make it difficult to reason about where a file is going to be on the machine. And that basically comes down to the artifact system, which I showed you briefly kind of in the demo there, where artifact, you know, this— SANS defines artifact as kind of a combination of description, location, and interpretation. I think I prefer, like, it's usually just that stuff I want that's on the machine. And artifacts, like, you know, if you go looking in, you know, kind of forensics forums and mailing lists where people talk about these sort of things, you'll see artifacts like these, like, oh, this is where the Chrome history lives, and this is where Dropbox installs to, and this is where mail is stored for this particular mail client on OS X, kind of stuff. But this is all kind of like really inconsistent and hard to use because all of this stuff is actually not filled in for you. So like, what is it, C drive or D drive or E drive that Windows is installed on? What, what users are on this particular machine? What user SIDs are in the registry for Windows? On OS X and Linux, you know, where are the home directories? What users have home directories? All of those things are really quite hard to address across a global fleet. I mean, how do you, how do you reference something that is inside a user's home directory is a really big problem because that's where most, or a lot at least, of the forensically interesting artifacts are. So one thing we've done with the artifacts effort is that we've come up with a common language to kind of reference some of that stuff. And we basically go looking for it with Go. So we'll build information about each machine that includes all the user accounts that are on it. As well as really useful environment variables like local app data on Windows. And so you'll be able to kind of simply reference these things without— and so Go is basically handling all the complexity of how you actually reference a home directory across a fleet of tens of thousands of machines. And so once you can do that, then you can start writing these simple forensic artifacts. So we've come up with this YAML format, which is human-readable and machine-readable., and this one's for a Windows event log. So this says this is where the Windows event log lives. This particular path is relevant for these particular versions of Windows. For older versions of Windows, you're going to, you're going to need a different artifact that tells you where it is. You can kind of label these into categories, and then this particular artifact is only supported on Windows, obviously. And here's a link to, you know, more information about what this actually is. So we built a whole library of these. So we've got about— and we actually split it out from GRR. So this is now completely independent of GRR. There's this library of YAML artifacts in this GitHub project. We're collaborating with the forensicartifacts.com guys who already had a library of artifacts. So we're basically making them machine-readable. So there's about 100 there. And then GRR itself has a few that are kind of GRR-specific ways to do things. So there's probably about 130 total in both locations. But this is reusable by any other tool, basically. You just need to be able to read YAML and then know enough about the machine to populate some of those things like the home directories and maybe some of the environment variables. It's used actively by us, so we use it in investigations, so we have a vested interest in keeping it up to date and accurate. And, you know, we obviously, you know, welcome other people contributing to that repository and reporting any problems or bugs and contributing new artifacts. All that's really welcome. So, okay, so we've aimed at being this, like, really really great collection tool, being able to collect from in lots of different ways. So once we've got all that information, how are we going to do our analysis? So basically, outside GIR is mostly the answer. So we don't want to reinvent kind of the analysis wheel. So we mostly think, like, for the most part, this is better suited to, you know, heavy statistical analysis packages or systems that are already designed to analyze big data in interesting ways. So we're basically making it easy as possible, as easy as possible to get the data out into systems that are going to be better at analyzing it. And this is how we use it internally too. So there's some HTTP RPC APIs you can use to move data out of Go, which are kind of just in the process of being developed now, but are getting to be reasonably complete. But there's also the— you saw during the demo there's this Hunt export plugin system. So the open source ones we ship with are CSV and the email kind of export. But I mean, in reality, if you wanted to do this at scale, you should probably write an Elasticsearch plugin or something like that. So we've got something very similar to this. So you might know the Google product BigQuery. We basically use something very similar to that internally. So we've written an output plugin to export to that format internally. And we're basically doing our analysis using that. But yeah, if you wanted to do this at scale, then Elasticsearch or BigQuery output plugin should be easy to write and a good choice. So that's most of what I have for how the tool works. So I wanted to give you a bit of information about what we're working on now and what's coming up. So one thing we want to make better— we do some of, but we want to make better— is having this event-triggered collection. So a really typical kind of scenario is there's a network monitoring— some network monitoring signal fires that says, you know, something bad's happening on this machine. And then an analyst looks at it and then schedules a GRR flow to collect something from that machine to help them figure out what's going on. Often, you know, because there's some time has passed before the analyst has looked at it, the machine maybe is now offline. So sometime in the future, that machine will kind of report in, report its results, and then an analyst will come and look at it again and try and figure out what's happened. So we basically want to crunch down all that waiting and remove as much of it as possible and, and have like, you know, where there's a signal that has a good investigative path that we know we're going to do X, Y, and Z, then we're going to just schedule that flow straight away. The machine's probably online because we just got an alert about it. We're probably going to get the results pretty much straight away, and then by the time the analyst is actually looking at the alert for the first time, there's already maybe the whole investigation, or at least some of the investigation and the context required to kind of analyze that alert is already complete. We're also working on and experimenting with a a C++ client to kind of be the always-on piece of Go that— so we're basically aiming at a smaller memory footprint and some more efficient, kind of hopefully faster, Protobuf serialization and deserialization and crypto operations and that kind of stuff. Hopefully speeding some of that up and getting some performance gains there. And basically bringing up Python whenever we need to do something more complicated that involves the rest of the capabilities. We're working on more powerful artifact collection. So at the moment, you can basically have an artifact that describes where stuff is and then a parser that is attached to that artifact. But that's basically it. So we want to be able to have an artifact that says, this is where stuff is, and then have a parser, and then have the output of that parser feed other artifacts, and that artifact feed another artifact. So you can kind of chain all these things together without having to write any sort of real code, just writing some YAML description of what you want to do, basically. And we also want to have kind of like a default collection mechanism that also gives the person writing the artifact some measure of control over how it's collected as well, too. And we're also— kind of in the process of making it easier to deploy GRR with lots of different labels into lots of different environments. So we're now collecting and kind of inside the UI reporting different statistics on, you know, different labeled clients so that we can answer questions like, oh, you know, we just installed it on 1,000 Ubuntu machines with this label. Do you see those reporting in? And make sure that we have we can see those reporting in. And we also want to kind of make building the clients a lot easier. So we've been doing some work to kind of automate the building of the clients and the labeling of them so you can build and label a whole bunch of clients quite easily. So, okay, great. You've really sold me on this, Greg. That's good. I want to try it now. So what do I need to do? Basically get a 64-bit Linux machine. It doesn't have to be Ubuntu, but it's probably going to make your life easier if it is, because then you can just run this quick start script. It's going to install a bunch of packages and ask you a few questions about kind of how you want things configured, and then it'll start the server. And so you just open a browser and then you can download and install the, the clients from, from the from the UI itself. It should— to kind of like get all this done is probably about 20 minutes of work. Once you have the machine kind of ready, then it's, yeah, sort of as long as it takes to download the packages and install them, basically. And so here's a whole bunch of links. So here's our home page and links to a whole bunch of other tools that either directly related or we use quite regularly or a part of our kind of like open source forensics tool suite. Yeah, so that's about all I have. I'm happy to take any questions you guys might have.

Awesome. GRR looks really cool. I have a question. It's not any— no, it's not for any specific use case or anything. I'm just curious if you can execute commands on the clients. From the server? If you want to do a one-off shell command.

OK. Yeah, so it's actually pretty useful. So yeah, we've kind of— because we've gone for this client that is less smart, it's not a super smart client, sometimes you want to do something that the client doesn't have the capability to do. And you want to do it quickly. You don't want to wait. Until whenever the next client release is going to be and build the functionality in. You just want to kind of run something and then get results back. And we do this kind of reasonably regularly. And it is a pretty useful capability. So the way it works is that you sign whatever bit of code it is. It might be like a shell script or a Python file or a Windows executable or a compiled C ELF or whatever. And upload it to the GRR server, and then you can run it, you know, on a single machine, or you can run it as a hunt across, you know, lots of machines or all your machines.

Okay, cool. Travis in the chat had a question. He wants to know if you can search for a known hash across your fleet.

Yeah, so we get asked this question a lot. So The short answer is yes. I'll give you the slightly longer answer though. So a lot of the time I think people share fairly weak indicators of compromise that are like not that useful. So just if you just have a hash, like you literally, that is the only thing you have, you don't have, say, like a file path where this, where this file might be, you don't have a file size. You don't have like any indication of what the file content might be. It's literally the only piece of information you have is a hash. Then yeah, you can go look for it, but you know, are you going to look for it across, I don't know, 100,000 machines? Are you going to hash the entire disk of 100,000 machines to look for the file? You know, it's, it's, it's a really kind of expensive way to look for things, and it's really fragile. So, you know, if you're looking for malware, then, you know, you're basically relying on getting exactly the same version of malware that someone else got. So maybe, you know, they compiled it, you know, after— like, the sample of malware you get is, like, you know, different because they just recompiled it, or they tweaked something slightly in the malware, or it has a campaign code built into it that's tailored to your particular environment. So, you know, that one bit difference means that you would then miss it if you are hunting by hash. So normally we tell people that this is not kind of the greatest way to do this, that, you know, a much stronger way would be to get, build some kind of content signatures. So you can grep inside files with Go, and you could grep across. So say you know that like inside the first 100 bytes of this file, there's pretty much always a byte string of blah. And so you could look, instead of like having to open every file, read the entire file, hash the entire file, you could just look in the first 100 bytes of these files. And then if you can limit it and you know, like say, oh, this always goes into system32, or this always goes into Program Files, or this always goes— this is a, I don't know, a launch daemon or something on OS X. If you can look in a particular place, then that's obviously also a lot faster than hitting the entire disk. But yeah, like, no one's really going to thank you if you hash their entire disk just because, you know, someone shared kind of bad indicators with you. So yeah, the short answer is yeah, you can do that, but the long answer is you probably don't want to.

Yeah. Also, if anybody has any questions that, um, that are remote, you can unmute yourself and then ask the question and then mute yourself again. Or you can paste your question into the chat and Greg can look at it or I can read it off too. I have another question. Let's see here. Sorry, I'm chatting at the same time. Oh, do you know anybody that's doing any sort of automation. I know you're talking about your event triggering in the future work, but can you like say you have the YAML file for the indicator, can you just somehow query the API? I'm assuming that may be available, I'm not sure, but to actually go ahead and get your fleet to look for that artifact? Basically I'm kind of thinking working with Cuckoo, if you know there's a known binary, getting some data from that and then using the API or maybe a command line tool that maybe includes with to actually do automated analysis?

Yeah, so I mean, the HTTP APIs are basically going to be the way to do this. So there is actually— I ran into a project that— I haven't actually talked to them, but someone out there has written a project that is attempting to do automated GRR collection triggered by Snort alerts. I saw this, you know, had kind of no input from us, or it just kind of turned up and was kind of interesting. So people are trying to do this, I think. So from memory, I think they probably use the console, like the command line interface, to do that stuff. But yeah, the way of doing this in the future will be the HTTP RPC API where you basically be able to call like, you know, Artifact Collector flow and I guess probably, I don't know, I think probably the way you want to do that is define your artifacts beforehand. So like, you know, if we see, you know, this particular type of alert, then we're going to fire this particular artifact. And then the RPC is just telling Go to do that, to collect that artifact. But we could maybe also— maybe APIs for dynamically adding artifacts might also make sense. OK, cool.

Yeah. Anyone else in the room have any questions? We get one more in the chat. Oh yeah, sorry, we got another question from the chat. Travis asked if there's any known integrations with YARA or OpenIOC.

So not right now. So Recall can actually hunt YARA signatures in memory, which is super cool, but the kind of more conventional, like finding stuff on disk. There are signatures we, we don't support. We might in the future. It's kind of up for debate about— so there's definitely one open source user who's a big OpenIOC fan, and they, they're very interested in getting OpenIOC to work. So I think they're going to work on that. I think we're more likely probably to do some combination of artifact collection and YARA maybe as a kind of, you know, here's where the data lives. And then YARA does kind of like the decision logic. So like if this thing and this other thing, then that's bad. But yeah, kind of not built in. I mean, you can't take a library of OpenIOX and run it on Go at the moment. You can run YARA signatures with Recall as long as using like recent Recall.

Yeah. Anybody else with questions? Okay, well, I think GRR looks really sweet. I would definitely like to deploy this. How many machines are you doing that over at Google? Your deployment size.

Yeah, so I mean, we've got more than 100,000 clients deployed.

Huge number. And I imagine you don't have any performance issues, like you were talking about earlier.

So I mean, it's a big complicated problem running any sort of collection at that kind of scale. So we're constantly refining how we do things. I mean, if you're following the repository closely, you will have seen, like, we completely reinvented how the queuing system works. And that was, that was to take some of the heat off some, some particular worker queues that were, you know, being really kind of hot rows in the database that had to get updated a lot. So we split out kind of like how the queues worked and I think that's been quite a good change. We're basically, yeah, I mean, we've been using this at a pretty significant scale for a number of years now. So we're pretty confident about the scale, and it was definitely designed for it. Kind of until recently, I also had to put a big caveat on that, that, you know, where this is based on our experiences with our own kind of internal data store. But now others are running tens of thousands of clients on the open source data stores, and it's performing well. So I'm more happy saying that the open source scales as well.

I'm curious, why don't you guys use MySQL or an open source database schema?

Just, you know, like if you've got an amazing, amazingly performant industry-leading database for free, would you use it? You know, it's kind of— we'd be crazy basically not to take advantage of some of the infrastructure that Google has, because, you know, it's very well maintained and well supported inside Google. It runs lots of other products, and we get a lot of benefits from using those. Those kind of products internally. So it's just a case of we'd be fairly crazy not to, basically.

I wasn't giving you shit. I was just wondering.

Yeah. Yeah, no, it's a valid question, definitely.

All right. Well, I think that summarizes it all. Greg, thank you for donating your time. I enjoyed it all. Yeah, no problem. I think we've— GRR looks really, really useful. We will have this video uploaded to YouTube. It'll be available online, also on Vimeo. What did you say? Can't confirm. Oh, yeah. William can't confirm how awesome it is. I guess that's it. Yep, so if you're not a part of the OpenNSM mailing list, do join. And also, you know, follow us on Twitter. And if you know anybody that is able to sponsor us, to help out so we can do our research and so that we can develop our lab, do let us know so that we can get in touch with them. Thank you guys and have a good night.

Great. Thanks, guys.