OpenNSM (Scaling OSSEC)

¡Hola a todos! My name is Jon Schipp. I'm a chair for the Open Network Security Monitoring Group at the University of Illinois Urbana-Champaign. Today we're going to have a presentation on the OSEC, Host Intrusion Detection System. And this comes about out of some work we've been doing at the National Center for Supercomputing Applications where we're kind of revamping our OSSEC deployment and kind of phasing out some other detection tools that we had such as SEC, the Simple Event Correlator, and Sagan, a multi-threaded log analysis engine. And so now we're primarily using OSSEC for our monitoring., and the monitoring that we're using it for includes, it's one of its main functions being log analysis and the other one being file integrity monitoring. So we get events from both of these, these types of monitoring, and we'll kind of correlate them in some way with various tools. So again, coming out of this for our work at the NCSA, we released a bunch of tools on GitHub. So if you go to, you can see on the screen here, github.com/nCSA/ossec-tools, we can take a look. We released a few decoders, some Nagios plugins, a number of rules that you can use and customize that we're using. Of course, some of those rules there that we're not publicly showing, etc. And then we have what I think is the bread and butter of it and is our revamped active response scripts. So we, you can see we developed a bunch of scripts to kind of do further analysis over what the actual rule set can do because there's some limitations in the rule set and sometimes it's nice to be able to script it to your needs. I'll kind of go over a few of these or whatever we have time for today. So first off, we run OSSEC in an open research environment. And if you're in the academic community, you might be familiar with the term Science DMZ. It's just a huge open environment where you really don't have firewalls in place because the pipes that you have to pass traffic over for large data flows used in scientific research, such as, you know, thousands and thousands or even millions of high-quality images used in physics or biology, and those are transferred over terabyte flows. And firewalls in that way, if they're in the middle between the flow and its destination, it will either slow down the research or obliterate the firewall in most cases. So, uh, to reiterate this, we are running this in an open environment, so we have to rely on the fact that not all of our systems can have the OSSEC agent running on them. You have many researchers coming to the university every day, lecturers, PhD students, and others, and just students in general where you don't have that control over their computer. So we rely on log analysis for those people that are in our organization. People have their systems that are running their computational things things that they are doing send their events, that is their logs, to our relay system, which is a kind of fault-tolerant rsyslog implementation, which relays these to different machines. And one of the machines that is relayed to is a log analysis system running OSSEC. And that just takes in those billions and billions of logs that we get every single year and runs all the rules through them. So we prime— so with our open environment, we rely heavily on log analysis. Now for machines that we do have a hold on, that is that we own, that we use, we can install in some groups, if they're under our control again, like this— like this— I'm on the security operations team, so our security operations servers we can have the OSSEC agent installed on those to give us some file integrity events because we own those servers. That's what we work with. So that's where our file integrity monitoring comes in, is for the few servers that we're able to— I guess it's not really a few, we have quite a few servers just for being in the security group. But one thing that we're kind of working on right now is talking to other groups such as network engineering and visualization groups, etc., and trying to get them to work with us to coordinate to actually deploy OSSEC in a fashion that's catered to the needs of those groups. That's kind of a work in progress and we're kind of doing that right now, but it's not quite there yet. So let's take a look at a few of the things that we are doing. So first off, the, the active response scripts in OSSEC are you know, they're really nice, but there's a way to extend a lot of them or to write new scripts that add more functionality. For example, it's not really— if we take a look at the command search, this one right here. Oops, didn't mean to do that. This script right here. So you want to be able to, based on the logs, look for commands inside of logs such as commands in sudo logs or anything else that may be a log event that could have some sort of thing that is, that you could recognize as like, you know, gathering of information such as people using the ID command or who or, you know, getting information from the databases that you have on your system with getent, etc. So in OSSEC there's not really a good way to do that with the rule language. You could do a database, a CDB. I don't know if you remember the content database that you can use in OSSEC where you can have a list of key-value pairs and you can then from a position in your log or a field that OSSEC can decode, compare that to what's in the CDB database. The problem is there's so many different variations of a command that can be sent in the command line. For example, in sudo logs, they put the absolute path. Sudo will get the absolute path for the command run. So to get that in the CDB, you have to have every possibility but to do the match. So that's kind of problematic. So instead of having millions of entries, we can just write an Active Response script that once an event is generated, it gets passed to this command search Active Response script, which can then look inside the log event and find all the instances of ID or wget or curl or FTP, etc., that we may be interested in. So that's just kind of one way of extending the functionality. So a few other ones that are really nice that we use are the Puppet DB lookup script. So if you have a Puppet implementation, and this is kind of how we scale OSSEC at the NCSA, we use the Puppet Configuration Management Tool to, you know, we have a manifest that will actually deploy OSSEC to all their agents and to deploy the server. And anytime we update these scripts, we can easily throw out these manifests, we can execute them on the actual client machines and they'll have that version of OSSEC with all our stuff. So right now we are able to scale to OSSEC to as many machines as OSSEC can actually handle using our Puppet environment. So one of the cool things about using Puppet though is if when you use Puppet you have it, we're using the Puppet in the server agent configuration or client server configuration, There's a thing called PuppetDB or Puppet Database. And anytime that you say, hey, this file should be provisioned to a machine, for example, you want to copy the maybe the fstab, the file system table to all the machines for certain boot parameters. Or the network configuration file, the networks file, /etc/hosts, whatever. You can do this. And then when Puppet knows that it's supposed to, you know, have this executed, it's in the Puppet manifest file, it's supposed to monitor this file for changes and then throw whatever you have in there, you can do a lookup into the Puppet database which records everything. So you can be like, Puppet database, do you have /etc/hosts? Is this a managed file for server xyz.com? And Puppet database will return either yes or no when you ask it the proper API query. And based on that information we can say, well, this file has changed and it's managed by Puppet, so it probably means that either A, we changed it using Puppet, or maybe the system was compromised and someone changed the file, but Puppet runs every 15 minutes in our case and would override it anyway. So we don't really care that much for those kind of files. So that's one of the really cool things you can do because When you have, you know, we manage tons and tons of files over hundreds of systems with our Puppet configurations, nice to be able to query and say, hey, OSSEC is, you know, does Puppet manage this file? And if it doesn't and the file is changed, that event should be escalated to a higher number, right? That's the one we care about. And we can kind of go further. And one neat thing we do is with the RPM lookups or the deb lookups, Whenever a package changes, one of the big things with OSSEC that kind of sucks, or in file integrity monitoring in general, and that is when you install a new package, which admins do all the time, latest security updates, need new tools to do different tasks, etc., each one of those packages comes a number of different files that are different than the ones before. So the next time that the integrity checking daemon runs, the hashes will be different, thus generating an alert to say, hey, the file has changed. Well, People get flooded with these things, especially when you have lots of systems. So something that we came up with was to do a lookup in the RPM database to see if, okay, so if a file has been changed on a system, we look it up to see if the file was managed by Puppet, okay? And then if it was not managed by Puppet, we look up in the Debian or in the RPM or Debian lookup in the package management script depending on what system whether it's a Red Hat or Debian-based system. Then we say, hey, is this the same hash that is in the package manager? Because the package manager has databases that store the hash of each file so you can verify that, hey, this file has changed from whenever I installed the package. For example, if you install OpenSSH server and when the key signing verification happens, blah, blah, it's a good package, you know. Well, then later on you get an event that the hash for /usr/bin/sshd has changed. Well, that's an issue. But in that case, what we do when we see it's changed is see if it was either someone actually updating the package or adding a new one or whether it was a malicious change by somebody editing the file or replacing it. What you can do in this case is look up inside the package manager to see if the hash is different than the one from the package that was installed whenever it did the metadata to the one on the file system. And if it's different, then that means someone manually modified the file most likely. And if we take these two events, we generate a log file for all these things, and we can correlate those two together. So PuppetDB, look up, will say, oh hey, there's a lot right in the log file that says, hey, this user bin/sshd is not managed by Puppet, and rpm_lookup.log says, "Hey, the hash change is different than what is in the RPM database." We can correlate across those two events for that file name, /usr/bin/sshd, and then we know when we get that alert that this is something that we really need to look at. In doing something like this, we've went from hundreds to even thousands of events on some days down to less than 20 in our case right now. So it's a huge, huge reduction in the number of events just by verifying the packet, the changes of the files against two different sources that we trust. So you have to have trust in these sources. It's not going to be foolproof, but we can say with a decent amount of guarantee that the RPM database probably is going to be fine and the Puppet database is probably going to return whether, or will return whether the file is managed or not. So with those two pieces of information, we can improve our detection. So I don't want to rant too much. I want to get more into a demo instead of just showing you the stuff on here. So each one of these scripts generates, let's go to a terminal here, generates log files. So we're going to go to our system here and I'll make this a little bigger. Okay, if you have any questions while I'm going on, you can unmute your microphone and just ask away or you can type in the chat and I'll try I'll try to address them. So here we're in our var/ossec installation directory and we're gonna take a look at the logs directory. And if you take a look at just all the log files here, you can see we have quite a few of them. And most of these are custom ones. Now ossec comes, whoops, ossec comes with ossec.log and it comes with the activeresponses.log. Well, we do a few more things. For every file that where a file integrity monitoring event is generated because of file change, we do things like Team Chimera malware hash registry lookups or VirusTotal lookups. So let's see if we got, figure our last rotation here. That one's empty, we don't have anything for that right now. That one's empty too. See, just take a look at the PuppetDB stuff. I cleared some of these out. No matching FQDN. That one, let's see here, one second. Yep, so here's an example of an event that happened recently. So we did an RPM lookup for var/ossec/rules and ncsa_rules and it passed verification. And this is one that we package up and push out ourselves, and that was looked up in the RPM database. So do take a look at some of the— if you run these scripts, take a look at the logs they generate. They're really nice. So this one, this time lookup one, will take every event and generate and see if their system clock was off. Because remember I said that we get all— our mainstay here is that we get all the logs from tons and tons of systems that our system analyzes. And so we use OSSEC as a syslog collector. Here we take it over UDP and we allow our stuff and that's taken in and then run all of our custom rules on it. And when this happens, you know, we'll get tons and tons of logs. What we do with our custom set, every log that comes in that matches certain criteria, we will launch another script on it, on the event. Now, so let's go down a little further here. Sorry, I'm doing all this live, so I'm kind of a little scatterbrained too. So anything that has, like, see, for the Debian RPM lookup stuff, you can see that the rules group, if anything matches an event from the SysCheck rules group and this level, then we execute this on the client. So this means that the Debian lookup script, or in the same case, the RPM lookup scripts reside on the agents. So whenever the agents have a file that has changed, they generate an event, is sent to the OS X server. Really, that's where the event is generated. And then it's passed to the Active Response system that says, hey, then execute the RPM lookup or Debian lookup script on those client side. One other thing that we do, because the Active Response stuff can get kind of complicated when you have so many of them, making sure the order— I found some issues with ordering. So what I've done is sometimes you don't want to execute— what you want to do is you want to test the execution of certain things inside the script. So we, we have these catch-all scripts called like ip_all and rule_all, and these will actually perform certain tasks once an event is handled— handed to them. So let's take a look at— actually, we could probably just go to the— go back to the GitHub here and just look at the code with the highlighting. I have the rule all, let's take a look at rule all. So we'll go into our rules, we'll go into, that's not the right directory, it's Active Response, and then we'll go to rule all. Okay, so we have a bunch of functions that check certain things, so everything for the rule all script is everything that's not a sys check should be handled here. So first we check if there's sys check as an argument that's passed because the Active Response, if you ever looked at Active Response log, we can quickly do that. Let's take a look. Da da da da da. There's a number of things that are passed and these are the arguments that are passed. So that's how we have to handle these arguments. So this is argument 1, this is argument 2, argument 3, 4, and so on. So if there's something that says system check in the argument, which is what happens for the system check events, then we skip it, we die. Otherwise, we run a bunch of scripts after. So we'll just kind of show you the flow of execution here. So here we have a function that checks if there's an IP available in the script, and that is because OSSEC passes an IP as the third argument. So if we go down here, here's where kind of the meat of the script is. So we check if our chat script's enabled, and these are all set with the paths up here, and if it is, then pass this information to the chat script. Which will then write it out to Slack or HipChat for us or whatever you guys use in your backend if you have chat operations style collaboration such as we do. Then we could also try to check if we have a CDB script and if there is a, if we do then in this rule matches we can start adding intelligence to their CDB script which is executed in the CDB script which is, you can look at the code for that here. And this is kind of how we're doing it. So we have, instead of having like 7 different blocks and inside the ossec.conf file we can all pass it to this one script and then based on the information that it gets we can be more customizable and launch various scripts after. So in this case, if, you know, the command script is executable then we can check for suspicious commands. And the two that are nice here are, you know, if there's an IP inside the event that was generated, which again the IP comes across in OSSEC on as the third argument here. Here's just an example right there. That gets passed right here to our BHR script, which black holes the system. That is, it is a perimeter-based system that uses BGP. It'll inject null routes, so anybody outside of the network that tries to contact it will be null routed. And then there's also the case for SIFT. Well, if these events match, has an IP, then we probably want to start adding it to our intelligence system. So then we have a SIFT script that just takes this, writes to a log file that our collective intelligence framework system kind of picks it up every night and generates the new intelligence feeds. Then we feed that back into other systems. So that's kind of the idea for how I handled the rule all. 'Cause I had a lot of issues with just doing many, many, many Active Response configure stanzas in the ossec.conf file. I couldn't get the execution order to work the way I wanted. So this was a better way to handle it in my opinion. Now for SysCheck, this SysCheckAll script happens, runs for everything that comes across that is a SysCheck event. So if a file integrity monitoring event happens, a file is changed on the system, then what we do is we execute the Puppet lookup script. Then if, you know, if the Puppet script returned true, in our case a 0 meaning that the file is managed by Puppet, then that means we don't really care anymore, then we just kind of exit out of the script, we don't really need to go on. But if the— that means we manage the file and every 15 minutes we'll overwrite the file anyway. So, but if that's not happening, Puppet's not managing the file, we want this to escalate to chat. We then want to do a Team Chimera malware hash lookup So for every file that comes across, we get the hash and we look it up in the Team Chimera malware hash registry. And then we also do the same thing and look it up in the VirusTotal hash database. And those are done through those two scripts. And that way, if there is a— I mean, this is looking for low-hanging fruit kind of stuff, but if there is something that's common enough that's going around, maybe, and it gets compromised and put on a system, maybe we'll be able to detect it. So it's just kind of another element to being as thorough as you possibly can. And that really calls this VirusTotal.py script that I wrote that takes in the hash and then checks if the entry is malicious or not. So it's good if there's no entry in the database or that the— is not malicious, otherwise it'll return false, and then in that case you'll get an event, you'll get a log that says, hey, we found this malware entry, or this file looked up, it was looked up and there's an entry in the malware database. So that generates a new log in the virustotal.log, and then we go, you know, investigate that. So the Team Chimera hash stuff is pretty simple if you ever query the Team Chimera hash registry, but you can do WHOIS or dig lookups. So a command as simple as this, dig, and then you put your hash as the beginning domain, the most subdomain, to mylr.hash.kymri.com, and if that returns as a, you know, 12.7.0 address, then that means the hash is malicious and has been found in the database. So you can just do that from the command line. All we do is pass the hash from, that we parse out from the from the system check event because it contains the hash. So those are a few things that we have that kind of makes things really nice. And let's take a look at the command search thing too now. So here's where we actually do the command search. We have basically a bunch of regex because that's kind of how you search for patterns and commands. So, you know, we can do a number of different things. We look for ID, bash, sudo, FTP, fetch, hashcrack. Or excuse me, Hashcat, CrackSniff, TCPDump, Wireshark, etc. Bunch of Metasploit stuff. And then if any of that's found inside the log event, then we generate a new log in the /var/ossec/logs directory, which then ossec reads in. So I talked about how all our, pretty much every single active response script that we have generates a new log. Okay, so you have to say, well, how do you make use of this? Well, what you do is you write decoders for your logs so that OSSEC can properly handle them and pull up the various fields, then you can do your rules on them. So let's kind of take a step back here and take a look at the decoders. So I have our Active Response decoder script. This kind of handle— this has all the decoders that we need for our Active Response scripts. So you can see that we have one for the RPM lookup, and you can see that we have a sample event that actually happened here. RPM lookup, it created an OK. /usr/bin/floppy passed the RPM verification, so it was looked up, the hash inside the database, of the RPM database, and checked out. So, and that's why it generated an OK. But if it did not pass verification, that is the hash for the file is different than the hash in the RPM database, then it would generate a warning. And that's when you get that log that says warning, then you need to go investigate that. So then we just decode these events as status and ID, where status is the OK or the warning, and the ID is the actual filename, and that gets decoded as that. So then in your rules, you can be like, you know, match this event, and if ID is, you know, that userbinflop, then hey, you can do something with it. So if you want to go a little bit further. Again, the Debian lookup is quite the same thing. I try to make the log files look exactly alike so you could reuse these decoders. And for the Puppet database lookup, you can see that this one generated a warning that a file was changed, javac— I hate when that keeps coming up, I keep double-clicking on accident— was not found for this host. And then again, we match the Puppet database in the program field of the syslog event. And then we get the file, we parse the file out to get its ID. And since we parse the files out as ID, this allows us to do what I talk about, the correlation between the puppetdb script and the RPM or Debian lookup script. Because we can match on to say, if the events or the IDs are the same for the puppetdb and for the RPM, then we can generate another event that says, hey, we need to investigate. So that's kind of nice. And then going further, you have the Team Chimera decoder just to show you what the logs were because I didn't actually have any logs on this system for today. This is what they actually look like. So it's a Kymrii lookup. Okay, no match found for /usr/bin/gpg, and then the log will actually contain the hash of it when it was gathered from the event at that time and tell you what host that binary or that file was residing upon. If there's a malicious hash found, then it says warning, and that's what we look for in our rule. We say status, you know, we decoded it here., but we want to know in our rule if the status is warning, then we escalate, we actually generate a new event. So that's kind of the state of the decoders. We also did a few other things where we, in my opinion, we improved a few decoders. One is, one I don't have released yet that's for Cisco ASA stuff, but one that I do have on here is for sudo. Now it does come with a sudo decoder, but some of the stuff is not added. You cannot decode a number of fields. That you can use in the logs. So I redid that so that you could decode a few extra fields such as the command. You can't do the command. So that with this you can actually do lookups in a database or something for like in the CDB like I talked about, but I don't recommend it because there's so many different possibilities and sudo uses absolute path in its log for the command that was run. We decode, we went ahead and decoded a status, but that way you can actually look for something specific if you need to. And then we also use other things. Like one thing you'll know, I found out that the extra data, you can't actually use that in rules. There's a, there's a, I think a bug, or at least it's not, or the documentation is wrong, one of the two, where you can't actually look for that stuff. So a lot of the rules are a lot of decoders that use extra data. You can't act on that in your rules. So I changed it for some things, but I acted upon things all that you can use. Like URL in this case is not a default in the default decoder.. In this case, we're using URL to decode the TTY— I'm sorry, not TTY, but the actual working directory of the user when they execute it. So like this one, the command was executed from— or the user was running in /var/ossec/bin, and that's decoded as the URL because that makes sense because it's a path. So there's kind of like things like that where you can then look at these in rules. You can say, hey, well, if URL is this for some reason, maybe you have a reason to look at a specific place. Then you can generate events on that. Otherwise, you would not be able to do that in the default sudo decoder. And one other one that we worked on, Vlad Grogorescu of the Bro team and NCSA did the top part of this one where he added the kuro stuff and then I added the kadmin stuff at the bottom here where we have decoders now for our Kerberos logs and in OSSEC that doesn't come by default. There is no decoder for Kerberos stuff, so we created our own. We can look for KSU events, you know, where authentication failed or we have authorization was successful, etc. And then we do the same thing for KAdmin where the admin would actually create new principals or modify or retrieve various principals, and we can look for queries for that by decoding these events. So that's just some of the stuff that we open sourced here. A few other things. We have some Munin plugins if you use the Munin graphing system, which is really nice. I'll show you what ours looks like. We can go down to the bottom real quick here. I have examples for a lot of these, but it's a very simple graphing system that out of the box you get tons of graphs. So usually when I'm on a network with systems, I need to get information. There's really no good visibility on the systems. First thing I'll do is set up Munin. And because it has out of the box tons and tons of graphs for various common things that you would be— that you would care about, such as disk latency, network latency, network throughput, processes, etc. So we created a bunch of Munin plugins that can work on the OS X various log files and graph data based on them, just basically just stats. So at the top here, you can take a look and you can see that this is a graph that documents the number, the count of execution per our Active Response scripts. Remember, we have a ton of Active Response scripts. And we can see based on what things match, because remember we have the rule all and the syscheck all and all that. So certain things, like whether it has an IP address, this script will run, or whether it has a syscheck event, then this script will run. So we can kind of track over time how many of these events are being, or these scripts are being called. So you can take a look at that and it gives you counts like command search was called 34,000 times or at least that was the last interval and that's a lot because, or at least in that window but the thing is we want to run that because we never know where to look for commands. They could be in any log event really I think. Can be injected in there or typoed or whatever that could indicate something. So we use command search across all the events that are generated. So that's why that one is specifically high. And that's this purple line right there. And then of course the rule all, because it's a catch-all rule, those and the SysCheck all should be at the very top and again they are. So the other thing we have is just alert counters, OSX stats by day. And this Munion plugin will just kind of show you a very simple count of rules, which is called all syscheck events or root check events, and kind of show you per the time interval these are going, which is 5 minutes, and over, over the period of a day, uh, how many of those syscheck events or root check events or a count of all the rules, uh, for events were generated. And there are some patterns. You can see it's kind of like a, whoops, a jigsaw, or not a jigsaw but a saw, when you look at this from the weekly view. Fairly standard baseline there. So going a little bit further, our OSSEC top group script shows you the top, okay, so it takes every rule, you can have a category or group tag, and what it does, is it aggregates them so that for any event that matches a specific tag such as access control authentication failed, authentication success, we graph the number of events by group so that you can find across all the events that have— that are related to a failed authentication. You can show the aggregate number instead of showing each event with a specific rule. You can show the group number for all those those events. And then we can see it here, the green line is the syslog event, which is usually like a lower event that is generated, and that's generated the most in our case, which is the most common because the rule hierarchy in OSSEC actually uses syslog at a lower point. So that's supposed to happen, essentially. So, and then moving along, Hope I'm not talking too fast. I'm actually running out of breath here. Let me grab a drink here. I do talk fast, so if you have any questions, again, just speak up or put it in the chat and I'll be happy to try to answer them. And then going forward, we have the os-set-top-rule script, or a Munin plugin that generates these graphs. Essentially, this is per event. It gives you the top 10 events and their count, whereas the one above shows you the top groups and the count of all the events in that group. This one gives you the top events. So instead of SSHD authentication success having an incremented counter for authentication success in the group above— sorry about that— it would actually just show you that the event is, in this case, the actual event name, which is SSHD authentication success, in the bottom. Hopefully that was clear enough here. And then it shows you like, because of this you can say, hey, well, there's— we've had this many integrity checksum changes, or this many new files changed or added, or this many failures for a specific rule that we care about that we want to see visually rather than grabbing through all the logs and counting them up. So that kind of covers the, uh, the Munin stuff that we've done. And going, uh, on the graphing, we recently instituted the Splunk, uh, plugin which I haven't had a whole lot of chance to play with it yet, but you can see here that we have graphs for that up on the screen. Again, just kind of some of it's similar to what Munition will report, but with Splunk you have a more advanced— well, you can't really do searches and stuff in Munition, you just look at the graphs, but in Splunk we can do lookups for specific field types in the logs, We can make really nice custom graphs such as pie charts, etc. And it gives you a lot more flexibility. So we can look at reported hosts, severities, top 10 users, or whatever, and then kind of look at that data further in detail to get specific information out of it when we ever have— when we have a— if we look at things in the aggregate or if we have a specific incident that we need to investigate, we can query the logs from the Splunk interface. So, and that's a free plugin, OSSEC Dashboard, that you can get in the Splunk App Store. It's pretty nice, it works fairly easily, just kind of works out of the box. Just send it your logs and should be done. So I recommend taking a look at that. I need to play with that more myself. So I'm going back to— whoops, find the other terminal. The other thing— see how we're doing on time, 6:41, I got about 20 minutes here. So the other thing that we have that we released is the Nagios stuff. So we want to monitor all our, our OSSEC infrastructure. And to do that, there's a number of things you really want to know about. And I didn't see any good, in my opinion anyway, good Nagios plugins on the web that would give you information that you would want. So I wrote two of them. This, the Python version kind of supersedes this shell version, which I wrote a few years ago. But some of the nice things that you can get with this is you can get the times that, if you take a look at the top here, this is a few examples. Actually, I wonder if I have, if we go back, I may have the examples inside this README. Yeah, I just have them here. Here's a, that's a better, just to look at that. Let's just do that. Dun dun dun dun dun. Didn't mean to do that. Go forward. Okay, sorry about that. Okay, so here we have a plugin check so we can take a look at the status of the OS X service that basically says, hey, is the system check daemon running? Is the rootkit check daemon running? Is agentd running or whatever else? And then it reports an error or critical event if it's not running. Well, you can also check that all your agents are connected and online. This is an important one because after time, sometimes agents will no longer be connected. And because of changes on the system, networking issues, etc., you want to know about these things. So you use the -t option. You say you want to use the connected check, and then you can— it'll check everything and check for anything that is not connected, or if there's something that is disconnected, such as a host that's down for maintenance for a long period of time, you don't want to get the events for it, you just say -s skip, and you give it the host, and then it'll skip that in the output. So you won't get an event if host 1 is down when you know it's down. Other things you can do that are really useful that I find a little bit more useful is the syscheck and rootkit-checktime. So one thing is you want to make sure that you're— which I've noticed, I've seen cases where things weren't completing on schedule. Like you have your times, right, where OS X file integrity monitoring needs to run every 6 hours, but for some reason it's not running at 6 hours. Has passed, but there's no changes or anything. And I didn't really find this out until I, I looked at this and wrote this plugin that sometimes you'll have cases where the daemon does not run on time. And this is usually because something in the operating system or scheduling issue or something caused it to slow down because maybe another process has run wild. Disk latency is really high, throughput, I/O, whatever, that it's being backed up and queued. Or it's going really slow to not actually finish the check time. And this can actually help you identify issues without— with those kind of problems. So in that case, we use the syscheck check type, and we say that all the agents, we want to get a critical event if an agent has not had its syscheck daemon run in 12 hours, and a warning event if it's only been 6 hours. So you can kind of customize that. And we do the same thing for the root check daemon. Again, we want— if it's more than 12 hours, we care, we want to get an alert. And if it's more than what it's supposed to, we don't need it, but it may be just because the system is slow, so it's still in that warning period. 6 hours might be fine, so we'll get a warning event for that. And I adjust these based on what your interval of running OS X syscheck daemon. So base it off of that. And then going a little further, you can check the status of certain agents that maybe are more critical than others, or you just need to know about them with the -a flag here in the actual, in the checkossec.sh script. That's the older one where you can give it a comma-separated list of the server names that you want to check for, and it'll only report if something in that list is not connected. So, and there's one more check that you can do on that one with the older script, and that is you can do if 3 agents are offline, then we want a critical event. If 1 agent is offline, then we get a warning event. If there's less than 1, then we get an okay, everything's good. So then you can do it by count. So there's kind of a number of different custom— a number of different ways to do your— the checking of your deployment of OSSEC to make sure everything's running smoothly. Let's see, is there anything else on the— on this that we want to discuss? I will do a write-up online in the future. I will get done faster if there's more pressure, if you guys are really interested on how to configure most of this. There's documentation here. It requires knowledge of OSSEC, so to adapt, or any of the other tools like MutedAudios. But maybe in the future I'll do a a document that can kind of help someone completely new to it to get started with all the custom stuff. So that is the OSSEC, our open source tools that we released, the wrap-up of that. Let's actually, no, let's take a step back, sorry. I forgot about the rules. We can just take a look at the rules real quick. So we talked about the decoders earlier. So we said we generated our own log files, right, for all our active response scripts. Well, this is where we can use the rules to identify things. Our log file for a VirusTotal lookup will have the text "Malicious hash found" in it. So all we have to do is write a rule because we parsed in our decoder the program name. If the program name is the name of our ActiveResponse script and we match the text "Malicious hash found" in that log, then we generate rule, this rule 120001. The same thing goes on for the other types of scripts. But the cool one that I want to show you that I talked about that I didn't actually show you the rule for it, which the neat thing where we correlate the PuppetDB database lookups with the RPM lookups. And if we take a look, PuppetDB, so it's this event right here. Alright, so let's just do, dun dun dun dun dun, that guy. If you take a look at this event, we match the event, which is 120,006, which happens to be the PuppetDB lookup script that says that a modified file was a file was modified but not managed by Puppet, so we care, right? Because something modified, that's something that's not being pushed out to our systems by our orchestration tool, so it's probably manually modified. Then we match on that event, but if within a time frame of about 1 second we get an event, because these things happen very quickly with the active response, 1 second actually works for us. This event, 120004, if we go scroll to the top, 24 is the RPM lookup script. And if this one matches the warning, meaning that a file has been modified that has a hash that's different than what was when the package was installed, that's indicated in the RPM database, then we generate that event. So when these two events are generated, we correlate them to— so we match these two things right here and for the same ID. Now remember, in the decoder, we actually decoded the file name as the ID field. So this says that if the file names are the same in both events, that is the ID field is the same, then we generate rule 120007, which says that a file has not— has been modified that is not in the Puppet database or in the RPM database. And that one gets a high level, that gets a 12, because we won't get the email on that that says, hey, you need to go investigate this because it was changed and we don't have a record of it in our systems, or at least they don't— they didn't record that change as being proper. Through our development operations workflow, etc. So that's what we would investigate. We've actually got a few alerts for these of just system admins making changes on files that were not in Puppet. So that's actually useful to us because then we can see, in some cases when it's not malicious, you can be like, oh well, maybe this file should be managed by Puppet. If you change it often, maybe it should be in revision control, right? So that can just help us from an admin perspective, just knowing that this file has been continuously modified, maybe every day or every week, etc. And that, you know, this should be under our control with— through— and, you know, for resilience and stuff with the version control system and pushed out by Puppet. So, and then we'll kind of go look through at the database stuff. So let's take a look at the CDB lookups. If you guys haven't used the CDBs, I'll go ahead and demo that real quick. So going back, we'll go into our list directory. And here we can actually, we have a list of various things. So let's take a look at MALLIST. MALLIST is a CDB database, actually it's kind of redundant saying database after CDB, that just contains a number of hashes that you can look up. We have a number of other ones. Remember they're just key-value pairs. So take a look at the domain database, a bunch of sketchy domains that are pulled from intelligence feeds and classified as malware. So OSSEC can look up a value in the key or in the value, and then we can do things based on these fields. So if we go back to our scripts, let's look for things that match lists. Let's do field equals. Okay, so in this case, we have a rule that wants to match a malware hash. You know, this is separate from our TeamCommary lookup and our Active Response for the VirusTotal lookup, we just do it from a database and we say if we get a match in the address match key lookup, so we're looking, we're making, essentially what this is saying is look in the key field in the list, malist file, and if it matches the ID field, then we can, and this rule is generated, there was a malware hash match from our intelligence feed. And of course you have to have things that are decoded as the ID. So you have to write the decode, modify your decoders to say that the field, the actual hash field, is decoded as ID. Um, see if we've got another example in here. I think I— let's go back to our NCSA. Whoops. Let's go to our organization rules here. So let's see if we've got our field here and just kind of look for that. So in this case, ah, so one common one that's nice to have is successful authentications to your system from a known bad IP address. I mean, that's one of the most obvious things you wanna look for, right? So, and you can actually do it very nicely now with OSSEC because of a rule like this. So all we have to do is have a database called ip.list in our case, so we called it, and take a look at the ip.list, and all it does is contain on the left-hand side in the key field the malicious IP addresses, on the right-hand side a tag, we could say that, hey, this is exploit, and we look them up against the source IP address, I'm sorry, against the source IP field of any log event that contains that field. And then we get an event, again a high pro— again an escalated event of level 12 that there was an authentication from a bad IP. And that's, I mean, this is a really useful rule, right? So if some system is compromised from some machines that are known to go out on the net and scan you and break into boxes, run exploits, etc., you can hopefully identify it using intelligence feeds. So that's just kind of one of the nice rules that we have here. Another thing that's nice is this rule right here. Where'd it go? BH, oh, there it is, okay. So with our black hole router, he blocks people that are doing lots of scans, various thresholds, and brute force attacks, et cetera. And it's blocked for a time interval. What if they come back? Well, this alert right here will look up that, all those events. So the black hole router actually generates a log file that has the IP addresses of things that were blocked. So we can pull that down at a set time, maybe hourly, etc., and feed that into our database. And then we look up the source IP against the logs against a previously blocked host, 'cause we blocked it before, so that means that host is pretty effin' persistent. It also could be the case that once that host got blocked, their timeout, they realize that if it was an automated host, for example, or it has some smart logic, it could say that, well, you know, I'm getting blocked, so I'm going to wait a period of time until and try again. And then once the timeout has passed, whether it's an hour or 6 or whatever, then it could try to do a slow scan. Well, if it comes back and if it does a slow scan, the BlackHole Router might not be able to detect it. So in that case, we can block it this way, and this will generate an event that gets then passed to that BHR script, the BlackHole Router Active Response script, and then it's blocked again. So that's kind of a nice way to kind of combat certain ways to, if it's not just a host that's just banging around for low-hanging fruit, it's actually a little bit more smarter than that, then we can kind of catalog that, create intelligence data out of it, and then use that to feed that back in. So, yeah, so that's just kind of a few rules. While we're here, I'm just gonna scroll through and see if anything catches my eye that you might think, that I think might be useful. And these are, I mean, other people have probably written these style of rules too, but we try to release a bunch of them that we think would be useful for many environments. So, so do have at it if you think these are useful. Um, let's go, let's go break from that because everybody gets, uh, understands what's— let's take a look at the, um, I think I got some SysCheck rules here. Yes. So one thing that I've discovered that kind of sucks with OSSEC is that you cannot, at least by default, no actually you can't at all, is you cannot match, okay, so in this case, you can take a look at this rule right here that matches a change, an event that indicates a change for the Kerberos keytab file, right? You would want to know that that has changed. It has your authentication data in it. So in this case, any system check event that has, that matches that file name will generate this specific alert as an escalate level of 11 that we need to notify about this. So you can put certain things in there like look for rootkit files like this Felix rootkit, etc. But one thing that would be really nice to do is just to be able to have a hash. You know, a malware hash or hash of a file in there instead of a file name. And that was my original goal was to do that, but it turns out the SysCheck decoder actually bypasses some things, if you look at the C source of OSSEC, and you can't actually get that data matched on there. It won't match on those hash fields. It's not actually passed up. So you actually have to do that in a different way, and that's why I wrote those malware scripts and did the CDB intelligence data to actually kind of circumvent that so we could actually do lookups on known bad files. So just be aware of that. There's a lot of gotchas that I found in OSSEC where either the documentation has either been— has not been updated, is wrong, or there might be a bug. And I'm actually compiling a list of all those things, and I'll release it in an article later because it's kind of interesting, and hopefully it can save some people some headaches. So just be aware of that. But yeah, so one of the nice things about this is, you know, any file that you have a particular interest in, you can lock down and look for an event for that. One other thing you might like to do is monitor certain things like all the CDB lists. One thing about the CDB lists is you notice that if you take a look at these things, take a look at, you know, startup list, these are all ASCII text files. Well, if we look at, go back and take a look at the ones that are .cdb, these are actually binary files. So what actually happens is you have to run this make program called makelist that comes with OSSEC that compiles these key-value pair ASCII lists lists into an actual database file. And you have to— every time you want to— every time you update these and you want to use the newly generated list, you have to recompile them. So something like this— oops, let's go ahead and click off here, show you this rule. This looks for any changes for a file that matches a /var/ossec/list in the file path, and then it does not log or anything because we expect it to be changing things. It's managed by Puppet in our case. We generate this event and then we have an active response script that looks for that event and then if a database has changed because we committed out to Puppet, it's pushed to the system, then we need to actually recompile it for the new entry to be analyzed in OSSEC. So this event will actually end up calling the active response script that rebuilds the databases. So stuff like that where you just kind of make this continuous integrated feedback loop. So yeah, that's the— that'll cover most of that. And let's go to— what time is it? 59. Okay, we're going to stop in just a second, but we'll quickly go over the Puppet stuff. I'll show you that. Give me one second. Just kind of show you how, if you ever use Puppet, it's a really sweet tool. There's other tools like Chef, CFEngine, SaltStack, etc., that you can use. Ansible is nice to manage things, but I like Puppet, especially now that we have the— we're using the PuppetDB, which actually stores a lot of information. Direct Queries to it. So in this case, we're going to go to— whoops, not what I wanted to do— OSSEC. And we have a manifest. This is— I'm going to make this full screen here, make it bigger so you guys can see it, and just kind of take a look at— so this is our Puppet manifest file. This is a Puppet configuration file. Basically, how we do this is we pass out this class to all our systems, and we have kind of like a doc or some hierodata that says whether a system is classified as what. And we default to if the type is an agent, then you get this agent class. If the type is a server, then you get the— include the OSX server configuration, which is a class too. And then, so in the case of agent, this one gets loaded because it has the same name, OSX agent. And what we do is we automate provisioning of all these hosts to OSSEC. Now, recently, I think in 2.8, they came out with this agent auth tool, which allows you to automatically connect to an OS server, negotiate the keys from the client, and then put them in the appropriate places on the server and the client. It is an automated configuration, so you don't have to generate the key manually, copy and paste it to the system, which is a pain in the ass, especially when you have a huge deployment. So in this case, our Puppet stuff will actually look and see if the clients.keys file is there. If it's not there, it will run this agentAuth to our OS X system, and DNS handles this translation here, and then it will generate that clients.keys, and this will never be ran again, and it will automatically be configured to the system. And since this actual tool is agentAuth tool speaks to the agent daemon on the server, and this are the agent auth daemon, and that actually communicates over TCP, so you can quickly lock this down when you provision a new host by deploying firewall rules that limit, um, OSSEC to only being contacted for every newly generated host on that port. So that way you can limit the communication so that you're not getting random machines connecting to that, to your OSSEC server and trying to generate keys to work on the system. So that's kind of a nice thing. And then what we can do is we set a bunch of other things with Puppet. Just kind of show you Puppet. So Puppet, you know, make sure that we have this merge file, make sure the ownership and group is set, make sure that there is an RPM lookup log file. We'll create it and then the log script will add after. We then deploy configurations to each client. And I don't know if some, some people might be aware, but you can use— OSSET comes with its own profiling system where you can deploy different profiles to the system without a tool like Puppet, a configuration orchestration tool. You can do it directly in OSSEC using its orchestration tool, which allows you to set different profiles for systems, and then the OSSEC manager, once the system connects to it, will push out the various files to the system. So we use a kind of a combination of both. And but before we go into that, let's take a look at the server. So the server, you set a bunch of ignore things that then get turned into— actually end up being available inside the OSSEC configuration file for things to ignore from SysCheck and stuff. We make sure we have all the users for deployment. We set up the certificates. We copy all our CDB lists. We have a directory. This is sourcing, basically copying from OSSEC list directory that contains our CDB databases and copies it to the for-OSSEC list directory. You know, we provision our own server configuration, etc. And then we set up our firewall rules. So if you go back and take a look at what I was talking about, the profiles. Let's go back a directory, take a look at— I think it's templates, and then we'll go into shared. So this file actually contains a bunch of stuff that OSSEC can use to profile and push out configuration to certain systems. So we have an agent profile called Linux, and you can see that the field this value is set to is Linux is set to— is set and stored or associated with this OS variable. So what this actually means, anytime there's a system that has the operating system Linux, OSSEC will include everything within the agent config. So that way you can deploy a set of things to all your Linux systems or to all your OS X systems or to all your BSD systems, right? Your FreeBSD systems or whatever. It's kind of nice. You saw all those ignore things earlier, um, like all this, this list of ignore items. We have that in Puppet. And then what happens is using an embedded Ruby template, we actually iterate through this. So for each item in that list, we create ignore and then we fill, we set substitute for each file name in there to ignore a bunch of them at once. Really nice. And then we set a bunch of other things that are common to all your OSSEC installs, such as monitor this file, monitor this file, monitor this file. And for every agent, we're going to monitor the lookup scripts that we generated from Active Response. We can say, hey, well, if your system is— has the profile of Apache, and you set these profiles on each agent inside the ossec.conf file, then you get a configuration that looks inside for the /var/log/httplogs. You look for the Apache logs. So this is an easy way to automatically include files that are, that are specific to certain systems in the service and the services they run. So it's a really nice way to do this. So we take a look at— give an example here. We do it, we deploy a very simple OSSEC configuration file on the actual clients, look at this. So if you take out comments, it's like 4 lines, right? So all we have is osxconfig, the client, and then the OS X server, again resolved by DNS, and that's it. In some cases, there'll be a config profile available that's below that kind of handles telling the OS X server what profile belongs to it so it can pass that information. Now we're here, and we take a look at this file, kevaro, sec, co, sec, conf. You can see the config profile is listed right here. And it says, hey, it's Apache. So that's how the server knows where— here, back to this and say the server knows Apache, it compares that, that value to the one in this profile value, and then it applies these configs, right? So it's a really nice way to deploy configurations with— I don't really see it used very often in the deployments that I've looked at. So, okey-dokey. So that's about all I have to say. We're a little past time. Does anybody have any questions? I know I rushed through. I'm sure I forgot a few things, but I'll try to in the future post more information about this so people that are further interested can look at it. So any questions? And this video will be available online. I'm gonna open the chat in case I missed anything. All right, looks like we don't have any questions. Thank you guys, thanks for attending. Again, you can always reach me on the Slack channel, my email address, on the mailing list, etc. If you have any questions about this, we can talk further. So thank you much for— thank you very much for your time, and you guys have a good evening.