Jon Schipp shows how to speed up tcpdump captures and reduce dropped packets. Disable DNS lookups with -n, increase the buffer with -B to 4 or 8 MiB, write to a file with -w instead of printing to console, and limit packet size with -s. Live vmstat output demonstrates how each change cuts CPU work and kernel drops.
Chapters
In this video, we're going to talk about performance with tcpdump, specifically capturing package life. So, um, a number of these options we'll talk about have been covered in other videos in the tcpdump module, so we're going to wrap them up quickly here. So 3 things we want to focus on. The first is domain name resolution. So tcpdump has the ability to resolve names. We use— just say we're sniffing on this interface. By default, it actually tries to resolve every IP address that comes across the wire, and for each one of those it has to resolve, it has to create a DNS request to the DNS— to your local resolver settings, to that DNS server, which will take some time. So it has to wait for every reply back before it can actually record that packet. So this causes extra problems. Also, in a network security monitoring setting, it could be an issue because if there's some traffic and you're reading it through it and it has some attacker's host in there that you're resolving, and maybe he has control over a DNS server, he may be able to see that you resolved or made an attempt to resolve his address. So as it makes its way to— as the DNS request makes its way to his server, thus alerting him that you're investigating his traffic. So that's another reason not to do— to use, uh, to keep the name resolution off. To do that, we use the -n -ni option. You can see now it's just the IP addresses. It didn't try to resolve anything. Just a quick demo of that again. Let's just print one packet here. As you can have the IP addresses much faster anyway. And now we have the resolution here, and it actually tried to use my hostname for this, so it pulled out the, uh, Vagrant, um, Ubuntu Trusty, uh, name for my hostname. So, um Next, we're going to talk about buffer sizes. So libpcap has a buffer and tspdump has the ability to set a value for that buffer to increase the size. So first of all, it's going to clear the screen. Let's just sniff on my interface again and press Ctrl+C and you can see that we've captured a number of them. That is packets that tcpdump was able to process. We also dropped a number of them by the kernel. That is the buffer overfilled. It was a buffer overrun. And the packets that were lost were 111, dropped by the kernel before they were able to pass the filter and then make its way to being captured and processed. So how do we prevent that? Well, one way is we can specify a larger buffer size. We can do that with the -b option, or you can do the GNU long format and do bash -buffersize, and the value is in kilobytes. So what we're going to do is we're going to set 4096, and that's equivalent to 4 megabytes. For the buffer size, and I want to do something to set the timeout. Timeout, uh, 5 seconds. So this is going to run TCPdump for 5 seconds and then it's going to quit. Okay, so we can see that 86 packets were dropped by the kernel. Now let's try to increase the size once more. Let's do 8196. This will be 8 megabytes. And this time we have 0 packets dropped by the kernel. And just check if that was just an anomaly, go ahead and run it again. Oh, just— we got 4 this time. So we can actually go further, but let's just do it one more time. Yep, 0. So the more we increase the buffer, the less likely we'll be able to drop packets. Okay, so, um, the other final thing is interrupts. Whenever you print, uh, data to the screen like this, that is, whenever the packets are printed line by line, you actually generate an interrupt, and the computer has to take control of the interrupt from the terminal. And to do— to write the stuff to the screen, so you have to go back and forth between context switches more often. So to prevent that, what we can do— or to demonstrate that, let's just show you, uh, in the same machine, I'm gonna run this tool called vmstat, which is going to print statistics for the system. We're looking for is under the system column, the in fields, and this stands for interrupts. So you can see that right, right about now we have interrupts are about less than 50. So we're going to go ahead and run tspeedup again, get rid of the timeout here. And you can see now that it went from 16 here— couldn't quit it— 16 to 200 or 2,623 interrupts per second, and then increased to 3,000, 4,000. So it's actually causing a lot more work for the system. You can also notice that the context switch column went up. It was, uh, 45, 118, 16 here, and then it jumped up to 18,000 and then 25,000, etc. So doing that causes a lot more work for your processor. So if we go ahead and run this again, and we're still, um, processing all these over here. Now we'll go ahead and quit this, and we can expect to see a drop-off now, as we have seen. So there was 3,699 interrupts, now we have, uh, we're down to 21, 11, 15, back to the normal rate. So to alleviate this when you're capturing packets, what you want to do is you typically just want to write it to a file. Just take everything and write it to a file. Um, let's call it bus.pcap, and just let it do its thing. It'll capture the packets, and then when you're ready, you can actually just read it in with tcpdump -nr. So let's talk about another option that can be used to increase tcpdump's performance. That is the snap length option. So first off, what we're going to do is we're going to run tcpdump, and we're going to specify the eth0 interface and Set a timeout and we'll count the number of packets lost here. Set it for 2 seconds. Okay, and we lost 129. And by default, tcpdump sets the maximum snap length value of 65,535. And this is the number of bytes from the end of the packet— or let's see, the number of bytes that tcpdump wants to keep. That is, it has to, has to process each one of those bytes. So if you limit that number of bytes to a lower amount, it has to process less, resulting in better performance. So by default it is 65,535. Let's go ahead and set it using the -s option to something smaller like 96. You can see here I have 0 packets now. We'll do it once more. And again, zero packets. So in cases where you know you don't need the full size of the packet, that is, TCPdump does not need to truncate it with the snap length option, then you can kind of specify which area, or based on the number of bytes, where in your packet you need to, um, keep. And it truncates it at towards the end to the beginning. So if we say, uh, 96 bytes, it's going to keep the first 96. Everything after is going to be removed. So that's just one way to, to keep track, or to lower the amount of data that tsnip has to process. It also lowers the— since it lowers the amount of data, when you write files to an output, when tsnip writes a file or new pcap file, it actually takes up less space because there's less bytes.
