P2P Security with Paul Marrapese

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Paul all about peer to peer security and other security topics. Paul is a security researcher from San Jose, California. His work has resulted in the discovery of critical vulnerabilities affecting millions of IoT devices around the world and has been featured in Krebs on Security, Forbes, Wired, znet, and several other security podcasts. Paul specializes in offensive security as part of the Red Team at a large enterprise cloud company. I recently saw a session he did at another conference and really loved this peer to peer topic that he covered. There's not a lot of information out there that I could find on this and I think Paul really addresses a lot of concerns people have around it. If you're new to security or you want to know more about peer to peer security, you don't want to miss this episode. Check it out.

It's no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution and that works with your current development tools and processes. With Fortify, you could start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security.

Hey Paul, welcome to the Guild.

Hey Joe, Good to be here.

Awesome. So before we get into Paul, is there anything I missed in your bio that you want the Guild to know more about?

No, I think that's about it. Security is kind of my life, so.

That'S what I do. A lot of nice. I don't think I've ever spoke to a security researcher. So what's the day in the life of a security researcher like?

Right. I don't think there's any hard or fast rule really. What tends to happen is say I'll be playing with a product or a piece of software or something like that and something will catch my attention, some sort of interesting behavior and I'll want to dig into that and sometimes that can cascade into an entire project. I basically just kind of whenever something catches my eye, something sounds a Little bit interesting to sort of poke at it a little bit, see if anything stands out as being a potential weakness. And with the case of these cameras, that really ended up being the case. They had some very interesting suspicious behavior and I did a little poking at it and it turned out to be a two year rabbit hole. Sort of an endless sequence of insecure things.

Nice. And I like how it began with cats. That's triggered this all for you.

It did if not for those little guys. And I'm not sure if I ever would have come across this.

Nice. So for the folks that don't know what is peer to peer.

So peer to peer in the context of IoT is basically a convenience feature. It comes from the fact that when people plug in a device, in most cases they would need to like forward a port on their router. But most people don't really know how to do that. So a lot of devices now are coming with these features to make it. So people don't need to do that. They can just plug their device in, get it on the Internet and they'll be able to access it immediately without any sort of setup.

Is this something a lot of people should be concerned with? How many devices is it just cameras or could it be anything? I know IOT is going to be supposed to be exploding. So do all type of IoT devices use peer to peer?

There's a very wide range, yeah. Cameras are definitely a popular feature that this appears in. But I mean I've seen this in everything from also baby monitors, alarm systems, smart doorbells, even DVRs and like network storage devices. So this is really something that's really ubiquitous in all around the world. There's probably over 50 million devices that have these sort of features, or 100 million devices that probably have these features.

What is some risk of using P2P? Is that inherently bad protocol or a way to communicate?

Well, it's not an inherently bad thing. I mean P2P by itself is just a technology. But there are several different providers of these peer to peer solutions basically. So a device manufacturer will basically get a P2P product to put into their own device. And it's when those products are not designed sufficiently that these problems start to manifest. The problem really starts to stem when companies don't implement these features properly. So in the case of these peer to peer features, there's two companies out of China and they're Shenzhen Uni, EyeLink, P2P and CS2 Network. P2P, both of these developers created these peer to peer products. That have these flaws. So it's not that peer to peer itself is inherently bad, it's that these particular implementations really have serious problems that are now exposing the devices that use them.

If you do find an issue with the device, how do you report it? And if so, do you get in trouble if you report it or if you announce it?

Well, reporting it is actually a pretty tricky thing. And the reason for this is because a lot of these devices that have these problems are actually white labeled. And what that means is there are these manufacturers in China that actually sell their equipment to resellers. And it's those resellers that put their branding on the devices. I'm sure a lot of people have seen, especially on something like Amazon, the same product, but it's listed under tons and tons of different brand names. So yeah, these are white labeled. And finding the actual manufacturer, the party that's responsible for the software that's on there, can be a pretty tricky thing. So if I want to report something, I'll usually need to dig a little bit and try to find a point of contact. From that point, I mean, I've reached out to a lot of companies and in a lot of cases I won't even get a response back. But for those that do respond, usually they're pretty friendly. I haven't ever been threatened or anything like that. There's really no way to get in trouble for doing this. I mean, I'm just reporting a vulnerability in their software. There's really nothing wrong about that.

In order to, I guess exploit a device or know more about it, I guess you need to know about the different parts that make up that particular thing that you're trying to, I guess, investigate. So what the architecture of a P2P type of device, I guess.

Right. So the thing that basically orchestrates all of these devices is called a peer to peer server. This can sound contradictory to people who know traditional peer to peer, which is decentralized. Whereas all of these devices all have these central servers that they report back to. The reason that these peer to peer servers exist is A, so they can manage the devices and B, so they can actually kind of control how this direct connection happens, allow the device bypass firewalls and allow these direct connections to come in on the Internet. There's hundreds and hundreds of these peer to peer servers for different vendors. And all of their devices will report back to them with these servers. As I said, like each vendor will have dedicated ones for their devices. What I started to do was just kind of see how they were configured. Basically these devices have these serial numbers. These UIDs are called unique identifiers. And I started to kind of poke to see can I get a little bit more information about what UIDs these servers support. And over time, by doing that, I was actually able to find some flaws that allowed me to predict the UIDs of devices. And what that means is I could then find any device and then connect into it. And by doing that, I was able to connect to over 3 million devices.

How did you figure that out? Is it like a sniffer? Like how did you know what to look for, what the traffic was in order to exploit it?

Yeah, it was a packet sniffer. Basically. I had picked up a cheap camera on Amazon to watch over my cats when I first got them. And I started watching the network traffic just to kind of see what this thing was up to. And I started to notice that like literally every minute this thing was pinging back to some servers and had a pretty constant flow of these cryptic messages that kind of indicated that it was up to something. So I started reversing the protocol and began to figure out that, yeah, like, these messages are what control the ability to connect to this thing. And they're what handle all of the video going in and out of this. So it really caught my interest and over time I started to develop some scripts to kind of play with the protocol and see what I could and couldn't do. And I started to realize, like, you know, these things can be hijacked.

Yeah, Nice. I believe. Did you use a tool like Wireshark in your session? You mentioned something about P2P dissector.

A dissector? Yeah. I used Wireshark to do all of this and I ended up writing a dissector which will basically analyze all peer to peer traffic and tell you what the message means and what all the fields within the message mean.

Cool. So what was the message? Was it. I believe also you mentioned some UDP hole punching. Is that a method you use in order to get that data also?

Well, UDP hole punching is actually a way to establish a connection and the protocol is used to make this happen. But UDP hole punching is actually just a technique to establish a direct connection. Even if there's a router at play or a firewall or the like. It basically happens when two sides want to connect to each other. There can be like a specific sequence of packets that they send at each other that will allow them to establish a direct connection. And the peer to peer protocol is used to perform this. This isn't necessarily something specific to peer to peer. UDP hole punching is something that's been around for a long time, but the protocol has this feature that allows this to happen.

Nice. So you are a security professional. I mean, how many people would know how to do this?

Well, so here's the thing. This was one of my first major reverse engineering projects like this. This was one of the first things that I wanted to fully disassemble everything and figure out how every part of it worked. I think a lot of people, given the right amount of persistence, could find similar things because there was a lot of clues that were really helpful in getting me in the right direction. For example, within this software, they left a lot of the debug information in. So in some cases, as these packets were going by, like, you could see log messages happening. And that will give you like a really great idea of, oh, okay, so this is what this message probably means. So if someone is like just getting started with reverse engineering, if whatever they're working on has logging messages or any sort of hints like that, then it's really going to make it a lot easier to figure out what's going on. Likewise, there wasn't a whole lot of real or effective encryption happening here. So pretty much everything was just going over in completely, essentially clear text. So there wasn't really extra steps of figuring out, okay, how do I have to decrypt this or decode this? Now it's just right there in the raw format, right on the wire.

So could it also be used for. I'm just thinking in the news is TikTok. They're saying that some countries are using TikTok to spy on you. Is this something similar or is this something totally different? Or could this be used by another country to spy on another country somehow?

Well, I really don't think that these problems are deliberate. I don't think that these vulnerabilities were put into place on purpose just, you know, so one country can spy on another. I really think that these are the result of really just companies being pretty naive when it comes to security. They don't think that people are going to figure out how these cameras work, and they think that that obscurity is enough. But, you know, obviously it's not. With that being said, however, there is a risk of people spying on each other. And I don't even really mean other countries. I mean, literally any stranger on earth can potentially spy on another person. And the reason for this is that other people's cameras on this network are actually sometimes used to help other people make connections. And what that means is that those cameras are actually tunneling other people's video feeds through. I actually did a demo at DEF con of this where basically if you capture enough packets on your camera for a long enough period of time, there is a pretty good chance that you will actually pick up someone else's session. And, and what you can do is you can take that session data and you can stream it straight into a video player and potentially view someone else's feed. In a lot of cases, this video is not encrypted, so the person that has this device that the traffic is going through can see it without any effort whatsoever.

So when you said a developer could be naive, is it that they don't know how to do security testing or because they don't have a security expert in house? Like, what do you recommend for people to avoid these type of issues in general before it gets in the field like this?

Yeah, it's a pretty good mixture of both. I think a lot of them have a very basic understanding of security and so the measures that they take are very basic. For example, I have seen some devices that do use encryption. However, they use a hard coded key. So that kind of defeats the point. You know, if someone looks into the firmware or the client and gets that key, then any traffic that they obtain, they're going to be able to decrypt it. So that kind of demonstrates a pretty severe misunderstanding of how cryptography works. So for that reason, yeah, I really kind of urgently recommend that a lot of these companies either hire someone internally or hire like a consultancy service to look over these products, because an expert in these matters is going to be able to quickly see that that sort of use of cryptography is completely ineffective and point it out and fix it before it's deployed to millions of devices all over the place, you know?

Absolutely. And I believe also you mentioned what people could try as a man in the middle attack, and I guess because these things are headless, that maybe developers think like, oh, no one's going to know, there's nothing to look at. But it's the traffic going on, you know, you could put on Fiddler or something, just intercept that traffic. So what's the man in the middle attack you were talking about? Especially how it relates to P2P.

Yeah, so that kind of goes back to, as I mentioned, you can just buy one of these off of Amazon and then start snooping on people's traffic. It's kind of funny, in one of the marketing decks for one of these products, they actually had a slide to address this concern. And they said, if you're worried about security, there's no way to spy on people because there's no interface for this. But even though the software doesn't have a feature that lets you do that, that doesn't necessarily matter, because if you have physical access to this thing, you can see every single packet that's going through this device, and you can conduct a man in the middle attack very easily. You don't even need any skill to do it, really. I mean, you can just buy a device, you plug it in, start capturing traffic, and you know the session data is going to be right there. And this vulnerability is a huge concern because anytime anyone connects to their device, there is a chance that they may be connecting to a malicious situation like this. It's literally a risk that someone's taking every single time that they connect to their camera. There's no way to alert you to say, hey, this is going through someone else's device. There's no way to say that they may be capturing your traffic. So it's really a pretty scary thing.

So once someone is in your device, the P2P device, what exactly can they do though? Can they. They can get you a video? They can. Are they able to get into other things on your network once they're in your P2P device?

Yeah, absolutely. And this is kind of another scary thing about peer to peer in general. These devices are often just running Linux under the hood. They run busy box Linux, and if you're able to exploit the device in a way that gives you access to the underlying operating system like that, then your potential as an attacker is really unlimited. At that point, you can start scanning around the network to see what else you can hit, or you can install other attack tools on the camera. Like someone could potentially install malware or something like that right on the device. I did another demo at def Con to demonstrate this exact problem. At this very moment, there are millions of devices on the Internet that have a vulnerability where as long as you have the camera's id, you can actually fully hijack that device. You don't need a username or password. It doesn't matter how strong the camera's password is, as long as you have the id, you can exploit this device to get full control like this. One thing that people often tell me is they don't really care if someone can hack into their video. They say it's just watching their yard or their dog or something like that. But really Viewing video is just the tip of the iceberg when it comes to the hazards of these things. Because when your device has a vulnerability like that, that is an entry point into your network and if any anonymous person on the Internet can exploit it, then they can get on your network and it can really be the catalyst to a much larger scale attack.

Well, I guess what's also scary is you mentioned in your session that you can't just shut down the vendor because that's not going to solve it. Because they white label it and it's everywhere. Basically. You may not even know what device is using it. I guess.

Yeah, that's been a real challenge throughout this whole project and even after the fact, because I have people who, they hear about this and they're worried they may have a little device that they got on Amazon and they want to know if it's affected or not. And if they just have the brand name, it's really not enough because there can be several layers of device manufacturers or resellers where this is going through. So really the only way to find out is to kind of watch the traffic on your network and see if this is going to any peer to peer servers. That is one thing that makes this a little easier where if you see anything going to UDP port 32100, then that's like a pretty surefire signal that this is using these vulnerable libraries. But that's not an easy thing for a lot of people to do. So I'm really trying to figure out a good solution to make it so people can more easily identify these things on their network.

Absolutely. And I guess another thing that was disturbing is you said patching won't necessarily fix the issue either. It's not like they can just release a patch and everything will be resolved.

Yeah, there's a lot of challenges there. A lot of these issues are actually not fixable. They're really pretty deeply rooted architectural problems where if they fix these problems now, it's going to break every unpatched device on the network. So frankly that's not going to happen. They can certainly move forward and start releasing new devices that are patched. But I mean, a lot of people, they buy a camera and then it's going to stay up for years. I've even seen some devices with firmware versions going back like, you know, five or six years. People just kind of plug it in and forget about it. Another problem comes down to the fact that these things are resold. So if the original device manufacturer actually comes out with a patch that's still going to need to propagate down through all of the resellers. And you as the end user, all you're going to know how to do is go to your reseller's site and if they don't offer the latest firmware download, then you're going to think that you're up to date and you may in fact remain vulnerable.

So at the end of your session you also gave a slide on reversing tips. So can you share a little bit those reversing tips?

Yeah, I think my favorite tip to give people is if you're doing reversing or you're analyzing something and you happen to stumble across like a magic string or something that looks a little funny, put that into GitHub. And the reason I do this is because it is absolutely mind boggling how many people, without realizing it, will push sensitive source code or documentation or anything like that up to GitHub. And of course at that point it becomes public. So magic strings or keys or identifiers like that are a real great way to hone in on those potentially sensitive repositories. And by doing that, I have found very valuable things such as firmware, source code and documentation just completely exposed. So as a reverse engineer, that's going to make your life a lot easier. You know, that's going to give you potentially function names and all sorts of stuff that you no longer have to figure out manually.

Nice. So, Paul, I'll be honest. I have a small farm. We just get some alpacas, goats and chickens and everyone's like, oh, you should install cameras. And I saw your session. I'm like, there's no way I'm installing a camera now. I mean, that is crazy. It's crazy what you could do. I'm amazed what you were able to find. So really awesome stuff.

Thank you. I do appreciate it.

So Paul, are there any resources people can go to to learn more about P2P?

Yeah, I actually have a small website up that people can go to that has some more information if you go to Hacked camera. Yeah, I have sort of an FAQ queue that people can look at that answers quite a few questions. There is a link to the DEF CON video. There is actually an interactive map that shows all over the world where these things are distributed. And there's a couple other resources on there as well that people can potentially find useful.

Awesome. Okay, Paul, before we go, is there one piece of actual advice you can give to someone to help them with their security testing efforts and what's the best way to find or contact you.

Yeah, I think if you're developing a product and you're not really an expert on security, or you just have a basic understanding, it can really go a long way to consult with someone who is an expert because they're going to be able to provide a lot of insight into potential flaws and help you resolve those early on in the development process. And obviously it's a lot better to do that pre release before something's out in the wild. And for those looking to get in touch with me, my contact information is right up on the website Hacked Camera.

Thank you Paul for your security testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com S30 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. And if you want to learn more about security, make sure you buy a ticket to the upcoming Secure Guild Online Conference, an online conference dedicated 100% to helping you succeed with all your security testing efforts. To register, all you need to do is head on over to guild conferences.com and click on the Register now button under the Secure Guild section. So that's it for this episode of the Test Guild Security Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack security testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.