Casey Cammilleri critiques the current state of penetration testing, calling traditional approaches "legacy" point-in-time tests that produce annual PDF reports and lengthy remediation cycles. He outlines limitations of crowdsourced models (uncertain tester identity, methods, and coverage) and fully automated solutions (cannot find business-logic and some tactical flaws), and argues for a hybrid approach combining automation with human-driven testing—an approach he is working to build.
Chapters
Yeah. The current state of penetration testing is still very much what we call legacy based, it's point in time pen testing. You would essentially hire a firm which equates to a penetration tester, human, that comes in for a week or two, however long, tests and tries to run through all the different attacker techniques and tactics. In this condensed period of time, they deliver you a PDF report. And then now I'm the business owner and I gotta fix all these things, right? Like, how am I going to fix all these things? So then you hand it off to an IT team and they go and attempt to remediate all these things. And you go back to the firm and you say, hey, I think we fixed all these things. Can you retest all this stuff? And then traditionally that's just done once a year in that recurring fashion of once a year. And that timeframe is just way too long. So working with that deliverable, that PDF that comes once a year, getting that tremendous amount of work that you have to do with your security teams to remediate and fix all those vulnerabilities is just not the best current state of testing. So there's a couple things that have changed since then. We have a crowdsourced model where it's, you know, hey, here's a bunch of bodies, they're all doing their own type of methodology and they're testing. But you can sign up on a platform, we can test whenever you want, but it's going to be just random people. The challenge with that is you couldn't determine what was tested, who tested it, when, and auditors kind of find that difficult as well as, you know, leadership finds that difficult to say, do we know if this was tested and by whom and under what methodology? So that makes, that makes it difficult. Then you have automated players that say, we've fully 100% automated penetration testing. And we all know that there's tactics and techniques that that cannot be 100% automated. There's vulnerabilities and business logic flaws that is extremely hard to automate. And so if you do 100% automated pen test, you're leaving something on the table which could be gaps that equate to breaches. So the current state doesn't really have a really strong hybrid penetration test where we use automation and we use human driven testing together to get to the best state possible. And that's what I'm working on.
