Team

Published August 1, 2023

I'll start this article with a bit of background around my experiences. Why I reached this conclusion, and why I believe that CISOs might needlessly have their hands tied by their compliance departments; and a possible solution.

I started my offensive security career in a boutique that was largely focused on penetration testing while also having some exceptional testers at the time, who were performing complex Adversary Simulation exercises around the birth of CBEST framework on some of the first CBESTs.

At the time, the business was still small and hired before the new wave of cyber security becoming headline news, and thus newly attracting those into the field that were seeking a career, rather than because they loved to problem solve. And so that firm was full of testers who had been hacking since their teens, numerous degrees in networking, computer science and a real passion for the craft. Many of those have gone on to become incredible code reviewers, Red Teamers, Head of Securities and more, at FTSE 100, FAANG and Fortune 500 companies.

I left that firm to move to KPMG's Cyber Defence Services practice, and at the time there was a myriad of senior testers, incident responders, commercial product assurance specialists and a leadership that had more years in cyber individually than I had been alive at that point, with M.O.D., Government and business leaderships roles throughout their career.

It was at KPMG that I really learnt how to Red Team, and was incredibly fortunate to work with some absolutely incredible professionals, with one in particular that refined my mindset and thirst for knowledge. As a team, we did all manner of Red Teams: CBEST pilots, TBESTs, Saudi Red Teams, six month Advanced Intrusion Tests, internal scenario based Red Teams where we'd come back to the same client over and over until their cyber maturity was enhanced by the extended KPMG team of architects, IAM specialists and many more.

I absolutely loved working for this team but in 2018 my life took a turn and due to numerous personal tragedies, I left the team and ventured out on my own moving back to Europe in 2019.

It was during this two year period that I was introduced to the concept of contracting in cyber security, running your own business and began to have my horizons expanded. I met a wide range of professionals who didn't fit the mold of large or boutique consulting firms. They were either those that had done both of the above, and had left to live a healthier work life balance, utilising their existing relationships to maintain a steady stream of specialist contracting work, or were multiple-degreed, PhD-level technical machines who'd been working in offensive security testing for at least a decade but in many cases more decades. I probably contracted to around 7 different businesses in this time - some in Italy, UK, the Middle East and also sold some work to end clients in Luxembourg and elsewhere. Working on one of these contracts through a company in Muscat, we were gathered in to do some work as subcontractors for the Abu Dhabi government through a UAE cyber giant, and it was during this time I realised that there are many offensive security folks out there, who are exceptionally talented, highly experienced, yet didn't fit the typical career path that you get to choose from: internal pen testers, big consulting firm pentester, or boutique pentester.

I pivoted the business at the time and began to assemble project teams under the brand of the company at the time, and at one point - before Covid wiped us out - had a large list of leads and proposals and contract negotiations midway with some huge entities because the level of testing capability that these testers represented together as a project team was on par with some of the leading security consultancies. Not that many security consultancies have 20 hands-on specialist consultants, all with 10-20 years experience, highly specialised in their own area: ICS, appsec, Red Team, mobile etc.

Needless to say my eyes were opened and despite being beset by tragedy again in my personal life and losing a lot, I was thankfully given another opportunity to work at KPMG as a Manager. While this time at KPMG my work revolved around code review work, a couple of CBESTs, a multi month Saudi Red Team, from a professional point of view - purely offensive security testing, I was not inspired in the same way as I had been before Covid and had a passion I wanted to pursue. The reality is that in big consulting firms there are some fairly rigid processes that are hard to shape and mould, and the reality is that YOU get shaped and molded by the machine. In my view by the time someone is a Manager, Senior Manager or Director in the Big4 or similar sized organisation - maintaining technical skills is not something they either have time or motivation to do, and by the time ML came onto the scene in a big way around 2020, EDR, XDR, MDR and associated defensive measure's complexity increased significantly, hybrid AD/Cloud environments became the norm, and for a technical manager to stay technically hands-on across all those domains, alongside the demands of the partners, clients, (wives and children), inflation, Covid; I don't know many who have stayed the course.

With this shift in investment into cyber in a big way becoming a major board issue, boutiques had to get much more technical, and the largest consulting firms I believe stopped focusing on technical hands-on innovation in the same way, where it had been passionate hackers loving their craft. Security testing is no longer niche, and represents a vast and growing market. Where the money is the driver, you will eventually find that those who can put money on the table thrive, and yet those who can find high complexity vulnerabilities in client environments are not the same level of importance.

With the growth of the industry, salaries also increased and so boutiques began to find it harder to attract testers as they wanted more money and more interesting/prestige work, more technology firms began developing Red Teams as an enabler to sell their technology, the CREST list of companies trebled in size if not more, countries like Saudi Arabia went from low cyber maturity to the second highest behind the US according to one study in a few short years, and the industry in some ways overlooked a huge cadre of testing capability that was masked in contractors, freelance testers, and small outfits - many small outfits have been performing security testing for 20+ years but whether because of lack of desire, lack of resource or lack of sales capability, have enormous testing expertise including in some really esoteric areas of technology - but the firms winning the large frameworks, having the ability to maintain accreditations, were now the largest businesses who had either built teams or acquired them.

I believe the current state of thinking in large firms is that artificial intelligence will take over the work of testers, which I believe is not going to happen for quite some time if ever, and so in terms of their strategic thinking they are unlikely to change their models, and thus attract the highest calibre of hands-on technical testing resource. The bug bounty model is attractive, but the reality is that CBEST, FEER, iCAST, TIBER, DORA and many other regulations out there cannot be met by these crowd sourced testers, and so what options do CISOs have?

1. The Big4, NCC other testing giants

No one ever got fired for using the Big4 is a phrase that is common in not just security testing, but all industries, but I genuinely believe that soon these firms will be unable to attract and retain passionate hackers. Because whatever you believe about security testing, the finding of issues is found by the passionate hackers, not the managers. The Big4 et al will continue to win business, as will the other large consulting shops, but from my personal discussions with testers and right up to the top of the those teams, given an ideal world without high cortisol around bills, mortgages and children many would leave to become contractors. Indeed as I look through my network I see more and more exceptional testers forming small security testing companies, preferring to take their wares for themselves.

2. Personal Relationships

There are plenty of CISOs out there that buy because of relationships irrespective of the capabilities of the companies their relationships run. We are human after all and people buy from people - of course the more mature an organisation the more testing companies are involved, and this is definitely not the case, given again third parties will often dictate which firms can even bid on these projects due the requirements to maintain accreditations (BoE, Ofcom etc). What is interesting is that many of these firms achieved their accreditations by the work of older consultants who've set up their own shops or doing other work, but the current testing resource is not likely to meet the level of expertise that existed in the past.

3. Take a chance

On one of these small boutiques that produces exceptional research, but, that may or may not meet the onboarding requirements that their compliance departments enforce.

Well now return back to the title of the article. Why can project teams outperform large scale and boutique consulting firms?

It's a fairly simple explanation, and why bug bounty firms have enjoyed so much success. With a larger and more specialised talent pool, not constrained by full-time hiring processes and candidate attraction, project teams can actually enforce specialist knowledge to meet specialist requirements - at scale.

Where a large firm has, let's say, forty testers, that does seem like it can handle most demands thrown at it. But the reality is that profit margins, employee churn, and scheduling mean that an ICS test that comes through might be performed by a tester who has done some cautious NMAP scanning of a PLC before, whereas in the world of contractors this would be performed by an ICS Specialist with experience doing hundreds of complex technical/paper OT assessments around the world, who maybe has developed specific tooling and more. Same goes for mobile testing - where you might have a senior tester in a boutique or consultancy have experience of doing five or six mobile tests in-between their application, infra, and other tests they've done, in the contractor world you will have a specialist who is an authority on Android, or iOS, having specialised for the last ten years in this domain. The same can be applied for many areas. Of course these large consultancies, and NCC have some phenomenal minds & testers, but those are not the ones performing the bulk of testing, day in day out, that generates the revenue for these businesses.

Thus I would ask a CISO to consider: would you rather have a firm that is excellent at selling their services, incredibly impressive leadership but the work is being done mid level (or even junior) generalists, or project teams where every requirement is met with resource that are specialists in those specialist areas?

Machine Learning is yet unable to the find the most complex bugs, even most bugs, end of development stage pen testing is not enough, and enterprises that don't provide their testing partners with adequate context, source code, and more (whether the fault of the testing company not asking, or the enterprise being unwilling / unable), or using bug bounties to scour your DMZ are not going to solve the problems with identifying security vulnerabilities on in your networks. Of course many huge enterprises have numerous testing companies onboarded, and entire teams to manage the vulnerability identification programme but many of these consultancies on these frameworks do match requirements with true specialism, test by test.

I believe assembling bespoke project teams to meet the requirements of the work, rather than sales (and a static team) to suit the client, is a more effective way to do security testing. I also believe that it is scaleable, as bug bounty has shown, and until Machine Learning can do it better than humans, an entire cadre of the freelance, the contractor, the micro-business highly experienced & specialised testers are being ignored, or merely being used as subcontractors - in a highly dispensible manner - to the those mentioned while molded into their process when a need arises. I don't believe this meets the demands of increasing complexity in IT systems, increasing threat actor sophistication, and an uncertain political climate where individual vulnerabilities can have profound economic, political and business consequences.

Acronym Reference

CBEST	Threat Intelligence driven regulatory led (by the Bank of England) security testing modelling sophisticated threat actors
FEER	Saudi Arabian equivalent to CBEST
ICAST	Hong Kong equivalent to CBEST
TIBER	European equivalent to CBEST
DORA	Digital Operational Resiliency Act - sets uniform requirements for security of network and information systems for the financial sector
PLC	Programmable Logic Controller
ICS	Industrial Control Systems
NCC	NCC Group (largest cyber security consultancy globally)
EDR	Endpoint Detection & Response
XDR	Extended Detection and Response
MDR	Managed Detection and Response

Read the full article on LinkedIn →