What Could Possibly Go Wrong?
I once contracted out a firm to perform a pentest to satisfy our annual pentest for our PCI-DSS requirements. We went through the phases of scoping out the testing and defining the rules of engagement (ROE) agreed upon for the annual examination. With all the pentest pleasantries aside, we gave the ‘OK’ to start the exercise. The contractor had already spent time performing reconnaissance on our organization to seed his scanning of targets. I remember the day well. The contractor started with what was thought to be a benign, non-invasive NMAP scan initiated during normal operations. First mistake. Mere moments after the scan commenced, three prodding, stern-faced network team members led by the director of network operations could be seen heading towards my cubicle. The contractor’s scans had brought down the entire network of a multi-billion dollar business. I channeled the rage of the director of netops directed at me onto the contractor – and ordered that the pentest halt immediately and sent him packing for the day. The rest of the day was spent recovering from the outage and trying to understand what went wrong. One thing was clear. A basic scan that any curious staff member or Internet opportunist could have initiated, caused an unintended consequence, and revealed a critical network flaw.
Fast forward – some years later, while standing in a long line for a session at Def Con, I happened to be chatting about THAT penetration tests and pentests in general with a fellow ‘Def Coner’ with whom I had made an acquaintance. I proudly recounted what went wrong that fateful morning and, with a bit of arrogance, boasted how I sent the pentester (now a dear friend) home that day. My new Def Con mate responded, “You should’ve sent the network team home. Your network team and infrastructure showed that it could not withstand an attack.” Wow. I didn’t see that response coming. My sole retort was “touché.” In a single moment, he knocked me off my high horse, transformed my perspective, and shattered long-held assumptions I had around the subject of pentesting.
Why Pentests Fail?
When I reflect back on the start day of that annual pentest, I wonder about what I could have done differently in preparing the organization for the exercise. I arrived at the company with past experiences from a respected research firm that sold expert pentesting services. Equipped with that background, I prided myself on the fact that I knew what to look for in a pentesting vendor. More than that, I was surrounded by an “elite” team of hackers and analysts that I was fortunate to hire (truth be told they found me) who were daily assessing the network and discovering zero days in products (true story). They would not have brought the network down had they been testing.
Should I have read the ROE more thoroughly? Restricted scans to off-hours? Researched and remediated what knocked over the network? To this day, I do not know what knocked the network over. What could I have done better? Chris Nickerson and I have been discussing responses to this question and other aspects of testing at length. In the sections that follow, I draw from Chris’ invaluable insights on pentesting and the penetration testing execution standard (PTES) to ponder over how I/we approached the exercise.
Specific considerations could have changed the Fates and reduced ‘fail’ during our annual testing. Here are some notable restrictions we should have spent more time discussing.
- Restrict the hours or length of testing – as a result of the downed network, I restricted the hours for the scanning phase to ‘end of business day’ out of fear of further impact. This decision, in turn, impacted the value of testing, not giving us a real perspective of our risks. Malicious actors tend not to operate on our schedules.
- Restrict testing to non-production networks or external assessments – We got a pass here, having scanned the internal and external production networks. Consequently, we discovered a resilience issue in our infrastructure.
- Improper Scoping – Looking back on the assessment, I am not sure that we correctly scoped the test or thought about follow-on testing, given the narrow focus on the PCI scoped environment.
Lack of focus on restrictions is just one category in which we could have done better in preparation and in establishing accountability. Lack of organizational ownership of the pentesting exercise hampered our engagement. It is not uncommon in our collective experience to see businesses regardless of vertical and size, hedge on awareness, negotiation, and management of testing. The following items tend to stand out most prominently.
- Lack of focus on BUSINESS risk and increased focus on technical issues – Our issue was solely based on the business risk of not satisfying our yearly compliance regulation. Compliance objectives often skew underlying risks. Businesses, in the effort to satisfy regulators, often underscope testing and fail to assess specific business risks. Similarly, pentesting should not cast solely as a technical exercise detached from the business context.
- Lack of follow-up – Our team never followed up on the cause of the outage and the resilience gap that was exposed as a result of the invasive scan. We did, however, manage to follow through on the remediation of medium to high-level findings validated through retesting.
- Lack of incident response ownership – The incident response component was a non-starter in our exercise. In fact, the incident response team was not part of the testing, so one could question whether we had an actual penetration test.
- Lack of collaboration – See above. Very little collaboration took place in the exercise other than the advanced team’s monitoring of the contractor’s attack activity (we were literally on the assessor’s box). No incident response, no legal resource, executive management, nor other relevant stakeholders were involved in testing.
Awareness and organization buy-in are critical aspects of a successful penetration testing. Chris is adamant that we do not do enough from a vendor perspective to help organizations prepare for testing, let alone accepting the results of engagements. As an industry, we can do better moving consumers of pentesting services from the vantage point of ‘victim’ to one of ‘advantage’ when working through findings. As a consumer of these services, I often felt at a disadvantage around remediation of findings. The annual engagement was siloed from the outset. I was unhappy with the lukewarm participation of management and the level of effort from supporting teams.
Chris talks about this phenomenon as commonplace in an organization because of “information asymmetry.” In preparation for the annual testing, I should have done a better job level setting expectations with stakeholders and management, especially given our culture with a low appetite for “truth.” In these situations, Chris recommends that organizations stand up “ask-me-anything #AMA” sessions with testing vendors. He also feels strongly that testing vendors invest time empowering stakeholders with the necessary talk-tracks to “train the corporation for testing.” Organizations tend to fail engagements when not armed with proper education. Awareness in the following areas can optimize an organization’s holistic focus in preparation for pentesting.
- Pre-prep for testing – Some organizations are guilty of stalling testing until there is a feeling of adequate testing, e.g., patching before testing. I have been a part of engagements in which the organization refused to start until the vulnerabilities were resolved. Needless to say, that engagement never happened.
- Vulnerability Fatigue – When organizations are already mired in vulnerability management, chances of successful engagements are significantly reduced.
- Disallow Exploitation of Systems – Organizations fear the risk of exploitation when there is insufficient effort preparation. I was trained well not to rely on FUD in convincing managers, but lack of insight into our business risk and threat landscape adversely impacted how we approached testing.
- Only allowing directed attacks (No social engineering/phishing) – In the case of my annual engagement, we did not consider any other testing in the scope. Social and physical testing were deemed out of scope. Transparently, compliance-related pentests were the only ones contracted in any given year.
Lessons in Fail and the Path to Wellness
Although not an exhaustive list, the factors enumerated above, when addressed, can lead to more optimal testing outcomes. Performing due diligence, and due care concerning the items discussed, empower both the testing company and consumer of the service. Characteristics of a company operating from an informed point of advantage take more care:
- Vendor vetting – careful consideration around the selection of a vendor and its practitioners and level setting expectations
- Achieving Repeatability in testing – The desire for consistent testing scope and process, so the results aren’t skewed by the way the test was done or the confines of the individual engagement (continuous defensive improvement through testing)
- Leveraging Feedback (e.g., observe-orient-decide-act (OODA)) – reliance on feedback to drive operational maturity (e.g., blue team capability), mitigate risk, and improve upon ROI
Call to Action
I could have avoided several points of failure, including the ire of upper management during that annual testing years ago. We ultimately ended up with a passing PCI-DSS report and satisfied regulators. What we did not receive was a pentest. We have collectively experienced and shared stories, just like mine, with peers and clientele over the years. By sharing our experiences, we aim to equip others for success and cast pentesting as a ‘necessary good’ to secure organizations across all sectors. Doing so affords us the ability to adapt our methodology, flexibility, and scoping to meet business requirements and achieve security outcomes and better products for our clients. Lares continues to improve on methodologies and service delivery through shared, continuous interaction with our customers, peers, and industry leaders. Our commitment to sharing reflects that two-way accountability and our resolve to see fewer ‘fateful’ days where pentests are concerned. Cheers.
I would love to hear from you. What are your thoughts? What could I have done better on that fateful morning? What can we collectively do to improve awareness? Let’s compare notes. Feel free to reach out to me at firstname.lastname@example.org or schedule time to chat.
This is the first in a series of blogs where we will hash out perspectives on pentesting. We will determine how my modalities as a managed service provider, consumer, seller, and tester jibe with Chris’ subject matter expertise and life-long musings on #AllThingsTesting. We hope to inform each others’ views and spark discussion on the subject of penetration testing.
Mark Arnold has a 15+ cybersecurity career, serving 8 of those years in leadership roles. As a transformational leader, Mark has built security teams and programs, authored maturity model blueprints to optimize risk management processes, and implemented security domain practices at large enterprises and service providers. Mark’s areas of interest include cloud security, threat intelligence, and vulnerability research, nation-state attack methods and related activities (e.g. information operations and disinformation campaigns) and their collective impact on nations and society. Mark recently completed an executive education cohort on the intersection of cybersecurity and technology at Harvard’s Kennedy School.