At the beginning of the year, I was approached by the infamous Marcus J Carey (@marcusjcarey) to help create content for a Red Team focused book. The format was something new to me but quite fun to contribute to. Marcus had a series of questions asked to a group of all star red teamers — such as Chris Gates, Rob Fuller, Marco Figueroa, Carl Vincent, and so many more — as a way to share information on many topics the industry tends to struggle with. I have to say, at first I was unsure how to respond without writing an entire book for each question 🙂
As I sat with each of the questions, I spent time trying to connect with their intent and what potential readers could gain from my answers. Somewhere between “sharing my experience” and “ranting” with a little bit of “precautions” seems to be where I landed. It was a fun exercise. It made me think and truly reflect. As a teaser and a conversation starter, I am putting my answers in full below. If you like where it is going and would like to see other masters of their craft give their thoughts a spin, I encourage you to purchase the book. I know a lot of us will glance over a pdf leak, or find value in searching an index, but I’d love to chat with more people who have read the entire book and start discussing some of these topics in more detail. So, for starters, here are the links to purchase the book.
- Wiley – Tribe of Hackers: Red Team Edition ($25 at time of post)
- Amazon Smile – Tribe of Hackers: Red Team Edition ($22.50 at time of post and supports the ASPCA)
- How did you get your start on a red team?
In my early days at Sprint, we had a task force put together to show the actual impact of vulnerabilities identified. Over time, the scope grew and began to include attacks from the Physical, Social and Electronic realms. Although this team was called a “Tiger Team” it was really my first professional exposure to mixed discipline attacking over multiple execution surfaces.
- What is the best way to get a red team job?
Earn it. The main reason to have a red team is to be able to simulate a wide range of adversarial tactics over the entire attack surface. The operator needs to have a broad background in problem-solving and quick thinking. Knowledge of one discipline can be tactical but is not sufficient when tasked with simulating many different adversarial models. The operator must be able to mimic not only tactics but thought patterns. With that in mind, the best way to get a red team job is to practice everything BESIDES red teaming.
- How can someone gain red team skills without getting in trouble with the law?
Practice in a controlled environment. This may come in many forms from picking your own padlocks to Creating pen-testing labs. The precursor to better at any craft is to understand it from as many viewpoints as possible. There are boatloads of classes at conferences and stand-alone. My advice would be to take a solid mix of classes in pen testing, social engineering, and physical security. On, and don’t forget to have mastery of your audience’s language. We can be the best red teamers in the world and will not get a single stitch of credit unless we can effectively and appropriately communicate our actions.
- Why can’t we agree on what a red team is?
It IS a fairly standard term in the military. Since the invention of the German 19th-century kriegspiel (wargame), it has had a few names but the sentiment remained the same. The Army said it well in (TR71-20 TRADOC) “A structured, iterative process executed by trained, educated and practiced team members that provides commanders an independent capability to continuously challenge plans, operations, concepts, organizations and capabilities in the context of the operational environment and from our partners’ and adversaries’ perspectives.” So, from that standpoint, the world has had a great definition of it. Why can’t “WE” agree? Well, we will have to ask who “we” is. If you mean “People in Cybersecurity” I would quickly point to the sales and marketing departments. Since such a small talent pool existed when the terms started to gain popularity they wanted to cash in on the interest. Just like “penetration testing” the term was watered down and manipulated to mean “whatever engineering talent I have to sell you.” Bruce Schneier captured this so elegantly while comparing the Information Security market to American economist George Akerlof’s paper “The Market for ‘Lemons.” A body of work which looks at markets where the seller knows a lot more about the product than the buyer. That’s partly the same issue we have in “red team” definitions. The customer doesn’t know what to expect and the sales team makes up a slick sales sheet that says the words they are looking for and the buzzwords that make them “feel” like they are doing it right.
- What is one thing the rest of information security doesn’t understand about being on a red team? What is the most toxic falsehood you have heard related to red, blue, or purple teams?
Red Teamers or Offense thinks they are “better” than any other teams. Look, I am all for having confidence in yourself and even more so… confidence in your team. BUT, there is no place or need for some pseudo-hierarchy. Members of this team should be the MOST similar. The should all be committed, passionate and mission-driven. They should be using their talents to drive progress. They should be the instrument of change no matter how hard it is or how long it takes. This isn’t a field for people that don’t want to make a difference. We’re all part of the same crayon box and when used in concert, can create something much bigger than our individual contributions.
- What is the appropriate point in the maturation of a security program to introduce a formal red team into an organization?
Coach Bear Bryant said it “Offense wins games…defense wins championships.” The key lesson here is that you NEED to have a defense to win. Having an active state Red Team requires an active state Blue Team. That’s not to say that one cannot get some value from “sparring” with the red side, but to realize the full potential of the organization’s ability to progress and improve through the challenge of red teaming, they must have a team dedicated to that constant improvement. This may be a “hunt team” or it may be a group of dedicated defenders charges with the proactive improvement of the environment. Either way, the commitment to challenging status quo is being able to act on it. The teams that are ready for a red team are the ones who are ready to put in the work to own the findings, measure the results and drive the organization forward.
- What has been effective at explaining the value of red teaming to a reluctant or non-technical client, or even to your own organization?
The “sparring” analogy is one that I usually find most people understanding. Whether it’s the Mike Tyson style “Everyone has a plan till they get punched in the face” or the idea that in fight club you join to see what it really feels like to be in a fight or something even deeper. The entire sentiment of Red Teaming is to challenge the status quo. Not through some type of theoretical or mathematical model, but to learn and evolve through experience. It makes me think of those silly t-shirts that say “There’s no patch for human stupidity.” They are totally wrong! The patches that we get are called “experience” and the more experience we get the more prepared we are. This applies back to the sparring partner analogy. If you are a beginner, you need to always be punching up above your skill level. Not too far, because you need to build your confidence. Let’s face it, if you jumped in the ring day 1 with Iron Mike… you’d likely hang up the gloves forever. As you progress, you need to move from sparring in your class to the next level above it. Each time you turn the dial to make the sparring partner a bigger challenge, you will build confidence, experience, and skill. Eventually, when you get to the pro level its no longer about fighting someone better its about someone that has a different style than you. You may be the baddest thing to ever hit the ring, but the variables you experience in the challenger or adversary, are the things that can catch you off guard. At this point in the game, you need a sparring partner that can act “just like your opponent.” It will prepare you for the inevitable fight ahead and give you trust in your skills. It will also point out opportunities in your game plan that may have never been tested. Not everyone is ready for a title fight, but the ones who are trained for it with every breath.
- What is the least bang-for-your-buck security control that you see implemented?
Training. Investing in your people will always beat your tools. It is a common misconception that you can buy your way secure. As you can see from common statistics, that idea isn’t working so well. There is nothing on this planet that can beat a dedicated and educated member of the team. The thing that most people don’t consider, is that security tools constantly have vulnerabilities just like everything else. So, at the end of the day, every time you buy a new tool your attack surfaces INCREASES not decreases. More things to attack, more openings, more everything. Now, with a highly trained team, you would be able to know that and engineer around the blind spots created. Without that training and investment in your team, you are increasing the likelihood that the very thing you bought to protect you will be the thing that gets you owned.
- Have you ever recommended not doing a red team and offered something else more suitable, even when a customer asked for one? What have you recommended?
Most people are not ready for a red team engagement. It is an extremely deep look into an environments real-world effectiveness to defend itself in multiple different modalities. Often times, the companies asking for exercises like this do not have the technology/staffing in place to even make progress on the findings. Back to the sparring analogy, they are an amateur trying to spar with Tyson in his prime. It’s counterproductive to the program, and something that we regularly guide companies to testing types that they are more prepared for. This may be pen testing or even a broader defensive controls analysis to determine what they are actually equipped to handle. It often doesn’t end well and they go with another vendor willing to sell them the “term” they are requesting. I still feel firm in the decision to only do what’s right for the client even if they don’t see it at the time. Now, has it happened where we did a red team job on someone that was woefully underprepared? Absolutely! How did it end? Just like we told them. A string of attacks that you aren’t staffed or engineered to deal with. Surprisingly enough about 10% of them recover from the shock and awe and actually make a massive change. The others… just don’t call back.
- What’s the most important or easiest-to-implement control that can prevent you from compromising a system or network? For example, how can we help small and medium businesses who have a smaller budget and staff?
2 Factor all the things. Passwords are the Achilles heel of modern access. It’s the most common way I see access and lateral movement happen. There are plenty of 2fa solutions out there that are free to low cost and provide a massive leap in security above the single password solution.
- Why do you feel it is critical to stay within the rules of engagement?
The rules of engagement aren’t just a contract but they are your word. The bond of trust that is required to allow a stranger to see the darkest secrets of your business is one that requires a far more emotional connection than words on paper. It requires honor, purpose, intent and overall trust. The foundations of that trust are memorialized in the rules of engagement. It’s a catalog of your mission and your commitment to the progress of the organization.
- Tell about a time you were busted on a penetration test (physical, network, social engineering, etc.). How did you handle it?
Oh boy, I have had some weird ones. One time I was working at this large health care facility. It was about 2am, and we had made our way into the IT operations building through a chat with the cleaning staff. As time went on, the cleaning staff left and we stayed on the network. After collecting the artifacts of access, the client requested we went exploring. We were looking for some final flags of raw patient information, data, persistent physical access, and additional bypasses. I found myself in the basement looking at a few boxes that looked JUST LIKE a key box. I swear. Anyway, I picked the box open and the alarm goes raging. I knew right away that this was the alarm central unit and the tamper switch had been triggered. I heard a faint yell from my teammate upstairs “Are you KIDDING ME? Nickerson what are you doing down there?” I ran up the stairs, beet red and totally embarrassed. We both convened at the alarm keypad to see if we could find a way to disarm it. We checked everything close to the panel and found nothing. As a last-ditch effort, we popped the external housing and on the inside of the casing was the number “4757.” Coincidentally that was also the number of the building we were in. We punched in the code and the cacophony of sirens immediately stopped. “Whew!!!! We good…” I laughed. I was met with the “I’m not impressed” look and we started to collect our things. Not more than 3 mins later, the road up to the facility was drowning in blue and red. “Well, here we go.” I said as I started to reluctantly unfold my engagement letter. We immediately went outside and waiting for the police to show up. Like any pride stung red teamer I thought to myself, maybe if I go first, they will think I am supposed to be here. So as soon as they were in the parking lot, I waved them to our side of the building. “Officer!!! I’m so sorry. I have been here all night working on this server being down and the cleaning lady locked me in. I just came out here for a smoke and the dang alarm started going off. I tried to turn it off but the call already went out.” He let out a faint giggle and grabbed a flashlight to shine in my face. After a few mins of questions, he actually bought the story. I helped him do a sweep of the area, looking for anything suspicious and he reassured me that “this kind of thing happens all the time.” Since I was worried about having to pay a fine for the emergency response call. They eventually left, we packed up and alarmed the building in good conscious that if anyone else broke in that night they were surely going to jail.
- What is the biggest ethical quandary you experienced while on an assigned objective?
Red Teaming is a really strange duality. On one hand, you are being paid to conduct a test in order to find a potential weakness and measure the successes of a program. On the other hand, you are doing all the things your mom told you to never do AND you are doing for MONEY?!? It’s a really strange place to be in. Also, there is a palpable intensity to it. You are breaking into a building. You are doing this highly criminal thing and no matter how you justify it, you feel those butterflies. But unlike a criminal, the intensity doesn’t stop there. You see, when a criminal gets caught, they know that’s the potential of the game they signed up for. When a red teamer gets caught, you no longer have the feeling of the criminal act or the intensity of going from hidden to exposed. You immediately switch to an entirely different fear. You go from “OMG, I don’t want to get caught.” to “OMG. I’m getting caught, I’m a phony. I’m terrible at my job. Everyone is going to know. My career is over. My peers are going to ridicule me. The client is going to think I suck. And on and on…” It’s vicious. There are other ethical aspects of the job that we come in contact with that have some effect or give me pause, but there is NOTHING like listening to my mother’s voice in my head, repeating the thing that I am in the process of doing is just wrong.
- Describe the “team” aspect of red teaming. How do you work together to get the job done, including documentation, reporting, and working with the blue team afterwards?
The team is everything. None of this can do it on our own. Even if it is a one-person job, the entirety of my team is there to support the operator every step of the way. You need a pump-up… we got you. Need someone on the cameras at 4 am… call, we will get on. Need someone in your ear to walk you through the office you should break into to try and get part of the safe codes? We will be there, digging through mailboxes and shared folders trying to help you narrow the 1000 offices down to the ones that likely have the code. This doesn’t stop at the operation. Afterwards, the same rules apply to our blue team counterparts. You need something, we are there. We are an extension of your team. There to make sure that when you tackle the next big issue, you know that we have your back. Findings, remediation, brainstorming sessions or just a late-night call to vent. We are just as invested in the program growing as they are. Together we make it happen.
- What is your approach to debriefing and supporting blue teams after an operation is completed?
We start the process as early as possible. If we are allowed, we let members of the blue team “ride-along” with us to gain both sides of the experience. I can remember some pretty famous faces in the InfoSec industry coming on a gig with us as a blue-teamer. Some of them used it as inspiration to create revolutionary defense programs, and some of them used it as a launchpad to move the offensive industry further than we could have ever expected. The one thing that was the same in every engagement was that the more we engaged, the more we all learned.
- If you were to switch to blue team, what would be your first step to better defend against attacks, and if it isn’t commonly done already, why do you suppose that’s the case?
Basic defensive inventory. Most companies have been bullied by compliance to buy tools and technologies they don’t even use. They have had Gartner in their ears for decades telling them WHAT to buy but not WHY. Situational awareness and “home field advantage” are the biggest assets to the defensive team. The best teams in the world use those 2 things to move mountains and respond to attacks in real-time, effectively.
- A lot of people complain about writing testing reports and a lot of customers complain about the quality of reports. What is some practical advice on writing a good report?
Collaboration. Work with your peers, work with your clients, and work with your team. Try a few different styles of reporting and see what resonated with the customer. Not everyone learns and communicated the same way. The more versatility you get in your reporting the more likely the customer is to understand the intent of the attack path as well as the remediation. If our goal is to grow, let’s push it from every possible direction.
- How do you ensure your program results are valuable to people who need a full narrative and context, instead of a showcase of their weaknesses vs. your skill set?
This is something that is crucial to the pre-engagement phase. Both sides need a roadmap. Both sides need to set clear expectations on what the exercise will cover and how it will be delivered. Don’t just spend time selling, spend time listening. Spend time learning about the business and the members of each team that is part of the test. Learn their differences and how they need communication to unfold. Define pathways to success and a picture of what success looks like. Throw out the SOW language for a little bit and just speak like humans. Ones that are about to embark on a deeply emotional and distinct journey together and make sure everyone is comfortable. That comfort and trust will be the foundation of the exercise and ultimately the thing that separates your success from your failure.
- How do you recommend security improvements other than pointing out where it’s insufficient?
Metrics. This is no longer a binary game. There is a way to distribute the information in a testing engagement without the F.U.D. We must first understand that the discipline of security is a capability. It is something that can be measured on a standard Capability Maturity Model Integration (CMMI) scale. As testers, we rarely know the actual impact to the organization beyond our theoretical impact. “OMG, I got Domain Admin! You are hosed.” Well…maybe, but is that a fact? Not always. It does, however, drive fear. Fear is the last thing want as an outcome. Remember back to the Tyson scenario. We can’t break the will and confidence of the team; our job is to show the opportunity for improvement and track its progress. That said, we need to use metrics to enrich the data over time. We can’t just say “Red Team wins” we need to measure the varying level of success of the program to protect and detect the threats we simulated. We then need to work together to improve our capability maturity tactic by tactic. Do you know what I think is insufficient? The graphs you see from every security tool tell you that it stopped “X number of threats or attacks” you know what it doesn’t tell you “Out of how many.” How many did it miss? How well is it REALLY doing? The red team is there to fill out the rest of those metrics for the defense team. The red team is not there to decide for them or even suggest what products they need to stop the attack. The red team is there to provide the metadata needed to empower the blue team to make the best possible decisions.
- What non-technical skills or attitudes do you look for when recruiting and interviewing red team members? Or, what is one of the most beneficial non-technical skills you use for red team activities?
Drive. Pride. Honor. Respect. The technical skills can be taught to just about anyone. The aforementioned traits are something that’s inside someone. The willingness to be behind the scenes, grinding, is the thing I see as the most valuable trait of a red teamer. Everyone is going to get pushed to give up, red or blue. The ones that can carry on in the face of adversity because of their dedication to the cause have the selfless nature that moves the needle. One breath at a time, one hack at a time, one collaboration at a time; They know that the mission is everything and will stop at nothing to get there.
- What differentiates good red teamers from the pack as far as approaching a problem differently?
Being able to switch modalities. Removing the route to the issue allows you to see it from a different perspective. It may not be a shot you can take from the outside. It may not be a phish that lands. It may not be a cred you can find or a pivot you can make from the outside. It may not even be the guard you try and trick your way past. A good red teamer is ready to execute on any and every part of the battlefield. A great red teamer is one that knows how to spot the weakness and knows how to leverage the collective power of their team to get the job done.
I hope you enjoyed reading over my answers. Again, I encourage you to purchase the book to get a look into the minds of some of the best red teamers in the business. Also, if you are reading this and are thinking “Boy, I need a red team test” I encourage you to reach out and schedule some time to talk to one of our experts.