Not Your Average Bug Bounty: How an Email, a Shirt, and a Sticker Compromised a High Security Datacenter

Introduction 

At NetSPI, we help our clients secure their applications, networks, and organizations against a broad range of attacks. Technical controls such as secure coding, configuration, and monitoring are all important parts of the security puzzle. However, as we demonstrated in a recent engagement, even the most sophisticated controls can quickly become irrelevant when they meet the real-world complexities of human interactions. What happens if an attacker can impersonate an employee or influence your employees to take dangerous actions?  

To address these types of risks, NetSPI performs social engineering penetration tests.  Through emails, phone calls, and in-person interactions, testers attempt to gain access to sensitive information and locations. Testers may impersonate customers, other employees, or almost anybody they need to get access. The purpose of these tests is not to fool, or “gotcha” employees, but to expose systemic issues in security policy or training which an attacker might exploit.  

In 2021, NetSPI performed an on-site social engineering penetration test against a high-security datacenter, which resulted in high-impact findings for the client. We hope sharing the details about this engagement will demonstrate how a little creativity and preparation are sometimes all that’s required to gain access to otherwise secure data.  

The Mission 

The client owned and operated an entire datacenter, the building it was in, and the grounds it sat on. They had put significant resources into hardening their security and wanted to understand how an attacker might attempt to physically breach the building and gain access to the data on the servers inside. We were given the authorization to perform pre-arrival social engineering via phone or email, with very few restrictions on what types of pretexts or techniques we were allowed to use.  

That was the good news.  

As we learned more about the location, the bad news piled up quickly. 

1. This was a minimally staffed building. Only two employees regularly work on-site, in addition to a third-party security guard.
2. The building is fully enclosed in an 8-foot-high barbed wire fence, with a single gate. Accessing the parking lot requires a badge scan, as well as a security code.
3. All the building doors, interior and exterior, are protected with badge readers, retina scanners, and security-guard controlled man-traps which require one door to close before the next will open. We needed to bypass these controls before even getting face-to-face with a human, and tailgating was going to be next to impossible.
4. This was an expedited engagement. We had less than one week to research the location, develop our pretext, and prepare. 

Compared to most business environments, this was a very hardened target, and was going to require some real creativity to breach. 

Preparation 

The first requirement for social engineering is a valid pretext. We needed a believable reason to be on-site, one that would give us access to the building.  

Research revealed that the client gives datacenter tours for prospective clients. If we posed as a fake organization, we might be able to get on the datacenter floor, and then break away from the tour to do a little snooping. Open, clear, and detailed communication with the client is critical during every step of this kind of assessment, which was demonstrated viscerally by the reply we received when we presented this pretext to our contact for approval: 

Thankfully we asked. During a hasty follow-up call with the client, we learned that apparently some of the client’s sales team members take physical security very seriously and have a history of taking situations into their own hands. We added that information to our list of bad news and, with two days before our flights, we went back to the drawing board.  

Real-world attackers aren’t limited by time-boxes. They have all the time they need to research and prepare. Since the timeframe of this test was shortened, we partnered with the client and had them provide us with some basic internal information, which a dedicated attacker would likely be able to obtain either through online research, or observation of the location. Included in that information were the names and email addresses of the two employees who work on-site full time. Also included in the provided information was a list of external vendors who came on-site.  

One of those vendors was a well-known, national pest control company. By lucky coincidence, one of our consultants had recently hired this same pest control company to perform services at their home, and still had all the registration and confirmation emails. Using these emails as templates, we quickly mocked up legitimate-looking scheduling and billing emails for our target location and date. 

Next, we registered a lookalike domain, similar enough to the client’s domain that they could easily be confused. We used this domain to send an email that looked like it had come from Employee #1 and sent it to Employee #2. The email notified Employee #2 of the appointment and asked that the message be forwarded to the security guard. 

The next morning, we got a simple reply from Employee #2:

Amazingly, the difficult part was done. 

Excited about having our “in,” all we had to do now was sell the pretext while on site. The pest control company we were impersonating has a recognizable brand, and “look” not only for their employees, but also for the vehicles they travel in. We purchased white polo shirts and had the company logo screen-printed on them. We rented the specific type of vehicle used by the pest control company and for extra flourish, acquired die-cut static cling logos for the side of the vehicle. Finally, when we arrived at the destination city, we swung by the local hardware store, picked up some tool bags, flashlights, pest control gear, and rented a ladder. Putting it all together, the result was fairly convincing for being pulled together in two days, and for less than $150. 

Execution

On the day of the test, we simply drove up to the gate with our branded rental truck and used the buzzer. Having been informed of our appointment in advance, the security guard opened the gate with very little explanation required. Employee #2 met us outside and we explained we were there for “winter pest proofing” (whatever that meant). He was expecting us as well, so without further questioning, he swiped his badge, scanned his retinas, and opened the doors for us. Within minutes, we were on the datacenter floor. 

Pretending to look for pests, we moved around the entire building, with our escort using his badge and eyeballs to bypass all physical controls for us. We’ve hunted for a lot of bugs during our careers, but never ones this literal. 

The final layer of physical security between us and the actual servers were cages on the datacenter floor, containing the actual racks of equipment. Our escort declined to let us inside the cages; however, we were able to set up our ladder and get into the ceiling tiles. Up there, data cables from the cages were easily accessible, and it would have been simple to splice network monitoring equipment directly into them or install microphones or cameras. While one tester was taking photos in the ceiling, the other was talking to our escort, eliciting information about the datacenter, their operations, and who their customers were.

After an hour of touring every inch of the building, we announced we had finished our work, and said our goodbyes. This probably would’ve been enough, but sitting back out in the truck, we discussed how we had gotten significant facility access, but wanted to push harder and get onto the network. After a quick discussion, we decided to dive back in.

From the truck, we called our escort and explained that we had forgotten to bring some paperwork we needed to have signed and asked if they had a printer we could use. Our escort agreed and let us back into the building, and even set us up with temporary credentials to access the network. Had this been a full red team engagement we may have tried to pivot to additional network resources, however, the scope of this test was strictly social engineering, so we stayed focused on that. 

After a little contrived hemming and hawing about how to best access the document and print it, we asked our escort if we could just email it to him and have him print it for us. He agreed, and we sent him an email with an attachment, which he was willing to open and print for us. Considering this a sufficient demonstration, we thanked him profusely for all his assistance (and patience) and left the site undetected. 

Evaluation

When evaluating a site’s overall security, it’s tempting to focus on any single employee who assisted us and point out things they personally could have done better, however, that would be a mistake. Not only would it be inaccurate, but it would also derail efforts to improve security and remediate underlying issues. 

In fact, in this case, the employee did not actually violate any company policies at all. He did not allow us to go unescorted on the datacenter floor (despite multiple attempts by the testers to split up) and he didn’t provide access to the actual cages. The information he provided in conversation had some value, but nothing sensitive or confidential. The network access he gave us was on a limited guest network, and opening email attachments is an unavoidable part of doing business, particularly if they came from someone you already know and trust.

The main vulnerability we exploited on this test was the fact that procedures for scheduling and confirming vendor visits were poorly defined. Without a policy or training to lean on, the employee simply received a reasonable sounding request from someone who he took to be his coworker, and then took reasonable actions to assist. He had no reason to suspect something was amiss. 

Ultimately, we did not exploit a flaw in a person, we exploited a flaw in policy.

Final Thoughts and Lessons

In the real world, there is no such thing as an “uncompromisable” target. What would be the point of a box that absolutely no one and nothing can open? Every physical and technical control can be bypassed by someone. Social engineering is, at its most fundamental, the act of finding that someone, and either impersonating them or enlisting their help. 

We have not yet encountered a penetration test where the employee was the vulnerability. Policy training, awareness, and compliance often need to be addressed, but true malice or incompetence is rarer than our natures lead us to believe. When evaluating the security posture of an organization, it’s important to stay focused on systemic issues, and not on individual people.

This test also drove home how communication between the client and the testers is key. If it hadn’t been, the outcome of this test may have been very different, and potentially dangerous. This type of work is not criminal, but it simulates criminal behavior. Criminal behavior involves inherent risks. The best way to mitigate those risks is to reduce surprises. When preparing for an engagement, make as few assumptions as possible, and don’t be afraid to ask for more information.

Similarly, it’s important to understand the difference between a penetration test and a red team assessment. Penetration testing is cool, but it’s not about being a secret agent or a ninja. A penetration test evaluates a specific set of policies and controls to determine if they are functioning as intended. When timeboxes are limited, it’s perfectly legitimate to work with the client to obtain internal information so you can stay focused on what’s important. In technical penetration testing, this is often referred to as a white-box or grey-box test. The same principles apply to social engineering. 

Ultimately, this test demonstrated the high impact social engineering can have, and the relative ease with which it can be used to bypass even the most sophisticated physical and technical security controls. Testing for gaps in training and policy is just as important as testing for gaps in technology. We learned a lot on this engagement and look forward to sharing more in the future. 

Ready to put your policies and security awareness to the test? Work with NetSPI on a social engineering penetration test.