GPU Cracking: Building the Box

Intro

This winter, we decided to create our own dedicated GPU cracking solution to use for our assessments. It was quite the process, but we now have a fully functional hash cracking machine that tears through NTLMs at roughly 25 billion hashes per second (See below). While attempting to build this, we learned a lot about pushing the limits of consumer-grade hardware.

We’ve recently updated this blog with more recent info – https://blog.netspi.com/gpu-cracking-rebuilding-box/

Goals

We set out to build a cracking rig with four high end video cards (AMD Radeon HD 7950) to run oclHashcat. We also wanted this solution to be rack mountable, so that it would be easy to store in our data center. As it turns out, there are not a ton of video card friendly server cases. We were only able to find a few GPU cracking friendly cases, but most of them cost more than the rest of our cracking hardware combined. If you have the money to spend, we would recommend going with the special case to save yourself from other issues, but this isn’t really an option for everyone. The reason why we recommend this is that the cards themselves do not take well to being lined up all together on a standard ATX motherboard. The fans tend to stick out further than they should and end up hitting the next card in the row. On top of that, the cramped conditions lead to overheating cards and cracking jobs stopping. The specialized cases have enough space to avoid these issues, making it easier to set up a box.

We opted for an “open air” configuration for our cracking box. This was primarily driven by trying to mimic the setups of bitcoin mining rigs that we had seen online. I will say that this is not the prettiest option for housing all of these cards. However, it is one of the most efficient ways to space the cards out for cooling. With the “open air” setup, we’re able to connect riser cables to two of the cards and keep the other two cards down on the board. These riser cables can have their own problems. We ended up opting for one (16x to 1x) riser cable and a different (16x to 16x) riser cable that has some modifications for voltage. The 16x to 16x cable has a 12 volt molex adapter soldered to the 12 volt pins on the riser slot.

While this looks a little hackish, it actually works quite well. We had to do this to supplement the voltage from the motherboard, as it was unable to pull proper voltage for all four cards (with two riser cables). I should also mention that there is some crafty engineering taking place to suspend the two cards above the board. This was accomplished with several zip ties and a modified piece of wire-mesh shelving.

I should also note that this whole rig is tied down (with stand-offs) to an old rack mount shelf. All in all, this setup works quite well. We can have all four cards running at full speed and the the hottest card will top out at 85° Celsius. We’re very aware of the fact that this looks insane. It’s hopefully a temporary solution. Eventually, we’re looking at securing a single rail to the rack to screw the cards into.

As for performance, here’s our current averages for hash cracking (OCL in Brute-Force mode):

MD5 – ~16000.0M/s
NTLM – ~25500.0 M/s
SHA1 – ~7900.0M/s

5 Tips for Building Your Own

So if you’re planning on putting together your own GPU cracking rig, here’s some steps that you may want to take to make it easier.

1. Look into a nice GPU server case and motherboard combo like this one https://www.newegg.com/Product/Product.aspx?Item=N82E16816152125
   1. These will be spendy (~$3,500+ for the combo, cards not included) but they are meant for this kind of setup.
2. Look at what the bitcoin miners are doing.
   1. Our “open-air” setup is actually pretty similar to most mining rigs that I can find.
   2. Replicate their parts list for your setup, if it works for them, it “should” work for you.
3. Plan everything out as best you can.
   1. From components and case layout to power and cooling requirements.
   2. Measure twice and cut once to avoid returns, repairs, and rebuying parts.
4. Devote a resource to the project
   1. Intern not busy enough? Have them build the cracking machine.
   2. Find the person that plays more PC games than you.
      1. They may know more about the cards and multi-GPU setups.
5. Don’t get discouraged if your set up isn’t working.
   1. We didn’t get it right on the first try, but we eventually got there.

Check out GPU Cracking: Setting up the Server by Eric Gruber on how to configure your cracking box to see all of the cards and run the cracking software.