A while back I put an article up on exposing a GPU up to a virtual machine for cracking password hashes (https://isc.sans.edu/forums/diary/Building+Your+Own+GPU+Enabled+Private+Cloud/16505).  This worked great for me for a while, but then it became evident that 1 or two GPUs just wasn't enough - each GPU adds a linear amount of processing power, so 6 GPUs will solve  problems 6 times faster than a single.  Problems like cracking wireless keys, windows passwords, passwords on documents or databases, any number of things (150 different hash types in the latest version hashcat).  

What I found when I added more GPUs to my ESX host was that there's a limit on VT-d (DirectPath I/O in ESX) - you can only assign up to 8 devices in ESXi 5.x.  Since each GPU represents 2 devices, that's only 4 GPUs.  (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789)

So I had to go to a physical server to get past 4.  What more is there to learn you ask?  First of all, the Linux drivers just don't cut it.  Getting more than a few GPUs to be recognized from one reboot to the next is a challenge, even if you use the exact OS Versions and drivers recommended.  Even getting lspci to see them all was a gamble - each time I powered the server on was a roll of the dice.

Windows drivers work fairly well - however, in Windows 7 there's a hard limit of 4 AMD GPUs (mine are AMD R9 280x's) buried in the driver - don't forget that these are supposed to be graphics adapters, and limiting a system to 4 PCIE x16 graphics card actually makes decent sense.  However, we're not using these for graphics!  You can fix this limit with some judicious registry edits, but these vary quite a bit depending on the GPU model and OS.  The fine folks at lbr.id.lv put together an executable (6xGPU_Mod) that builds the reg changes for your setup - find it here:
   https://lbr.id.lv/6xgpu_mod/6xGPU_mod.html

But wait, there's more!  OCLHashcat requires a specific version of the AMD drivers to work correctly.  Again, these are graphics cards, and the newer versions of the driver don't lend themselves to computation apparently (a bug that doesn't affect graphics affects mathematical calculation).  Today's recommendation (for oclhashcat) is to use AMD driver version 14.9 (exactly), and no other.  This version recommendation does change - refer back to the documentation for whatever tools you are using for driver version recommendations.

Also, don't skimp on power supplies.  I have 2500W available (2x1250) for these 6 GPUs and the powered risers that feed them, plus the power supply for the system unit.  If the cards don't have enough power, either they'll just run slower, or they won't run - either way it's an easy fix.  And if you have issues during the build (everyone does on these), ruling out power problems is a good start in resolving these problems.  I budget 300W per card - likely at least a bit overkill, but I'd rather have a bit extra than be a bit short.  The old proverb "when in doubt, max it out" is a good one for a reason.

At long last though, I now have 6 GPUs dedicated to cracking whatever encrypted information I need to throw them at!

One final note - yes, I do know that you can spin up an AWS instance with GPUs to perform similar functions.  In my practice though, I'm not comfortable cracking customer passwords on someone else's server.  Also, in my previous rig, it was not uncommon to see password cracking runs for a typical list of hashes take 5-7 days, with 2 GPUs running flat-out - depending on the list and the hashing algorithm, this can run up to some serious computation time, which costs real dollars in a cloud service.  Bumping the count up to 6 GPUs in my own build cuts the time for me down by a factor of 3 for a pretty low cost, and still keeps the password hashes (and cracked passwords) in my own rack of servers.

If you've found other gotcha's in this sort of implementation, or if you've had good luck using a cloud service for stuff like this, please, use our comment form and let us know how you've fared !

===============
Rob VandenBrink
Metafore