A security researcher Jeremi Gosney (a.k.a epixoip) demonstrated his GPU hash cracking rig at the Passwords^12 conference in Oslo, Norway on Monday.  It leverages the Open Computing Language (OpenCL) framework and Virtual Open Cluster (VCL) to run HashCat across a cluster of five, 4U servers equipped with 25 AMD Radeon GPUs and communicating at 10 Gbps and 20 Gbps over Infiniband switched fabric.  Jeremi was able to attempt 348 billion Lan Manager (LM) password hashes per second in a test.  For example, a 14 character Windows XP password hashed using LM would be cracked in just 6 minutes.  The clustered GPU's also clocked in impressive speeds against more sturdy hashing algorithms as well, including MD5 at 180 billion attempts per second, 63 billion/second for SHA1 and 20 billion/second for passwords hashed using the NTLM algorithm. So called “slow hash” algorithms fared better. The bcrypt (05) and sha512crypt permitted 71,000 and 364,000 per second, respectively.  Another interesting note is that Gosney mentioned that VCL makes load balancing across the cluster – once an arduous task that required months of custom scripting – a trivial matter. As a result, Gosney said that his team is at a point where their implementation of Hashcat on VCL could be scaled up far above the 25 GPU rig he has created – supporting “at least 128 AMD GPUs.