There is no easy way to say it - I cannot believe my luck. I set up the hard drives that the premiums sites were backed up onto using software raid - something very standard we have been using here from the beginning. Unlike the hardware RAID, it has been rock solid and reliable.
All variations of RAID will save data if one disks fails, and all will lose data when 2 fail.
Well guess what happened - the power dropped on 2 of the disks in the set, making them appear to have "crashed". The disks went down, taking the data with it.
After talking with peeps at the colo, it's looking like a UPS in our section of the building is to blame. Overloaded from very poor power conditions due to the WTC tragedy (the colo is only a couple of blocks from there), it looks like this UPS was not able to provide full power to us.