Take 2: Amazon S3 file deletion FAIL

The S3 bucket key deletion saga continues. I had two Ec2 XL instances running 24/7 for about a month, running ~150 parallelized DELETE requests on s3 bucket keys (each machine), and dented my S3 usage by only 10%. I spent ~$1,500 (money went to Amazon obviously) on those dedicated XL instances in the process, while only reducing my S3 bill by ~10%, and that doesn't include the operational/engineering overhead to manage the deletion software/machines.

I'm left with only one option to wipe out the data on S3 that I don't need anymore; closing my Amazon account, and opening a new one. While I have high hopes that I'll be able to use the same keys/tokens/secrets in the new account, I'm doubtful. Thus, I'll incur several thousand more dollars re-wiring Gnip's vast network of machines, and RightScale, to work w/ the new account.

Lesson here is that you have to be very conscious when dealing with very large data sets on Amazon's S3.