Take 2: Amazon S3 file deletion FAIL

The S3 bucket key deletion saga continues. I had two Ec2 XL instances running 24/7 for about a month, running ~150 parallelized DELETE requests on s3 bucket keys (each machine), and dented my S3 usage by only 10%. I spent ~$1,500 (money went to Amazon obviously) on those dedicated XL instances in the process, while only reducing my S3 bill by ~10%, and that doesn't include the operational/engineering overhead to manage the deletion software/machines.

I'm left with only one option to wipe out the data on S3 that I don't need anymore; closing my Amazon account, and opening a new one. While I have high hopes that I'll be able to use the same keys/tokens/secrets in the new account, I'm doubtful. Thus, I'll incur several thousand more dollars re-wiring Gnip's vast network of machines, and RightScale, to work w/ the new account.

Lesson here is that you have to be very conscious when dealing with very large data sets on Amazon's S3.

Jud Valeski

Jud Valeski

Parent, photographer, mountain biker, runner, investor, wagyu & sushi eater, and a Boulderite. Full bio here: https://valeski.org/jud-valeski-bio
Boulder, CO