Sunday, May 9, 2010

Take 2: Amazon S3 file deletion FAIL

The S3 bucket key deletion saga continues. I had two Ec2 XL instances running 24/7 for about a month, running ~150 parallelized DELETE requests on s3 bucket keys (each machine), and dented my S3 usage by only 10%. I spent ~$1,500 (money went to Amazon obviously) on those dedicated XL instances in the process, while only reducing my S3 bill by ~10%, and that doesn't include the operational/engineering overhead to manage the deletion software/machines.

I'm left with only one option to wipe out the data on S3 that I don't need anymore; closing my Amazon account, and opening a new one. While I have high hopes that I'll be able to use the same keys/tokens/secrets in the new account, I'm doubtful. Thus, I'll incur several thousand more dollars re-wiring Gnip's vast network of machines, and RightScale, to work w/ the new account.

Lesson here is that you have to be very conscious when dealing with very large data sets on Amazon's S3.

1 comment:

Anonymous said...

But even if youjr salon iis a smoke free environment,
the smoke from straightening and curling irons ccan make the air thick.
This requires extra sills in accounting and managerial aspects to keep
on top of the business ffor smooth organisation, continued
growth and profit. In fact look around and you will see several successful business ideas already operational as franchises in tthe market place and all you need to doo is to sign the franchise agreement and get
started, instead off searching 101 vendors, suppliers, how too do’s,

Visit my web page; beauty hurts beyonce,,