Sunday, May 9, 2010

Take 2: Amazon S3 file deletion FAIL

The S3 bucket key deletion saga continues. I had two Ec2 XL instances running 24/7 for about a month, running ~150 parallelized DELETE requests on s3 bucket keys (each machine), and dented my S3 usage by only 10%. I spent ~$1,500 (money went to Amazon obviously) on those dedicated XL instances in the process, while only reducing my S3 bill by ~10%, and that doesn't include the operational/engineering overhead to manage the deletion software/machines.

I'm left with only one option to wipe out the data on S3 that I don't need anymore; closing my Amazon account, and opening a new one. While I have high hopes that I'll be able to use the same keys/tokens/secrets in the new account, I'm doubtful. Thus, I'll incur several thousand more dollars re-wiring Gnip's vast network of machines, and RightScale, to work w/ the new account.

Lesson here is that you have to be very conscious when dealing with very large data sets on Amazon's S3.

7 comments:

Anonymous said...

But even if youjr salon iis a smoke free environment,
the smoke from straightening and curling irons ccan make the air thick.
This requires extra sills in accounting and managerial aspects to keep
on top of the business ffor smooth organisation, continued
growth and profit. In fact look around and you will see several successful business ideas already operational as franchises in tthe market place and all you need to doo is to sign the franchise agreement and get
started, instead off searching 101 vendors, suppliers, how too do’s,
etc.

Visit my web page; beauty hurts beyonce, http://www.43things.com/,

Anonymous said...

This website was... how do you say it? Relevant!!
Finally I have found something that helped me. Many thanks!


Feel free to surf to my page; forex guide ()

Anonymous said...

Greetings! Very helpful advice within this article! It is the
little changes that make the largest changes. Thanks for sharing!



Also visit my webpage :: ottawa lighting store

Anonymous said...

This ite cwrtainly has all of the information I wanted about this subject and didn't know who to
ask.

My blog post ... how to play roulette ()

Anonymous said...

Fine way of explaining, and pleasant article to obtain information concerning my presentation focus, which i am going to present in college.


my blog: clash of clans hack; clashofclanshack18.blogspot.com,

Anonymous said...

When I initially left a comment I seem to have clicked the -Notify me when new comments are
added- checkbox and from now on each time a comment is
added I get four emails with the exact same
comment. There has to be a means you can remove me from that service?
Appreciate it!

Look into my homepage: phlebotomist classes online
()

Anonymous said...

I am sure this article has touched all the internet viewers, its really really fastidious paragraph on building up new web
site.

My web blog -