Saturday, November 22, 2008

Anatomy of a shared memory node failure.

After my previous post about teams, I quickly received several requests to provide more details. Voici!

Gnip's redundant; you can walk up to any of our cloud instances, vaporize it, and Gnip chugs along its merry way. We use shared memory (via TerraCotta) to replicate memory across nodes. As you can imagine, shared memory across network nodes isn't all that cheap. Just like anything else, when its over used, things can melt down.

One of our customers started injecting hundreds of thousands more actors into their Filter rules than we'd tested for in a long time (or... ever, in the true production environment (there's a "you can never actually replicate production conditions in your staging/demo/review environment" blog post brewing in me). This caused one of the nodes to start working really hard to build the objects to support the additional actors. In turn, TerraCotta had to keep up its replication, going on its own merry way. The number, and size, of objects we were asking TC to manage (across clients, and three TC nodes as well (one primary, one secondary, and a third for good measure) caused too much lock contention across the system, and TC clients started dropping (heartbeats couldn't be kept up between clients and servers) because they were spending too much time processing locks. Once a TC client drops out of rotation, it has to be bounced in order to reconnect to the TC server. (in shared memory situations, you can't let your objects between client and server get "too" far away from eachother, otherwise you have bigger problems).

So, a node was dropping out of the TC network, we'd bounce it, it would come back up, try to recreate all the objects again, and crater. We'd restart it, it'd come back up.... rinse repeat, rinse repeat. Viscous cycle.

We resolved the issue by dramatically (several orders of magnitude) reducing the number of objects TC was managing in this code path. We optimized the object model to only keep the bare minimum in TC in order to keep our cherished clustered approach; the rest of the state stays put in local VM space, and is not shared.

There were other side effects floating around which got cleaned up in the process which was nice. We reduced some function call times from 45 minutes at their worst, to 45 seconds. We reduced our TC data set size from 16G to a few hundred meg. In the process, we also upgraded to TerraCotta 2.7 which further reduced in-memory, and on-disk, data set sizes.

125 comments:

Anonymous said...

frank kern http://www.facebook.com/pages/Living-Our-Dreams/170531582961171%3Fsk%3Dapp_10442206389 Fundpopog [url=http://www.youtube.com/watch%3Fv%3Dngjho0uaJz8]Scam Frank Kern[/url] Stype Scamming Frank Kern Frank kern There are simple systems you can put in place to help you focus on your business.

Anonymous said...

frank kern http://www.youtube.com/watch%3Fv%3DJVfEpBWVZR8 Fundpopog [url=http://www.entrepreneurs-journey.com/tag/frank-kern/]Frank Kern[/url] Stype Frank Kern Mass Control 2 Scam Frank Kern Better question, how much money do you desire to make a day!!!

Anonymous said...

online payday loan Stype [url=http://loans.legitpaydayloansonline1.com]payday loans online[/url] Flallododebag http://loans.legitpaydayloansonline1.com Fundpopog Most consumers have no takes literally local pay deposit will cost more than a payday loan.

Anonymous said...

order xanax no prescription xanax withdrawal ocd - xanax online no script

Anonymous said...

xanax online drug schedule of xanax - xanax and alcohol interaction

Anonymous said...

buy cheap tramadol tramadol hcl 50 mg shelf life - tramadol online apotheke

Anonymous said...

buying xanax online no prescription .5 xanax effects - xanax 1mg day

Anonymous said...

tramadol no prescription cheapest tramadol online no prescription needed - tramadol zydol 50mg

Anonymous said...

generic xanax buy xanax ativan - xanax bars to buy

Anonymous said...

buy tramadol online tramadol withdrawal panic attacks - tramadol for dogs dosage

Anonymous said...

order tramadol overnight tramadol 50 mg wiki - tramadol 50 mg

Anonymous said...

xanax online xanax bars white mg - cheapest xanax online no prescription

Anonymous said...

how to buy xanax online legally xanax generic substitute - xanax 1 mg yellow

Anonymous said...

generic xanax xanax tablets 0.5mg alprazolam - zoloft for xanax for anxiety

Anonymous said...

buy tramadol online generic tramadol no prescription overnight - buy tramadol cod fedex

Anonymous said...

buy tramadol online where can i buy tramadol over the counter in the us - tramadol hcl acetaminophen par

Anonymous said...

best buy tramadol tramadol 10mg - tramadol hcl for

Anonymous said...

buy tramadol online no prescription cheap tramadol withdrawal restless legs - can you buy tramadol over counter

Anonymous said...

buy tramadol online without a prescription buy tramadol online no prescription mastercard - buy generic tramadol

Anonymous said...

buy cheap tramadol buy generic tramadol - tramadol dosage 60 lb dog

Anonymous said...

xanax online no prescription xanax overdose effects brain - buy xanax with paypal

Anonymous said...

carisoprodol 350 mg carisoprodol soma watson brand - soma carisoprodol tablets 250 mg dosage

Anonymous said...

xanax no prescription online can buy xanax vietnam - alprazolam ratiopharm 0 5 mg nebenwirkungen

Anonymous said...

generic xanax is xanax 1 mg strong - much does 2mg xanax cost

Anonymous said...

generic xanax xanax bars empty stomach - xanax kratom

Anonymous said...

buy tramadol online tramadol 50 mg strength - tramadol hcl 50 mg tab amn

Anonymous said...

buy carisoprodol order carisoprodol online no prescription - carisoprodol pills

Anonymous said...

xanax online is 1 mg of xanax a lot - xanax 2mg u94

Anonymous said...

generic xanax order xanax online mastercard - long does 1mg xanax kick

Anonymous said...

carisoprodol 350 mg do they drug test carisoprodol - long does carisoprodol take kick

Anonymous said...

xanax online xanax pills for anxiety - generic xanax uss

Anonymous said...

carisoprodol 350 mg carisoprodol 350 mg online - carisoprodol images generic

Anonymous said...

buy carisoprodol side effects for carisoprodol - carisoprodol en espanol

Anonymous said...

xanax online generic xanax greenstone - jpee xanax 2mg

Anonymous said...

buy tramadol online how to buy tramadol - tramadol hcl obat

Anonymous said...

buy tramadol online tramadol kali 083 - best place buy tramadol

Anonymous said...

buy cialis online usa cialis benefits - eli lilly cialis price

Anonymous said...

xanax online drug interactions xanax naproxen - buy xanax online spain

Anonymous said...

cialis online buy+cialis+online+without+prescription+in+usa - order cialis paypal

Anonymous said...

generic cialis no prescription cheap cialis for sale - cialis soft

Anonymous said...

buy cialis online cialis 5 mg online-apotheke - generic cialis buy us

Anonymous said...

cialis online buy cialis online pay with paypal - order cialis in us

Anonymous said...

buy cialis online cialis information - buy cialis us pharmacy

Anonymous said...

xanax online round generic xanax - xanax drug class pregnancy

Anonymous said...

buy cialis online cialis online next day - best price cialis

Anonymous said...

buy xanax online cheap xanax and alcohol and pot - caffeine pills xanax

Anonymous said...

cialis online cost daily cialis nz - cheap cialis once day

Anonymous said...

xanax no prescription buy xanax online legally - lorazepam or xanax for anxiety

Anonymous said...

buy cialis online buy cialis in vancouver bc - what is generic cialis called

Anonymous said...

buy cialis online cialis daily vs. 36 - buy brand cialis online us

Anonymous said...

buy real cialis online cialis retail cost - cialis tadalafil 20mg reviews

Anonymous said...

cialis online cialis online 24 ore - cialis online no rx

Anonymous said...

xanax online xanax pictures - safest place order xanax online

Anonymous said...

cialis online cialis online australia - cialis price us

Anonymous said...

xanax online xanax 0.5 mg street value - buy 3 mg xanax online

Anonymous said...

cialis online nhs price cialis - cialis online overnight shipping

Anonymous said...

cialis online buy cialis las vegas - cialis risks

Anonymous said...

buy tramadol online no prescription overnight tramadol hydrochloride 50 mg get you high - buy from online texas tramadol

Anonymous said...

http://landvoicelearning.com/#23561 high from tramadol - high with tramadol

Anonymous said...

buy tramadol india buy tramadol online paypal - tramadol withdrawal pregnancy

Anonymous said...

tramadol cheap order tramadol online usa - tramadol hydrochloride 50 mg get you high

Anonymous said...

buy tramadol where to buy tramadol cheap - buy tramadol overnight saturday delivery

Anonymous said...

http://blog.dawn.com/dblog/buy/#41639 tramadol abuse - tramadol hydrochloride online

Anonymous said...

http://landvoicelearning.com/#63987 does tramadol high feel like - buy tramadol online cod only

Anonymous said...

http://landvoicelearning.com/#97734 tramadol 50 mg with ibuprofen - best place buy tramadol online

Anonymous said...

buy tramadol tramadol street value - 100mg tramadol overdose

Anonymous said...

Conduct synchronize avec le treatise de [url=http://www.abercrombieandfitch4s.us]abercrombie and fitch[/url] la plage sud, peut etre decrit comme tres accrocheur couleur, les tons de bleu avec du rose, du gaz Sao, [url=http://www.monclerfrancesmagasinfr.fr]moncler[/url] mais n'a pas encore ete annonce ce chaussures modeles masculins ou feminins modeles, bien sur, les petits Hutchison est un espoir que les deux hommes et les [url=http://www.abercrombieufrancea.info]abercrombie and fitch[/url] femmes (rires)! Eh bien, l'homme de peu de mots, ou passable vous-meme en profiter!
Christian Louboutin chaussures a talons hauts a ete [url=http://www.christianlouboutin7spascher.com]louboutin[/url] recherche base les celebrites de nombreux artistes, la derniere serie 2012 printemps et en ete, les concepteurs utilisent le represent de l'ultra-haute avec [url=http://www.hollisteruonlineshops.com]hollister online shop[/url] en added sealed a l'eau de sorte que vous pouvez immediatement mettre sur le champ de gaz pleine. Une tete de poisson chaussure a lacets, imprimes animaliers et des inserts decoratifs [url=http://www.hollistercosale.us]hollister[/url] metalliques integres dans Gengrang suffisamment accrocheur et elegant.
Vêtements protège les gens contre beaucoup de choses qui pourraient blesser le fuselage of men humain découvert. Vêtements agir en tant que coerce contre les éléments, y compris la pluie, la neige, le inlet et autres conditions météorologiques, [url=http://www.hollisterpascher7s.com]hollister pas cher[/url] ainsi que du soleil. Toutefois, si les vêtements est trop scant, mince, petit, serré, etc, l'effet de defence est réduite au minimum. Vêtements également de réduire le niveau de risque au cours de l'activité, comme le travail [url=http://www.hollisteruksale5.co.uk]hollister sale[/url] ou le sport. Vêtements parfois se porte comme une clutch cover contre certains risques environnementaux, tels que les insectes, les produits chimiques toxiques, des armes, et le coupling avec des substances abrasives. Inversement, les [url=http://www.hollisteruksale5.co.uk]hollister sale[/url] vêtements peuvent protéger l'environnement de l'utilisateur vêtements, comme des médecins portant gommages médicaux.
[url=http://svyatogor.kiev.ua/forum/index.php?topic=110698.new#new]Ses guichets fermes pompes a prendre colorblocking avec multicolore zig zag et 4.7 talons. Supervised[/url]
http://[url=http://www.northernunion.net/forum/viewtopic.php?f=8&t=5132]La haul down obliterate arbore un dessus noir contraste second-rate le Jumpman rouge et un dessous en cement-splatter gris the[/url]
http://[url=http://61w.forum-akm.pl/viewtopic.php?TopicID=1703&orderby=NumViews&direction=ASC&arc=&page=4&temp=1]La 2e lieu Revient a juin chaussure de Stuart Weitzman au talon aiguille en metal, Suivi the unsurpassed garnishment Jimmy Choo sandales juin. the[/url]

Anonymous said...

tramadol online overnight tramadol webmd - where to buy tramadol online usa

Anonymous said...

learn how to buy tramdadol physical addiction tramadol - order tramadol cod overnight

Anonymous said...

buy tramadol ez tramadol order status - order tramadol cash delivery

Anonymous said...

tramadol 100mg tramadol withdrawal suboxone - tramadol hcl 50 mg street price

Anonymous said...

http://landvoicelearning.com/#23561 tramadol 50 mg and pregnancy - what does tramadol generic look like

Anonymous said...

buy tramadol tramadol 50mg tablets narcotic - tramadol for anxiety

Anonymous said...

can you buy tramadol online legally tramadol no prescription mastercard - buy tramadol online in usa

Anonymous said...

http://buytramadolonlinecool.com/#28875 can you buy tramadol internet - ultram tramadol hydrochloride tablets

Anonymous said...

http://landvoicelearning.com/#30896 good online pharmacy tramadol - buy tramadol 24x7

Anonymous said...

buy tramadol can you buy tramadol over the counter usa - buy tramadol online in australia

Anonymous said...

buy tramadol buy tramadol online australia no prescription - tramadol for dogs same as for people

Anonymous said...

http://landvoicelearning.com/#23561 tramadol buy no prescription - is it legal to order tramadol over the internet

Anonymous said...

learn how to buy tramdadol buy tramadol from usa - tramadol quick release

Anonymous said...

order tramadol online overnight buy discount tramadol - tramadol 50 mg side effects

Anonymous said...

learn how to buy tramdadol can you buy tramadol over the counter usa - tramadol 37.5 addiction

Anonymous said...

buy tramadol cod best place order tramadol online - tramadol 50 mg bluelight

Anonymous said...

learn how to buy tramdadol tramadol 100mg pills - tramadol for dogs 25mg

Anonymous said...

buy tramadol online tramadol hexal 100 mg - tramadol withdrawal exercise

Anonymous said...

ways to buy ativan online ativan 4mg im - ativan overdose in children

Anonymous said...

buy ativan online ativan .5 high - ativan no prescription

Anonymous said...

buy tramadol online illegal get tramadol online - tramadol 50 mg bluelight

Anonymous said...

http://buytramadolonlinecool.com/#96430 tramadol hcl-acetaminophen zen - tramadol buy online

Anonymous said...

buy ativan online ativan 6 weeks - buy ativan online pharmacy

Anonymous said...

http://reidmoody.com/#96318 buy lorazepam online us - ativan nursing

Anonymous said...

ways to buy ativan online lorazepam 1 mg price walmart - long break ativan addiction

Anonymous said...

http://reidmoody.com/#51208 ativan and xanax equivalent - ativan dosage routes

Anonymous said...

buy xanax online xanax vs ativan - xanax drug test duration

Anonymous said...

cheap tramadol no prescription tramadol 200mg online - buy tramadol health solutions network

Anonymous said...

buy tramadol 100mg does tramadol overdose feel like - tramadol 50mg

Anonymous said...

order tramadol no prescription tramadol from usa no prescription - tramadol and ultram recovery room

Anonymous said...

http://staam.org/#50589 37.5 mg tramadol high - buy tramadol online saturday delivery

Anonymous said...

buy tramadol online buy tramadol online with cod - venta tramadol online

Anonymous said...

buy xanax online xanax online usa - xanax 3mg

Anonymous said...

http://staam.org/#37708 tramadol hcl percocet - tramadol hcl 50mg buy

Anonymous said...

http://staam.org/#56821 best place buy tramadol online - tramadol hcl 50 mg what is it for

Anonymous said...

http://staam.org/#21236 tramadol hcl nursing implications - tramadol hcl humans

Anonymous said...

http://ranchodelastortugas.com/#51462 xanax does it show on drug test' - 2mg of xanax high

Anonymous said...

buy xanax online watson 658 generic xanax - xanax for sleep

Anonymous said...

http://ranchodelastortugas.com/#30416 xanax and alcohol withdrawal dosage - drug interactions between xanax and ambien

Anonymous said...

buy xanax online some generic names xanax - generic equivalent xanax

Anonymous said...

buy xanax online xanax side effects in children - where can i buy xanax online legally

Anonymous said...

buy cheap xanax buy xanax online without prescriptions - buy xanax online europe

Anonymous said...

http://bayshorechryslerjeep.com/#1833 xanax online stores - alprazolam 0.5mg shelf life

Anonymous said...



Here is my web-site :: buy quality facebook likes

Anonymous said...

cialis achat cialis comprar viagra

Anonymous said...

buy tramadol in florida order tramadol no prescription - can you buy tramadol online

Anonymous said...

tramadol online overnight buy tramadol no prescription overnight - buy tramadol no rx

Anonymous said...

order tramadol online cod buy tramadol 100mg online - tramadol no rx

Anonymous said...

australia cialis generic cialis generic cialis cialis online

Anonymous said...

cialis on-line cialis achat france

Anonymous said...

Check for the vitamin and mineral content in sugar
from any source, you will find it to be either nil or minute.
You have already spotted the dress of your dreams, but there is a problem…how to get
in it. ALTADRINE Fat Burner Tablets is the solution to
your problem; it’s a specific product that contains extracts that are capable of
burning your surplus fat while producing an energy available for all the body.

Of course, our bodies are different and some of us may react negatively to fat burners while others will experience no effects whatsoever.


Also visit my blog post: top fat burning foods :: :
:

Anonymous said...

Shell cameos have a thin concave spine, using the exception of abalone and mommy of pearl which might be normally flat around the back and somewhat thicker.
Nevertheless males are a little way behind and also jewelry for males is one on most idealistic pieces of jewelry worn by men.
The campaign was rolled out in 1950; within a few years, diamond jewelers could barely
keep up with the demand for diamond rings - including mens diamond rings.



My web page homepage

Anonymous said...

It is less flashy because of a smaller single diamond,
but still elegant on the finger and of which your
lady can weary comfortably. Some stores possess a attractive style, but don't hold the correct dimensions accessible. You do not need to settle for "almost great enough.

Also visit my homepage - promise rings size 9.5 - ifgoiano.edu.br -

Anonymous said...

Right after you have chosen the ring that you might
be planning to present for your partner, you must decide what you're moving to possess inscribed about the ring. They will have amazing diamond rings that they are more than willing to sell for a reduced cost. These designs are each bit as exotic and also a great deal a lot more economical than a matched strand.

Also visit my blog ... promise rings her jared's ()

Anonymous said...

This is a great place to start looking for a ring because you will have questions and a
jewelry store may have more answers than a department store.

It definitely puts the whole matter of wearing a divorce ring to rest.
After the removal of the pornographic application known as "Hottest Girls" (designed by
app developer Allen Leung) for the increasingly popular Apple
i - Phone, it seems the multinational consumer electronics
giant is seeking to clear its name of any connection
to sexually explicit material.

Here is my blog post ... homepage

Anonymous said...

(If it makes you feel better you will have time to upgrade the ring while she sleeps).
The form of your pearl is additionally crucial and also the rounder the pearl,
the greater. Therefore you can go for detailed and eloquent designs of
band to surprise your lady with.

Also visit my webpage; promise rings under 20 dollars

Anonymous said...

Great article, totally what I needed.

Take a look at my blog post wizard101 crown generator []

Anonymous said...

The outer layers of the pores and skin turn out to be infected, ensuing in burning,
itchy rashes. Examine your dog frequently for ticks, even following walks.
Sometimes, it suddenly appears anyplace on the physique.


Also visit my website - std testing portland or ()