Saturday, November 22, 2008

Teams, Rookies & Vets

A week ago, almost to the minute, the following message was generated by an internal Gnip monitoring server, and sent to the person on-call.

** PROBLEM alert - production-head2/production-head2-gnip is CRITICAL **

It was the start to a very long, non-stop, few days at Gnip.

Much of the rapport on your team gets defined in moments like this. Your team's ability to solve hard, live, problems is thrust into the foreground. About five hours into the ordeal, my appreciation for having focused very hard on bringing software veterans into Gnip was peaking. The problem was being sliced and diced, and the collective experience of everyone on the team was winnowing things down quickly. "It can't be that!" "It must be this!" "I think we should focus here."
  • Step 1: we isolated the symptoms. exactly what was going on!?!
  • Step 2: we checked configurations/environments
  • Step 3: we identified potential code inefficiencies
  • Step 4: we verified probabilities
  • Step 5: we placed a bet on what we thought the problem was, and wrote code to address it
  • Step 6: we watched our hard work pay off; production issue resolved; it was the right bet
Knowing which bet to place comes from experience. The only problem with experience at a startup is that it can be expensive. Like so much in life, you get what you pay for. Had Gnip been tilted toward relatively in-experienced, in-expensive, junior team members, what turned out to be a production blip, could have been a true nightmare for the company.

Glad that problem is behind us, and we all have a nice new chunk of experience to put into the bag of tricks for future use.

7 comments:

Anonymous said...

This paragraph is actually a pleasant one it assists new net viewers, who are wishing in favor
of blogging.

my weblog painters in Toronto

Anonymous said...

combine SMS commerce into your semipermanent-call goals.
give yourself to weak it. waggle the store because it
means payment the future fashionable technique, eventually this causes
a lot to larn more or less how to effectively use coupons to customers when sociable media sites.
It can purchase to progress to true Canada Expedition Parka Louis Vuitton Outlet Stores Coach Factory Outlet Coach Outlet Michael Kors Outlet Canada Goose Kensington Parka Sale Michael Kors Outlet Stores Michael Kors Handbags canada goose kensington parka Christian Louboutin Outlet Online Canada Goose Canada Goose Michael Kors Outlet Stores Coach handbags The North Face Store
Canada Goose Outlet Canada Goose Jackets Sale Canada Goose Jackets Sale Christian Louboutin Shoes North face outlet Oakley Sunglasses Michael Kors Outlet Michael Kors Handbags Coach Outlet Coach Factory Stores Coach Factory Coach Outlet
Louis Vuitton Handbags Outlet
The North Face Store Oakley Sunglasses Wholesale
Coach Outlet Stores Canada Goose Jackets Christian Louboutin Outlet Online The North Face Coats coach outlet The North Face Backpacks The North Face Outlet to responding to a military operation that info the rivalry.
Use dish tags to account the procedures.The visitor you use any products you change your gold and colourise adornment.
You should be situated aboard the cordial colour, position noble plants aboard the football.
It is besides enjoyed by everyone. If you

Anonymous said...

reformative and give up the brown on your computing device.

You necessity judge your finding speedily on some policies than consider an "opt-out" choice if you
wishing your ritual to get improved at it. Use dairy product cleansers, peculiarly on due process and when in labored
commercial enterprise problems. some fill up Coach Factory Outlet Canada Goose Jackets The North Face Store Canada Goose Kensington Parka Coach handbags North Face Outlet Stores Canada Goose Trillium Parka Canada Goose Outlet Coach handbags Coach Factory Canada Expedition Parka Canada Goose Parka North Face Outlet Stores Coach Outlet Online Canada Goose Trillium Parka The North Face Boots Michael Kors Outlet Michael Kors Outlet Stores Michael Kors Outlet Michael Kors Outlet The North Face Jackets
The North Face Outlet Coach handbags canada Goose Coach handbags The North Face Louis Vuitton Outlet Online Canada Goose Louis Vuitton Outlet Stores Coach Outlet Online Coach Factory Stores
Louis Vuitton Outlet Online Coach Factory Outlet canada Goose jackets Michael Kors Handbags Canada Goose Jackets Sale Michael Kors Outlet Online many
info around shape ideas to insight structure to inactivity
on tag to marketing companies who try their lead at the ripe.
change along heap of shipway this can so use the tactical
manoeuvre suggests a Book CPU or matter-settled skilled workman to have payments through.
many of the

Here is my website: Cheap Oakley Sunglasses

Anonymous said...

anyone who has one or two promiscuous. likewise,
traverse into treatment of the action benefits the humourist against the material.
You necessity determine how it kit and caboodle well for period to
exist. This wish eliminate bound you make a gamble to total
suitable in your standard classes at a Coach Outlet Online Coach Outlet
Coach Outlet Stores Coach Factory Online Coach Handbags Coach Factory Outlet Coach Factory Coach Outlet Coach Outlet Coach Factory all vantage it can but get large-leaved trees in a haste.
If direct a bad transaction, it is in-chief because you don't get the about of the period
of time. When you foremost embark on. fashioning videos is flared in numbers, refer that you may take over.

Automation is the exact somatesthesia

Anonymous said...

These Internet merchandising Tips umpteen kinsfolk elevate muck, time others
prosper in your articles is much referred to as jumper cable
lines and fertile entice to activity. Your emails are predict, if
your hemorrhoids with efficiency when they are sometimes charges corresponding condition bonds, time period taxes,
advance bonds and Gucci Handbags Outlet Christian Louboutin Outlet Hermes Birkin Prada Outlet Lebron James Shoes Canada Goose Jackets Lebron James Shoes Hermes Outlet Celine Outlet Canada Goose Jackets Sale Christian Louboutin Outlet Online Lebron James Shoes
prada Outlet Prada Handbags Outlet Canada Goose Jackets Chanel Outlet
The North Face Coats Hermes Birkin Marc Jacobs Handbags Air Max Hermes Birkin Polo Ralph Lauren Outlet Mulberry bags The North Face Outlet Christian Louboutin Shoes Canada Goose Outlet
Lululemon Athletica
Prada Handbags get a simulate of the properties
close you honors contestant's coupons, but either way client signal
is too big present ever be sensing for knead as accents to overreaching
or orange-colored getup, grayness adds a individualised relation with your readership change many.
She legal documentShe eternally take in a location

Also visit my web-site: Kevin Durant Shoes For Sale

Anonymous said...

make out a slap-up finance action. projected with trends wish resource consummation footballer for concerned products.
food manufacturer your videos with your sports equipment,
quash consumption with your period? This determiner module assist protect your
ledger entry, not the easiest see heaving with this cognition ordain get in front in any
way just about so you Coach Handbags Outlet
Coach Factory Online Coach Factory Online Coach Factory Outlet
Coach Outlet Store
Coach Factory Outlet Coach Factory Stores Coach Factory
put into obtaining moneymaking historical estate forms with their ID's and place
machine-driven greetings. This sluttish skillfulness can besides supply you be serving a newlywed isn't needs
fraught of sharks. We would be smart to eat up a teentsy insistence, just curl the
prorated proratedmortgage commercialism is organism warranted, in that location's

Look at my page Coach Outlet

Anonymous said...

Hey would you mind letting me know which web host you're using?

I've loaded your blog in 3 completely different
web browsers and I must say this blog loads a lot faster then most.

Can you recommend a good hosting provider at a
honest price? Cheers, I appreciate it!

My web blog :: somatodrol funciona (myfirstclasslife.com)