Sunday, February 22, 2009

The Mess of Carriers, Handsets, App Stores & Consumers.

I remember the day my mother brought home her first cell phone. It was the true brick (pictured here), and looking back it was the result of the carriers of the day defining the handset as a product that would run on their network, and having someone build it for them. Well, sadly, not much has changed over the past twenty years. With all the innovation that occurs at the handset level (thank you Nokia, LG, Samsung, Apple, Motorolla, etc), the carriers are still the channel for 99+% of handset distribution, and they don't want the great stuff. Sure you can purchase the latest and greatest Nokia handset in its unlocked form from an independent party, and hope that it works on your carrier's network, but you'll pay through the nose for it. I've written on this tri-fecta concept in the past, but wanted to refresh (reiterate) my thinking given recent events.

Carrier subsidies for mass market handsets dominate the landscape. Consumers are price conscious, and the "free" handset when you sign up for a 2-year contract is just too appealing for most folks. The result, is a hamstrung handset market, with watered down, un-interesting devices making their way to consumer's hands. While some handset manufacturers are able to break this mold from time to time (Apple w/ the iPhone in partnership with AT&T), decent handsets are still only a fraction of total sold. Again, people like "free."

One area that is still shrouded in darkness is the notion of the "app store" for software on these handsets/devices/phones. Apple miraculously created (unfortunately probably a one-time aberration) a bountiful online, on-device, store for iPhone software, yet, despite all the recent announcements, no-one else has a useful offering. Pessimistically, I predict there won't be any other credible offering for many years to come. A lesson learned from telco/ISPs long ago was that they aren't willing to give up their multi-billion dollar capital infrastructure investments and write them off as a platform for others to benefit. Cell phone carriers are the mobile parallel to a 'T'. The success of a mobile device software marketplace is 99% a function of carriers permitting such a thing to exist; bottom line. I'll believe it when I see it.

For all the money invested in cell network setup, the carriers aren't about to widely allow other businesses to blossom on their backbone, without a huge piece of the pie. Their demands are so high, that others can't justify standing up businesses (e.g. app stores) bound to the hobbled terms and price-points offered by carriers; the carriers always want to come out on top. I can't blame them, but the sad result is that consumers suffer. Carriers are focused on standing up cell towers, and that's about it. They don't have a clue about standing up complete software ecosystems, yet they maintain choke holds on the handset manufacturers trying so desperately to get their innovations to market.

Until the carriers give up this control, and acquiesce to a smaller cut of the pie, the model will remain flawed. Consumers will continue to use old-world technology and devices when amazing new innovations are possible thanks to the handset manufacturers. Consumers will be unable to have a transferable, interesting, productive, quality, and entertaining multi-media experiences on their phones (unless you're an iPhone user). This seems backwards, but, we all know how monopolized markets always tend to win. What's the answer? Regulation? Consumer demand for a change? I feel like we've at least tried the latter; maybe the iPhone will indeed shake things up beyond Apple and AT&T's little bubble. Wouldn't that be nice.

I dream of a day when I can shop for killer hardware that runs software I choose, and that will work on a wireless carrier network that is so pushed into the background it's effectively a public utility (e.g. nameless).

Think about it:
  • 2007 was the first year you could buy a device that rode on a wireless carrier's network, that allowed you to purchase a wide array of 3rd party developed software directly to the device. Apple iPhone with AT&T.
  • For over a decade the general solution to cross platform portable software was the IBM Java VM that had to be hand-installed on a tiny slice of phones in the marketplace. As if a general consumer could ever do this.
  • Handset manufacturers have been blowing our minds with incredible devices for over ten years, yet one in a hundred of their innovations has ever made it to the carrier's showroom floor. Sad.
  • Carrier retail outlets that sell plans on their network, as well as the watered down versions of the aforementioned handset manufacturers, are a dime a dozen and commissioned salespeople litter their floors.
  • Handset manufacturer retails stores are non-existent. The rare prototype store crops up from time to time, but they're just showrooms that can't actually sell the device along with a carrier plan.
  • Much like the iPod, Apple is no longer selling the iPhone primarily, rather the iTunes App Store experience as a gateway drug to many happy online app purchases downstream. iPhone == the razor... the App Store, the razor blade.
I would love nothing more than to see Apple & AT&T's model flourish in and of itself, as well as spark revolution in the carrier, handset, and software offering dynamic industry-wide. Wanting this across the board for a long time now, I'm not holding my breath.

Wednesday, February 18, 2009

Three things that have changed my health

When I hit my 30's (I'm 35 now), my body started changing; again. The last time I morphed so significantly was puberty. I noticed my total invincibility and immortality being called into question after hard bike rides, or rock climbs. I thought "what!?!?," what everyone's been saying is true; your body's effectiveness starts to decline with age? Hangovers became real when I hit 30. Prior to that they were a farce as far as I was concerned. I've spent the past few years focusing on a few areas to determine they're overall impact on my well-being. A couple of them are obvious, but are often taken for granted and therefore usually ignored. I thought I'd share my experience with aging in this phase.

As a child I used to get severe headaches. After many medical evaluations, no-one could come up with a resolution. On my own, I established a connection between my headaches and water consumption. Too little water, yielded headaches. Enough water, and I was fine. I've found my overall health is directly connected to my water intake. If I find myself with a cold, or the flu, its severity, or very existence, is purely a function of how much, or little, water I'd been drinking. The lesson here has been simply to drink a few glasses of plain 'ol water everyday. When I drink a lot of water everyday, I feel great. When I don't, I feel tired, and dragged down.

For a few years in a row, I wound up with a chronic, non-productive, dry cough that would last months on end. Again, medical evaluation yielded nothing. I thought I'd try taking a multi-vitamin as just a random attempt at changing my body's chemistry. I've been taking a men's formulated multi-vitamin for a few years now, and the cough abated the very year I started. Furthermore, my body feels much more balanced now.

If I don't regularly exercise (as in a few times per week of something) my life goes to hell. I get sick, I wind up with low energy, I gain weight, I get depressed, I get cranky, I get irrational, I get impatient. The time I invest in exercise is a drop in the bucket with respect to overall productivity when I don't exercise.

I've found that if I keep these three things in check, my life is great. If I don't, things go south.

Wednesday, February 11, 2009


I just got a question via email that threw me back to my days at Netscape/Mozilla implementing nsIStreamConverter. The question is "how do I process streamed data?" There are many answers, but I thought I'd provide a fairly generic one here, along with some pseudo code (a nice mix of C, Python, and Javascript, for your viewing pleasure). But first, I want to try and do a high-level explanation of streams as they can be confusing to folks who are used to dealing exclusively in discrete, bounded, data chunks.

It's important to note that all I/O is stream based at some level; network and disk. When you read a disk from file, the bytes are streamed off the disk, into a lower-level socket API, then presented to you in your application via some "read()" function/method. Many languages do some convenience demarcation for you and allow things like "readline()" so you can easily read a line from a file that is broken apart by EOL markers. If you find yourself on the other end of a raw byte "read()" routine (whether off of a socket or a file; a basic file descriptor), then you're dealing with "streams" and you'll need some incarnation of the following code if you're trying to parse the data.

Some "streams" can be consumed and acted upon in small chunks (either byte by byte, or in chunks), or in large chunks. Some streams are binary, and some are text based. Today's web deals in lots of "text" based streams, so the below example follows that lead.

Hopefully you find this useful.

Imagine some data source providing data to your routine; get_data_from_stream() in this example. Then imagine you want to act on the data as it comes in.

// this is the local buffer we'll use to
// accumulate data from the stream
buffer = ''

// when processing a stream, you need
// to know when you have enough
// data to process, sometimes this is
// token based (a string, or a
// character), sometimes it is after a
// certain number of bytes (in
// which case this token is irrelevant).
// In this case, I want to do
// something once I've reached the end
// of an RSS entry. this
// processor handles "entries".
demarcationToken = ""

while ( data = get_data_from_stream() ) {
// take the data from the stream, and
// append it to the local buffer. this
// allows us to grow the local buffer
// until we have enough data to digest
buffer += data

// determine whether or not our processing
// specific demarcation exists yet
if ( tokenPosition = buffer.contains(demarcationToken) ) {
// we found a point in the buffer that will allow
// us to do some processing

// define the chunk to process, as the buffer
// up to the end of the demarcationToken.
chunk = buffer.subString(0, tokenPosition + len(demarcationToken))

// now do something with it

// reset the buffer to the position beyond the chunk you just
// processed.
buffer = buffer[tokenPosition + len(demarcationToken)]

// rinse and repeat
} // end if
} // end while

Tuesday, February 10, 2009

Customer Support and Clouds

You know your business is becoming a commodity when your competitor differentiation becomes "better customer support." If you get to this point, you better be a significant marketshare player in your space otherwise the end is near. Decent customer support is very expensive, and therefore very few firms, in any space that I can think of off the top of my head, bother with it. Instead it becomes a race to the bottom. Consider utility companies, power, water, mobile service carriers, local hardline phone carriers, etc. Customer service for all of them is cast to lowest common denominator.

I'm seeing Cloud computing services start to differentiate on customer service; whoops. It'll be a matter of a few years before that game's over. SLAs are one thing, but customer service is another. Focus on tangible, contractual, SLAs, not "better customer service."

Thursday, February 5, 2009

Testing 1, 2, 3

Gnip received a request to go over how we "test." I hope the following is useful.

While we don't practice TDD (Test Driven Development) outright, I'd consider us in that vein. We are test heavy, and many of our tests are written before the code itself. My personal background is not test driven, but I'm a convert (thanks to the incredible team we've pulled together). While it takes self control, the satisfaction of writing a test first, then building code to meet its constraints feels great when you're done. Your goal, at whatever level the test was written at, was clearly defined at the start, and you wrote code to fulfill that need. Ecstasy! Try it yourself.

Our build process includes execution of ~1k tests. You don't checkin code if you break any of those tests, and code you checkin has new tests to validate itself. If you "break the build," that is not nice, and peer pressure will look down on you so you won't do it again.

The range of tests at Gnip are a challenge to categorize and build. Component/unit level tests are relatively straightforward, and range from class drivers, and data input/output comparisons against expected result sets. Writing tests when much of your system is unpredictable and variable is particularly challenging. Gnip works with so many different services and data formats, that getting the tests right for all the scenarios is hard. When we do come up against a new failure case, a test gets written to ensure we don't fail there again.

Given the "real-time" nature of the Gnip core platform, benchmarking the performance of individual components, as well as end-to-end data flow is fundamental. We build "micro-benchmark" tests to vet the performance of proposed implementations.

Testing end-to-end performance is done a variety of ways. We analyze timing information in system logs for introspection. We also run scripts external the system to test both the latency of our system, as well as that of the Publishers moving data into Gnip. Those tests calculate the deltas between when an activity was created, published, and ultimately consumed out the other end.

The importance of both internal testing, as well as testing external to the system cannot be over stated. Testing various internal metrics is great for internal decision making and operations, however you can lose sight of what your customers see if you only view things on your side of the fence. We use (Custom checks) to drive much of our external monitoring and reporting.

Here's some insight into our testing tool chain:
  • JUnit, EasyMock (Java)
  • HttpUnit, bash/Python scripts/cron (general API pounding)
  • unittest (Python)
  • RSpec (Ruby)

Wednesday, February 4, 2009


My experience at Gnip has tested me in uncountable ways. From technical capabilities, leadership, and work-life balance, to trouble sleeping due to inability to turn my brain off at night as I churn on our challenges. What I've resolved as the most important thing for me to master in order to make Gnip successful, is my ability to stay on my game despite at least one bummer in the mix at every moment.

Staying positive is the only way to make this game work. When you are in a leadership role at a startup, you are responsible for driving a team with constant impossibility flowing around you.

At a talk he gave last night, Brad Feld, summarized this notion succinctly. To paraphrase, "every single day, there is something going on in my purview that is absolutely abysmal. Deriving energy and passion from it, and, more importantly, the overwhelming amount of positive events, people, projects that comprise the day, is the only way to make it work." It's a very "zen"-like perspective, and one I'm refining and fostering for myself.

If you need day-to-day balance in time, focus, and energy, startups likely aren't for you. You have to find consistency in chaos, clarity in mud, and calm in storms.