2013-11-24

Do you even lift?

If you want to do important things, you should find something that is either on, or beyond, our current capabilities and do what was borderline impossible up until that point.  If you are more concerned with making money, you should lower your ambitions and shamelessly pump out highly derivative work in the hope that something is going to stick.

I have been in the software industry for a few decades now and it appears to be a constant that at any given time the vast majority of the industry is stuck in a group-think.  Everyone agrees on what is important.  In the 80s it was 4-gen systems.  In the 90s it was OO-everything.  In the 00s it was pragmatism (as a backlash to the insanely tedious process oriented nonsense of the mid to late 90s).

The current craze appears to be a mish-mash of business development ideas applied to software engineering in order to squeeze tons of cash out of shallow ideas that have a limited life-span.

Sure, it is exciting when people spend 3-6 months building something they can sell for a cubic meter of Benjamins.  I'd love to do that.  Mostly because it would mean I could pay people to do things I find important without having to convince anyone -- which is a tedious, undignified affair.  But I think it is fundamentally unhealthy when this is what we aspire to. Because it has more in common with winning the lottery than accomplishing anything really important. The ugly truth is that most startups do not get acquired for a billion dollars, most shallow products are just worthless nonsense and most of the people who build them aren't really that brilliant.  They are just lucky.

Beware of having lucky people trying to rationalize their luck into repeatable process postfact.

One of the greatest, truly worthless ideas of its time was Pointcast.  In essence a service that pushed advertising to screen-savers.  For a while in 1996, this was the biggest idea around and everyone was terribly excited.  At some point it was even one of the biggest traffic generators on the Internet -- pushing terabytes of advertising to screens that nobody was looking at.  I remember lots of people building copycat products and a guy I was working with felt we should drop everything we were working on and bet the farm on this.

Pointcast was, at one point, valued at a quarter of a billion dollars.  Then 100 million.  Then 10 million. And then nothing.

Pointcast wasn't important.  Anyone with half a brain back then should have been able to see that the product was bullshit and that it brought us absolutely no breakthroughs of any kind and it was trivial to rip off.  But most people didn't.  Most people were unable to tell a win in the lottery of Internet fads from creation of lasting value.

During the same years, Page & Brin wrote bunch of papers(1) on data-mining and IR and eventually built a search engine.

And before people who know sod-all about what Page & Brin did:  there was nothing easy about what they did.  They didn't start off with a minimum viable product and then iterate it into something that was successful.  They started by solving a handful of hard problems.  And then they solved a bunch more.  Including problems that every single one of their competitors found to be outside the scope of what they were doing.

(One example is their internal infrastructure from the early 00s which allowed them to speed up innovation.  Much of which was very un-traditional and required engineers and leaders with a flexible mind to do.  But also in terms of things like AdWords and AdSense -- which was invented independently multiple places at roughly the same time, but which was in some cases shouted down as "irrelevant distractions" by leaders without vision).

At the time there was no shortage of people who were ready to piss on what they were doing.  I can remember much of what was said about Google in the early years by people in the industry.  How pompous executives made fun of them because they didn't conform to the orthodoxy of the time -- which was portals.  One stop shops to capture and hold audiences.   Horribly ugly things filled with desperate salesmanship and terribly packaged content.

Everyone knew this was the way to do Internet services.

You can, as I mentioned earlier, make a lot of Benjamins by winning the Internet lottery and become an overnight hit.  With audiences numbering in the hundreds of millions, and sometimes billions, things that become popular can become very valuable.  But it is a lottery.  And we tend to focus only on the winners -- not the thousands upon thousands of losers.

And we do not focus at all on the substance these "winners" leave behind.

If you are a young developer I would advise you to seek out deep challenges.  Solve hard problems. Use hard science, or at least apply your education to practical problems where you might have an edge.  Sure, you should buy the occasional lottery ticket, and sure you can have some commercial success by timing Internet fads, but if you can, you should try to do some heavy lifting every now and then.

(1) The first time I heard about Sergey Brin was in 1997 when I read his paper on "Dynamic Itemset Counting": http://www2.cs.uregina.ca/~dbd/cs831/notes/itemsets/DIC.html . At the time a friend of mine was building an online book-store and I spent a weekend figuring out how he might implement efficient recommendations for he bookstore.  Of course, he never did, but at least I learned something by writing a people-who-bought-this-book-also-bought-these-books recommender based on what I learned from reading those papers.


4 comments:

  1. I really liked your post, it got me thinking!

    I agree with your base point that you have to solve a 'hard' problem that is 'important': far too much time is spent on shallow problems. Ultimately, if you want to do something compelling you also have to solve it in a way that makes a fundamental difference to your (prospective) user. Anyway, solving an easy problem isn't likely long-term interesting and you only have so much time on this earth so do interesting things!

    Where I slightly disagree is that when you've identified a 'hard' problem that you want to solve in a way that will 'make a difference to the user' - well at that point the speed of iteration and experimentation is needed so you can see how it works for users. So at least in that regard 'minimal viable product' and that methodology is useful - it's a baby versus bathwater thing!

    Steve

    ReplyDelete
  2. I agree that iterating without anything substantial or deep behind it is pedantry. Make something important and deep, and they will not come. That´s when you need to experiment and not complete the supertanker construction based on your initial hypothesis.

    I have spent quite some time the last months in what can be called the Oslo Startup Scene, and while there are some star startups and some activity seeking to pick the next stars, there is a lot of valueless noise happening. A surprisingly large industry of scrambling for the side effects of having a "Startup Scene". The "industry" surrounding the startups consisting of entrepreneurs and wannabe´s offering advice, help in seeking (early stage, preferrably government) funding, arranging events and offering office space is vibrant.

    Securing funding for the more serious rounds is very hard. Not "not all startups get funded"-hard. More like "there is no venture capital to compete for"-hard. The funnel seems to be on steroids, and only the rare cases with real, deep, hard problem solving have a shot at funding for scaling.

    ReplyDelete
  3. Steve: There are two kinds of "hard". There is the kind where you use a sledge-hammer to turn big rocks into gravel, and then there is the kind where you design and build a machine that can turn the rocks into gravel for you. Now, both approaches lend themselves to iteration, but you will probably not end up with a rock-crushing machine by starting with a sledge-hammer and then iterating since the mechanisms are completely different.

    Iteration is a discovery- and refinement process. And it is an absolutely necessary part of discovering how products and users interact.

    In the software industry you have the same. There are products that represent a lot of work and then there are products that represent very deep and specialized knowledge (and a lot of work). You can always get hold of people who can do what is, in practice, manual labor. But it is hard to find people to solve hard problems. This manifests itself quite visibly: products that involve solving hard problems are rare, while products that do not really solve any fundamentally hard problems are abundant.

    CAM software to compute toolpaths for CNC machines is a good example of a hard problem where even a minimal viable product requires you to solve problems that very few programmers are able to solve.
    Google search is an immensely hard problem. 15 years ago, web scale search was just hard. Now you have to spend billions just to be on par with the market leader. (Just look at how much Bing is costing - and I can guarantee that they have a lot of bright people working on it).
    Facebook is a hard problem due to its scale -- initially it was not a particularly hard problem.
    Ditto for Twitter -- it was trivial to prototype, but due to its scale it now represents a hard problem.

    There tends to be no shortage of the things that require little up-front effort to produce a MVP. There is an almost disastrous shortage of the things that start off with one or more fundamentally hard problems.

    I can`t remember which VC said this, but he essentially said that he never invests in what everyone else is doing precisely because if everyone else is doing it, it is probably too easy and you will be overrun by competitors that can do what you do marginally better.

    ReplyDelete
  4. This comment has been removed by a blog administrator.

    ReplyDelete