2009-07-29

On the Bing / Yahoo! deal.

I worked in the search engine industry for a little over 10 years. From 1999 until 2009.

Around 2003/2004 my feeling was that the web search market was no longer as open for new players as it used to be. In short: to build a competitive web search engine you would need to be extremely well funded. The amount of data you needed to index, the sheer amount of processing you would have to do, and the time-frames you would have to do that processing in, required huge capital expenditures as well as considerable amounts of cash to cover running costs.

In addition, although web search was becoming an area with more usable scientific papers being published (a luxury we did not have in 1999), you still needed that technological X-factor that defines successful web-scale companies. The X-factor you will never find anywhere but in companies that have to invent their own infrastructure to even have a slight hope of being able to do what they do at a reasonable cost.

Although I was working for Yahoo! at the time (and then later Google) I saw this high barrier to the search market as a huge problem. You really do not want to find yourself in a market where you either have no competitors or just one competitor.

On one hand, monopoly or near-monopoly, removes the sense of urgency and incentive to push hard for improvement. Given that the web is still exploding in size and complexity, the problem of web search is only getting harder. We need continual improvement just to remain where we are.

On the other hand, you get all these hassles with governments poking their fingers where they don't belong. You are no longer free to do as you please and it has very direct consequences that are noticable all the way down to the lowliest engineer.

It saddens me if Yahoo! are throwing in the towel for good. Yahoo! has a lot of raw talent and it is somewhat puzzling that they have not been able to deliver on web search after so many years of trying. I hope that this represents a more temporary deal and that Yahoo! will spend their time working on their re-entry into the business of building a web search engine.

It would be dreadful to have only two web search engines. For everyone.

No comments:

Post a Comment