2009-07-29

On the Bing / Yahoo! deal.

I worked in the search engine industry for a little over 10 years. From 1999 until 2009.

Around 2003/2004 my feeling was that the web search market was no longer as open for new players as it used to be. In short: to build a competitive web search engine you would need to be extremely well funded. The amount of data you needed to index, the sheer amount of processing you would have to do, and the time-frames you would have to do that processing in, required huge capital expenditures as well as considerable amounts of cash to cover running costs.

In addition, although web search was becoming an area with more usable scientific papers being published (a luxury we did not have in 1999), you still needed that technological X-factor that defines successful web-scale companies. The X-factor you will never find anywhere but in companies that have to invent their own infrastructure to even have a slight hope of being able to do what they do at a reasonable cost.

Although I was working for Yahoo! at the time (and then later Google) I saw this high barrier to the search market as a huge problem. You really do not want to find yourself in a market where you either have no competitors or just one competitor.

On one hand, monopoly or near-monopoly, removes the sense of urgency and incentive to push hard for improvement. Given that the web is still exploding in size and complexity, the problem of web search is only getting harder. We need continual improvement just to remain where we are.

On the other hand, you get all these hassles with governments poking their fingers where they don't belong. You are no longer free to do as you please and it has very direct consequences that are noticable all the way down to the lowliest engineer.

It saddens me if Yahoo! are throwing in the towel for good. Yahoo! has a lot of raw talent and it is somewhat puzzling that they have not been able to deliver on web search after so many years of trying. I hope that this represents a more temporary deal and that Yahoo! will spend their time working on their re-entry into the business of building a web search engine.

It would be dreadful to have only two web search engines. For everyone.

Learning from user behaviors.

Years ago I visited a university campus and was struck by how conveniently the paved footpaths were arranged. The paths seemed to follow a very natural route between buildings -- the sort of route one would take if there had been no paved paths to enforce some landscaping architect's idea of how to get from one building to another.

Also, the paths even had features that seemed directly aimed at encouraging the use of roller-blades and skateboards; which is rather unusual given the institutional hostility usually displayed towards any and all boisterous behavior in public spaces.

I was told that not only was this the result of a deliberate effort, but that the process at which the architects arrived at the precise arrangement was rather unusual. After the buildings had been erected the construction of the footpaths were put off; only to be finalized and eventually paved once paths had formed on the lawns between buildings.

In other words: the landscape architects had made a conscious decision to observe how people wanted to move between buildings and then turned those paths into paved paths.

Figuring out what users want is hard. Mostly because users have no idea what they want. We are terribly bad at articulating what we want and often when we try, we get it wrong. An oft cited example of this is if you ask people what sort of coffee they prefer. Most of us will use words like "strong" and "dark" -- yet tests show that most of us prefer nothing of the sort. We prefer weak and mild coffee. It takes a lot of experience and discipline to correctly identify and articulate what you want.

There are numerous research companies that attempt to figure out what the consumer wants. At first glance their methodologies often look sound, yet time and time again we are reminded that much work done in this area is relatively fruitless. I won't pretend to understand much of what they are doing, but after reading some literature on the topic, listening to talks given by numerous innovators, and attempting to observe my own behavior when choosing products, I have this gut feeling that much of what they do is merely identifying "local maxima".

In the algorithm world what these companies often do reminds me of Hill climbing:
Hill climbing can be used to solve problems that have many solutions, some of which are better than others. It starts with a random (potentially poor) solution, and iteratively makes small changes to the solution, each time improving it a little. When the algorithm cannot see any improvement anymore, it terminates. Ideally, at that point the current solution is close to optimal, but it is not guaranteed that hill climbing will ever come close to the optimal solution.
If you squint, a lot of activities that go into refining products and services seem to behave like hill climbing. The problem with this approach is that it often misses maxima that are distant or that the effort is not conducted in parameter-spaces that matter the most.

It is striking how the content industry seems so spectacularly incapable of observing the paths that have formed where their product is consumed -- largely outside their control. Not only does the media industry fail to make note of why these paths form, or how, but they expend considerable effort in the provably futile effort of herding their consumers back into the pen.

It is tempting to conclude that the content industry is more obsessed with having consumers access their product on their terms than they are obsessed with actually earning money.

One prime example of this is the sheer dumbness of regional licensing.

There exists various rationale for regional licensing. For instance it may be impractical to release the product at the same time in all markets due to adaptations that need to be made for each market; the germans usually dub all their movies, norwegian releases are subtitled, and in the US you have to make sure that under no circumstance are minors exposed to even partial nudity or even mildly colorful language.

Additionally a lot of attention is paid to preserving outdated licensing arrangements that prevent the same content from being distributed in other ways than the rights holder is capable of. This is the case for Formula One, meaning that most Formula One fans in Norway either have to shell out serious money for marginally interesting, yet expensive cable plans just to watch F1 or download the content from unauthorized distributors on the Internet. Legal access to reasonably priced F1 coverage on the Internet does not exist in Norway because the rights holders either were too cheap to buy the additional rights or they were incapable of offering the service.

The prime motivator behind illegal distribution of content on the Internet is convencience. Those who control the rights to content are not doing a very good job of reaching their markets. Occasionally you will see them try; with much fanfare and self-congratulatory announcements, but for the most part these attempts end up being feeble efforts that fail to deliver the convenience consumers wish.

Meanwhile the consumers are creating their own paths.

Years ago, getting your hands on a movie or a TV show on the Internet required some work and you needed to have friends in dark places. Today it is fairly easy for unsophisticated consumers to do the same. Not only that, but the effort needed is radically diminishing.

In a couple of weekends most people moderately adept at searching the web would probably succeed in setting up an automated system that acquires the content they want automatically and makes it available in their media library -- ready for playback on their device of choice at any time.

The reason people do this is not because they are evil or because they are solely motivated by getting stuff for free, but because it is a much better user experience. It is more conventient.

iTunes offers much of this already. The problem is that since I live in Norway the rights holders are more keen on slamming on the brakes, insisting on pointless, unintelligent litigation and show very little interest in removing the obstacles for doing business with me. If I could get TV shows and movies through iTunes I would.

I think the content industry needs to work a lot harder to adapt to the changing times. The archaic models they cling to do not work and it is infuriating that they insist on behaving in this destructive, and not least self-destructive manner.

The content industry needs to stop focusing on litigation and start paying attention to the paths that have been built by the consumers. They need to make an effort to understand why these paths form and be more ready to accept the fact that their notions of how business is done may be severely outdated. They have to stop thinking that it is about getting things for free. It isn't. The focus on getting stuff for free is only a by-product of the content owners refusal to provide convenience and reasonable terms. However, by encouraging and incentivizing the consumer to find their own paths while being largely unable to offer consumers reasonable service, they have indeed created a whole generation of people who are used to getting stuff for free.

Content owners need to learn from the very consumers they hate. The ones who have created the paths that actually work and provide the level of convenience we all want.

2009-07-18

Digital distribution woes.

According to David Pogue it would appear that Amazon went to the drastic measures of deleting books off of their customers eBook readers.

Upon further investigation the story does not seem as cut and dried as in Pogue's article. It would appear that there are ways to publish eBooks through Amazon without actually having the rights to do so and that this has happened before.

Still, I think this was handled very badly.

In my mind this is exactly the sort of underhanded stunt that greatly reduces the credibility of Amazon as a company, the Kindle as a product and the publishers as entities with whom one should bother doing business. If I buy a book, for a moment ignoring the legal fine print of what rights I actually gain in doing so, I have certain expectations. One of them being that the book does not simply vanish once I have paid for it and it is, for all practical purposes, in my posession. If I have bought and paid for a book in good faith from a vendor I have no reason to be suspicious of, it would seem that the right thing to do is to leave me alone.

At the very least, this is a very rude thing to do. It is probably legal though; very few people know what terms they are accepting when using a service because the terms are almost always unreasonable both in scope, intent and form. But they are the terms. Take them or leave them.

I am sure Amazon did not like having to do this. They obviously are not stupid enough to think that this would not reflect badly on them. But it is their own fault. First for selling an illegitimate product and second for being foolish enough to make it technically possible to delete content from devices without the explicit consent of the owner.

(One can envision an attack on Amazon's reputation where some party injects large amounts of illegitimate books, sneaking them past whatever controls Amazon have, thereby forcing Amazon to act in this unpopular manner.)

I think the lessons are:
  1. Don't believe everything you read. David Pogue seems to have reported a somewhat narrow view of what has transpired.
  2. Digital distribution of content is far from being mature enough not to offer up nasty surprises even from reputable parties. Expect more pain and more sillyness in the next decade too.

2009-07-16

The KLM website. Still rubbish after all these years.

A few years ago I got angry enough about the KLM website to actually sit down and write them a letter. In these times nobody seems to bother inscribing flattened trees with strings of symbols as a means for communication anymore, but I did indeed send them a proper letter. On paper.

The thinking was that since they so obviously have not understood this web thing, it would be necessary to communicate with them by means of more traditional technology -- in line with the somewhat crusty and stale impression they project if you have the misfortune of being exposed to their woeful "web presence".

Well, to cut a long story short: they didn't respond. It would appear that nobody is minding the store. Perhaps I should have used papyrus or stone tablets instead.

Today I had the silly idea that I should log in and check my frequent flyer miles, after KLM sent me one of their garish, pushy marketing emails. Of course, when I clicked a link in the email I was greeted with a HTTP 500 type error message indicating that some permanent-looking internal error had occurred. How professional of them to send me an email with links in it that do not work.

I was eventually able to dig up my Flying Blue member number, but what my password was set to is anyone's guess. Rather than offer a sensible way to recover passwords they insist on being difficult. I had to answer a question which I can't remember the answer for anymore, plus feed the interface an email address. As if I remember which email address i used when I registered, years ago.

After three guesses I was told that my account was disabled for 24 hours.

How come KLM, who are demonstrably free of talent when it comes to designing web sites insist on this "added security"? It does not add any security. It merely makes an already miserable experience even more miserable.

I know the airlines are experiencing hard times, but if anything this is precisely the reason why you want to set yourself apart from the competition by having a competently designed and implemented web site.

Come on KLM. If you wonder why I have not booked my vacation through you it is because your stupid website keeps getting in the way -- and to be quite honest, my life has enough annoyances for me to bother dicking around with your worthless website when there are other companies that are more than happy to take my money.

2009-07-12

McBook Pro

For the next edition of the MacBook pro I wish Apple would start thinking in terms of ergonomics. The front endge of the thing is not particularly user friendly. The edge digs into my wrists if I have the thing on my lap.

Also putting a glossy screen on it was a horrible idea. I spent some time outside in the sun today; writing on my new MacBook Pro. For much of the time I was just typing away, not being able to see what I was typing.

I wonder if there is some sort of matte film you can attach to the screen in order to make it usable in the sun?