2008-01-15

Ancient history, Part 3: more on the origins of FAST Search

Back in the day when FAST was a startup, there was a lot of talk in the press about the origins and nature of our technology.

Most of what was presented as "fact" in the press was, at best, misleading and often just wrong. You can't really blame anyone for this. Whenever people from our management talked to the press, the journalists would invariably get the details wrong or they'd take something out of context and run with it as far as possible. Or worse, journalists would try to distill facts by skimming inaccurate articles by other journalists, thus accumulating the number of bogus claims.

One myth I heard quite often was that Web Search was based on FTPSearch. That we had somehow taken the original FTPSearch code and "adapted" it for use on the web. This, of course, is nonsense.

Web search and FTPSearch were in fact two completely separate and un-related code-bases.
At the time of writing even the wikipedia entry for Alltheweb has gotten this wrong.

FTPSearch, which was an independent implementation of the Archie search system, was originally written by Hugo Gunnarsen while he was doing his masters thesis at NTH (or NTNU as it is called today). The project grew out of a discussion Hugo had with Anders Christensen (always a great source of inspiration at PVV) about what one might do with the MS160 search chip that Professor Arne Halaas had done. The original FTPSearch ran on a 50Mhz 486 running Linux 0.99 and was developed during the fall of 1993. Tegge acted as mentor for Hugo while working on the project although he wasn't very involved in the project itself at that point.

The first time I heard of the project was when I almost destroyed the ISA board with the search chip, by accidentally dropping a large pizza on top of it. Hugo was not amused, but still kind enough to explain what the chip could do. All while suspiciously eyeing my pizza as if he expected me to drop it on his thesis project. Again. (Hugo insists I spilled coke in his keyboard rather than drop a pizza on the board while it was lying around the coffee table, but I only spilled coke in a keyboard once at PVV; and that was the indestructible keyboard of the IBM RS/6000).

After Hugo got his degree he left NTH and started working for SGI in january 1994. Hugo later joined FAST in 1999 and now works for Yahoo!

The project was taken over by Tor Egge -- "Tegge". Tegge ported FTPSearch from Linux to NetBSD some time in 1994-1995 and the project grew on. An early casualty of the port to NetBSD was the support for the MS160 chip. According to Hugo, Tegge never ported the MS160 code to NetBSD and thus the search chip was dropped. The main reason being that the new machine (a Pentium, insert "ooh"-noises) would run searches significantly faster without using the MS160.
In addition to improving the service, he added more features and grew the index to cover most ftp servers worth indexing. One of the neat things he added was proper regular expressions.

During the same time Stig Bakken developed a web interface to FTPSearch in Perl. Up until then, FTPSearch had only been available to users of special search clients that used the Prospero protocol to talk to Archie servers. With the increasing adoption of the World Wide Web, Stig figured that a web interface would be useful. The first interface was pretty crude, but he kept improving it until it was a lot easier to use than an Archie client. This interface was eventually replaced by a web interface written in C by Tegge.

The IT department at NTNU graciously let Tegge host the service at the university. It consumed quite a bit of bandwidth since it was a popular service on the net. In fact, the web interface for FTPSearch was the most popular web site in Norway for quite a while -- something the corporate types in the budding Internet industry in Norway disliked intensely as it wasn't what they thought of as a "proper website". They'd routinely avoid mentioning FTPSearch when talking about the most popular websites in Norway -- even though it was seeing a lot more traffic than the rest of the sites. This was the source of much merriment.

I don't think the IT department at NTNU has ever been properly credited for lending a hand. Especially Odd Arne Haugen and Jan Ole Waagen. Not only did FTPSearch get a box to run on and much needed bandwidth to serve up searches, but they also let Tegge use "Storm" for indexing the data. "Storm" was a four-CPU SGI machine with 1Gb of memory, which was an almost obscene amount of RAM back in those days. The IT department didn't even kick the project out when Tegge managed to completely hose the "unbreakable" journaling filesystem on the machine during an index build.

Eventually Fast Search and Transfer ASA bought the rights to FTPSearch. However, the FTPSearch code had already been released as open source, so long after FAST bought the rights, you could still download the source code of an older version, compile it and run it.

In fact, the source is still available from:

http://ftpsearch.archive.sunet.se:8000/search-info/software.html


As mentioned earlier, Web Search was a completely different code base. I can understand how some people might be tempted to conclude that it was the same code, but it wasn't.

The first version of Web Search were the fruits of Knut Magne Risvik's Masters thesis work. For all practical purposes, it was a research prototype and not so much a finished product. However, Knut Magne wanted to see if it could be taken further -- if it could form the basis of a large scale search engine. Knut Magne interrogated Tegge on the subject, and Tegge was only "mildly sceptical to the whole idea" -- which, as these things go, is almost equivalent to a golden seal of approval. A ten page memo on the feasability of building Alltheweb was hammered out by Knut Magne and sent off to Espen Brodin, who then got Robert Keith to cut us a check in order to make it so.

(Although it wasn't called "Alltheweb" back then. If you can believe it, the internal name was even more awkward: "TheWorldsBiggestSearchEngine" or TWBSE for short. Full marks for ambition -- a smack in the back of the head for the ghastly name)

The next revision of the search engine grew out of the combined efforts of Knut Magne and Tegge. As the project, the company and the workforce grew, more and more people were involved in developing the search engine, but Tor Egge and Knut Magne continued to play a key role in its development.

Our first real customer was Lycos. We provided the search services for their portal. Our search results were presented, dressed up in drag, on their portal pages. Portals were all the rage at the time. All the rubbish any normal person can't possibly be bothered to look at all crammed into one site. The density of STUFF on these pages was obscene. Every square millimetre was full of blinky stuff, links, advertising and even more links.

But hey, everyone was doing it that way, so they could be no better.

After getting the regular web search up and running we started to work on Image Search. In every sense of the word an interesting project. I'm no longer sure if this was something Lycos pushed for or something our people said we could do (or more likely, practically had in the can, ready to ship), but all of a sudden we had to figure out how to do it.

A lot of interesting pieces of code got written to do this. For one, libraries for encoding and decoding image formats were not really written by people who knew a whole lot about writing robust software, so the libraries we used had a tendency to crash. I didn't envy the guys who had to deal with this.

Also I vividly remember a small HTML parser someone wrote inside Finn Arne's original HTML parser, which was able to parse HTML backwards. The idea was to locate IMG elements in the HTML code and then extract N characters of context that would have been rendered as text in front of and behind the IMG tag. I can still remember Finn Arne pursing his lips and shaking his head in disgust. I guess he felt somewhat violated for having his neat HTML parser defiled in this manner.

Another last minute hack was thumbnail serving system (when you do a search on any search engine today that has image search, it will show the hits in the form of a grid of thumbnails). Earlier on I had written an Apache module for accessing crawled pages in our crawler storage. Our first crawler stored one web page per file, but even in the early, limited size document crawls, we soon discovered that this wouldn't work. You'd end up with an awful lot of files, and thus an awful lot of random accesses on the disk during processing. So we added our own, rather simple, storage format which allowed us to to processing (eg. indexing) more efficiently -- by exploiting the fact that sequential access over lots of data is generally quite fast. In essence we stored content inside very primitive miniature filesystems on top of the UNIX filesystem. If course, this meant that you couldn't access them without using the library we made handling this file format, so to make things easier I wrote an Apache module to provide browser access to all the content we had crawled.

One day, Rolf popped his head in my office and asked how much trouble it would be to serve images through the Apache module. Not a lot really. I just had to add something to set the correct content type and it would work. It wouldn't be fast, but it would do the job. We could always replace it with a more suitable system later...

I think it took two or three years before we judged it to be enough of a pain to actually replace it with a proper thumbnail serving system for Image Search. Which, to be quite honest, was a lot longer than I had thought it would last.


For years there was much talk of how the FAST search technology was a hybrid solution -- a search engine powered by special search chips. There was even rumors that we were using polymer memory technology from Opticom -- originally FASTs parent company. Entertaining rumors for sure, and I can only imagine how they messed with the minds of the competition. As entertaining as they were, there was no truth to these rumors. The last time a search chip was used in any of the services was in FTPSearch -- and Tegge stopped using it long before FAST entered the scene.

FAST was actully developing a search chip, but it was never used to power any of our search engines. As far as I know we intended to use it when it was finished, but we never did. In 2002 the department working on the chip was spun off into a separate company (Interagon).

2008-01-14

Ancient history, Part 2: using open source.

The first time it dawned on me how oddly mistaken most people were
about what sort of tools and technologies we used to develop Web
search systems in FAST back in the early days, was when Doc Searls
told me that he thought we were running everything on Windows NT (I
think this must have been back in late 1999 or early 2000).

I have to admit I was a bit shocked.

To me, it was odd that someone would even assume that we ran
everything off Windows NT, so told him a bit about the sort of systems
and tools we used; that we used FreeBSD for our servers, that most of
us at the time were running Linux or FreeBSD as a desktop operating
system. A few ran Windows, and I think one guy used a Mac. But most
developers ran Linux or FreeBSD.

Our entire code-base was built using the GNU tool-chain and various
open source languages and systems.

In all fairness, eventually some commercial tools and libraries were
used. The first commercial development tool I can remember us having
was a system for finding memory leaks. I also remember that we had
some libraries to handle parsing text in some languages. But by and
large, everything was built using open source tools.

The first thing people think about when they hear "Free Software" is
"free" as in "doesn't cost anything". Of course, our management was
very happy that we didn't need to spend money licensing operating
systems, compilers, editors and whatnot -- but that wasn't really why
we used Open Source systems. We used them because they were much
better suited to solve the sort of problems we needed to solve.

First of all, we were intimately familiar with these systems. Most of
us had been using them for years and when you are going to do
something that is a bit hard, you tend to stick to the tools you know
well and trust.

Second, we didn't really have the time to deal with closed source
vendors. If something broke, we needed to understand the problem and
come up with a solution. If you have access to the source and you
have competent developers on-staff, fixing show-stopper issues is a
lot easier than when you have to rely on a third party to make time
for you and then perhaps come up with a solution in a week or two.

We had competent people on-staff and we did fix our own problems. When
we ran into problems in the OS we had at least two or three people who
could look into it. Several of them used to contribute fixes to
FreeBSD, Linux etc., so whatever problems we came across and fixed,
the rest of the world would benefit from as well. It was the same for
innumerable other systems and tools.

Some people also took part more directly in open source projects. For
instance, my long-time colleague Stig took part in the PHP project as
a member of the core team for several years. Since we made heavy use
of PHP for a lot of front-end work, having him working on PHP and
being part of the PHP project was of great benefit to us.

This active use of open source tools and participation in open source
was by no means unique to FAST. There were many companies that had a
very intimate relationship to open source, and today their numbers are
even greater. In fact, all companies I have worked for since (with one
notable exception) have been big believers in open source. Look at
any of the great Internet brands and chances are they are using,
and/or contributing to open source.

I am not sure how people come to believe that open source tools are
somehow not suited for developing large scale, cutting edge systems.
My experience of this was very different. The sort of systems we
built were not Enterprise-scale, they were Web-scale.

Web-scale systems are very different from Enterprise-scale in that
they start off having to handle traffic and data a few orders of a
magnitude larger than enterprise systems, and then have to handle
exponential growth from there on. If you can't design a feature in a
way that'll scale for that sort of growth, you can just forget about
it. If the feature is really important, you have to realize it in a
way that allows it to scale. If that means developing some custom
technology for dealing with it: so be it.

I often see people complain about how some huge web service is missing
some feature and that implementing it would be "trivial". Often this
simply isn't true, but it is an easy mistake to make when you're not
familiar with systems that need to scale really well and where every
millisecond counts in response times.

In enterprise-scale systems you can afford greater feature richness.
You can afford to use traditional systems like databases for data
storage, and really heavy, feature rich frameworks. Your users are
numbered in the tens of thousands -- not in tens of millions, or some
extreme cases, hundreds of millions. The growth rates of your data
are more predictable.

This is why some systems designed for web scale won't necessarily work
for the enterprise, and vice versa.

For most people "enterprise" systems means "really large" -- but the
enterprise concept of "really large" is different from that of "web
scale". As are the expectations for features offered. This doesn't
mean that one is harder than the other, but it does mean that the ways
you set about working on such problems are often fundamentally
different.


So let's summarize some main points here.

Open Source plays a big role in the Internet industry. Serious people
create serious systems using Open Source.

It is not only possible to create large, cutting edge systems with
nothing but open source tools, but it has been done many times and it
continues to be done today. That isn't going to change any time soon.

Open source gives you more control. When problems arise, you can
address them. This is far more valuable than any support contract you
can get for a closed source system. It also insulates you from the
misfortunes and follies of other companies -- if the vendor for a
critical closed source component you rely on goes out of business, you
are toast. The worst that can happen to an open source project is
that people lose interest in it and stop contributing to it. You'll
still have the source and you'll still have people willing to work on
it for money.

The most important resource you need in order to innovate and create,
is good people. If you want to create the next big thing on the net,
you start with a small team of good developers. People who are not
afraid to dive all the way down and build things from the ground up if
need be.

2008-01-13

Ancient history, Part 1: early years.

The other day Microsoft announced they were going to acquire Fast
Search and Transfer ASA -- a company that I used to work for.
Although I was hardly surprised by this turn of events I would
probably never have guessed it just 12 years ago. Or 16 years ago,
when this adventure actually started. I figured that perhaps this would be
a good time to share some stories, so I decided to write about some of the
things leading up to Trondheim becoming associated with search technology.


About 12 years ago, me and three other friends founded a company. We
were young, single and what we lacked in experience and business
sense, we made up for with pure hubris and a firm conviction that all
problems are solvable.

Most of us originally met at what is today NTNU in 1992. Back then it
was called NTH and people who graduated before the re-organization
tend to make a point of that. Three of us were students and one just
hung around the UNIX machines on campus long enough for someone to
finally put him to work.

At the centre of our universe in those days was PVV -- a computer club
that mainly attracted people interested in playing around with UNIX
computers. PVV had a bit of a reputation for being detrimental to
your academic progress. A disproportionately high number of the
regulars at PVV had...unconventional academic records. From people
who spent several years extra in getting their degrees to people who
simply got bored with academia and dropped out to pursue different
avenues. There was even a member who managed to screw up the process
of printing student records one year because the system had not
anticipated that anyone would fail a class more than N times (where
N was some "reasonable" number).

The more well kempt students would often frown if PVV came up in
conversation.

If you were interested in UNIX, it was the place to be. We had lots
of different UNIX platforms to play with -- a few of them so obscure
that it is very unlikely you've heard about them. Most serious
vendors would make sure we had at least one of their machines. You
could get your hands dirty playing sysadmin on them, you could write
software or you could just kill time there. If you were hiring a UNIX
sysadmin in those days, THAT was the place to go. That was where
you'd find people who had been exposed to up to a dozen different UNIX
flavors and varieties.

But what was probably more important were the people. At the time I
became a member of PVV there was a mix of really great people there.
People you could learn from. People you could have discussions
with. It was during these years (1992-1993) I think the free software
mindset started to become clear to me -- the why's and how's. This
was well before the term "open source" was coined, but the Gnu project
and the Free Software Foundation were around at this time. A lot of
people at PVV had deep admiration for the idea, although I think even
back then that Richard Stallmann was seen as a bit of a nutcase by
some -- deeply respected, but still viewed as bit of an oddball.

I had many fruitful discussions with people at PVV about not only
computer science, networking and UNIX, but also about Free Software,
the legal complexities of computing, politics as well as ethics. I
think everyone who was somewhat involved with PVV at that time became
infused with a set of values and exposed to ideas that were
later to become very important when the Open Source movement came
creeping out of the woodwork and into the mainstream.

In the original sense of the term "hacker", PVV was a thriving hacker
culture.

One key aspect of the hacker culture was a can-do attitude. If a
piece of your system was misbehaving or not working as you wanted to,
it was not all that far-fetched an idea to simply rip the system apart
and have a look inside. Today, this is an almost alien concept for
most -- as it was back then. Software is something people buy, and
when it doesn't work, they call support and then, maybe, after much
yelling, things get fixed. If not, someone gets sued, reimbursed,
apologized to or throws a computer out of a window. Most of the time,
things do not get fixed. Or it ends up taking forever and costing
truckloads of money.

A good example of this culture was a friend of mine: Tor Egge, or
"Tegge" (since most of us referred to each other by our usernames).
He discovered some bug in one of the operating systems we were
running. His way of solving it was to write a disassembler, disasseble
the kernel binary, read through the assembly code, find the bug, fix
it, run the thing through the assembler to build a new binary and test
it.

The fix worked.

If memory serves, he also did some experiments with the C-compiler,
feeding it code and looking at the assembly generated by it. He then
used this knowledge along with the debug information embedded in the
kernel to reconstruct the original C code that contained the bug, as
well as what the correct code should be. Someone said he sent in a
patch to the vendor and got it accepted without any questions asked,
but I'd have to ask Tegge if this is accurate.

Another friend of mine, Arne, did similar things. We had an early SGI
system that for some reason had ended up with a corrupted root
filesystem. The machine would not boot. So Arne used the ROM editor,
loaded in blocks from the filesystem into memory and had a long, hard
stare at them. As far as I know he didn't have any documentation on
the exact binary layout of the file system, but armed with some
general knowledge on how UNIX filesystems are organized, he edited the
blocks to correct the errors, wrote them back to disk and rebooted the
machine successfully.

When surrounded by people like that you tend to learn from them and
you slowly start to realize that there is nothing mystical about
computers and software and that you CAN fix things. Most things
require a lot less knowledge and effort than in the above examples
though.


It was here I first got to know the people I later would start a
company with. After a breakup I found myself single and with a lot of
time on my hands. A fellow PVV member (Stig, or "ssb" as he was known
to most people at PVV) found himself in the same situation, so he
moved out of the appartment he used to share with his former
girlfriend and camped out in my livingroom.

The place was a mess. We spent most nights partying or discussing
starting a company. Usually we did both. Indeed, the name of the
company came to me while on the dance floor at a club nearby, and with
the name the ideas started so solidify.

The next important point on the agenda was to find some more people to
start the company with. I don't remember if there was a list of
candidates, but I do remember that we gave this a lot of thought and
probably pissed off some people who felt that they should have been a
part of it but weren't asked. Finally we settled on Finn Arne
(finnag) and Alexander (astor). Both good programmers, both solid
UNIX people and last but not least: they were people that were fun to
be around -- which is important when you're starting a company.
Another aspect I think was important was that we complemented each
other really well.

Getting Finn Arne on board was easy. We just dropped by PVV, pitched
the idea and he was in. Alexander took a bit more work. Not because
he was unwilling, but because he was nowhere to be found. He had just
dropped off the face of the earth and vanished. We emailed him, tried
to call him, asked people he used to hang out with. Nothing. In the
end we figured that we'd give it a week and if he didn't turn up, we'd
just go ahead with starting the company and then catch up with him
later.

He eventually turned up, surprised that we had made such an effort
looking for him. He'd been at home in his studio appartment. With
his new stereo. Of course, none of us knew where he lived, so we
hadn't looked there. It turned out that he had been spending some
time at home, listening to his new stereo. When presented with the
idea of forming a company he promptly sold the stereo to free up
enough cash to chip in his part of the needed capital.

And then we were four.

Guardian Networks AS, as the company was called, was an odd sort of
company. Our original business plan, and I cringe when I call it
that, was to produce a hardened Linux for use in firewalls. The idea
was that Stig and I take on various consulting jobs to fund the
development and that Finn Arne and Alexander worked on the Linux
kernel as well as put together a minimal software distribution. To
make a long story short, there was not a lot of demand for our product
back in 1996. In fact it was pretty close to zero. Also, although a
lot of our ideas on how to make an OS suitable for firewalls looked
good on paper, they turned out to be rather impractical. The idea was
that an intruder should be able to gain root access on the system and
still not be able to do much damage. Configuration changes required
the machine to be shut down, booted up with a kernel that had no
networking, you'd change the config and then take the machine back up
again. As I found out when I ran the OS on an IRC server in a
datacenter I didn't have easy physical access to, this was a royal
pain in the ass.

Fortunately we managed to build a reputation as a sort of consulting
company. Most of our customers were people who had UNIX systems and
wanted them whipped into shape. Companies like HP would rent out
"sysadmin" types at exorbitant hourly fees to do upgrades, installs
and maintainance. Supremely incompetent people who had absolutely no
idea what they were doing. Poaching customers from such companies was
easy. We were cheaper, faster and when we left the site, everything
would work and the customer would also have generous helpings of free
software installed.

It may be hard for people these days to imagine how terribly bad a
typical default vendor installed UNIX was back in those days, given
the luxury of today's obscenely rich Linux environments we enjoy
today, but believe me, a UNIX box was usually something you'd get to
run just one or two special applications. Other than that it didn't
even have a proper command shell.

We also did a lot of odd software development projects. Projects for
which there didn't seem to be any natural choice in the more
traditional software-developers-for-hire shops. On some we learned
the hard way exactly how inexperienced we were. On other projects we
did surprisingly well. We did stuff for a bank, for various
engineering companies, for the university and for a long list of
companies I have since forgotten about. We worked on a really, really
broad range of problems.

In 1998 we were approached by a company called Fast Search and
Transfer. I had heard about FAST -- they had bought the rights to
FTPSearch, which had been developed mainly by Hugo Gunnarsen and then
Tor Egge, when Hugo left for a job SGI. Stig had even written a web
frontend for it at one point and I had played around with the code a
bit to port it (back) to Linux (why, I can't remember). We knew two
of the people working for them: Knut Magne Risvik and Tor Egge. Knut
Magne used to work at the user helpdesk at the university while
studying and was part of the helpdesk/sysadmin crowd at NTNU (most
UNIX sysadmins were actually students at NTNU).


Fast was going to build a web search engine, but they needed a
crawler. So they came to us. Foolishly assuming that it would be a
piece of cake, we set about doing it. It couldn't possibly be as hard
as writing the indexing and searching software, right?

Finn Arne did most of the design and implementation on the first
version of the crawler. The basic design and implementations were
simple, tidy and quite sound. I liked it a lot. The thing was that
Finn Arne had had very little exposure to web software. While he was
good at writing very efficient networking and IO software, he had
never bothered with HTML. So for instance, when he asked me which
elements would contain URLs I told him. All of them. Including URLs
in FORM elements. It didn't occur to me why he was asking (he was
writing the link extraction code).

Hilarity ensued when we unleashed the beast onto the web and people
started getting empty submissions to their guestbooks. The crawler
was following the ACTION URLs in the FORM tags. You only needed to
type in "braindamage" and "fastcrawler" into Dejanews to survey the
damage.

We fixed that one quickly.

Developing a web crawler turned out to be a much harder problem than
we had thought. Not only does it have to be capable of handling vast
amounts of information fast, but it traverses the web -- a universe
filled with the craziness and laziness of a million people who create
all sorts of complexities. From an almost incredible inability on
part of web server implementors to get even date formats correctly (I
think I counted 19 broken formats that existed in web servers), via
bad configuration that would look like an infinite number of web pages
on a site, to whatever curve-balls the users would throw at us in
terms of unwieldly and hard to process contents.

Somewhere along the line FAST acquired Guardian Networks AS, and we
became part of FAST. I was actually quite happy about this. I was
seriously tired of the annoyances of running a company. I made a note
about never starting a company without having at least one dedicated
"suit" on board.

2008-01-10

FAST, Google, Yahoo and Microsoft.

Recently Microsoft announced they would be buying FAST. This made me think about the past 10-15 years here in Trondheim. What started with a few geeky hackers and then ended up with people in Google, Yahoo and Microsoft (provided the deal closes). I doubt anyone would have guessed back in 1993 that we'd all be in roughly the same line of business, working for big name companies.

It is kinda funny that most of my old friends and colleagues all ended up in Google, Microsoft and Yahoo!.

I started scribbling down some notes on ancient history yesterday. Not sure when or where (or if) I'll publish them, but it was fun to remember how things came about.

2008-01-07

A flickr wish

I tend to browse Flickr on a daily basis to see if people in my contact list have added any interesting shots. For me, Flickr is in part about learning photography and being inspired by the pictures of others, and in part about keeping in touch with people -- like a blog, without all the boring verbiage (hey, I'm a simple person. I like to look at the pictures :-)).

Something that really annoys me is all the tacky "award" icons that people put in the comments. It just takes up a lot of space and looks....well, tacky and cheap.

How if Flickr added some special functionality for this? For one so the people who like to spam other people's comments with these icons can get a more convenient mechanism for it. And second, but most important, so I can choose to filter it away when I am reading the comments to pictures I am actually interested in. Or at least have it formatted in a more visually pleasing manner.

2008-01-04

Pixelmator: promising start, but that's about it.

The other day I stumbled across a program called Pixelmator. I played around with it a bit in order to try to perform some of the tasks I usually need to perform on photos I've shot.

The first thing I noticed was that it doesn't read RAW files. OK, no big deal, I can make Bibble spit out something that Pixelmator can read. Then I tried to do some standard color tweaking; fiddle around with levels, contrast, saturation etc. Even before I had gotten to dig deeper into the app, things ground to a halt.

No, I mean the thing literally just ground to a halt.

The thing spins the dreaded rainbow mini-pizza (for those of you not using OSX: the "I'm busy" cursor) constantly and even things that should be quick and simple, like drawing a gradient or opening the "levels" dialog box, take forever.

Huh!? Why the hell is it doing that?

The short verdict: not useful software. Yet. But I'd check back in 6 months to a year.

However, if someone had labeled it an alpha version, called i a work-in-progress I would have had completely different expectations. As an alpha version of an image editing program I would have said that it looks promising, but it still needs significant amounts of work. Heck, if the makers had asked for donations instead of selling the software I would have chipped in because I would have had more reason to believe that eventually there will be a decent, working product.

Selling this as a finished product just makes me suspicious.

2008-01-01

Why using Wikipedia in schools is a good idea.

In 2007 I read quite a few articles on the topic of libraries and educational institutions banning the use of Wikipedia. Many of them on the grounds that Wikipedia is not usable for academic work because it contains many inaccuracies and much false information.

I don't think people who contribute to or work on Wikipedia would deny that; much information in Wikipedia is in fact inaccurate, and there are people who actually go out of their way to inject falsehoods into it.

So what? Many encyclopedias are no different. Especially those that were made with some political or religious agenda. For instance, in eight grade I remember my teacher giving my class an assignment where we had to have a look at a handful of topics in three different encyclopedias. This was a real eye-opener as at least one of the encyclopedias wanted to promote an alternate view of the world. It was a politically slanted encyclopedia filled with value judgements that, at least I, found insulting to the reader's intelligence.

And this is in the presence of an agenda. There are still what I would refer to as "honest errors" in more traditional encyclopedias and textbooks.

I think it is critical for students to understand this: that all sources of information need to be evaluated. That books, web-sites and people are fallible. In order to arrive at a probable truth you need to investigate claims and analyze sources. In this sense, Wikipedia is the perfect teaching tool. Not only does it contain more articles on more topics than any encyclopedia, but it is a living work -- a constant work in progress and a work that anyone can monitor.

If you want to teach people a sensible approach to uncovering facts, just banning Wikipedia because the information in it might be wrong, is not only silly, but academically suspect. It presupposes that there are infallible sources -- a way of thinking that has no place in academia.

Encourage kids to use Wikipedia. Encourage kids to routinely check sources. Encourage kids to seek out conflicting statements of fact. Encourage kids to think for themselves.