2008-01-13

Ancient history, Part 1: early years.

The other day Microsoft announced they were going to acquire Fast
Search and Transfer ASA -- a company that I used to work for.
Although I was hardly surprised by this turn of events I would
probably never have guessed it just 12 years ago. Or 16 years ago,
when this adventure actually started. I figured that perhaps this would be
a good time to share some stories, so I decided to write about some of the
things leading up to Trondheim becoming associated with search technology.


About 12 years ago, me and three other friends founded a company. We
were young, single and what we lacked in experience and business
sense, we made up for with pure hubris and a firm conviction that all
problems are solvable.

Most of us originally met at what is today NTNU in 1992. Back then it
was called NTH and people who graduated before the re-organization
tend to make a point of that. Three of us were students and one just
hung around the UNIX machines on campus long enough for someone to
finally put him to work.

At the centre of our universe in those days was PVV -- a computer club
that mainly attracted people interested in playing around with UNIX
computers. PVV had a bit of a reputation for being detrimental to
your academic progress. A disproportionately high number of the
regulars at PVV had...unconventional academic records. From people
who spent several years extra in getting their degrees to people who
simply got bored with academia and dropped out to pursue different
avenues. There was even a member who managed to screw up the process
of printing student records one year because the system had not
anticipated that anyone would fail a class more than N times (where
N was some "reasonable" number).

The more well kempt students would often frown if PVV came up in
conversation.

If you were interested in UNIX, it was the place to be. We had lots
of different UNIX platforms to play with -- a few of them so obscure
that it is very unlikely you've heard about them. Most serious
vendors would make sure we had at least one of their machines. You
could get your hands dirty playing sysadmin on them, you could write
software or you could just kill time there. If you were hiring a UNIX
sysadmin in those days, THAT was the place to go. That was where
you'd find people who had been exposed to up to a dozen different UNIX
flavors and varieties.

But what was probably more important were the people. At the time I
became a member of PVV there was a mix of really great people there.
People you could learn from. People you could have discussions
with. It was during these years (1992-1993) I think the free software
mindset started to become clear to me -- the why's and how's. This
was well before the term "open source" was coined, but the Gnu project
and the Free Software Foundation were around at this time. A lot of
people at PVV had deep admiration for the idea, although I think even
back then that Richard Stallmann was seen as a bit of a nutcase by
some -- deeply respected, but still viewed as bit of an oddball.

I had many fruitful discussions with people at PVV about not only
computer science, networking and UNIX, but also about Free Software,
the legal complexities of computing, politics as well as ethics. I
think everyone who was somewhat involved with PVV at that time became
infused with a set of values and exposed to ideas that were
later to become very important when the Open Source movement came
creeping out of the woodwork and into the mainstream.

In the original sense of the term "hacker", PVV was a thriving hacker
culture.

One key aspect of the hacker culture was a can-do attitude. If a
piece of your system was misbehaving or not working as you wanted to,
it was not all that far-fetched an idea to simply rip the system apart
and have a look inside. Today, this is an almost alien concept for
most -- as it was back then. Software is something people buy, and
when it doesn't work, they call support and then, maybe, after much
yelling, things get fixed. If not, someone gets sued, reimbursed,
apologized to or throws a computer out of a window. Most of the time,
things do not get fixed. Or it ends up taking forever and costing
truckloads of money.

A good example of this culture was a friend of mine: Tor Egge, or
"Tegge" (since most of us referred to each other by our usernames).
He discovered some bug in one of the operating systems we were
running. His way of solving it was to write a disassembler, disasseble
the kernel binary, read through the assembly code, find the bug, fix
it, run the thing through the assembler to build a new binary and test
it.

The fix worked.

If memory serves, he also did some experiments with the C-compiler,
feeding it code and looking at the assembly generated by it. He then
used this knowledge along with the debug information embedded in the
kernel to reconstruct the original C code that contained the bug, as
well as what the correct code should be. Someone said he sent in a
patch to the vendor and got it accepted without any questions asked,
but I'd have to ask Tegge if this is accurate.

Another friend of mine, Arne, did similar things. We had an early SGI
system that for some reason had ended up with a corrupted root
filesystem. The machine would not boot. So Arne used the ROM editor,
loaded in blocks from the filesystem into memory and had a long, hard
stare at them. As far as I know he didn't have any documentation on
the exact binary layout of the file system, but armed with some
general knowledge on how UNIX filesystems are organized, he edited the
blocks to correct the errors, wrote them back to disk and rebooted the
machine successfully.

When surrounded by people like that you tend to learn from them and
you slowly start to realize that there is nothing mystical about
computers and software and that you CAN fix things. Most things
require a lot less knowledge and effort than in the above examples
though.


It was here I first got to know the people I later would start a
company with. After a breakup I found myself single and with a lot of
time on my hands. A fellow PVV member (Stig, or "ssb" as he was known
to most people at PVV) found himself in the same situation, so he
moved out of the appartment he used to share with his former
girlfriend and camped out in my livingroom.

The place was a mess. We spent most nights partying or discussing
starting a company. Usually we did both. Indeed, the name of the
company came to me while on the dance floor at a club nearby, and with
the name the ideas started so solidify.

The next important point on the agenda was to find some more people to
start the company with. I don't remember if there was a list of
candidates, but I do remember that we gave this a lot of thought and
probably pissed off some people who felt that they should have been a
part of it but weren't asked. Finally we settled on Finn Arne
(finnag) and Alexander (astor). Both good programmers, both solid
UNIX people and last but not least: they were people that were fun to
be around -- which is important when you're starting a company.
Another aspect I think was important was that we complemented each
other really well.

Getting Finn Arne on board was easy. We just dropped by PVV, pitched
the idea and he was in. Alexander took a bit more work. Not because
he was unwilling, but because he was nowhere to be found. He had just
dropped off the face of the earth and vanished. We emailed him, tried
to call him, asked people he used to hang out with. Nothing. In the
end we figured that we'd give it a week and if he didn't turn up, we'd
just go ahead with starting the company and then catch up with him
later.

He eventually turned up, surprised that we had made such an effort
looking for him. He'd been at home in his studio appartment. With
his new stereo. Of course, none of us knew where he lived, so we
hadn't looked there. It turned out that he had been spending some
time at home, listening to his new stereo. When presented with the
idea of forming a company he promptly sold the stereo to free up
enough cash to chip in his part of the needed capital.

And then we were four.

Guardian Networks AS, as the company was called, was an odd sort of
company. Our original business plan, and I cringe when I call it
that, was to produce a hardened Linux for use in firewalls. The idea
was that Stig and I take on various consulting jobs to fund the
development and that Finn Arne and Alexander worked on the Linux
kernel as well as put together a minimal software distribution. To
make a long story short, there was not a lot of demand for our product
back in 1996. In fact it was pretty close to zero. Also, although a
lot of our ideas on how to make an OS suitable for firewalls looked
good on paper, they turned out to be rather impractical. The idea was
that an intruder should be able to gain root access on the system and
still not be able to do much damage. Configuration changes required
the machine to be shut down, booted up with a kernel that had no
networking, you'd change the config and then take the machine back up
again. As I found out when I ran the OS on an IRC server in a
datacenter I didn't have easy physical access to, this was a royal
pain in the ass.

Fortunately we managed to build a reputation as a sort of consulting
company. Most of our customers were people who had UNIX systems and
wanted them whipped into shape. Companies like HP would rent out
"sysadmin" types at exorbitant hourly fees to do upgrades, installs
and maintainance. Supremely incompetent people who had absolutely no
idea what they were doing. Poaching customers from such companies was
easy. We were cheaper, faster and when we left the site, everything
would work and the customer would also have generous helpings of free
software installed.

It may be hard for people these days to imagine how terribly bad a
typical default vendor installed UNIX was back in those days, given
the luxury of today's obscenely rich Linux environments we enjoy
today, but believe me, a UNIX box was usually something you'd get to
run just one or two special applications. Other than that it didn't
even have a proper command shell.

We also did a lot of odd software development projects. Projects for
which there didn't seem to be any natural choice in the more
traditional software-developers-for-hire shops. On some we learned
the hard way exactly how inexperienced we were. On other projects we
did surprisingly well. We did stuff for a bank, for various
engineering companies, for the university and for a long list of
companies I have since forgotten about. We worked on a really, really
broad range of problems.

In 1998 we were approached by a company called Fast Search and
Transfer. I had heard about FAST -- they had bought the rights to
FTPSearch, which had been developed mainly by Hugo Gunnarsen and then
Tor Egge, when Hugo left for a job SGI. Stig had even written a web
frontend for it at one point and I had played around with the code a
bit to port it (back) to Linux (why, I can't remember). We knew two
of the people working for them: Knut Magne Risvik and Tor Egge. Knut
Magne used to work at the user helpdesk at the university while
studying and was part of the helpdesk/sysadmin crowd at NTNU (most
UNIX sysadmins were actually students at NTNU).


Fast was going to build a web search engine, but they needed a
crawler. So they came to us. Foolishly assuming that it would be a
piece of cake, we set about doing it. It couldn't possibly be as hard
as writing the indexing and searching software, right?

Finn Arne did most of the design and implementation on the first
version of the crawler. The basic design and implementations were
simple, tidy and quite sound. I liked it a lot. The thing was that
Finn Arne had had very little exposure to web software. While he was
good at writing very efficient networking and IO software, he had
never bothered with HTML. So for instance, when he asked me which
elements would contain URLs I told him. All of them. Including URLs
in FORM elements. It didn't occur to me why he was asking (he was
writing the link extraction code).

Hilarity ensued when we unleashed the beast onto the web and people
started getting empty submissions to their guestbooks. The crawler
was following the ACTION URLs in the FORM tags. You only needed to
type in "braindamage" and "fastcrawler" into Dejanews to survey the
damage.

We fixed that one quickly.

Developing a web crawler turned out to be a much harder problem than
we had thought. Not only does it have to be capable of handling vast
amounts of information fast, but it traverses the web -- a universe
filled with the craziness and laziness of a million people who create
all sorts of complexities. From an almost incredible inability on
part of web server implementors to get even date formats correctly (I
think I counted 19 broken formats that existed in web servers), via
bad configuration that would look like an infinite number of web pages
on a site, to whatever curve-balls the users would throw at us in
terms of unwieldly and hard to process contents.

Somewhere along the line FAST acquired Guardian Networks AS, and we
became part of FAST. I was actually quite happy about this. I was
seriously tired of the annoyances of running a company. I made a note
about never starting a company without having at least one dedicated
"suit" on board.

8 comments:

  1. The original business idea

    Have you forgot all about our original business idea? We were going to make software for internet banks using 128 bits encryption, which would be at least 4000 billion billion times more secure than the puny 56 bit DES encryption everyone else used!

    I distinctly remember walking around NTH talking hushedly about "the customer" (DNB I think?) who would buy this and make us insanely rich, as long as no one else heard about it first and stole the brilliant idea from us...

    ReplyDelete
  2. Come to think of it, I can't actually remember when the plan to make a hardened Linux kernel became The Plan. I always thought that "The Customer" was a short-lived side-track which went away after a few weeks of hushed meetings, but you might be right.

    But I do remember that they had a brilliant chicken sandwhich at the cafe where we used to meet the representative for The Customer :-).

    ReplyDelete
  3. I think Tegge's vendor patch was for a SCSI card manufacturer.

    ReplyDelete
  4. Mikkel,

    Perhaps. Wouldn't surprise me.

    Tegge had a habit of patching a lot of systems. If I remember correctly the one in the blog posting was a DEC (not sure if it was the somewhat exotic OSF/1 on MIPS, or an Ultrix).

    I do know for sure that he disassembled and modified the DolphinOS kernel quite a bit. Mostly to extend and fix the NFS implementation. Among other things I remember him fixing an NFS security problem I discovered. (Fixing the problem was done quickly -- rebooting the machine took forever). The DolphinOS box (flipper) ran a kernel that was so heavily modified that people jokingly called it Tegglix :-).

    The most impressive hack I saw him do was to write a boot-loader with a menu system for Endre Angelvik's PC. Endre was trying out a bunch of different operating systems, but couldn't get any available boot loader to let him choose between them. (Endre had the x86 version of NeXTSTep on it among other things, and boy was I jealous!). So Tegge just sat down, booted DOS, fired up debug.com, wrote a boot-loader on the fly, wrote it to the boot sector (probably after moving things around a bit) and after a bit of fiddling, it worked.

    ReplyDelete
  5. To be more specific, I think there was a bug in a SCSI card manufacturer's firmware or driver (not sure which) that was only revealed when putting enough load, i.e. trying to power a large web search engine...

    I also vaguely remember someone trying to run disk stress tests before putting machines in production, only to find that the search software would reveal errors that the dedicated stress test software didn't. :)

    ReplyDelete
  6. Mikkel,

    Ah, ok. The patch you are referring to was probably something he did in the late 90s (if it was while at IDI/IDT). I remember discussing the issue with Tegge some time after PVV moved to SB-II and occasionally at FAST when we were located at the pier.

    The one I am talking about was early 90s. Probably around 92, but no later than 93.

    ReplyDelete
  7. it was really fun reading your ancient history entries... this was around the time we met, so it brought back some good memories. although i wasn't a part of pvv, some of these names sound very familiar (finnag, stig) since i met up with lots of you on irc. geez... those were the days.

    i specifically remembering you not having money then, but somehow scraping up enough to come see me in bergen. what fun!

    ReplyDelete
  8. Nice story. Tor helped me to connect to PVV via modem, log on to flipper.pvv and even sync mail using uucp over to my mac. And after some years I had the pleasure to hack the UKA (student festival) billing system with Alexander. He fixed the broken SCO system by moving it to (SCO-compatibility under) Linux and created a shell-script /sbin/init for the linux terminals. So brilliant people.

    ReplyDelete