History of Search Engines

My last post has got me thinking about the business relationships of Search Engines. The history of Search Engines is actually quite fascinating, with most of the big engines (with the exception of AltaVista) coming from students and academics, not big tech companies.

In 1990, the web was mainly comprised of academic papers, reports, technical documents, and other non-main-stream items stored on computers. To access these documents, you would need to know the computer's name, location (think IP address), and document name. In addition, you would not to type this information in the proper syntax.

ARCHIE
Hoping to make this process a little bit easier, Alan Emtage, a McGill University Student, created ARCHIE in 1990 - the first search engine. ARCHIE had a user interface for search and was mainly based on FTP. Archie's architecture used many features that we still see today, including web-crawlers scurrying around gathering information from titles of pages, reports, documents, etc. to create an index. The index would tell the user which computers had the document, and the user would connect to the computer and try to find the document.

AltaVista
After Archie, there were a couple more student based search engines, but the first big search engine came from Monier, a researcher at DEC. At the time, DEC was a computer manufacturer (like DELL). They had just come out with a super Alpha processor, and wanted to test it's might. Monier suggested that a search engine might be the answer, and even against the desires of DEC's executives, created AltaVista. Monier truly loved the search engine, and stuck with it through thick and thin for many years. AltaVista was truly an innovative search engine, because its crawlers created an index of the entire web, including text on the actual websites. This was different from the 10-20 other search engines currently out there, who were mostly looking at indexing titles. Eventually, DEC let AltaVista go public, and through word of mouth alone, it became a success. In the first day, it had 300,000 hits alone, and by the end of the first year, it had 4 Billion Queries. Even though the search engine was powerful, DEC executives did a poor job of supporting it, as they still believed that mini-computer manufacturing was the key to their business. Through terrible management, a failed spinoff, it's eventual sale of Compaq, and three failed IPO's, Monier eventually left due to sheer frustration of poor management.

Excite
Excite was founded by 6 Stanford Alumni in 1994. Originally developed with the intent to be sold to large corporations to mine internal databases, these 6 friends had a lot of hope. Coming under the guidance of Vinod Khosla, a VC from Silicon Valley, they started to focus on the Web as a portal for consumers. Taking his $1.5 million in Seed money, they hit the ground running. Unfortunately, their youth proved their inexperience, as they they didn't listen to Khosla's advice. He tried to set up mergers between, Yahoo and and Excite & Lycos and Excite, both of which failed. Excite's true contribution to the internet came in the form of MyExcite, allowing users to customize their homepage for free. In addition, they were the first major search engine to offer free e-mail. At it's climax in 1998, Excite had two offers, to merge with Yahoo or to be bought out by @Home, an internet broadband company. Unfortunately, Excite took the money and was bought out by @Home, and through bad management and the internet bubble bursting, eventually went bankrupt. Today Excite still exists, but it's market share is a former shell of itself.

Yahoo
Yahoo was founded by two PhD candidates at Stanford. According to many stories, these two really started their search engine by trying to mine basketball statistics for a fantasy basketball league. They successfully mined their data, won the pool, and realized that they had come up with a pretty unique search engine. At the time, they were procrastinating and did not want to write their thesis, so they spent a lot of time on the internet. They would write down interesting web site they found, and eventually started a website telling about these sites - essentially a guide of interesting websites. Eventually, they adopted the advertising model from "Wired Magazine" and put banner ads on their website. They expanded their site by putting up directories, such as "science, art, etc." Popularity spread through word of mouth, and became a big success. These two young guys also needed supervision, and eventually had to bring in Tim Koogle to be their first CEO. Interestingly enough, when Google was still a research project, Yahoo had an opportunity to buy the company and declined.

Why the name Yahoo?
Ya - Yet another
Yahoo - Yet Another Hierachical Officious Oracle

I think Google is too big of a phenomenon to describe here, and requires a separate post of its own.

Do you notice any trends amongst these search engines? There are many, but I think the following three similarities are extremely interesting:
1) Search Engines were all developed by people who were passionate about their projects. However, their passion did not turn into immediate business success, as most venture capitalists demanded they get supervision (by hiring experienced CEO's). Many search engines (excluding Yahoo) eventually found their downfall because of poor management as senior management mishandled search engine's promise.

2) If you look at the progression of search engine history, most search engines had the opportunity to buy each other out. Excite almost merged with Lycos and Yahoo, Yahoo and Microsoft had the option of buying Google, etc. Did these companies fail to see the potential in these up and coming rivals, or am I just playing Monday Morning Quarter Back a little too much? Either way, it's interesting to think of what the industry could look like today if any of these deals would have completed.

3) Until Google, search engines weren't considered to be imperative products in an internet company's business model. Most companies were drawing up business models with portals being at the center of media enriched sights. Excite, Yahoo, and Lycos were all trying to create portals (think of iGoogle) where you could find anything you needed on your homepage. Google was truly the first company to realize that Search can be the center product, even if it is the center of your portal.

*Thanks to John Battelle, Co-founder of Wired, for writing "The Search" - which is where I got a lot of my information on the history of Search Engines.