I was on stumble upon when i found this, an archive of the internet has been kept on a cluster computers and servers in 3 locations. although the one in the picture does not look as snazzy as a data center for Microsoft, there are two more including one with blade servers.
"
November 04, 2002 | Tucked away in one of the seediest neighborhoods of San Francisco is a roomful of over two hundred computers with a terabyte of data stored on every three. Stairs from the street lead up an intimidating hallway that opens into a room with 15-foot ceilings and just-this-side of hip ductwork in the ceiling. To the right is a storage area with a single desk, to the left are Baker's racks tightly packed with off-the-shelf HP desktop machines, each turned on it's side to maximize the space. Somewhere in all that ductwork, a fan is squeaking painfully. Walking into this echoey, over-warm warehouse space, it's easy to be underwhelmed until you realize what you are looking at: spinning away on these computers is nothing less than a copy of the Internet from 1996 until today.
 |
stairs leading to the Archive |
Hyperbole is easy to generate: over 10 billion pages are held here. The content of a single computer is equivalent to the entire Library of Congress. Over 250 gigabytes of data are added daily. Over 12 terabytes are added every month and there are a total of over 120 terabytes of storage available. As a copy of the entire publicly accessible internet, it is also certainly the worlds largest collection of pornography in a single room.
The Internet Archive was founded in 1995 by Brewster Kahle with the intent of preserving what is arguably the fastest growing archive of human expression ever created. The Library of Congress, and other analog equivalents, keep copies of the thousands of books published every year, helping to preserve our paper history, but the amount of content on the internet has grown to dwarf that repository. Also, unlike books on the shelves of libraries, web pages are in a constant state of flux: the average age of a web site is only 19 months with the average page changing every 100 days. The amount of information created and lost every day is staggering. The current state of digital technology and the internet makes it feasible for the Archive to reach it's stated mission of universal access to human knowledge.
The Internet Archive has several locations, the Mission street facility is their co-location site. This no-frills building is where the data is stored. The other primary site is located at the Presidio in the shadow of the Golden Gate Bridge. That San Francisco location is where the most of the staff are and where the day-to-day business of the Archive takes place. There is another site in Egypt that holds a duplicate of the data on the Mission Street hard drives.
Computing Power on a Shoestring budget
A walk through any other co-location site would be a testament to the Cisco and Sun sales force: high-end servers with terabytes of storage all connected together in a network that has packets flying so fast you can feel your fillings melt if you get too close to the wires. The Archive, on the other hand, is a poster child for function over flash. The machines on the racks are the cheapest possible—the ideal computer is one with a reasonable CPU, a gigabyte of RAM and case for under $300—loaded with a free operating system. The racks look to be straight out of Costco and you won't find an Aeron chair anywhere on the premises.
Gathering this many computers is certain to point out even the smallest problems inherent in the hardware. There is a failure of one kind or another almost very other day. Hard drives are a common problem, failing long before the manufactures' specification claims they should. The summer-time temperatures in the Archive facility hover slightly above what would be covered by the warranty, and a failed drive is potentially a piece of internet history lost. Motherboards, power supplies: anything that can go wrong, will. In those cases, a crash cart is rolled over, the problem diagnosed and hopefully fixed.
So why isn't the Archive using the latest and greatest Sun servers in a temperature controlled co-location facility where the air and power are filtered? The answer is simple: Librarians don't drive Porches. The Archive budget is not overly generous, and the total cost of ownership for Sun servers, proprietary operating systems and top-of-the-line routers and switches can be onerous. Filtering and cooling the air would approximately double the operating cost of the Archive. Under these restrictions it makes more sense to use commodity hardware and have layers of redundancy rather than a single server worth thousands of dollars.
Crawling
Anyone familiar with the concept of a web spider will understand the heart of the Archive's operation. Spiders are programs that traverse the internet and glean information from web sites. Search engines such as Google, Yahoo and Lycos rely on spiders to gather information to feed their search engines. The quality of these spiders and the quality of the indexing determine the success of your searches.
The Internet Archive does essentially the same thing in what it calls a "crawl." Crawls are carried out by the Archive itself or crawls are donated by other entities. Most of the data currently in the Archive has been donated by Alexa internet, also founded by Brewster Kahle. The Archive does two kinds of crawls: broad and narrow. A broad crawl is an attempt to archive a wide range of sites as completely as possible while a narrow crawl is designed for complete coverage of selected sites or selected topics. Both types of crawls have their own inherent challenges.
Broad crawls can create a number of different problems. While it may be easy to create a crawler that takes full advantage of a 100Mbs link to the internet, it becomes increasingly difficult to keep the crawler fed. Extracting unique URLs becomes computationally demanding as the database of indexed sites increases. A broad crawl can encompass over 150 million web pages in a week and run for 40-60 days total. Each page encountered needs to have the links on it extracted and followed. However, those links first need to be checked against the database of previously visited sites. If it's the first time this URL has been hit, then it will need more attention while a repeat can be ignored. This comparison is done in RAM, so if the database is large enough to exhaust the RAM on the computer and force it to page to disk, the speed of the crawl slows down significantly.
Another issue is for crawls is politeness—not all web sites are able to handle the load imposed by a high-performance, multi-threaded crawler. In these cases, there are only two outcomes: either the crawler is smart enough to back off and reduce the strain on the server, or the server will likely crash. A crawl that is looking up tens of millions of web sites per day can also having a devastating effect on DNS servers. These servers also often run out of RAM quickly as the lookups accumulate.
Broad crawls are a necessity because in the vast pool of web pages, it's impossible to know what information should be preserved. A broad crawl is specifically designed to copy as much information as possible over a wide range of web sites. Since there are a lot of duplicate pages out there, and the task of discerning them on the fly is far to difficult, the Archive ends up with about a 30% duplication rate on broad crawls.
Narrow crawls may require less storage and less bandwidth, but they have their own challenges. On a topic-driven crawl, the most obvious is the programming involved in assuring that the crawl has achieved its goal. Making certain that topics such as the attacks on September 11th have been covered completely goes far beyond finding pages with those keywords on them.
Taking care not to overwhelm servers becomes even more of an issue in a site-based narrow crawl. Attacking an underpowered web site with the full force of the Archive is certain not to make friends. In order to cover the sites completely, it is also vital to be able to pull out links from the pages that are not strictly HTML—links buried in Flash, Shockwave, JavaScript or created dynamically in other ways. In the same vein, the robot exclusion file that some sites use to declare parts of the site out-of-bounds to normal spiders can be ignored by the Archive if they are doing a crawl on behalf of an authority such as the Library of Congress. In that case, the webmaster will receive a notification, suitable for framing, explaining that they should be suitably honored that the exclusion files are going to be ignored so that this site can be added to the Library of Congress Web Archive.
Sure it's cool, but what can it do?
The Internet Archive is for web developers what home movies must be for celebrities. Preserved for posterity are our bad web designs, animated gifs and blinking text. The Wayback Machine is a glimpse into the pages held in Mission Street: enter a URL and it will sweep you back as far as 1996 to look at the early days of the internet explosion.
The Wayback Machine is a popular site, it receives about five million hits per day. While it is clearly in heavy use, it's not known what research, if any, the people using the Wayback Machine are doing. Other researchers are using the Archive, but it's not an easy task. While there is plenty of data to look at, there isn't an easy interface for accessing it.
The types of research that are waiting to be done are even more interesting. The Archive contains over ten thousand news sites, various archives of e-mail lists, and a growing number of blogs. This is the chatter of the world, and as time goes on it can provide a wonderful glimpse into the psyche of the time. Nefarious uses are equally easy imagine: the effects of advertising campaigns, product releases or political debates. Having the opinions of millions of people clearly documented, easily accessible and quantifiable would be a boon to market researchers and anthropologists alike. Blogs and personal web sites may be of even greater interest than mainstream news site since they are an unfiltered view of the public at large.
Bigger, Better, Faster, More
The Archive can only continue to grow, but obtaining more raw data isn't the only goal. The tools for accessing the data have to be improved before research can become commonplace using the Archive's stores. The quality of the collections needs to be addressed also: duplicates, missing pieces, new sites and new technologies all must be dealt with as the collection continues to grow.
The future also holds more cooperation between the Archive and other organizations. The Library of Congress and other national libraries have a keen interest in adding web archives to their collections, so a teaming makes perfect sense.
The speed of crawls is also due for an upgrade. Alexa Internet is doing the majority of the crawls for the Archive, but soon they will be able to do their own. When that is put in place, the collective speed of crawls will effectively double from the current 45Mb/sec to almost 100Mb/sec. That also means an increase in the speed of the LAN at Mission Street and the WAN between Mission and the Presidio.
There is a great deal more to the Archive than what is covered here.
The Prelinger collection of advertising, educational, industrial, and amateur movies is a fantastic, scary and sometimes downright hilarious remembrance of "duck and cover" type movies from the 1950s and on. The collection covers everything from patriotism to personal hygiene with a pre-"Bewitched" Dick York in a number of roles. Most of the movies are unintentionally, jaw-droppingly funny. (See: Dick York as the
Shy Guy, Dick York as a
runner battling insomnia, Dick York as a
Navy recruit with insomnia. Geez, did this guy ever sleep?)
Explore the Archive's web site often, this article has barely scratched the surface of the material available. The Internet Archive Bookmobile, a mirror of the Project Gutenberg files, the Orphan Films collection and the special collection of September 11th related web material are a small sample of the overwhelming store of information available. And it's free, just like any good library."
Sometimes, the strange foods in speculative fiction can make your mouth water. These delicious, fantastical items seem so good, you wish you could take a bite. Here are some of our most-wanted fictional snacks.
