IBM has launched an ambitious initiative, called Project Kittyhawk, aimed at building “a global-scale shared computer capable of hosting the entire Internet as an application.” Forget Thomas Watson’s apocryphal remark that the world may need only five computers. Maybe it needs just one.
The Register’s Ashlee Vance points to a fascinating white paper about the IBM program. The effort focuses on expanding the company’s Blue Gene supercomputer to handle web-scale applications of every imaginable stripe – to create a “generic” computing platform, incorporating millions of processors, that can essentially run anything you throw at it. The authors argue that the reigning, Google-style model of web-scale computing – big clusters of cheap servers – was born of necessity, rather than choice, and has fundamental flaws:
At present, almost all of the companies operating at web-scale are using clusters of commodity computers, an approach that we postulate is akin to building a power plant from a collection of portable generators. That is, commodity computers were never designed to be efficient at scale, so while each server seems like a low-price part in isolation, the cluster in aggregate is expensive to purchase, power and cool in addition to being failure-prone. Despite the inexpensive network interface cards in commodity computers, the cost to network them does not scale linearly with the number of computers. The switching infrastructure required to support large clusters of computers is not a commodity component, and the cost of high-end switches does not scale linearly with the number of ports. Because of the power and cooling properties of commodity computers many datacenter operators must leave significant floor space unused to fit within the datacenter power budget, which then requires the significant investment of building additional datacenters.
Many web-scale companies start in a graduate lab or a garage, which limits their options to the incremental purchase of commodity computers even though they can recognize the drawbacks listed above and the value of investing in the construction of an integrated, efficient platform designed for the scale that they hope to reach. Once these companies reach a certain scale they find themselves in a double bind. They can recognize that their commodity clusters are inefficient, but they have a significant investment in their existing infrastructure and do not have the in-house expertise for the large research and development investment required to design a more efficient platform.
IBM believes that it has a better way, a means of building a computing platform that “is an order of magnitude more efficient to purchase and operate than the commodity clusters in use today”:
Companies such as IBM have invested years in gaining experience in the design and implementation of large-scale integrated computer systems built for organizations such as national laboratories and the aerospace industry. As the demands of these customers for scale increased, IBM was forced to find a design point in our Blue Gene supercomputer technology that allowed dense packaging of commodity processors with highly specialized interconnects and cooling components … We postulate that efficient, balanced machines with high-performance internal networks such as Blue Gene are not only significantly better choices for web-scale companies but can form the building blocks of one global-scale shared computer. Such a computer would be capable of hosting not only individual web-scale workloads but the entire Internet.
The researchers report that early tests of the platform in running Web 2.0 programs and other web apps and related software “are promising and show that it is indeed feasible to construct flexible services on top of a system such as Blue Gene.”
IBM sees the future in IBM terms, while Google sees the future in Google terms. But the IBM paper makes for fascinating reading for anyone interested in the future of computing. It underscores the fact that the design of the shared computer that we’ll all be using in the future remains up for grabs. In the long run, this is a battle that will prove of far greater import than Microsoft and Google’s tussle over Yahoo.
“Then God said, “Let us make man in our image, after our likeness.”
-Gen 1.26.
this project sounds to me like a brain – one that will allow multitudes of thought processes to take place within it.
there will be scarcity on this computer at some point – my guess is that the capacity of this machine will not be infinite.
when we reach the point where there is no capacity for adding a process without dropping an old one, there will be a battle for a dominant thought or set of thoughts – much like what you have in your own head today.
the difference between our brains and this brain will be one of scale – no human will be able to compete in a thought battle with this one.
though we created it in our own image, the manifestation of that image will be, in some ways, stronger than the creator.
Hi Jim, one would have to define what you mean by thought battle!
“The cluster in aggregate is expensive to purchase, power and cool in addition to being failure-prone.”
I had gained the impression that the big cluster, server farm model was not failure prone!
But they had been designed so that any single failed component would be automatically by-passed without interruption in services.
Alan
>> One computer to rule them all
If true then it would seem to make the current multi-billion dollar Microsoft/Google server farm arms race/build out kind of a waste of time and money. Fast forward ten years, after creating hundreds of expensive data centers around the world containing millions racked mounted PCs, IBM unveils KittyHawk, a machine the size of a refrigerator, that can process 1 billion times the amount of traffic at 1 billionth the cost. Might hurt their stock price a bit…
a machine the size of a refrigerator …
I just hope nobody trips over the power cord.
> I just hope nobody trips over the power cord.
seriously, the world as we know it is a network of decentralized components and knowledge, what is wrong with geographically distributed computer systems, one computer/machine == single point of failure. am i missing something?
Skynet, anyone?
I just finished The Big Switch last night. This is definitely essential reading for 2008! I have to admit that Nick Carr was one of the names in the Gillmor Gang 3 years ago I had never heard of ;) Waiting for a copy of “Does IT matter?” now.
So, it’s like AS3, but instead of storage they provide BG-powered computing power, right?
Stan – I believe that the system IBM is imagining would be distributed. Lots of refrigerators, lots of power cords. Nick
Ugh… speaking of false prophecies…
http://www.johnmwillis.com/redmonk/demystifying-clouds/
Assuming all the resource usage numbers are correct in the IBM
sales pitch, the data center owners (e.g. Google ) may still prefer
paying more running many PC’s rather than few Blue-Gene’s. The
business strategy choices sounds very much like in software, with
Microsoft offering a out-of-the-box solution for a known price as opposed to the Linux free-but-maybe-expensive-administration-or-coding.
The logic of the strong: when faced with a choice, choose both.
Hey Google: Buy a few Blue-Gene’s, put them in your Dalles compound, but keep using PC’s also. That way you are not dependent on one supplier mafia leverage.
An IBM whitepaper on this subject needs to be read with more skepticism, since IBM obviously has a vested interest in selling its big iron and has so often been ham-handed in conceiving and marketing real products. A lot of people are working on the very real problem of datacenter efficiency, and even IBM has multiple efforts in this area, including one that it spun off, Seval/IceCube.
IBM has also been trying for awhile to re-brand its old mainframes as web servers, which it now calls “mainframe servers” and can run a version of Linux. (Is there a single trendy technology that IBM hasn’t tried to embrace, usually without results?)
I’ll never forget attending an IBM product announcement many years ago when TCP/IP was exploding onto the scene in the internet’s early days. An audience member asked the team of IBM engineers on stage whether the new product supported TCP/IP, and the IBMers had clearly never heard of it, bringing howls of laughter from the crowd.
The company has been forced to change, but it’s still hard to imagine IBM taking charge in this market. More likely they’ll license technology to someone who can actually build and market products efficiently.