Google Had the Most “Massive Parallelized Redundant Computer Network” in the World

(p. 198) . . . by perfecting its software, owning its own fiber, and innovating in conservation techniques, Google was able to run its computers spending only a third of what its competitors paid. “Our true advantage was actually the fact that we had this massive parallelized redundant computer network, probably more than anyone in the world, including governments,” says Jim Reese. “And we realized that maybe it’s not in our best interests to let our competitors know.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added.)

Silicon Valley Is Open to Creative Destruction, But Tired of Taxes

(p. A15) Rancho Palos Verdes, Calif.
When the howls of creative destruction blew through the auto and steel industries, their executives lobbied Washington for bailouts and tariffs. For now, Silicon Valley remains optimistic enough that its executives don’t mind having their own businesses creatively destroyed by newer technologies and smarter innovations. That’s an encouraging lesson from this newspaper’s recent All Things Digital conference, which each year attracts hundreds of technology leaders and investors.
. . .
In a 90-minute grilling by the Journal’s Walt Mossberg and Kara Swisher, Apple Chief Executive Tim Cook assured the audience that his company has “some incredible plans that we’ve been working on for a while.”
Mr. Cook’s sunny outlook was clouded only by his dealings with Washington. He was recently the main witness at hearings called by Sen. Carl Levin, a Michigan Democrat, who accused Apple of violating tax laws. In fact, Apple’s use of foreign subsidiaries is entirely legal–and Apple is the largest taxpayer in the U.S., contributing $6 billion a year to the government’s coffers.
Mr. Cook put on a brave face about the hearings, saying, “I thought it was very important to go tell our side of the story and to view that as an opportunity instead of a pain in the [expletive].” Mr. Cook’s foul language was understandable. “Just gut the [tax] code,” he told the conference. “It’s 7,500 pages long. . . . Apple’s tax return is two feet high. It’s crazy.”
When the audience applauded, Ms. Swisher quipped, “All right, Rand Paul.” A woman shouted: “No, I’m a Democrat!” One reason the technology industry remains the center of innovation may be that many technologists of all parties view trips to Washington as a pain.

For the full commentary, see:
L. GORDON CROVITZ. “INFORMATION AGE; Techies Cheer Creative Destruction.” The Wall Street Journal (Mon., June 3, 2013): A15.
(Note: ellipsis between paragraphs added; italics in original; ellipsis, and bracketed words, within next-to-last paragraph, in original.)
(Note: the online version of the commentary has the date June 2, 2013.)

Larry Page: “At His Core He Cares about Latency”

(p. 184) Speed had always been an obsession at Google, especially for Larry Page. It was almost instinctual for him. “He’s always measuring everything,” says early Googler Megan Smith. “At his core he cares about latency.” More accurately, he despises latency and is always trying to remove it, like Lady Macbeth washing guilt from her hands. Once Smith was walking down the street with him in Morocco and he suddenly dragged her into a random Internet café with maybe three machines. Immediately, he began timing how long it took web pages to load into a browser there.
Whether due to pathological impatience or a dead-on conviction that speed is chronically underestimated as a factor in successful products, Page had been insisting on faster delivery for everything Google from the beginning. The minimalism of Google’s home page, allowing for lightning-quick (p. 185) loading, was the classic example. But early Google also innovated by storing cached versions of web pages on its own servers, for redundancy and speed.
“Speed is a feature,” says Urs Hölzle. “Speed can drive usage as much as having bells and whistles on your product. People really underappreciate it. Larry is very much on that line.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.

Dohrmann and Quevedo Survive Creative Destruction of Inacom

DohrmannHokampQuevedoCosentry2013-10-07.jpg “Cosentry, an Omaha-based provider of data center storage and managed technology services, has a new CEO, Brad Hokamp, center. With him at the Cosentry data center in Papillion are company founders Kevin Dohrmann, left, and Manny Quevedo.” Source of caption and photo: online version of the Omaha World-Herald article quoted and cited below.

Innovation through creative destruction brings us the new products and processes that make our lives longer, richer and more satisfying. The major downside of creative destruction is the job loss of those working for firms that are creatively destroyed. Sometimes, in class, I use Omaha’s Inacom as a concrete example. Inacom was a value-added retailer of computer equipment. They would buy PCs from IBM, Compaq and the like, then add software and hardware, and re-sell and install for firms, at a mark-up. They were creatively destroyed by Dell’s process innovation of customizing and selling direct, at much lower prices than Inacom charged. When I arrived in Omaha, Inacom was one of a handful of Fortune 500 firms. Now Inacom is gone. But just because a firm is creatively destroyed does not imply that all those who worked for the firm are creatively destroyed. Dohrmann and Quevedo were executives at Inacom. They had the skills, knowledge, resilience and work ethic to create their own entrepreneurial startup that has thrived. Not everyone can do what Dohrmann and Quevedo did. But everyone should be able to improve their skills, knowledge, resilience, and work ethic, so that if creative destruction destroys the firm that employs them, they will still survive and possibly thrive.

(p. 1D) Cosentry’s regional data center footprint has grown far from its “humble beginnings” 12 years ago of just 4,000 square feet in the old Southroads Mall in Bellevue.

“Everyone saw it as a mall that was in deterioration, and I walked in and saw the most beautiful building in Omaha,” co-founder Manny Quevedo said, (p. 3D) remembering solid walls and below-grade space for computer systems.
Investments from Omaha firms Waitt Co. and McCarthy Capital along the way helped the firm grow; it was sold in 2011 to Boston private equity firm TA Associates but still has its headquarters at 127th Street and West Dodge Road.
. . .
The company’s workforce has approximately doubled in the last five years to nearly 200, more than half of them in Nebraska, and will continue to grow gradually with the expansion as Cosentry hires more engineers and technicians, Quevedo said.
Today the company has six data centers, including two each in the Kansas City and Sioux Falls, S.D., metropolitan areas. If you use utilities or health care services or do any shopping or banking in the region, there’s a chance some of your information has been stored or processed through Cosentry’s servers.
Cosentry started with what Quevedo said was a handful of clients and grew to hundreds within its first five years.
. . .
(p. 3D) Cosentry Timeline
2001: With investment from Waitt Co., Cosentry is started by Manny Quevedo and Kevin Dohrmann, former employees of InaCom, the former Omaha Fortune 500 computer dealer that began as a division of Valmont Industries but merged with VanStar of Atlanta in 2000 and later declared bankruptcy. Cosentry creates a data center in Bellevue.
2005: Cosentry, also called IPR Inc., sold its IP Revolution division to a Kansas firm, Choice Solutions. IP Revolution sold voice and data communications services and systems. Cosentry doubles the size of its Bellevue data center and expands to the Kansas City and Sioux Falls, S.D., markets.
2008: Omaha investment firm McCarthy Capital invests in the firm. At the time, Cosentry had 95 employees.
2010: Cosentry cuts the ribbon on the $26 million Midlands Data Center in Papillion, a joint project with Alegent Health, which uses the center to store electronic medical records.
2011: Boston investment firm TA Associates buys Cosentry for an undisclosed amount from McCarthy and Waitt. The local management team continues to operate and have an ownership stake in Cosentry. The firm expands with second data centers in both the Sioux Falls and Kansas City markets.
2013: Cosentry refinances its credit facilities to provide up to $100 million to enable expansion, including the expansion of the Midlands Data Center. Today, Cosentry has nearly 200 employees and six data centers in three metropolitan areas.

For the full story, see:
Barbara Soderlin. “A Growing Tech Footprint: As Businesses’ Data Storage Needs Expand, Cosentry Adds to Its Papillion Center.” Omaha World-Herald (MONDAY, AUGUST 26, 2013): 1D & 3D.
(Note: ellipses added; bold in original print version of article.)
(Note: the online version of the article has the title “As Businesses’ Data Storage Needs Expand, Cosentry Adds to Its Papillion Center.”)

CosentryScottCappsAtPapillionDataCenterCoolingSystem2013-10-07.jpg

“Scott Capps of Cosentry’s Papillion data center with the cooling system that helped Cosentry earn an Energy Star certification, which is given by the Environmental Protection Agency based on energy efficiency and lower emissions. It’s the only data center in Nebraska with the certification.” Source of caption and photo: the archive online version of the Omaha World-Herald article quoted and cited above.

Google’s Redundant, Fault-Tolerant System Worked with Cheap, Low-Quality, Failure-Prone Equipment

(p. 183) Google was a tough client for Exodus; no company had ever jammed so many servers into so small an area. The typical practice was to put between five and ten servers on a rack; Google managed to get eighty servers on each of its racks. The racks were so closely arranged that it was difficult for a human being to squeeze into the aisle between them. To get an extra rack in, Google had to get Exodus to temporarily remove the side wall of the cage. “The data centers had never worried about how much power and AC went into each cage, because it was never close to being maxed out,” says Reese. “Well, we completely maxed out. It was on an order of magnitude of a small suburban neighborhood,” Reese says. Exodus had to scramble to install heavier circuitry. Its air-conditioning was also overwhelmed, and the colo bought a portable AC truck. They drove the eighteen-wheeler up to the colo, punched three holes in the wall, and pumped cold air into Google’s cage through PVC pipes.
. . .
The key to Google’s efficiency was buying low-quality equipment dirt cheap and applying brainpower to work around the inevitably high failure rate. It was an outgrowth of Google’s earliest days, when Page and Brin had built a server housed by Lego blocks. “Larry and Sergey proposed that we design and build our own servers as cheaply as we can– massive numbers of servers connected to a high-speed network,” says Reese. The conventional wisdom was that an equipment failure should be regarded as, well, a failure. Generally the server failure rate was between 4 and 10 percent. To keep the failures at the lower end of the range, technology companies paid for high-end equipment from Sun Microsystems or EMC. “Our idea was completely opposite,” says Reese. “We’re going to build hundreds and thousands of cheap servers knowing from the get-go that a certain percentage, maybe 10 percent, are going to fail,” says Reese. Google’s first CIO, Douglas Merrill, once noted that the disk drives Google purchased were “poorer quality than you would put into your kid’s computer at home.”
(p. 184) But Google designed around the flaws. “We built capabilities into the software, the hardware, and the network–network– the way we hook them up, the load balancing, and so on– to build in redundancy, to make the system fault-tolerant,” says Reese. The Google File System, written by Jeff Dean and Sanjay Ghemawat, was invaluable in this process: it was designed to manage failure by “sharding” data, distributing it to multiple servers. If Google search called for certain information at one server and didn’t get a reply after a couple of milliseconds, there were two other Google servers that could fulfill the request.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added.)

Gates Did Not See that Gmail’s 2-Gig Storage Would Beat Hotmail

(p. 179) About six months after Gmail came out, Bill Gates visited me at Newsweek‘s New York headquarters to talk about spam. (His message was that within a year it would no longer be a problem. Not exactly a Nostradamus moment.) We met in my editor’s office. The question came up whether free email accounts should be supported by advertising. Gates felt that users were more negative than positive on the issue, but if people wanted it, Microsoft would offer it.
“Have you played with Gmail?” I asked him.
“Oh sure, I play with everything,” he replied. “I play with A-Mail, B-Mail, C-Mail, I play with all of them.”
My editor and I explained that the IT department at Newsweek gave us barely enough storage to hold a few days’ mail, and we both forwarded everything to Gmail so we wouldn’t have to spend our time deciding what to delete. Only a few months after starting this, both of us had consumed more than half of Gmail’s 2-gigabyte free storage space. (Google had already doubled the storage from one gig to two.)
Gates looked stunned, as if this offended him. “How could you need more than a gig?” he asked. “What’ve you got in there? Movies? PowerPoint presentations?”
No, just lots of mail.
He began firing questions. “How many messages are there?” he demanded. “Seriously, I’m trying to understand whether it’s the number of messages or the size of messages.” After doing the math in his head, he came to the conclusion that Google was doing something wrong.
The episode is telling. Gates’s implicit criticism of Gmail was that it was wasteful in its means of storing each email. Despite his currency with cutting-edge technologies, his mentality was anchored in the old paradigm of storage being a commodity that must be conserved. He had written his first programs under a brutal imperative for brevity. And Microsoft’s web-based email service reflected that parsimony.
The young people at Google had no such mental barriers. From the moment their company started, they were thinking in terms of huge numbers. Remember, they named their company after a 100-digit number! Moore’s Law was as much a fact as air for them, so they understood that the expense of the seemingly astounding 2 gigabytes they gave away in 2004 would be negligible only months later. It would take some months for Gates’s minions to catch up and for Microsoft’s Hotmail to dramatically increase storage. (Yahoo Mail also followed suit.)
That was part of my justification for doing Gmail,” says Paul Buchheit of its ability to make use of Google’s capacious servers for its storage. “When people said that it should be canceled, I told them it’s really the foundation for a lot of other products. It just seemed obvious that the way things were going, all information was going to be online.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

Rising Google Stock Prices Led Googlers to Be Wary of Innovation

(p. 156) . . . Googlers were affected by stock ownership. (They were, after all, human.) Bo Cowgill, a Google statistician, did a series of studies of his colleagues’ behavior, based on their participation in a “prediction market,” a setup that allowed them to make bets on the success of internal projects. He discovered that “daily stock price movements affect the mood, effort level and decision-making of employees.” As you’d expect, increases in stock performance made people happier and more optimistic– but they also led them to regard innovative ideas more warily, indicating that as Googlers became richer, they became more conservative. That was exactly the downside of the IPO that the founders had dreaded.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added; italics in original.)

“SEC Rules Demanded Complexity”

(p. 152) Google had considerable experience with pleasing users, but in the case of the auction, it could not create a simple interface. SEC rules demanded complexity. So the Google auction was a lot more complicated than buying Pokémon cards on eBay. People had to qualify financially as bidders. Bids had to be placed by a brokerage. If you made an error in reg-(p. 153)istering, you could not correct it but had to reregister. All those problems led to a few postponements of the start of the bidding period.
But the deeper problem was the uncertainty of Google’s prospects. As the press accounts accumulated–with reporters informed by Wall Streeters eager to sabotage the process– the perception grew that Google was a company with an unfamiliar business model run by weird people. A typical Wall Street insider analysis was reflected by Forbes.com columnist Scott Reeves, who concluded that Google’s target price, at the time pegged to the range between $ 108 and $ 135 a share, was excessive. “Only those who were dropped on their head at birth [will] plunk down that kind of cash for an IPO,” Reeves wrote.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.

Nanny Feds Take Revenge on Zucker for Trying to “Save Our Balls”

ZuckerCraigBuckyballsEntrepreneur2013-08-31.jpg

Craig Zucker. Source of caricature: online version of the WSJ article quoted and cited below.

(p. A11) Mr. Zucker is the former CEO of Maxfield & Oberton, the small company behind Buckyballs, an office toy that became an Internet sensation in 2009 and went on to sell millions of units before it was banned by the feds last year.

A self-described “serial entrepreneur,” Mr. Zucker looks the part with tussled black hair, a scraggly beard and hipster jeans. Yet his casual-Friday outfit does little to subdue his air of ambition and hustle.
Nowadays Mr. Zucker spends most of his waking hours fighting off a vindictive U.S. Consumer Product Safety Commission that has set out to punish him for having challenged its regulatory overreach. The outcome of the battle has ramifications far beyond a magnetic toy designed for bored office workers. It implicates bedrock American notions of consumer choice, personal responsibility and limited liability.
. . .
In August 2009, Maxfield & Oberton demonstrated Buckyballs at the New York Gift Show; 600 stores signed up to sell the product. By 2010, the company had built a distribution network of 1,500 stores, including major retailers like Urban Outfitters and Brookstone. People magazine in 2011 named Buckyballs one of the five hottest trends of the year, and in 2012 it made the cover of Brookstone’s catalog.
Maxfield & Oberton now had 10 employees, 150 sales representatives and a distribution network of 5,000 stores. Sales had reached $10 million a year. “Then,” says Mr. Zucker, “we crashed.”
On July 10, 2012, the Consumer Product Safety Commission instructed Maxfield & Oberton to file a “corrective-action plan” within two weeks or face an administrative suit related to Buckyballs’ alleged safety defects. Around the same time–and before Maxfield & Oberton had a chance to tell its side of the story–the commission sent letters to some of Maxfield & Oberton’s retail partners, including Brookstone, warning of the “severity of the risk of injury and death possibly posed by” Buckyballs and requesting them to “voluntarily stop selling” the product.
It was an underhanded move, as Maxfield & Oberton and its lawyers saw it. “Very, very quickly those 5,000 retailers became zero,” says Mr. Zucker. The preliminary letters, and others sent after the complaint, made it clear that selling Buckyballs was still considered lawful pending adjudication. “But if you’re a store like Brookstone or Urban Outfitters . . . you’re bullied into it. You don’t want problems.”
. . .
Maxfield & Oberton resolved to take to the public square.On July 27, just two days after the commission filed suit, the company launched a publicity campaign to rally customers and spotlight the commission’s nanny-state excesses. The campaign’s tagline? “Save Our Balls.”
Online ads pointed out how, under the commission’s reasoning, everything from coconuts (“tasty fruit or deadly sky ballistic?”) to stairways (“are they really worth the risk?”) to hot dogs (“delicious but deadly”) could be banned.
. . .
. . . in February [2013] the Buckyballs saga took a chilling turn: The commission filed a motion requesting that Mr. Zucker be held personally liable for the costs of the recall, which it estimated at $57 million, if the product was ultimately determined to be defective.
This was an astounding departure from the principle of limited liability at the heart of U.S. corporate law.
. . .
Given the fact that Buckyballs have now long been off the market, the attempt to go after Mr. Zucker personally raises the question of retaliation for his public campaign against the commission. Mr. Zucker won’t speculate about the commission’s motives. “It’s very selective and very aggressive,” he says.

For the full interview, see:
SOHRAB AHMARI, interviewer. “THE WEEKEND INTERVIEW with Craig Zucker; What Happens When a Man Takes on the Feds; Buckyballs was the hottest office game on the market. Then regulators banned it. Now the government wants to ruin the CEO who fought back.” The Wall Street Journal (Sat., August 31, 2013): A11.
(Note: ellipses, and bracketed year, added.)
(Note: the online version of the interview has the date August 30, 2013, and has the title “THE WEEKEND INTERVIEW; Craig Zucker: What Happens When a Man Takes on the Feds. Buckyballs was the hottest office game on the market. Then regulators banned it. Now the government wants to ruin the CEO who fought back.”)

When Google Earned a Profit, Sergey Brin “Felt Like We Had Built a Real Business”

(p. 94) . . . , Google was reaping rewards, and 2002 was its first profitable year. “That’s really satisfying,” Brin said at the time. “Honestly, when we were still in the dot-com boom days, I felt like a schmuck. I had an Internet start-up– so did everybody else. It was unprofitable, like everybody else’s, and how hard is that? But when we became profitable, I felt like we had built a real business.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added.)

Yahoo Valued “Marketing Gimmicks” More than Search Speed

(p. 44) Google had struck a deal to handle all the search traffic of Yahoo, one of the biggest portals on the web.
The deal–announced on June 26, 2000–was a frustrating development to the head of Yahoo’s search team, Udi Manber. He had been arguing that Yahoo should develop its own search product (at the time, it was licensing technology from Inktomi), but his bosses weren’t interested. Yahoo’s executives, led by a VC-approved CEO named Timothy Koogle (described in a BusinessWeek cover story as “The Grown-up Voice of Reason at Yahoo”), instead were devoting their attention to branding–marketing gimmicks such as putting the purple corporate logo on the Zamboni machine that swept the ice between periods of San Jose Sharks hockey games. “I had six people working on my search team,” Manber said. “I couldn’t get the seventh. This was a company that had thousands of people. I could not get the seventh.” Since Yahoo wasn’t going to develop its own search, Manber had the task of finding the best one to license.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)