Google’s Redundant, Fault-Tolerant System Worked with Cheap, Low-Quality, Failure-Prone Equipment

(p. 183) Google was a tough client for Exodus; no company had ever jammed so many servers into so small an area. The typical practice was to put between five and ten servers on a rack; Google managed to get eighty servers on each of its racks. The racks were so closely arranged that it was difficult for a human being to squeeze into the aisle between them. To get an extra rack in, Google had to get Exodus to temporarily remove the side wall of the cage. “The data centers had never worried about how much power and AC went into each cage, because it was never close to being maxed out,” says Reese. “Well, we completely maxed out. It was on an order of magnitude of a small suburban neighborhood,” Reese says. Exodus had to scramble to install heavier circuitry. Its air-conditioning was also overwhelmed, and the colo bought a portable AC truck. They drove the eighteen-wheeler up to the colo, punched three holes in the wall, and pumped cold air into Google’s cage through PVC pipes.
. . .
The key to Google’s efficiency was buying low-quality equipment dirt cheap and applying brainpower to work around the inevitably high failure rate. It was an outgrowth of Google’s earliest days, when Page and Brin had built a server housed by Lego blocks. “Larry and Sergey proposed that we design and build our own servers as cheaply as we can– massive numbers of servers connected to a high-speed network,” says Reese. The conventional wisdom was that an equipment failure should be regarded as, well, a failure. Generally the server failure rate was between 4 and 10 percent. To keep the failures at the lower end of the range, technology companies paid for high-end equipment from Sun Microsystems or EMC. “Our idea was completely opposite,” says Reese. “We’re going to build hundreds and thousands of cheap servers knowing from the get-go that a certain percentage, maybe 10 percent, are going to fail,” says Reese. Google’s first CIO, Douglas Merrill, once noted that the disk drives Google purchased were “poorer quality than you would put into your kid’s computer at home.”
(p. 184) But Google designed around the flaws. “We built capabilities into the software, the hardware, and the network–network– the way we hook them up, the load balancing, and so on– to build in redundancy, to make the system fault-tolerant,” says Reese. The Google File System, written by Jeff Dean and Sanjay Ghemawat, was invaluable in this process: it was designed to manage failure by “sharding” data, distributing it to multiple servers. If Google search called for certain information at one server and didn’t get a reply after a couple of milliseconds, there were two other Google servers that could fulfill the request.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added.)

Covey Was Amazon’s Entrepreneurial CFO

CoveyJoyAndSonTyler2013-09-25.jpg

Joy Covey and son Tyler. Source of photo: was posted on Joy Covey’s Google+ page: https://lh6.googleusercontent.com/-3CNdHv-7W3A/Thx9kuMHEPI/AAAAAAAABKs/lX9H2JlJ_lg/w763-h762-no/J+%2526+T+snowbird.jpg

(p. D8) As Amazon’s first chief financial officer, Ms. Covey helped take the company public and was an independent-minded advocate for Amazon’s plans to ignore Wall Street and invest for the future. That notion, radical in its day, was the foundation for Amazon’s growth into a $61 billion retailing and entertainment behemoth.

In its early days, Amazon prided itself on its unconventional hires, telling staffing agencies to “send us your freaks.” Ms. Covey did not have a traditional background. She dropped out of high school at 15 and worked as a grocery clerk. She attended Cal State Fresno and later Harvard Law School, where, she said, she did not fit in.
“We’d go to lunch and people would talk about their favorite 17th-century poets, and I’d be thinking, ‘Could I even name five poets? From any century?’ ”
But after joining Amazon in late 1996, when its annual revenue was less than $20 million, she thrived. She sold Wall Street the debt that the company needed to expand. The company went public on May 14, 1997, with an initial offering price of $18. Shares this week were selling for more than $312. Her own wealth is estimated at more than $200 million.

For the full obituary, see:
DAVID STREITFELD. “Joy Covey, 50, Top Executive in Amazon.com’s Early Days.” The New York Times (Sat., September 21, 2013): D8.
(Note: the online version of the obituary has the date September 19, 2013, and has the title “Joy Covey, Top Executive in Amazon.com’s Early Days, Dies at 50.”)

Gates Did Not See that Gmail’s 2-Gig Storage Would Beat Hotmail

(p. 179) About six months after Gmail came out, Bill Gates visited me at Newsweek‘s New York headquarters to talk about spam. (His message was that within a year it would no longer be a problem. Not exactly a Nostradamus moment.) We met in my editor’s office. The question came up whether free email accounts should be supported by advertising. Gates felt that users were more negative than positive on the issue, but if people wanted it, Microsoft would offer it.
“Have you played with Gmail?” I asked him.
“Oh sure, I play with everything,” he replied. “I play with A-Mail, B-Mail, C-Mail, I play with all of them.”
My editor and I explained that the IT department at Newsweek gave us barely enough storage to hold a few days’ mail, and we both forwarded everything to Gmail so we wouldn’t have to spend our time deciding what to delete. Only a few months after starting this, both of us had consumed more than half of Gmail’s 2-gigabyte free storage space. (Google had already doubled the storage from one gig to two.)
Gates looked stunned, as if this offended him. “How could you need more than a gig?” he asked. “What’ve you got in there? Movies? PowerPoint presentations?”
No, just lots of mail.
He began firing questions. “How many messages are there?” he demanded. “Seriously, I’m trying to understand whether it’s the number of messages or the size of messages.” After doing the math in his head, he came to the conclusion that Google was doing something wrong.
The episode is telling. Gates’s implicit criticism of Gmail was that it was wasteful in its means of storing each email. Despite his currency with cutting-edge technologies, his mentality was anchored in the old paradigm of storage being a commodity that must be conserved. He had written his first programs under a brutal imperative for brevity. And Microsoft’s web-based email service reflected that parsimony.
The young people at Google had no such mental barriers. From the moment their company started, they were thinking in terms of huge numbers. Remember, they named their company after a 100-digit number! Moore’s Law was as much a fact as air for them, so they understood that the expense of the seemingly astounding 2 gigabytes they gave away in 2004 would be negligible only months later. It would take some months for Gates’s minions to catch up and for Microsoft’s Hotmail to dramatically increase storage. (Yahoo Mail also followed suit.)
That was part of my justification for doing Gmail,” says Paul Buchheit of its ability to make use of Google’s capacious servers for its storage. “When people said that it should be canceled, I told them it’s really the foundation for a lot of other products. It just seemed obvious that the way things were going, all information was going to be online.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

Brazilian Entrepreneur Inspired by “The Men Who Built America”

HangLucianoArrivesAtFlagshipHavanStoreInBrusque2013-09-29.jpgThe co-founder of the Havan chain, Luciano Hang, arrives at the chain’s flagship store, which is in Brusque, Brazil. Source of photo: online version of the NYT article quoted and cited below.

(p. 6) “My philosophy is pro-capitalism, so of course the best symbols for this come from the United States,” said Mr. Hang, who flies around Brazil on a Learjet to visit the nearly 60 stores in his chain, called Havan. “I tell people that we’re about freedom: the freedom to stay open when we choose, the freedom to work for us and the freedom to shop,” he added. “I know this can be controversial, but I think those who disagree with my approach are few and far between.”
. . .
The son of textile factory workers, descended from German and Italian immigrants, Mr. Hang said he admired European culture but preferred the United States. He said he was inspired by a show on the History Channel, “The Men Who Built America,” about industrial titans like John D. Rockefeller and Cornelius Vanderbilt.
“I couldn’t sleep after I saw that program,” he said.
His business model is partly based on Walmart, whose small-town origins he admires, as well as its method of turning economies of scale into low prices.

For the full story, see:
SIMON ROMERO. “Reshaping Brazil’s Retail Scene, Inspired by Vegas and Vanderbilt.” The New York Times, First Section (Sun., September 15, 2013): 6.
(Note: ellipsis added.)
(Note: the online version of the story has the date September 14, 2013.)

Larry Page’s Very Tough Love: “I’d Rather Be Doused with Gasoline and Set on Fire than Use Your Product”

(p. 171) Caribou took forever to develop. Part of the problem was that Larry and Sergey were so invested in the project. They adopted it as their primary email system and would often drop by to give criticisms and suggestions. Buchheit would often take a working prototype to the weekly Google product strategy meeting, where product managers submit their products to a human wind tunnel of executive criticism. Products have been known to die at GPSs; there are stories of teams entering the conference room, exhausted and hopeful after long hours of getting a demo just right, and Page saying, “You’re wasting our time” and ordering the project dismantled. Larry and Sergey liked Caribou too much to kill it but dished out very tough love. At one point Page told the group, “I’d rather be doused with gasoline and set on fire than use your product.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

Innovative Entrepreneurs More Likely to Have Engaged in Illicit Activities as Teens

(p. C4) What does it take to be a successful entrepreneur? The signs are obvious in future moguls’ teenage years: brains, confidence–and illicit activities.
Those are the surprising findings of a new working paper by economists at the University of California at Berkeley and the London School of Economics. The researchers argue that merely being self-employed isn’t a particularly good indicator of entrepreneurship, in the sense of taking big risks and mobilizing capital to create new goods and services.
. . .
. . . the professors sorted the self-employed into those who were incorporated and those who were not, with the researchers regarding the former as the genuine entrepreneurs.
. . .
Despite . . . dubious youthful pursuits, the incorporated tended to come from stable, well-educated families with high incomes in 1979. These entrepreneurs were much more likely to be white, male and well-educated than were salaried workers or the unincorporated self-employed.

For the full story, see:
DANIEL AKST. “The Bad-Boy Entrepreneur.” The Wall Street Journal (Sat., August 17, 2013): C4.
(Note: ellipses added.)
(Note: the online version of the review has the date August 16, 2013.)

The working paper discussed is:
Levine, Ross, and Yona Rubinstein. “Smart and Illicit: Who Becomes an Entrepreneur and Does It Pay?” NBER Working Paper # 19276, August 2013.

Rising Google Stock Prices Led Googlers to Be Wary of Innovation

(p. 156) . . . Googlers were affected by stock ownership. (They were, after all, human.) Bo Cowgill, a Google statistician, did a series of studies of his colleagues’ behavior, based on their participation in a “prediction market,” a setup that allowed them to make bets on the success of internal projects. He discovered that “daily stock price movements affect the mood, effort level and decision-making of employees.” As you’d expect, increases in stock performance made people happier and more optimistic– but they also led them to regard innovative ideas more warily, indicating that as Googlers became richer, they became more conservative. That was exactly the downside of the IPO that the founders had dreaded.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added; italics in original.)

Immigration to the U.S. Is the Story of Hope, Achievement, Youth, Freedom and Creation

ToAmericaWithLoveBK2013-10-04.jpg

Source of book image: http://ecx.images-amazon.com/images/I/51VTjY0xVbL.jpg

(p. C6) In his new book, “To America With Love,” the British critic A. A. Gill attempts to make up for his fellow Britons’ grouchiness, sending the United States a frilly, funny valentine.
. . .
Perhaps the most provocative thing in “To America With Love” is Mr. Gill’s European take on our history of immigration. He argues that America over the years has been a magnet, drawing “the young and the strong from Europe; the adventurous, the clever, and the skilled.”
In the United States, “immigration is the story of hope and achievement, of youth, of freedom, of creation,” he writes. “But all entrances on one stage are exits elsewhere. In Europe it is loss. Every one a farewell, a failure, a sadness, a defeat.” Between 1800 and 1914, he says, “more than 30 million Europeans immigrated to the New World: one in four Irishmen, one in five Swedes, three million Germans, five million Poles, four million Italians. There is not a country, a community, a village or household that wasn’t affected by the lure of the West.”
As Mr. Gill sees it, much of the bitterness that animates trans-Atlantic relationships (Europeans, he says, patronize America “for being a big, dumb, fat, belligerent child”) can be traced back to this dynamic. “The belittling, the discounting, the mocking of the States is not about them at all,” he writes. “It’s about us, back here in the ancient, classical, civilized continent.”
Europe’s view of America, he contends, “has been formed and deformed by the truth that we are the ones who stayed behind, for all those good, bad and lazy reasons: because of caution, for comfort, for conformity and obligation, but mostly, I suspect, because of habit and fear. We didn’t take the risky road.”

For the full review, see:
MICHIKO KAKUTANI. “BOOKS OF THE TIMES; A Rebellious Trans-Atlantic Infatuation: Take That, Mrs. Trollope!” The New York Times (Thurs., August 22, 2013): C6.
(Note: ellipsis added.)
(Note: the online version of the review has the date August 21, 2013.)

The book under review is:
Gill, A.A. To America with Love. Reprint ed. New York: Simon & Schuster, 2013.

GillAA2013-10-04.jpg

“A. A. Gill” Source of caption and photo: online version of the NYT review quoted and cited above.

Google’s Calico Company Seeks to Expand the Human Life Span

(p. B4) Google Inc.is backing a new company to research aging, taking an unusual business swing at the burgeoning science of extending the human life span.
The venture, which is long on goals but short on specifics, is known as Calico, and will operate separately from Google, the online search giant said on Wednesday.
“We believe we can make good progress within reasonable time scales with the right goals and the right people,” Google CEO Larry Page said in a blog post. “This is clearly a longer-term bet.”
. . .
Google provided scant details about how Calico would operate or how it would tackle its ambitions of improving the health of “millions of lives.” But Jay Olshansky, an expert on aging at the University of Illinois of Chicago, said one potentially promising path is to research therapies that target the aging process itself.
While medical research typically focuses on finding treatments and cures for individual ailments such as cardiovascular disease and cancer, “if you’re going to have an impact on human health and longevity in the future, the way to go is to go after aging itself,” Dr. Olshansky said.
A founder of a consortium called the Longevity Dividend Initiative, Dr. Olshansky said [he gave a talk at conference (sic) in 2010 in which he said that finding a cure for cancer would only extend human life span by about three and one-half years. The reason is, he said, is (sic) it would “expose people who were saved from dying of cancer to all the other diseases and disorders” that are the result of aging.]
. . .
{He said Mr. Brin attended the meeting and asked him questions about the talk. He hasn’t discussed Calico with anyone at Google–the first he’d heard of the venture was Wednesday– though he described the formation of Calico as “great news.”}

For the full story, see:
GREG BENSINGER and RON WINSLOW. “Google Backs New Venture to Research Aging.” The Wall Street Journal (Thurs., September 19, 2013): B4.
(Note: ellipsis added; square-bracketed words are in the print, but not the online, version of the article; curly-bracketed words are in the online, but not the print, version of the article.)
(Note: the online version of the story has the date September 18, 2013, and has the title “Google Backs Venture to Research Aging.”)

Former Economics of Entrepreneurship and Economics of Technology Student Luis López Voted Crowd Favorite at Straight Shot Startup Accelerator Demo Day

LopezLuisPitchesCardioSys2013-10-05.jpg “Luis López pitches his startup, CardioSys, to investors during Demo Day at Aksarben Cinema this week. The event was the culmination of a 90-day Straight Shot startup accelerator program that offered new companies networking opportunities, advisers and investment dollars. Seven startups were in the inaugural accelerator class.” Source of caption and photo: online version of the Omaha World-Herald article quoted and cited below.

Luis López, the entrepreneur who is featured in the article quoted below, was a student of mine in both my Economics of Entrepreneurship and my Economics of Technology seminars (and before that, in micro principles). I cannot say that I taught him everything he knows, but it appears that I did not do him much harm.

(p. 1D) The same day Luis López and his brother, Danny, were accepted into Omaha’s Straight Shot startup accelerator for their new company, corporate America called.

The 25-year-old Central High grad had received a job offer from Gallup. But he turned it down, choosing to take an entrepreneurial risk over a predictable salary and benefits.
“I can always apply for a job in the corporate world,” he said, but it’s not every day that one’s company is accepted into an accelerator program that offers $20,000 in investment, more than 300 mentors and more than $75,000 in in-kind services.
The risk paid off, López said last week as the 90-day program wrapped up. The López brothers’ startup, CardioSys — which uses predictive analytics to calculate a person’s risk of developing conditions like heart disease and diabetes based on factors such as age, blood pressure and lipid profiles — came out of the program with a group of nine advisers.
. . .
(p. 2D) Luis López said CardioSys is hoping to land some investment in the next month or two, and is now looking at applying for a short-term health industry-focused incubator program in California, which the founders were connected with via Straight Shot.
In the long term, however, López said that with its strong community of medical and insurance providers, Omaha is CardioSys’ home. At Demo Day, the startup was voted crowd favorite. “I was surprised. It’s an honor to have people excited about what we’re doing,” he said.

For the full story, see:
Paige Yowell. “Straight Shot at Success; Accelerator’s First Startups Make Their Pitches.” Omaha World-Herald (SATURDAY, OCTOBER 5, 2013): 1D & 2D.
(Note: ellipses added.)
(Note: the online version of the story has the title “Straight Shot Accelerator’s First Startups Make Their Pitches.”)

LopezLuisCoFounderCardioSys2013-10-05.jpg

“Luis Lopez, who with his brother Danny Lopez, created CardioSys, gives his pitch at Demo Day.” Source of caption and photo: online version of the Omaha World-Herald article quoted and cited above.

“SEC Rules Demanded Complexity”

(p. 152) Google had considerable experience with pleasing users, but in the case of the auction, it could not create a simple interface. SEC rules demanded complexity. So the Google auction was a lot more complicated than buying Pokémon cards on eBay. People had to qualify financially as bidders. Bids had to be placed by a brokerage. If you made an error in reg-(p. 153)istering, you could not correct it but had to reregister. All those problems led to a few postponements of the start of the bidding period.
But the deeper problem was the uncertainty of Google’s prospects. As the press accounts accumulated–with reporters informed by Wall Streeters eager to sabotage the process– the perception grew that Google was a company with an unfamiliar business model run by weird people. A typical Wall Street insider analysis was reflected by Forbes.com columnist Scott Reeves, who concluded that Google’s target price, at the time pegged to the range between $ 108 and $ 135 a share, was excessive. “Only those who were dropped on their head at birth [will] plunk down that kind of cash for an IPO,” Reeves wrote.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.