Excerpt from "Checklist Manifesto"

‘Airline pilot’ protocols in finance

By Atul Gawande

Published: January 16 2010 00:04 | Last updated: January 16 2010 00:04

In the late 1990s, the early go-go years of the internet, a psychologist at Claremont Graduate University, east of Los Angeles, began a study of 51 venture capitalists. Geoff Smart wanted to know how they decided whether to give entrepreneurs money or not. The venture capitalists must yea or nay high-risk, multi-million-dollar investments in unproven start-up companies. The ideas might be little more than scribbles on a sheet of paper or clunky prototypes that barely work – but then again, that’s how Google and Apple started out. You would think venture capitalists’ decisions would centre on whether an entrepreneur’s idea was a good one. But finding a good idea is apparently not all that hard, Smart learnt. Finding an entrepreneur who can execute a good idea is a different matter entirely. You need a person who can take the idea from proposal to reality, work the long hours, build a team, handle the pressures and setbacks, manage technical and people problems alike, and stick with the effort for years on end without getting distracted or going insane.

Such people are rare – and extremely hard to spot. Smart identified half-a-dozen different ways the venture capitalists he studied decided whether they’d found such a person. These were styles of thinking, really. He called one type of investor the “art critic”. He or she assessed entrepreneurs almost at a glance, the way a critic can assess the quality of a painting – intuitively and based on experience. “Sponges” took more time gathering information about their targets, soaking up whatever they could from interviews, on-site visits, references and the like. Then they followed their gut.

The “prosecutors” aggressively interrogated entrepreneurs; “suitors” focused more on wooing people than on evaluating them; “terminators” saw the whole effort as doomed to failure and skipped the evaluation part. They simply bought what they thought were the best ideas, fired entrepreneurs they found to be incompetent and hired replacements. And then there were the “airline captains”. Studying past mistakes and lessons from others in the field, they built formal checks into their process. They forced themselves to be disciplined and not to skip steps, even when they found an entrepreneur they “knew” intuitively was a real prospect.

Why were these last, methodical types labelled “airline captains”? Four generations after the first aviation checklists went into use – lists deemed necessary to get giant, complicated aircraft off the ground and over oceans and continents with minimum risk to passengers – people are starting to apply that methodology to a range of fields. In three years of research, I found that checklists were being used to improve patient outcomes after surgery, keep buildings from collapsing and ensure consistently excellent food at popular and high-quality restaurants. I also found that checklists seem able to protect their users against failure where they might have made mistakes – even the experts.

Sure enough, when Smart tracked the venture capitalists’ success over time, it became clear that the airline captains had by far the most effective style. Those investors taking the checklist-driven approach had a 10 per cent likelihood of later having to fire senior management for incompetence or concluding that their original evaluation was inaccurate. The others had at least a 50 per cent likelihood. The results showed up in their bottom lines, too. The airline captains had a median 80 per cent return on the investments studied, the others 35 per cent or less.

Those with other styles were not failures by any stretch – experience does count for something. But those who added checklists to experience proved substantially more successful.

Smart’s most interesting discovery was not the relative success of the airline captains, in fact. Rather, it was that most of his subjects were either art critics or sponges – intuitive decision-makers instead of systematic analysts. Only one in eight took the airline captain approach. Now, maybe the others didn’t know about the airline captain approach. But even knowing seems to make little difference. Smart published his findings more than a decade ago. He has since gone on to explain them in a best-selling business book on hiring called Who. But when I asked him, now that the knowledge is out, whether the proportion of major investors taking the more orderly, checklist-driven approach has increased substantially, he could only report, “No. It’s the same.”

We don’t like checklists. They can be painstaking. They’re not much fun. But I don’t think the issue here is mere laziness. There’s something deeper, more visceral going on when people walk away not only from saving lives but from making money. It somehow feels beneath us, an embarrassment, to use a checklist. It runs counter to deeply held beliefs about how the truly great – those we aspire to be – handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism needs updating.

. . .

Smart certainly isn’t the only one noticing society’s – and industry’s – resistance to change. Recently I’ve met three investors who have taken a page from aviation to incorporate formal checklists into their work. Mohnish Pabrai is managing partner in Pabrai Investment Funds in Irvine, California, and runs a $500m portfolio; Guy Spier is head of Aquamarine Capital Management in Zurich, a $70m fund; the third did not want to be identified or to reveal the size of the fund where he is a director. But it is one of the biggest funds in the world and worth billions.

These three people are value investors, buying shares in under-recognised, undervalued companies. They don’t time the market. They don’t buy according to a computer algorithm. They do intensive research, look for good deals, and invest for the long run. Over the past 15 years, Pabrai has made a new investment or two every quarter, and he’s found that each one requires in-depth investigation of 10 or more prospects. The ideas can bubble up from anywhere but most drop away after cursory examination. Every week or so, however, he spots one that starts his pulse racing. It seems surefire. He can’t believe no one else has caught on to it yet. It could make him tens of millions of dollars if he plays it right. “You go into greed mode,” he said. (Spier called it “cocaine brain”.) And that, Pabrai said, is when serious investors try to become systematic. They focus on dispassionate analysis, on avoiding both irrational exuberance and panic. They pore over the company’s financial reports, investigate its liabilities and risks, examine its management’s track record, weigh up its competitors, consider the future of the market it is in.

The patron saint of value investors, of course, is Warren Buffett. Pabrai has studied every deal Buffett and his company, Berkshire Hathaway, have made – good and bad – and read every book he could find about them. He even pledged $650,000 at a charity auction to have lunch with Buffett. “Warren,” Pabrai said – and after a $650,000 lunch, I guess first names are in order – “Warren uses a ‘mental checklist’ process” when looking at investments. So, that’s more or less what Pabrai did from his fund’s inception. And he did very well following this method – but not always. He also made mistakes, some disastrous.

These were not mistakes merely in the sense that he lost money on his bets or missed making money on investments he’d rejected. That’s bound to happen. These were instances where he had miscalculated the risks involved, made errors of analysis. For example, looking back, Pabrai noticed that he had repeatedly erred in determining how leveraged companies were. The information was available; he just hadn’t looked for it carefully enough. In large part, he believes, the mistakes happened because he wasn’t able to damp down the cocaine brain. No matter how objective he tried to be about a potentially exciting investment, he found his brain working against him, latching on to evidence that confirmed his initial hunch and dismissing the signs of a downside. “You get seduced,” he said. “You start cutting corners.” Or, in the midst of a bear market, the opposite happens. You go into “fear mode” and overestimate the dangers.

He also found he made mistakes in handling complexity. A good decision requires consideration of so many different aspects of a company in so many ways that, even without the cocaine brain, he was missing obvious patterns. His mental checklist wasn’t good enough. “I am not Warren,” he said. “I don’t have a 300 IQ.”

So he devised a written checklist.

. . .

Pabrai made a list of all the mistakes he’d spotted (even Buffet made them) as well as his own – about a dozen. Then, to help him guard against them, he devised a matching list of checks – about 70 in all. Similarly, the anonymous investor I spoke to – I’ll call him Cook – made a checklist. But he was even more methodical. He enumerated the errors known to occur at any point – during the research phase, during decision-making, during execution of the decision and even in the period after making an investment. He then designed detailed checklists to avoid the errors, complete with clearly identified pause points at which he and his team would stop and run through the items together. He has a Day Three Checklist, for example, by which time, the list says, the team should confirm that they have gone over the prospect’s key financial statements for the previous 10 years, including specifically checking for items in each statement and possible patterns across the statements. “It’s easy to hide in a statement. It’s hard to hide between statements,” Cook said.

One check requires the members of the team to verify that they’ve read the footnotes on the cashflow statements. Another has them confirm they’ve reviewed the statement of key management risks. A third asks them to make sure they’ve looked to see whether cashflow and costs match the reported revenue growth. “This is basic basic basic,” Cook said. “Just look! You’d be amazed by how often people don’t do it.”

The checklist doesn’t tell him what to do, he explained. It is not a formula. But it helps him be as smart as possible every step of the way, ensuring that he’s systematic about decision- making, that he’s talked to everyone he should. With a good checklist in hand, he was convinced he and his partners could make decisions as well as human beings are able. And as a result, he was also convinced they could reliably beat the market.

Cook would not discuss precise results – his fund does not disclose its earnings publicly – but he said he had already seen the checklist deliver better outcomes for him. He had put the checklist process in place at the start of 2008 and, at a minimum, it appeared that he had been able to ride out the subsequent economic collapse, avoiding disaster. Others say his fund has done far better than that, outperforming its peers.

After about a year working with a checklist, meanwhile, Pabrai’s fund was up more than 100 per cent. This could not possibly be attributed just to the checklist. With it in place, however, he observed that he could move through investment decisions far faster. As the markets plunged through late 2008 and stockholders dumped shares, there were numerous deals to be had. And in a single quarter he was able to investigate more than a hundred companies and add 10 to his funds’ portfolios. Without the checklist, Pabrai said, he couldn’t have completed a fraction of the analytical work or have had the confidence to rely on it. A year later, his investments were up more than 160 per cent on average. He’d made no mistakes at all.

. . .

On January 15 last year, US Airways flight 1549 took off from La Guardia Airport in New York with 155 people on board. It then struck a flock of Canada geese over Manhattan, lost both engines and famously crash-landed in the icy Hudson River. Everyone survived, and Captain Chesley B. “Sully” Sullenberger III, a former air force pilot with 20,000 hours of flight experience, was hailed the world over.

But Sullenberger said from the first of his interviews that the outcome was the result of teamwork and following procedure as much as of any individual skill he may have had. Before the pilots – Sullenberger was joined by first officer Jeffrey Skiles, whom he’d not met before – started the aircraft’s engines at the gate, they ran through their checklists. They did a short briefing, discussing the flight plan, potential concerns and how they’d handle any problems. And by adhering to this discipline – the kind most other professions lack or avoid – they not only made sure the aircraft was fit to travel but also transformed themselves from individuals into a team, one systematically prepared to handle whatever came their way.

I don’t think we recognise how easy it would have been for Sullenberger and Skiles to have disregarded those preparations. People fear rigidity if they adhere to protocol. They imagine mindless automatons, heads down in a checklist, incapable of looking out of their windscreen and coping with the real world in front of them. But what you find, when a checklist is well made, is exactly the opposite. The checklist gets the dumb stuff out of the way.

About 90 seconds after takeoff, US Airways flight 1549 was climbing through 3,000ft when it crossed the path of the geese. The aircraft came upon the birds so suddenly Sullenberger’s immediate reaction was to duck. Jet engines are designed to be able to shut down after ingesting birds – which aircraft hit all the time – without exploding or sending metal shrapnel into the wings or the passengers on board, and that’s precisely what the A320’s engines did. They immediately lost all power.

Once that happened, Sullenberger made two key decisions: first to take over flying the aircraft from his co-pilot, Skiles, and second to land in the Hudson. Both men had decades of experience but Sullenberger had logged far more hours flying the A320. All the key landmarks to avoid hitting – Manhattan’s skyscrapers, the George Washington Bridge – were visible from his left-side window. And Skiles had just completed A320 emergency training and was more familiar with the checklists they would need.

“My aircraft,” Sullenberger said, using the standard language as he took the controls.

“Your aircraft,” Skiles replied.

There was no argument about what to do next, not even a discussion. And there was no need for one. The pilots’ preparations had made them a team. Sullenberger would look for the nearest, safest possible landing site. Skiles would go to the engine failure checklists and see if he could relight them. But for the computerised voice of the ground proximity warning system saying “Pull up … Pull up… Pull up… Pull up”, the cockpit was virtually silent as each pilot concentrated on his tasks and observed the other for cues that kept them co-ordinated.

The aircraft had only three and a half minutes of glide in it. In that time, Skiles needed to make sure he’d done everything possible to relight the engines while also preparing the aircraft for ditching if it wasn’t feasible. But the steps required just to restart one engine might typically take more time than that. He had some choices to make. Plunging out of the sky, he judged that their best chance of survival would come from getting an engine restarted. So he decided to focus almost entirely on the engine failure checklist and run through it as fast as he could. The extent of damage to the engines was unknown, but regaining even partial power would have been sufficient to get the aircraft to an airport.

In the end, Skiles managed to complete a restart attempt on both engines, something investigators later testified to be “very remarkable” in the time frame – and which they found difficult to replicate in simulation.

Yet he didn’t ignore the ditching procedure, either. He didn’t have time to do everything on the checklist but he sent the distress signals and he made sure the aircraft was configured for an emergency water landing.

“Flaps out?” asked Sullenberger.

“Got flaps out,” responded Skiles.

Meanwhile, the three flight attendants followed through on their protocols for such situations. They got the passengers to brace for impact. Upon landing, the flight attendants gave instructions to don life jackets. They made sure the doors were swiftly opened when the aircraft came to a halt, that passengers didn’t waste time grabbing for their belongings. One flight attendant, Doreen Welsh, stationed in the back, had to wade through ice cold, chest-high water leaking in through the torn fuselage to do her part. Just two of the four exits were safely accessible. Nonetheless, working together they got everyone out of a potentially sinking aircraft in just three minutes – exactly as designed.

The entire event had gone shockingly smoothly. After the aircraft landed, Sullenberger told the press, “First officer Jeff Skiles and I turned to each other and, almost in unison, at the same time, with the same words, said to each other, ‘Well, that wasn’t as bad as I thought.’”

. . .

In aviation, everyone wants to land safely. In the money business, everyone looks for an edge. If someone is doing well, people pounce like starved hyenas to find out how. Almost every idea for making even slightly more money – investing in internet companies, buying tranches of sliced-up mortgages – gets sucked up by the giant maw almost instantly. Every idea, that is, except one: checklists.

I asked “Cook”, our anonymous investor in Switzerland, how much interest others have had in what he has been doing these past two years. Zero, he said – or actually, that’s not quite true. People have been intensely interested in what he’s been buying and how, but the minute the word “checklist” comes out of his mouth, they disappear.

Even in his own firm, he’s found it a hard sell. “I got pushback from everyone. It took my guys months to finally see the value,” he said. To this day, his partners still don’t all go along with his approach and don’t use the checklist in their decisions when he’s not involved. “I find it amazing other investors have not even bothered to try,” he said. “Some have asked. None have done it.”

This is an edited extract from Atul Gawande’s ‘The Checklist Manifesto: How to Get Things Right’, published January 28 by Profile Books.