Driving out failure

77june2006

Driving out failureProduct recall notices in shops, newspapers and elsewhere inform us now and then of children’s toys that might be dangerous or electrical goods that could catch fire. You or I might describe that as faulty design. Tim Davis calls it an “escaped failure mode”—and he is against it. His world is not toys or electrical goods, but vehicles, where failure modes and recalls are more expensive and usually more serious. As an example: in 2001 the failure of certain Firestone tyres on Ford SUVs led to rollover accidents in which close to 300 people died. Nearly 20 million tyres needed to be recalled. Industry estimates put the cost at around $3 billion. It was Tim Davis who was called in to find out what had gone wrong. Julian Champkin interviewed him.

Dr Tim Davis, statistician and engineer, holder of the Royal Statistical Society’s Greenfi eld Medal, is also the Henry Ford Technical Fel-low for Quality Engineering. In the company’s 100-years-plus history he was only their tenth Technical Fellow. It makes him a very big man indeed at Ford Motors. His thing is quality, and how to embed it in engineering design. Th at is why Tim Davis is an advocate of statistics in engineering: statistics as a route to reliability.

His defi nition of reliability, though, might surprise some statisticians. He has to make re-liable cars, and sell them, in the real, unreliable world. His statistics is correspondingly real and practical.

“If you pick up most statistics textbooks you will get a defi nition of reliability that is cast in the language of probability. It will say something like: ‘Reliability is the probability that the system performs its intended func-tion under particular operating conditions for a particular time.’ Th is is quite convoluted. Ac-tually it is also quite useless because you can’t measure it until the failure mode has escaped. Th e defi nition we use at Ford, and I think else-where in engineering, is quite simply ‘reliability is failure mode avoidance.’ What does it take to avoid failure modes in your design? Of course, you need statistics for that, but also you need things like designed experiments and robust design, and also other more down-to-earth

things like prevention of mistakes, to make sure your design is as failure-mode free as pos-sible before you even get to a point where you are going to test it and then manufacture it in large numbers.”

You don’t want your new car to leak cool-ant all over the road. If it does, a failure mode has escaped into the great outdoors. “Its root cause should have been captured at source, which is the design stage, so the counter-meas-

3(2)_10 lifeinstats_Davis.indd 773(2)_10 lifeinstats_Davis.indd 77 09/05/2006 16:34:0209/05/2006 16:34:02

78 june2006

ure could have been embedded in a revised design. Failing that, testing ought to reveal it. Certainly the failure mode should have been pounced upon by the time the manufacturing line is set up.”

Th at, fundamentally, is Tim Davis’s phi-losophy. He even believes that failure modes (with their corresponding counter-measures) could one day be treated as fundamental en-tities of engineering, just as subatomic parti-cles are fundamental to physics, electron pat-terns to chemistry and genes to evolution. He is still working on that one, with engineering colleagues at the Massachusetts Institute of Technology (MIT). For the present, though, we will stick with cars.

He has an offi ce in North America, trou-bleshoots in Germany, Sweden, and visits most continents most years, but we met at his UK base in Ford’s Design and Engineer-ing complex in Gaydon, Warwickshire, just off the M40. Design offi ces for Land Rover, Jag-

uar and Aston Martin are dotted around the compound, with big logos on their facades and the latest models outside. Mobile phones with cameras are banned from the complex, for fear that you might take away early evidence of an even newer model, and tape recorders need to be cleared by security. Once inside, his offi ce is rather more friendly.

“I guess I am a rare individual in that I am not only a chartered statistician but also a chartered engineer; so I have this kind of sym-metry to my professional credentials, which nicely refl ects the work that I do at Ford.” Th e symmetry is there on his window sill. At one end of the window stands the Donald Julius Groen Award from the Institute of Mechani-cal Engineers, for his work on reliability; at the other end is the RSS Greenfi eld Industrial Medal, which is for statistical contributions to manufacturing and industry and of which they give only one, or at most two, a year.

Nevertheless, he has on occasion felt that engineering may not be the most favoured profession among statisticians, even—breathe it low—in some parts of the RSS. “I am sur-prised,” he says, “that a career in industry is not mentioned on the RSS website, and I seem to have trouble getting my papers published in our journals.” One of his latest is entitled “Sci-

ence, Engineering and Statistics”. It is fl oating around on the Web.

“I gave a talk at an industrial seminar at the University of Michigan, and David Cox was in the audience, and after the lecture he said ‘Write it up. Industrial statistics in the UK needs a boost.’ So I wrote the thing, and he read it and sent me back a couple of good comments; and the RSS promptly rejected it because there was too much engineering and not enough statistics in it! Of course, it is pos-sible that I am rubbish at writing papers—but they do tend to get published somewhere in the end …”

Could it be that engineers and statisti-cians have not yet reached a proper under-standing in this country? Or is it the low status of engineering generally here? Some and some, he reckons:

“I think people who aren’t involved in engineering and manufacturing in this coun-try genuinely don’t understand the depth of intellectual rigour that is required to apply statistics in an engineering and manufacturing environment. Th ey get misled by the appar-ently simple mathematics that underlies some of the key ideas. Take Shewhart control charts as an example. Shewhart was an economist, not a statistician, and his maths is not hard; but actually some of his ideas are. How do you manage a system to diff erentiate between com-mon-cause and special-cause variability? How do you get people to react to them in the right way—in other words, if it is common, react as if it is common; if it is special, react as if it is special? Trying to do that in a large organisa-tion requires skills and methods that cannot be expressed in mathematical formulae; and yet the scientifi c ideas that you are trying to embed in your organisation are really quite profound, and often unintuitive.

“And I think that a lot of the mathemati-cal statisticians don’t quite see it in the pro-found way that it ought to be seen. Maybe it is a symptom of statistics being swallowed by mathematics generally. Th e statistics depart-ment at Aberystwyth where I did my under-graduate degree no longer exists; nor does the department at Birmingham where I did my PhD. Th ey both ended up in the maths de-partment.”

If statistics is not maths, engineering is not, by defi nition, a natural science. In a way it is the reverse.

“Science is explaining nature; engineering is changing nature. A scientist will do every-thing he can to remove sources of variation in his laboratory, so that he can understand the law he is trying to explain. Th e opposite is true

in engineering. You actually introduce variabil-ity into your lab, because you need to verify that whatever you have cooked up as your lat-est design is going to work across a wide range of operating conditions. Maybe we should call engineering an unnatural science! In addition, engineering is primarily about selection, not prediction, which is a distinction not always appreciated.”

Now-defunct university departments were not his only introduction to statistics. His father’s West Midlands bakery had some-thing to do with it as well.

“I got into statistics, like most people, by accident. I went to university to do maths, but I found myself taking all the statistics options because I enjoyed them. Aberystwyth is in ru-ral Wales so all of the datasets were on plant breeding and agriculture, whereas my passions were Meccano sets and messing around with bikes. But still they were solving real problems, even if they were not engineering problems. I suppose I had this visceral feel for the useful-ness of statistics.

“At the same time I was working in my fa-ther’s bakery in the university holidays; in fact, I ran it for a while when he was away. Bread is highly variable, because of all the natural proc-esses that are going on—the yeasts, cutting up the dough, and so forth. But the law wouldn’t recognise that. My father used to complain that the Royal Mint was allowed more toler-ance in the manufacture of coins than we were allowed in the manufacture of bread. Th ere was a statutory weight for a small loaf and for a large loaf, and the inspectors would come in from time to time and pick loaves at random and weigh them, and if they found even one that was below the statutory weight you could get fi ned. It occurred to me that some sort of statistical approach to the legality of the bread we were making and selling might not be a bad idea. But I was too shy and inexperienced and unqualifi ed to take on the government inspec-tors, so I let that one pass …”

And he carried on with his degree in sta-tistics.

“I got an upper second in the end, which was actually the best thing that could have hap-pened to me because if I had got a fi rst I would probably have stayed on at Aberystwyth. As it was, it was the early 1980s, and without a fi rst there was not a lot of money around then for grants. So I decided to go into industry.”

He ended up with Dunlop in Birming-ham, in their tyre testing department.

“And that was a revelation, because here we were, doing all this tyre testing, with people recording the time it took for the tyre to fail,

“Science is explaining nature; engineering is changing

nature”


79june2006

then writing down the reason why it failed; and all these data were being recorded but no one was actually doing anything with them. Nobody had a clue what the data meant or what to do with them. It wasn’t being fed back into making or designing better tyres. It was just data.

“To be fair, Dunlop had sort of realised this, which was why they hired me. And so I started to look at all these data and one thing I recognised straight away was that hardly any-thing I had learned at university seemed very helpful! Th e data were not Gaussian. Some of the tyres were removed unfailed. So they did not come in to the test results, even though they were obviously the best tyres of all. Th e engineers would do what seemed to me silly things like changing the conditions of the test halfway through. I would say, ‘You are not sup-posed to do that …’. I read somewhere that ‘real data doesn’t read textbooks’, and I realised at Dunlop that it was true.”

A far-sighted boss suggested that he enrol for a part-time PhD at the University of Bir-mingham to sort out these problems, which was manna from heaven. His thesis was on reliability and survivability with emphasis on competing risks, and when it was fi nished he realised that making cars is very like medicine, and nothing at all like making aeroplanes.

“It never occurred to me until after I fi n-ished my PhD and looked at my list of refer-ences at the back: three-quarters of them were papers dealing with medical survival data. It was because the contexts were so similar. In medicine you have got lots of people in the fi eld, experiencing uncontrolled stresses, with poor data on the people who are not failing or dying—they don’t come back to be treated if they get well. In cars, you’ve got thousands of customers who buy your car, stress it in individual ways you don’t know about, and only come back to complain when it has gone wrong. Once a car leaves your factory you re-ally have very little follow-up. You are very data-poor about success.

“So the context is very similar to medicine, and quite diff erent from the aerospace indus-try, which is the usual benchmark that people try to judge us by. Boeing has only ever made about 2000 747 planes. We have factories in America that build 2000 vehicles a day. Planes come in small numbers and they are moni-tored all their lives. Th ey don’t get lost to fol-low-up; their stress regimes may be severe but are pretty much known—and they have huge safety factors, not just because the law requires it but because there is revenue in it. You and I are quite happy to pay a bit extra for our airline

ticket knowing that the airframe has a safety factor of three. If we engineered cars like that, we would not be able to sell them. Th ere is no revenue in excessive car safety margins.

“I am not saying that a car is only as safe as it needs to be and no safer. We always engineer our cars to be safer than the law requires. But the point I am making is that if you achieve safety through what we might call over-engi-neering it is a very quick way to go out of busi-ness. Th ere are things that you demand from your vehicle; there are things that it is capable of doing. People have worked out how to posi-tion the demand/capacity curves to be as close together as they need to be, without the aero-space luxury of having clear daylight between them. And, of course, that is a fantastic link to statistics, because if you want to manage this very close relationship between demand and capacity, you have got to employ a statistical approach to engineer and manufacture cars that are failure-mode free.

“I suppose that was a conclusion I had reached by the time I fi nished my PhD. I had joined Ford by then, but before that, while I was still with Dunlop, I went to Japan, and when I saw what they were doing in the sta-tistical approach to quality I was just bowled over completely.

“I heard George Barnard give a lecture in London on W. Edwards Deming and the way his statistical approach revolutionised quality improvement and management philosophy, and I realised it was Deming’s infl uence that I had seen so much in Japan. Th e standards of statistical skills in engineering there was then, and still is now, much greater than in Europe.

“Th e Japanese were using tyre testing in a much more proactive way. Th ey were doing ex-periments, where we were just collecting data for verifi cation. My section in Dunlop was called the Tyre Testing Section; my equiva-lent department in Japan was called the Tyre Experimental Group—the word ‘experiment’ was in the title. And I thought, ‘Th at is fan-tastic. Th ese guys are doing experiments. All I am doing is just describing routine data; I am not doing any experiments at all.’ And it really brought home to me the way to combine sta-tistics and engineering.

“I remember they had a problem with one particular customer, a big quarrying company which was, basically, overloading its tyres till they failed in a particular way. Based on my re-search up to that point, I was able to set up a test for them that jiggled around with pressures, loads and speeds till it replicated that failure mode. Th ey could then work on the counter-measure plan. Th at sort of analysis stood me in

good stead very many years later at the time of the Firestone crisis. So all that early work has paid dividends throughout my career.”

Before that, though, he spent 20 years ris-ing through the ranks at Ford, at the Dunton Engineering Centre in Essex, then in Germany and North America as quality manager of the Focus, Mondeo, Scorpio and Galaxy range; and then, in Detroit, as quality director for the entire truck group, where he was respon-sible for, among other things, the F-150 pick-up truck—which most of us may never have heard of, but which is actually the world’s best-selling pickup and which won industry awards for reliability.

“We build and sell a million F-150s a year. I did the quality plan for the engineering of that. To put it into context, if you start produc-ing the vehicle and you have made a mistake

in the design and you don’t spot it for 2 years, potentially that is 2 million vehicles you have to recall.” If Tim Davis is obsessive about reli-ability, you can understand why.

It was while he was in North America that the Firestone problem emerged. Ford Ex-plorer vehicles fi tted with Firestone tyres as standard equipment were experiencing tread separation and some vehicles were then over-turning; nearly 300 deaths resulted. Th e tyre makers blamed the car; the car makers blamed the tyre. It turned out to be the tyres that were at fault.

“I was perfectly positioned to help with that because I had a background in tyres, and by then I had been noted in Ford as a prob-lem solver. Actually, I think that the Richard Feynman approach is the most effi cient way of solving problems. It is ‘fi nd the person who knows what the answer is and go and talk to him.’ In a large organisation somebody some-where usually does know the answer, and you have just got to fi nd him or her. Th e reason the Firestone case was so challenging was that the people who knew were deep inside Firestone and we at Ford couldn’t get to them very eas-ily, so we had no choice but to reverse-engi-neer the problem, and in so doing used a lot of statistical methods, like proportional hazards analysis, response surface methods and so on.

“Fundamentally it turned out that these particular tyres were very sensitive to a high-

“The Japanese were doing experiments. We British were just collecting data”


80 june2006

speed temperature-stress profi le. Usually tyres are very robust in that region—so much so that the standard industry testing for tyres did not usually explore that part of the stress regime. Th e particular design of this tyre—by a combination of geometry, the chemistry of some of the rubber compounding, and the shape of the tread patterns, among other things—made the diff erence. It was a four-factor integration and so, once the tyre was in the fi eld, it started to see all aspects of the stress regime; and, where it saw the particular point in the stress space that causes failure, of course it failed. So we had to act very quickly to get that sorted out. It is an example of good use of statistics to get the problem understood and fi xed. Unfortunately it just happened very late. Th e failure mode was not detected until the tyre was in the fi eld, and the consequences were dreadful. Th e silver lining to that cloud is that the industry standards for tyre testing, and tyre quality, have now been changed. I suppose one draws a small crumb of comfort from that.”

Besides the lives, the episode cost Ford an estimated $3 billion and ended the historic alliance between Firestone and Ford that had begun with Henry Ford’s fi rst vehicle in 1906. He and Harvey Firestone used to go camping together.

“Th e one thing I am proud of is that as a result of the work that I led with our team, we actually recalled a second batch of tyres something like 6 months before the govern-ment asked us to, which was unprecedented at the time. In our hazard analysis on the fi rst batch we realised that the failure rates increased with time. In other words, this sen-sitivity that I mentioned got worse over mile-age. When the tyre was new it worked, but as it degraded, as the ozone attacked the rub-ber and so forth, it was just enough to reach a tipping point, and then the failure rate really started to accelerate. And when we repeated that analysis with a newer set of tyres we real-ised that the same thing was going to happen. So at that point we said we are going to have to bring these back too. It took a lot of very careful analysis, and of course I had to per-suade very senior people in the company that this was the right thing to do—because, if we did not do it, more people would probably be killed. And in the end we did it. I am actu-ally very proud of that. In fact we got the law changed. I spent a lot of time in Washington DC with the National Highway Traffi c and Safety Administration explaining our analysis to them. Th ey now understand hazard analy-sis and are now able to force a manufacturer

into a recall before the failure rate has started to accelerate to a so-called unacceptable level. Before, they never had the power to do that. So the result of our eff orts is that lives will be saved in the future.”

“In the end the US government agreed with our fi ndings on the tyres. Our approach to failure mode avoidance has now been rec-ognised, and not just inside Ford. Th e US Department of Defense have been looking at the way they commission military equipment and how they specify reliability requirements. A couple of US Army generals recently came to see me in Detroit to talk about that. And I was the only non-American engineer to be in-vited to a reliability workshop at the National Academy of Sciences to discuss implications for military equipment.

“What I am really trying to do is to en-courage the engineering profession to leave behind the probabilistic defi nition of reli-ability, because that is only useful once the

failure mode has escaped. Th e probabilistic defi nition, on the face of it, looks very useful and attractive if you are happy just to analyse failure rates, but stopping the failure modes themselves from being created in the fi rst place needs a diff erent approach. Th ere is a bit of frequentist versus Bayesian undercurrent here, because the probabilistic defi nition is frequen-tist by its nature, whereas before you release a design into the fi eld you should ask yourself, ‘What is the probability that my design is free of failure modes, given that I have got all this test data and other evidence?’—which is a Bayesian-type question.

“It does remind me of what someone once said to me at a conference in Australia. I had explained that the reason I am pas-sionate about what I am doing is that if I am wrong I am personally accountable. And that makes a diff erence. As one of the Australians in the audience said, ‘You’ve got to have skin in the game.’ It is an expression I like. It means that I am never going to make a decision just based on signifi cance levels and p-values. We didn’t recall the second batch of Firestone tyres because of p-values and confi dence lim-its; we recalled them because we understood the implications of the projected failure rate and developed a lab test that confi rmed the

presence of the failure mode, and the under-lying failure mechanism. Getting this analysis wrong would have been a serious career-alter-ing event. Having skin in the game makes you think in a completely diff erent way, focuses you on what is important and what you need to do.

“If I have been able to think out of the box in engineering and make some fundamental contributions, it is because I am lucky: I have not been encumbered by any formal engineer-ing education! Th e same is true of statisticians: many of the fundamental contributions in statistics have come from people who did not have a statistics degree to begin with. George Box was a chemist, Deming was a physicist. David Cox always impresses me because he talks about all his ideas from the perspective of the scientifi c problem he is working on; there is never anything abstract about it. ‘What is the scientifi c problem you are involved with, why does statistics help with it?’ He impressed upon me over lunch at the Michigan seminar the importance of context when it comes to statistics. I think most engineers have been taught very bad statistics because it is out of context.

“I am not the only one who thinks that not enough statisticians go into engineer-ing, but it seems to me there is no point in teaching engineers statistics if all you are go-ing to do is shuffl e cards and fl ip coins or pull balls out of urns. Th at doesn’t connect prop-erly with the problems that engineers have to solve. Statistics has to be taught to engineers in the context of engineering. Th at is true for every application of statistics. You want to teach medical statistics from the perspectives of medical problems, survey statistics and gov-ernment statistics from the point of view of surveys and government. I do not believe that the statistics profession is somehow immune to the scientifi c context and that we can just fl it around. Having this magic statistics degree does not mean that one day I can analyse some data from an engineering experiment and the next day I can analyse data from medicine and then I can give advice to a pharmaceutical company on how to do a drug survey. I just don’t subscribe to that at all.

“Th e vast majority of us, if we mean to be eff ective statisticians, need to carve out a niche in whatever area of knowledge drives us, whether it is climate change or the ecol-ogy of songbirds or, as in my case, engineering. Th at way you add value to your own life, and do some good in the world. It comes down to that Australian expression again. We need a lot more statisticians with skin in the game.”

“You have got to have skin in the game. It focuses you

on what is important”


Documents

Driving out failure