Reliability Engineering anti System Sqfety 33 (1991 ) 265-276

A Survey of Expert Opinions on the Effectiveness of Non-destructive Examination of Pressure Vessels

G u s t a f O s t b e r g

Engineering Materials, University of Lund, PO Box 118, 221 00 Lund, Sweden

{Received 20 January 1990; accepted 18 June 1990)

ABSTRACT

There is an increasing awareness of the fact that our knowledge about the risk of not detecting defects in heavy section, welded pressure vessels is unsatisfactory. There is only limited understanding of what confidence can and should be given to non-destructire examination. In order to improve the basis for further development as well as for probabilistic assessments, an inquiry into the efficiency of non-destructive examination of pressure vessels was made. Six qualified and experienced specialists on non-destructive examination were interviewed in order to determine their opinion about the efficiency of defect detection. The observations showed that such specialists do not possess the knowledge of statistics that is desirable for a trustworthy assessment of probabilities. The)' proved to be very uncertain about the efficiency in question, often referring to the assumed high quality of the manufacture of the vessels as a guarantee of safety. In the light of these observations, the current views on the efficiency of defect detection therefore, appear less trustworthy than commonly believed.

P R O B L E M A N D AIM

Steel pressure vessels may fail due to propagation of cracks. In order to prevent such failure the vessels are inspected by non-destructive examina- tion using ultrasonic techniques. It is assumed that this inspection may not be absolutely effective, leaving a number of crack-like defects undetected in a vessel.

265 Reliability Engineering and System Safeo" 0951-8320/91/$03.50 © 1991 Elsevier Science Publishers Ltd, England. Printed in Great Britain

266 Gustaf Ostberg

Failures of pressure vessels usually have serious consequences. This is true in particular of those vessels in nuclear power stations that encase the radioactive core of the reactor. It is commonly postulated that pressure vessel failure in a nuclear reactor can generate missiles which may break the containment. As a consequence, large amounts of radioactivity are instantaneously released from the power station to the surroundings.

It is obviously of interest to know the probability of failure in cases like nuclear reactors as well as for other installations of heavy wall steel pressure vessels. This probability can in principle be calculated on the basis of statistical information about the properties of the steel, the mechanical stresses and the presence ofcracklike defects. The problem to be dealt with in this paper is that the information about defects is not satisfactory. In fact, the number of remaining, undetected defects in pressure vessels can be considered rudimentary, particularly in comparison with the quality of other factors comprising the basis for probabilistic failure analyses.

There are no statistics from experience or experiments of value in a normal so-called frequentistic sense to establish the probability function; the only statistical evidence available is obtained by extrapolation of information from pressure vessels of similar but different kinds. The technical differences between nuclear pressure vessels and those classified as similar, however, is so great that the value of such extrapolations are highly doubtful, to say the least. Therefore, one has to resort to the judgement of experts on non-destructive testing obtained by interviews. This is in fact the present state. The sole basis for taking the effectiveness of defect detection into account in probabilistic analyses of catastrophic failure of nuclear pressure vessels is one single questionnaire asking twenty-one experienced operators about their opinions. ~-3 Clearly, this limited base is very unsatisfactory in view of the seriousness of a failure of pressure vessels, in particular the main vessels in a nuclear power plant.

Furthermore, as will be elaborated elsewhere, the procedure used in the only previous expert judgement was such as to call for a complementary study. In the earlier investigation referred to the interviews contained only one type of question, asking for the opinion of the probability of detecting cracks of different sizes in a particular testing situation. The answers ranged between relatively low values of the effectiveness, sometimes of the order of 40 and 100%. The distribution between the extremes was skewed; often about half the number interviewed specialists claimed 100% effectiveness. In the evaluation of this distribution, the average was used as a representative value for the subsequent probabilistic analysis of the rate of failure of nuclear pressure vessels.

There is thus only limited understanding of what confidence can and should be placed on estimates based on non-destructive examination. In

Non-destructive examination of pressure vessels 267

fact, special investigations of the effectiveness of non-destructive examina- tion have shown that the detection of defects by ultrasonic techniques may be far less reliable than usually assumed, even with due consideration of improvements in recent years in the effectiveness of examination for intergranular stress corrosion cracks. 4

This deplorable state of affairs is considered true not only for stress corrosion cracks, but also for the main pressure vessels in nuclear power plants. Concerning the latter, discrepancies between inspection results obtained in the manufacturer's workshops and after installation of the pressure vessels indicate that the effectiveness of non-destructive testing is a problem.

There are reasons, therefore, to question the general validity of the present basis for judgement of the probability of detecting cracklike defects in pressure vessels. Any inquiry into this matter would improve confidence in the precautions against catastrophic failure.

GENERAL APPROACH

At first glance the most straightforward approach to the present problem should be a further attempt to collect frequentistically valid information by tests and experiments. A consideration of the effort needed to acquire a statistically satisfactory minimum of data shows, however, that this is not feasible. While it is true that on-going or planned research on the effectiveness of ultrasonics with respect to location and sizing of defects is likely to produce some additional information about detection, the number of observations will still be far from sufficient for regular statistical purposes.

The same limitation applies to recent developments of the ultrasonic technique under laboratory conditions which have promised some improvement. The results so far obtained cannot be translated into practical performance until considerable additional experience has been gained with the aim of accounting for all the psychological conditions in actual nuclear power plants.

Against this background and the experience of previous research on so- called man-machine problems and risk handling the only way found to proceed was to make a more extensive and elaborate study of subjective estimates of the probability of (not) detecting defects, in particular in order to assess the dependability of their opinions. For this purpose a number of experts on non-destructive examination have been subjected to an inquiry into the basis for their judgement about detection of defects. The emphasis has been on ultrasonic inspection of pressure vessels, but X-ray examination has also been included when appropriate.

268 Gustaf Ostberg

M E T H O D and P R O C E D U R E S

The interviewing

In the present study the inquiry was made by normal interviews. The number of questions was large and their scope was very wide, with the aim of elucidating and understanding the background and reasons for the claims of the individual specialists interviewed. The total number of questions was 67, including a few explicit ones on the percentage of detection of specific defects. All of these questions were not addressed in all interviews, but even then all the subjects in the questionnaire were touched upon in the conversation during the interviews.

The questions were not shown to the specialists beforehand. Some questions related to subjects that made it necessary to refer to drawings or graphs.

The interviews were made with one specialist at a time, with the exception of two who were interviewed together. The duration was some two to three hours. No tape recording was made but notes were taken of the answers. In some cases, a second interview was made with no specific questions, in order to provide an opportunity for the specialists to express second thoughts on different subjects as well as on the interviews as such.

The minor differences between the interviews with respect to the ways the questions and subjects were addressed were not intentional or systematic but a natural consequence of the individual differences between the persons interviewed, i.e. their experiential background and interests, as well as the situational conditions during the interviews.

The specialists

The specialists interviewed can be classified (with respect to their competence for the purpose of delivering subjective estimates on detection probability) as senior inspectors whose experience was not limited to interpretation and evaluation of indications of defects in heavy wall welded steel pressure vessels, including those installed in nuclear reactors. They were selected from the Swedish communi ty of specialists in this field on the basis of their reputation of belonging to the top rank of their profession with competence recognized nationally and internationally. Among the six specialists interviewed, individual differences were no ted- -no t only of opinions about the subjects addressed in the various questions. Their general attitudes to the issue of defect detection also differed. There were some who at first expressed reluctance to engage in any interview, while others were rather eager to voice opinions about the efficiency of their technique.

Non-destructire examination of pressure ressels 269

The questions

For the sake of brevity only some of the questions and answers can be stated in full. Furthermore, the questions were designed and the interviews were structured so as to reveal the mental background for value-related opinions rather than producing data on the effectiveness. The object of the investigation was the specialists' way of reasoning as an indication of the trustworthiness of their opinions. This approach may be called humanistic since it does not avoid 'soft', non-rationalistic issues that are commonly assumed to be essential for the professional competence.

In the following examples, the questions are given under the headings used for subdivision of the issue.

(1) On accounts of defective detection in general and the robustness of the specialist's opinion. In the graph shown to you the crosses represent your own opinion. If another specialist's opinion is plotted like the circles, would this make you change your mind about defect detection?

(2) On the application of statistics. Nowadays, the large piece of a nuclear pressure vessel can be made without welds. This requires correspondingly large steel ingots which in turn might possibly increase the number of defects in comparison with welded vessels made of pieces from ingots of normal size. Assume for the sake of argument that the number of such defects is eight per vessel. These defects can be detected with an efficiency of 99% . On the other hand, the number of defects in welded vessels is assumed to be four. What do you think about 98% effectiveness of detection in the latter case, assuming that the probability of detection of different cracks is independent?

(3) On the conditions during testing. It is true to say that the cause of deficiences of the detection of defects is to be found in the technique and performance of the testing, or does the reliability rather depend on the system for quality assurance?

(4) On the occurrence of defects. In the bottom head of the vessel of a boiling water reactor, defects may occur in the weldments of the nozzles for the control rods. Is there a certain probability of defects remaining after inspection due to the fact that the geometry makes inspection difficult, or can one rely on the particular care taken by the welders to make the welds perfect?

(5) On the importance of defect detection. Does it make any difference for an inspector ifa beam of an overhead crane or a pressure vessel is considered?

270 Gustaf Ostberg

(6) On the operators' performance during changing conditions. Assume that a new testing procedure is to be introduced. For this purpose operators have to pass an examination. Of the outcomes of this examination the following two cases will be considered for the sake of elucidating opinions about evaluations of new detection practices. One is that half of the number of operators fail. This is explained as caused by the mere fact that they are subjected to examination; on a second examination, they could be expected to do better, as they will also do in practice. According to the other assumption, the outcome is such that the operators perform better during the test than during actual work because they try harder when examined. Which alternative do you consider more likely?

(7) On the interviewed specialist's own experience. Have you ever seen (detected) a crack or have you heard of cracks that have escaped detection?

RESULTS AND INTERPRETATION

This section contains a generalized, interpretive account of the result of the interviews under headings corresponding to those used for systematizing the many questions. The implications of the results will be presented in a separate discussion following this account.

Figures for defect detection

Naturally, several questions sought to achieve values for the effectiveness of defect detection comparable with those found in the literature. Generally, no precise answers were given to such explicit, specific questions on the basis of graphs on the effectiveness as a function of defect size. In the first place it appeared that the specialists interviewed were not familiar with such graphs. This is remarkable as such; this important way of representing and applying the result of their work does not seem to have been made known to them other than occasionally.

It is against this background that the following answers should be judged. When specialists were asked their opinions about the various relationships between effectiveness and defect size that have been published, only a few qualified answers were given. This might be interpreted as a general disbelief in such estimates; possibly these vague answers reflected the opinion that the published values may have been produced by people possessing as limited knowledge as they themselves.

Non-destructive examination of pressure vessels 271

Concerning views on the importance of detecting defects of different sizes, small defects were stressed. This might indicate that the specialists interviewed were interested in problems related to non-destructive examination rather than in the consequences of remaining defects with respect to failure. This attitude is, of course, quite natural and respectable. On the other hand, however, it appeared that the specialists, in the back of their minds, mostly had a more or less precise idea of the implication of defects with respect to the structural integrity of the vessels. Again, this should also be considered inevitable and justifiable as long as it did not adversely affect their judgement.

Uncertainty of detection

Again it is noteworthy that these specialists on non-destructive examination were not very familiar with the statistics used in this context. The answers given to differently formulated questions on the same subject were not always consistent.

An issue raised more than once was whether the scatter of results of non- destructive examination should be considered stochastic or not. Most answers expressed a belief in a stochastic nature of the scatter, but there were also some respondents who thought that varying competence among the operators could explain the scatter.

As an overall conclusion it may be stated that the specialists interviewed in this study believed only in a very limited precision of any claims on the effectiveness ofdefect detection, the uncertainty they referred to when shown literature values was the largest ever reported.

Trust in efficiency figures

A searching question was raised to find out what confidence the specialist interviewed had in a figure given for the efficiency. He was asked to make a bet on either a roulette game with a certain chance of winning or a case of defect detection claimed to have the same outcome. This is a type of question often used in studies of people's belief in statistical outcomes. Almost invariably, most people bet on the statistically defined alternative. In this case, unexpectedly, the answers of the specialists on non-destructive examinations varied, and there was a slight majority in favour of betting on the detection of defects.

When they were asked to draw a line for a 'lower bound' in a graph with a large variation among the data points, this line usually passed through the lowest values, or sometimes even somewhat higher. Apparently, the view that a "lower bound' should take further scatter into account did not occur.

272 Gustqt" Ostberg

Another observation that should be noted is the belief that the accuracy of a value for the effectiveness of defect detection is greater the smaller the scatter. High precision is considered equivalent to high accuracy.

Finally, there was a question whether one could trust someone who claimed that 100% effectiveness could be achieved. The opinions were divided, but some believed, inconsistently, in this possibility even after having expressed doubts about the effectiveness in answers to other questions.

Existence of defects

The most remarkable observation with respect to the underlying beliefs of the specialists interviewed was made during the conversation in connection with the question about defects in vessels manufactured from large and small ingots, respectively (question (2) above). The immediate answer to this question was mostly hesitant; the simple arithmetic was seldom applied.

In addition to this, however, there were comments implying that in fact, there were never any defects in nuclear pressure vessels. There was no hesitation or ambiguity in this statement. Obviously, in the back of the minds of specialists on non-destructive inspection is the firm belief that there are no defects there to be detected. The purpose of the inspection should then be only to confirm the absence of defects.

This view on the role of non-destructive examination in relation to other conditions and measures for preventing failure was revealed also by other answers. For instance, the probability of defects (not) remaining at penetrations in the bot tom head of a vessel in a boiling water reactor was attributed to the quality of the welding procedure rather than the efficiency of the non-destructive examination.

Experience and importance of defects

When questions related to the importance of defects were asked, the answers mostly indicated consciousness and understanding of the critical role of defects being detected. There were exceptions, however, for instance in the answers about possible differences between pressure vessels and beams for cranes according to question (5) above.

The response was limited when the question approached the subject of personal experience of defects that had escaped detection. While it may be considered common knowledge that cases have occurred when cracks have been found after examination, this never became apparent in the answers given to questions aiming specifically at this issue. Information on this point would, of course, have made a significant contribution to the assessment of

Non-destructive examination of pressure vessels 273

defect detection. Apparently, such information has to be inferred from other evidence.

Factors determining the effectiveness

A number of questions addressed factors that could determine the effectiveness of non-destructive examination of pressure vessels such as examination techniques, technical parameters of the object and access to it, environmental conditions, mangement, organization and administration, and finally competence of the operator. The answers given on these points were rather conventional. Of most interest were the comments on operator competence.

While the importance of the system for quality assurance as a whole (question (3) above) was stressed by some but not all, it was quite clear that the operator's personal competence was considered crucial. The common answer to questions related to possible improvements of the efficiency was also educational. The major recommended effort was making the operators more knowledgeable about the welding procedures.

All the specialists interviewed shared the view that not much could be achieved with respect to efficiency by further automation of the testing procedures.

DISCUSSION AND CONCLUSIONS

A question of paradigms

Before entering the discussion of the results and their interpretation as presented in the previous section, a general comment will be made on this study and its methods. More precisely the following remarks refer to the criticism that may be advanced against applying other than so-called frequentistic methods in assessments of probabilities. The points about subjective estimates by experts made below are by no means meant to be exhaustive but are only mentioned to relate briefly the observations in this survey of some general aspects of the problem of expert judgement. For a more complete treatment of this subject the reader is referred to the literature on probabilistic safety analysis, s'6

In particular, this criticism relates to subjective estimates. It is true that probabilistics arrived at by asking people about their opinions differ in nature from so-called frequentative data, such as figures for the outcome of games of dice. Admittedly, subjective probabilities cannot be compared with

274 Gustaf Ostberg

frequentistic probabilities in a strict statistical sense. Nevertheless, they possess a certain value as a basis for considerations and decisions of risks, which is the ultimate purpose of fracture mechanical probabilistic assessments. This value is relative to the more arbitrary subjective assessment that should otherwise be necessary by those who have to make the ultimate decisions about the acceptance of nuclear power. From this point of view the subjective estimates of defect detection, in spite of their deficiencies, represent the only evidence of some statistical quality that may be used.

It has been argued that inference from investigations of the detectability of defects (to be distinguished from detection of defects) might be used as an input in probabilistic analyses. In order to make such information useful for this purpose, however, one has to treat it in ways that will prove to be even more pragmatic than the application of subjective estimates. It therefore appears as if--at least for the time being--we have to make do with subjective probabilities.

In the literature on probabilistic safety analysis referred to above one can find, among other things, an analytical instrumentalism to be utilized in order to obtain measures on reliabilities of subjective probabilites. While such information would also be desirable regarding the detection of defects in nuclear pressure vessels it appears that the uncertainty in question is of such a nature that the present approach using a less strict method may be justified. Accordingly, the aim of this survey has been to identify and characterize the uncertainties rather than to evaluate and measure them. In order to illustrate this critical aspect of the work by reference to an example, attention is called to the observation mentioned above that some of the interviewed specialists expressed the opinion that there are in fact no defects present in nuclear pressure vessels.

Therefore, the present study does not aim to give a complete and final answer to all possible questions about the reliability and trustworthiness of non-destructive examination of heavy wall steel pressure vessels. The principal aim has been to elucidate our knowledge in certain respects and make it more obvious or transparent. Still there probably remain several aspects that need to be treated using the same paradigm as in this study, namely to consider qualitative as well as quantitative phenomena.

While the technique for non-destructive examination as such should in principle be assessed on the basis of its 'hard', quantifiable qualities, its use in a system comprising people, and organizations made up of humans, certainly includes 'soft' quantitative elements and procedures. The balancing of these factors is not provided by their weights as determined independently, but rather by their importance for the ultimate purpose. Therefore, a development of the assessment of the efficiency of non- destructive examination should also involve the decision makers.

Non-destructive examination of pressure vessels 275

Suggestions for improvement

The main conclusion of this study is, obviously, that the trustworthiness of current opinions about the efficiency of non-destructive examination of nuclear pressure vessels is less than usually assumed. A first, immediate step towards a more satisfactory view would be to revise the figures used previously for the probability of defect detection, and hence the probability of pressure vessel failure.

The next step should be to subject the observations made in this study to discussions jointly by ultimate decision makers, i.e. licensing bodies and local opposing groups, and organizations responsible for inspection of pressure vessels. The form for such discussions needs to be established with great care in order to balance the diverse interests in a way that serves the purpose of aiding the decision makers in assessing the risk of catastrophe due to pressure vessel failure.

In addition to these conclusions, about the general implications of the present study for risk assessments, a number of improvements could be made independently with respect to the non-destructive examinations as such. The weakest point in this system appears to be the competence and performance of the operators. Possible avenues of improvements could be further development of their understanding of the manufacture of pressure vessels, in particular the welding. Another extension of the operators' attitudes towards their work should be to make them aware of the presence of defects and the importance of detecting them.

ACKNOWLEDGEMENTS

The principal investigators have been the author and Mr Stig Dahn, a senior specialist on non-destructive testing with experience from inspection of nuclear installations, from Scandinavian Quality Assurance Services (formerly STK Konsult Company) and SD Utvecklingskonsult KB, Sweden. As consultants, a work psychologist, Mr Bo Rydnert of SYNTECO, and a statistician, Mr Per Ofverbeck of the Swedish State Power Board, have also been engaged; both have previous experience of similar studies on reliability and protection against failure.

During the planning of this study, advice and encouragement have been received from Drs G. Dau and M. Behrawesh of the Electric Power Research Institute. USA, as well as from Dr Spencer Bush of Battelle Richland, USA.

REFERENCES

1. An assessment of the integrity of PWR pressure vessels. Report by a Study Group Chaired by Dr W. Marshall, UKAEA, HMSO, 1978.

276 Gustaf Ostberg

2. An assessment of the integrity of PWR pressure vessels. Second Report by a Study Group Chaired by Dr W. Marshall, UKAEA, March and June 1982.

3. An assessment of the integrity of PWR pressure vessels. Addendum to the Second Report of the Study Group, since 1982 under the Chairmanship of Professor Sir P. B. Hirsch, FRS, April 1987.

4. Dau, G., Behravesh, M. & Doctor, S. R., Status of stainless steel pipe inspection. In Fourth International Seminar on Assuring Structural Integrity of Steel Reactor Pressure Bounda O' Components, SMIRT 8 Post Conference Seminar 2, Ispra, Italy, 26-27 August 1985.

5. Apostolakis, G., Expert judgement in probabilistic safety assessment. In Accelerated Life Testing and Experts' Opinions in Reliability, ed. C. A. Clarotti &

D. V. Lindley. North-Holland, Amsterdam, 1988. 6. Ostberg, G., On the meaning of probability in the context of probabilistic safety

assessment. Reliability Engineering and System Safeo', 23 (1988) 305-8.