1 Chris Haught, Instructor for NCS Code Validation (ANS-8.24)

1

Chris Haught, Instructorfor

NCS Code Validation (ANS-8.24)

2

SCOPE OF TOPIC:

• Purpose

• Requirements

• Issues

• Process

• Examples

3

PURPOSE:

(1) To establish the adequacy of a computer, code and cross section set (“the computational method”) for its intended use,

And

(2) To establish the calculational bias and uncertainties, and an acceptance criteria such that there is confidence that the calculated results thought to indicate subcriticality reliably indicate subcritical systems.

4

BASIC REQUIREMENTS (from ANS 8.24)

• Determine calculational bias

• Determine uncertainties

• Establish the area of applicability

• Establish a margin of subcriticality

• Document work and conclusions

5

CASE-SPECIFIC VALIDATIONS

• Based on a few (but usually >10) critical experiments.

• Experiments very similar to system being evaluated.

• Limited area of applicability.

• For operation-specific or unique situations.

6

GLOBAL VALIDATIONS

• Based on a wide variety and large number (>50) critical experiments.

• Wide area (range) of applicability.

• For general use, is the frequently preferred, economic approach for large-scale process facilities.

7

BIAS

• A measure of the systematic difference between the experimental data and the calculated results.

• Bias may not be constant (especially for global validations); it may be a function of one or more independent variable(s) due to trends.

8

TWO COMMON TECHNIQUES FOR DETERMINING (EXPRESSING) BIAS

•Taking the difference between the lowest of the calculated results (in a group) and unity. (bias is a single, fixed value).

OR

•Taking the difference between a regression of the calculated results, as function of some independent variable(s), and unity (bias not constant, is a function of one or more variables).

9

BIAS UNCERTAINTY - THREE SOURCES

(1) The critical experiment:

• Material specifications - fuel and hardware (composition, assay, mass, density, etc.).• Dimensional specifications - fuel and

hardware (experimental apparatus, surroundings, etc.).

• Experimenters' manipulation and adjustments to the reported data.

10

UNCERTAINTIES - THREE SOURCES (Cont’d)

(2) The computational technique

• Convergence criteria (deterministic codes) or statistical variation (Monte Carlo codes) (usually negligible). • The mathematical equations solved and approximations used.• Cross-sections and manipulation of the cross-sections.• Limitations of the calculational technique (geometry limitations, multiple fissile nuclides, etc.).

11

UNCERTAINTIES - THREE SOURCES (Cont’d)

(3) The code validator

• Geometric modeling and code options used.• Interpretation of the results.

12

ERRORS

User errors (input description, coding error, etc.) are not uncertainties within the scope of validation efforts.

Programmer errors (error in cross section set, error in coding) are within the focus of validation efforts, but do not necessarily factor into the value of kUSL.

User beware of programmer validations! Code validation is the user’s responsibility.

The value of kUSL is never intended to compensate for errors!

13

ERRORS

Properly validating a safety code for its intended application is important.

However, error-free code use and proper application of the results is even more important.

14

AREA OF APPLICABILITY (Type of System)

Examples -• Fissile Material Composition• Uranium (high or low enriched)• Plutonium (high or low 240Pu content)• Solid or liquid• Metal or compounds (oxides, nitrates, etc.)• Moderation (dry, water, carbon, Be, etc.)• Homogeneous or heterogeneous• Geometry configuration• Single unit or interacting units (arrays)• Reflected or unreflected

15

RANGE OF APPLICABILITY(Defined in terms of physical or nuclear characteristics, usually with numerical limits)

Examples - • Moderator ratio (H/X = 10 to 1000)• Average energy group (AEG = 6.5 to 21.3)• Energy at average lethargy (E = 0.025 to 300 keV)• Dimensionless parameters such as ratio of total to thermal fissions (Fth/F = 1.0 to 5.0), ratio of total to thermal absorptions (Ath/A = 0.2 to 0.8), thermalness factor (CF = Ath

2/(A.Fth) For dimensionless parameters: Reference Y. Nomura and T. Shimooke, “A Multiple Regression Equation for Calculated Keff Bias Errors by Criticality Code System,” Nuclear Technology Vol. 65, 1984.

16

ACCEPTANCE CRITERIA FOR CALCULATED SAFETY-BASIS RESULTS

kUSL = 1.0 + Bias - Bias Uncertainty – MSM

where• kUSL = maximum allowable keff accepted as being subcritical, with some defined confidence (“Upper Subcritical Limit”). • Bias = estimate of expected difference between computed and experimental results for benchmarks (constant or a function)• Bias uncertainty = variability/uncertainty in the bias estimate (constant or a function also)• MSM = minimum safety margin, generally ranges from 0.02 to 0.05 (same as “MMS” in NRC terms).

17

NOTE: A fissionable material system with keff ~ kUSL (ksafe), in general, does not necessarily represent a “safe” system.

It (should) represent a system that is reliably subcritical provided no other changes to the system occur.

Typical criterion statement:

If k(process upset) + 2calc < kUSL,

then the postulated process upset configuration is reliably subcritical.

18

WRITTEN REPORT SHOULD ADDRESS:

• Description of the critical experiments - Materials - Dimensions

• Description of the computational models and the computational method - Differences between experiments and models - Calculational technique - What cross section set was used - Cross section processing options used - Other code options

19

WRITTEN REPORT SHOULD ADDRESS: (Cont’d)

• Description of the validation technique - Determination of bias, bias uncertainty - Range/area of applicability - Margin of subcriticality (recommended kUSL)

20

http://icsbep.inel.gov

Pedigreed descriptions of many critical experiments are available in The International Handbook of Evaluated Criticality Safety Benchmark Experiments (IHECSB). This allows the validation report to cover these areas by reference, representing both an effort reduction and quality improvement tool for the NCS analyst.

Updates and additions are performed each year,DVD versions free of charge to NCS personnel.

21

Frequently raised issues about the validation:

• The statistical methodology used.

• Whether a "positive" bias can be credited.

• Adequacy of the margin of subcriticality (kUSL and what its basis is)

• Adequacy of cross section data for certain materials

22

Frequently raised issues about use of the validation:

• How close does the system of concern need to "fit" the range of applicability?

• Can a very broad range of applicability be inferred?

• Are statistical processes even applicable? (Each critical experiment measurement is a unique measurement of some combination of materials, configuration, and condition. Statistics views all experiments as a common effort to measure the “same” condition (keff = unity.)

23

PROCESS

An example decision process for statistical treatment.

Validation ExperimentsSelected and Modeled

Develop Single SidedLower Confidence

Band

Clear Trend in Dataidentified?

Yes

No

Good PhysicsUnderstanding?

NormalDistribution?

Validate With IncreasedSafety Margin or Non-parametric techniques

Yes

No

Develop Single SidedLower Tolerance

Band

Develop Single SidedLower Tolerance

Limit

No

YesSufficient Amount of

Data Available?

Yes

No

Tabulate Data and PerformRegression Fit

Leastconservative

Mostconservative

Consider what thedesired area of applicability is

24

SOME DEFINITIONS OF TERMS:

Population:There are two types of populations, (1) the actual or observed population (seen), and, (2) the true or parent population (not seen).

The actual or sample population is the set of calculated keff values for the available experiment set. The true or parent population is the statistical extension of the observed population to a hypothetical infinite number of experiments within the area of applicability.

25

TERM DEFINITIONS: (Cont’d)

Deviation: - A deviation is the difference between a sample point and the mean of all sample points.

Variance: - Average of the squares of the deviations about the mean or keff correlation.

Standard deviation: - The square root of the variance for a normal (Gaussian) distribution.

26

IMPORTANT DEFINITIONS FOR LIMIT TYPES:

Tolerance Limit: - A single, lower limit above which a defined fraction of the true population of calculated keff values are expected to lie with a prescribed confidence within the area of applicability.

The tolerance limit is widely used by NCS practitioners. It is more conservative than the implied requirements of ANS-8.24, in that it focuses on lower-bounding individually computed keff values, not on the bias or the uncertainty of the bias as indicated by the benchmark set. Most likely approach for a case-specific validation.

27


Tolerance Band: - A band about a keff correlation above which a defined fraction of the true population of calculated keff values are expected to lie with a prescribed confidence (the fitted curve lower- bounds the "true" keff population). A tolerance band requires a regression fit to the data; limited extrapolation beyond the area of applicability is permitted.

The tolerance band is also widely used by NCS practitioners. It is more conservative than the implied requirements of ANS-8.24 because of focus on individual keff values. Most likely approach for a global validation.

28


Confidence Band: - A band about a keff correlation that lower bounds the true correlation with a prescribed degree of confidence (the fitted curve lower-bounds the bias and accounts for uncertainty of the bias). The confidence band requires a regression fit to the data; limited extrapolation beyond the area of applicability is permitted.Although the confidence band is not widely used by NCS practitioners, its focus on bias and bias uncertainty is more in line with the requirements of ANS-8.24. It is the least conservative of the three limit types and requires more practitioner judgment as its usage plus the choice of MSM. Apparently not endorsed by the NRC; it is omitted from NUREG/CR-6698, “Guide for Validation of Nuclear Criticality Safety Calculational Methodology.”

29

ILLUSTRATION:

Tolerance Limit, Tolerance Band, and Confidence Band Note 1: Limits shown do not include MSM. Note 2: Observe that some data points have keff values below that of the confidence band correlation.

0.970

0.975

0.980

0.985

0.990

0.995

1.000

1.005

1.010

0 200 400 600 800 1000 1200

Independent Variable

Calc. keff

Range of Applicability

Correlation or Mean of Population

Tolerance Limit Confidence Band Tolerance

Band

30

SOME QUESTIONS:

What to do if the benchmark experiments have keff values not equal to unity?

For statistical processing, use normalized keff values: keff(normalized) = keff (calculated)/keff (experimental)

What/how many independent variables should be used for developing the fit?

Problem dependent. Usually the fewer variables involved, the better that the bias and area of applicability are understood.

31

SOME QUESTIONS: (Cont’d)

What type of curve fits will work?

Problem dependent. The simpler the better. Use one variable and a straight line fit wherever practical, or even a non-statistical method if practical.

How does the analyst know there is a "good fit"?

Various statistical tests are available: linear correlation coefficient (provided by EXCEL), F-Ratio are two examples.

32

EXAMPLES:

1 Lower tolerance limit

2 Nonparametric limit

3 Lower confidence band

4 Non-statistical method

33

EXAMPLE 1: Lower Tolerance Limit

Need: A kUSL value for low-enrichment, stainless-steel clad UO2 PWR fuel rods, as single water-submerged lattices, or groups of lattices separated by thin-wall steel plates

34


Code and cross section, computer:

KENO-V.a as SCALE 4.4a routine CSAS25,ENDF/B-V – based 238-neutron-energy group libraryWS5, a controlled and verified workstation at ORNL

35


Code Options:

“Latticecell” for cross-section processing.

Standard specifications for the KENO-V.a case inputs included:

GEN=550 (number of generations)NPG=10000 (number of neutrons per generation)NSK=50 (number of generations to skip)SCT=3 (order of scattering, P5)

In general, each KENO-V.a result is based on ~5,000,000 neutron histories.Flat neutron start, no albedoes or reflector biasing options used.

36


Selected Benchmark Experiments:

The LEU-COMP-THERM-001 experiments had square-pitch lattices of 2.032 cm, and a 235U enrichment of 2.35 %. In one experiment, all involved fuel rods were in a single cluster. In the remaining seven experiments, the rods were arranged in three separate clusters, with spacing between the clusters ranging from ~4 cm to ~ 12 cm.The LEU-COMP-THERM-002 experiments had square pitch lattices of 2.54 cm and a 235U enrichment of 4.31 %. The experimental material was obtained from N. S. Savannah Core II fuel assemblies; the fuel pellets were of the same diameter as for the lower enrichment (3.9%) Core II fuel.The LEU-COMP-THERM-007 experiments involved a 235U enrichment of 4.738 %. Both square and triangular lattices were involved, with fuel rod pitches varying from 1.26 cm to 2.52 cm.

37


Selected Benchmark Experiments: (Cont’d)

The LEU-COMP-THERM-009 experiments involved a 235U enrichment of 4.31 %, using the same experimental rods as for LEU-COMP-THERM-002, in clusters of three fuel rod lattices similar to those considered in LEU-COMP-THERM-001. In the LEU-COMP-THERM-009 experiments, absorber plates of various materials were located between the two outer clusters and the central cluster. Square lattices of 2.54 cm pitch were involved. Only the first four experiments were selected for inclusion in this validation; those experiments involve mild steel plates of 0.485 cm (0.191-inch) or 0.302 cm (0.119-inch) thicknesses as the absorber plates.

The LEU-COMP-THERM-019 experiments involved a 235U enrichment of 5.00 %. Three experiments in triangular lattices were involved, with fuel rod pitches of 0.7, 0.8 and 1.4 cm. These experiments are distinct from the other experiments utilized here, in that the fuel rods are fabricated using stainless steel as the cladding. (All other referenced ICSBEP low-enrichment fuel reports involve aluminum-clad fuel.)

38


BenchmarkResults:(30 points)

Average Energy at Computed Benchmark Normalized

Case Fission the Average keff Model keff keff

Group Lethargy (eV)

lct001a 208.78 0.0985 0.9948 0.0003 0.9998 0.9950lct001b 208.86 0.0979 0.9936 0.0004 0.9998 0.9938lct001c 208.96 0.0970 0.9930 0.0004 0.9998 0.9932lct001d 208.90 0.0974 0.9941 0.0003 0.9998 0.9943lct001e 209.06 0.0962 0.9914 0.0003 0.9998 0.9916lct001f 208.95 0.0972 0.9939 0.0004 0.9998 0.9941lct001g 209.16 0.0953 0.9937 0.0004 0.9998 0.9939lct001h 209.02 0.0965 0.9919 0.0003 0.9998 0.9921lct002a 207.41 0.1156 0.9939 0.0004 0.9997 0.9942lct002b 207.42 0.1156 0.9959 0.0004 0.9997 0.9962lct002c 207.46 0.1152 0.9951 0.0004 0.9997 0.9954lct002d 207.55 0.1143 0.9944 0.0004 0.9997 0.9947lct002e 207.73 0.1127 0.9926 0.0004 0.9997 0.9929lct007a 197.57 0.2529 0.9897 0.0004 1.0000 0.9897lct007b 207.40 0.1121 0.9944 0.0004 1.0000 0.9944lct007c 212.69 0.0719 0.9966 0.0003 1.0000 0.9966lct007d 214.61 0.0611 0.9975 0.0003 1.0000 0.9975lct007e 196.38 0.2783 0.9886 0.0004 1.0000 0.9886lct007f 207.32 0.1128 0.9943 0.0004 1.0000 0.9943lct007g 212.72 0.0717 0.9976 0.0004 1.0000 0.9976lct007h 197.14 0.2614 0.9909 0.0004 1.0000 0.9909lct007i 207.28 0.1133 0.9935 0.0004 1.0000 0.9935lct007j 212.67 0.0721 0.9967 0.0003 1.0000 0.9967lct009a 207.45 0.1153 0.9941 0.0003 1.0000 0.9941lct009b 207.50 0.1148 0.9943 0.0004 1.0000 0.9943lct009c 207.47 0.1151 0.9936 0.0004 1.0000 0.9936lct009d 207.53 0.1145 0.9948 0.0003 1.0000 0.9948lct019a 193.87 0.3329 1.0074 0.0004 1.0000 1.0074lct019b 202.48 0.1647 1.0030 0.0004 1.0000 1.0030lct019c 215.52 0.0545 1.0047 0.0004 1.0000 1.0047

39


No obvious or significant trend in data if stainless steel clad experiments are omitted.

Apply Shapiro-Wilk test for normality to the 27 points (with stainless clad experiments omitted).

NOTE: Normality test fails with stainless steel clad experiments included, as expected.

40


Terms for Shapiro-Wilk Normality Test

41


Processing of 27Data Points

42


Application ofShapiro-Wilk Testfor Normality ofData

Shapiro-Wilk Test for Normal Distribution

ai values from Table A.2 of NUREG/CR-6698

kn+1-I values are ranked from lowest value upward,

kI values are ranked from highest value downward.

index (i) an+1-i kn+1-i ki an+1-i*(kn+1-i-ki)

1 0.4366 0.9886 0.9976 0.0039292 0.3018 0.9897 0.9975 0.002354

3 0.2522 0.9909 0.9967 0.001463

4 0.2152 0.9916 0.9966 0.001076

5 0.1848 0.9921 0.9962 0.0007586 0.1584 0.9929 0.9954 0.000396

7 0.1346 0.9932 0.9950 0.000242

8 0.1128 0.9935 0.9948 0.000147

9 0.0923 0.9936 0.9947 0.000101

10 0.0728 0.9938 0.9944 0.000044

11 0.0540 0.9939 0.9943 0.000022

12 0.0358 0.9941 0.9943 0.000007

13 0.0178 0.9941 0.9943 0.00000414 0.0000 0.9942 0.0000 0.000000

Y = San+1-i*(kn+1-i-ki) = 1.0543E-02

Y2 = 1.1115E-04

W = Y2/S2 = 0.94497

Test statistic values are obtained from Table A.5 of NUREG/CR-6698.

W, the test statistic, exceeds the required value (for 27 points) of 0.923.

The data therefore may be assumed to have a normal distribution.

43


Determination of USL (kUSL)

KL = keff – USp

(For NRC, credit for positive bias not permitted:If keff > 1, then KL = 1 - USp)

USL = kUSL = KL – MSM,

where MSM > 0.02.

44


Determination of SP2 (Variance about the mean):

Datapoint uncertainty

Variance about the mean

Average total uncertainty

Similar terms apply for non-weighted application.

45


Determination of SP2 (Variance about the mean):

Continued

Weighted average keff

Square root of pooled variance

Similar terms apply for non-weighted application.

46


Determination of USL (kUSL) (Nonweighted Example)

Calculation of lower tolerance limit

The variance about the mean, Sp2 = 4.5241E-06

Sp = 0.0021270

For a confidence factor of 95%, and a proportion factor of 95%,and a data set of 27 points, Table 2.1 of NUREG/CR-6698 provides

a One-Sided Tolerance Limit Factor: U = 2.292.

KL = kaverage - USp = 0.98912

For a minimum safety margin (MSM) of 0.02:

kUSL = KL - MSM = 0.96912

47

EXAMPLE 2: Non-Parametric Limit

Need: A kUSL value for highly enriched uranium as single or multiple units, in well-moderated forms (i.e., mixed with water).

In the process application, moderating reflectors of concern are water or concrete (or both).

48


49


Picked many highlyenriched uranium(HEU) solution experiments…

50


51

EXAMPLE 2: Non-Parametric LimitCheck for Bias as Function of AEG

52

EXAMPLE 2: Non-Parametric LimitCheck for Bias as Function of EALF

53

EXAMPLE 2: Non-Parametric LimitCheck for Bias as Function of H:235U Ratio

54

EXAMPLE 2: Non-Parametric LimitCheck for Bias as Function of Nitrate Content

55

EXAMPLE 2: Non-Parametric LimitCheck for Bias as Function of 235U Concentration

56

EXAMPLE 2: Non-Parametric LimitCheck for Qualitative Bias

57

EXAMPLE 2: Non-Parametric Limit90 Cases Fail Chi-Square Normality Test (k=0.004)

58

EXAMPLE 2: Non-Parametric LimitFails Chi-Square Normality Test Again (k=0.002)

59

EXAMPLE 2: Non-Parametric LimitGuidance from NUREG/CR-6698

60

EXAMPLE 2: Non-Parametric LimitGuidance from NUREG/CR-6698

61


Application

There are n = 90 data points.The confidence level that 95% of similar critical systems will have computed keff values greater than the lowest computed keff value is

= 1 – qn = 1- (0.95)90

= ~ 99%

62


Our degree of confidence is ~ 99% (for 95% of the population); an NPM value of 0.0 is appropriate.

63


Lowest computed keff value is 0.9913,with calc = 0.0005 and expt = 0.0025;

total = 0.00255.

kUSL = smallest keff value - uncertainty of smallest keff value - non-parametric margin - MSM

kUSL = 0.9688 for MSM = 0.02.

64

EXAMPLE 3: Lower Confidence Band

Equations are more complicated, especially if data uncertainty (weighted values) are used. From NUREG/CR-6698:

Note: Omission of this term results in the confidence band fit

65


66


All 76 thermal and epithermal 233U solution benchmarks in the 2001 IHECSB Handbook: keff vs AEG

67

EXAMPLE 3: Lower Confidence BandAll 76 thermal and epithermal 233U solution benchmarks in the 2001 IHECSB Handbook: keff vs EALF

68

EXAMPLE 3: Lower Confidence BandAll 76 thermal and epithermal 233U solution benchmarks in the 2001 IHECSB Handbook: keff vs 233U Concentration

69

EXAMPLE 3: Lower Confidence BandAll 76 thermal and epithermal 233U solution benchmarks in the 2001 ICSBEP Handbook: keff vs H:233U atom ratio

70

EXAMPLE 3: Lower Confidence BandAll 76 thermal and epithermal 233U solution benchmarks in the 2001 IHECSB Handbook: keff vs NO3 content

71


Final confidenceband expression

72


Final confidenceband tabulatedvalues

73


74

EXAMPLE 3: Lower Confidence BandArea of Applicability Statement

75

EXAMPLE 4: Non-Statistical Method

Need: A kUSL value for hydrided U-Zr alloy rods sealed within Hastelloy N tubes. The alloy is ~10 weight % U(93) and ~90 weight % Zr, hydrided to ~6.5 x 1022 H atoms/cm3 (solid-moderated fissile material, H:235U ~ 50).

The fissile material is in a space reactor configuration (4.6 kg 235U in 8" dia by 14" core vessel). The process is to safely defuel the reactor. Potential process-condition reflectors include water, concrete, and personnel. Potential moderators include water.

76


Selected Benchmark Experiments:

A total of 73 experimental benchmarks were derived from NAA-SR-8490 and NAA-SR-9871 (these experiments are not in the IHECSB) .

These two reports detail 56 critical and 17 subcritical arrangements of the precise fissile material items (hydrided Zr-U alloy rods) as involved in the process application.

Many of the experimental configurations are highly comparable to the potential off-normal process conditions of concern.

77

0.86

0.87

0.88

0.89

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Number of Fuel Elements in Core Vessel (N)

Ca

lcu

late

d k

-eff

+/-

Es

tim

ate

d T

ota

l Un

ce

rta

inty

Quantifiable SNAP 10A/2 Measurements Normalized to Critical (56)

Experimentally Sub-Critical, Un-Normalized SNAP 10A/2 Results (17)

Data points may be artificially separated around integer values

of N to enhance their legibility


Results:

Note: The indicated uncertainty value for each point is ~ + 0.008; this value is inferred from the variability of experimentally critical test results. (The uncertainty for the subcritical points should be somewhat larger than shown.)

78


Conclusions:

No strong bias is evident in the benchmark results. Whatever bias that may exist is expected to be of the same magnitude and direction for both the benchmark results and the application results.

The overall spread of the computed keff’s for the benchmark configurations is taken to represent the inherent uncertainty of the method for this application.

A limit for the use of the computational method is specified such that conservatively derived values of kcalc + 2calc < 0.975 for hypothetical conditions of interest are interpreted to represent sub-critical systems.

The area of applicability is specifically limited to: disassembly and defueling of the space reactor core, handling and repackaging of the fuel rods into DOT-approved shipping containers.

79

Essential References

• * ANSI/ANS-8.24-2007 Validation of Neutron Transport Methods for Nuclear Criticality Safety Calculations

• NUREG/CR-6698, "Guide for Validation of Nuclear Criticality Safety Calculational Methodology," 2000.

• FCSS ISG-10, Nuclear Regulatory Commission Interim Staff Guidance 10, "Justification for Minimum Margin of Subcriticality for Safety," 2006.

* Copyrighted – must be purchased from ANS or obtained via organization/site license

Test• Know the reasons why NCS evaluations are performed• Know what the process analysis requirement is• Know what the double contingency principle is (not a requirement!)• Be able to recognize whether certain configurations meet PA and

DCP• Understand the steps in performing a CSE• Given a one-line diagram, identify how certain components and

interfaces should be controlled• Know the basic requirements in ANS 8.3 and 8.23• Understand the interface between ANS 8.3 and 8.23• Know how the “minimum accident of concern” and “bounding

accident” are used with respect CAAS and IEZ calculations• Understand the goals of an IEZ• Understand the benefits and consequences of a CAAS• Concept of drills and exercises• Know terminology associated with code validations• Know the conditions under which certain statistical methods can be

applied80

Documents

1 Chris Haught, Instructor for NCS Code Validation (ANS-8.24)