Robert Brient

Quality Assurance Specialist

Center for Nuclear Waste Regulatory Analyses

Southwest Research Institute®

San Antonio, Texas, USA

Presentation Outline

Review Objective

Review Approach

Code Documentation and QA Reviews

Review Findings

SKB Code QA Program Review and Findings

Conclusions

Back-Up Slides

Preliminary Initial Review Findings, 2012 May 21–23 2

Review Objective

Determine whether SKB code documentation

and quality assurance (QA) practices are

adequate to provide sufficient confidence that the

SR–Site is supported by valid and accurate

modeling and calculations.

Determine whether additional information

regarding SKB code documentation and QA

practices should be provided by SKB for

subsequent SR–Site reviews.

Preliminary Initial Review Findings, 2012 May 21–23 3

Review Approach

Code documentation and QA reviews

— General review of all codes used in

SR–Site

— Detailed review of selected codes

Supplemental review of SKB code QA

program requirements

Preliminary Initial Review Findings, 2012 May 21–23 4

Code Documentation and QA Reviews General Review

— Primary Source: SR–Site Model Summary

Report TR–10–51

— There are 22 codes described, covering topics such as code

suitability; usage; development process and verification; and

passing data between models

— Codes are categorized by source for applying graded

quality assurance

– Commercial (10 codes)

– Modified-commercial (3 codes)

– Project-specific (11 codes)

– Several codes are in more than one category

Preliminary Initial Review Findings, 2012 May 21–23 5

Code Documentation and QA Reviews

(continued)

Detailed Reviews

— Cross-section of codes in various categories selected

— Code_Bright, CONNECTFLOW, 3DEC, and MATLAB–FPI

— Model Summary Report references are primary sources for

detailed reviews

Preliminary Initial Review Findings, 2012 May 21–23 6

Review Findings

Information regarding suitability of the codes for their

SR–Site applications appears to be adequate

User’s manuals are available where appropriate

Information for the development process and verification

varies widely

— Software standards applied to development

are identified for Abaqus and ConnectFlow

(commercial) and MARFA (project specific);

none mentioned for other codes

Preliminary Initial Review Findings, 2012 May 21–23 7

Review Findings (continued)

— Model Summary Report identifies comprehensive

verification and validation documents for 3DEC, Abaqus,

CODE_BRIGHT, ConnectFlow, DarcyTools, FARF31,

MARFA, MATLAB–COMP23, TOUGHREACT

— PHAST and PHREEQC descriptions indicate extensive

testing, but without references (documentation is

available at U.S. Geological Survey websites)

— Other codes had references for verification activities

Data exchange information provided is adequate for

all codes

Preliminary Initial Review Findings, 2012 May 21–23 8

SKB Code QA Program Review

and Findings

Based on the variability in code documentation and QA

among codes noted in the general and detailed reviews, the

reviewers examined SKB code requirements.

— SKB requirements focus on the Model Summary Report

format and content rather than directly providing

requirements for codes, particularly for the verification

and validation necessary to demonstrate that codes

produce correct results for the purpose for which they are

being used.

Preliminary Initial Review Findings, 2012 May 21–23 9

SKB Code QA Program Review

and Findings (continued)

— SKB applies code development and verification requirements only to modified commercial and project-specific codes. However, the Model Summary Report has information for commercial codes as well. Verification information, in particular, is important for commercial codes and should be developed where needed.

— SKB requirements are much less detailed than recognized software QA standards (i.e., NQA–1, Subpart 2.7 or ISO 9001). Additional guidance in SKB instructions may reduce variability and enhance overall confidence in codes producing correct solutions.

Preliminary Initial Review Findings, 2012 May 21–23 10

Conclusions

Confidence in SKB code documentation

and QA practices could be enhanced with

comprehensive verification and validation

documents for each code, including

commercial codes. DarcyTools verification

and validation is an excellent example.

SKB instructions may need to be

revised for additional detail in order to be

effectively implemented

Preliminary Initial Review Findings, 2012 May 21–23 11

Backup Slides

12 Preliminary Initial Review Findings, 2012 May 21–23

Observations: Commercial Codes

Code Proper Usage

Development Process

Followed Appropriate

Procedures

Produces Accurate Results

ABAQUS User’s manual ISO 9001, U.S. Nuclear QA Information on web site

CODE_BRIGHT User’s manual References in MSR

ConnectFlow User’s manual ISO 9001, TickIT References in MSR

Ecolego User’s manual References in MSR

ERICA Tool User’s manual Affirmative statement, but no

references in MSR

MATLAB None identified in Model

Summary Report (MSR), but

User’s manual found in web

search

No references in MSR, but

wide-user base should identify

and correct errors

MIKE SHE User’s manual References on DHI web site

PHAST User’s manual affirmative statement, but no

references in MSR

PHREEQC User’s manual Affirmative statement, but no

references in MSR

TOUGHREACT User’s manual Affirmative statement, but no

references in MSR

Preliminary Initial Review Findings, 2012 May 21–23 13

Observations: Modified Commercial Codes

Code Proper Usage

Development Process Followed

Appropriate Procedures

Produces

Accurate Results

3DEC User’s manual includes coverage

of the FISH language for

modifications

Development process for

modifications not identified in the

Model Summary Report (MSR)

References in MSR

ABAQUS Available user guides are likely to

cover user-defined subroutines

Development process for

modifications not identified in

MSR

User defined subroutines verified

by using simple test examples, no

references provided in MSR.

CODE_BRIGHT‡ CODE_BRIGHT user’s manual

does not appear to cover

modifications

Development process for

modifications not identified in

MSR

References in MSR

Preliminary Initial Review Findings, 2012 May 21–23 14

Observations: Project-Specific Codes

Code Proper Usage

Development Process Followed

Appropriate Procedures Produces Accurate Results

Analytical model for quantification

of buffer erosion and canister

corrosion

Simple user instructions included

in the file

Excel-based code should not

require a formal development

process

References in MSR*

DarcyTools User’s manual MSR indicates no attempt has

been made to show that

DarcyTools conforms to any

international QA standard

References in MSR provide

comprehensive documentation of

verification, validation, and

demonstration

FARF31 User’s manual Development procedures not

identified in MSR

References in MSR includes

validity document, regression

tests of code changes

MARFA User’s manual NQA–1–2000 based software QA

program

Validation tests in user’s manual,

reference in MSR

MATLAB–COMP23 User’s manual Development procedures not

identified in MSR

References in MSR includes

validity document

MATLAB–FPI No user’s manual mentioned in

MSR

Development procedures not

identified in MSR

References in MSR

MATLAB–Pandora Internal user’s manual, code is

documented in a reference

Development procedures not

identified in MSR

References in MSR

Numerical GIA model No user’s manual, but users work

closely with developer

Development procedures not

identified in MSR

References in MSR

Numerical permafrost model No user’s manual, but

descriptions of the code are

referenced

Development procedures not

identified in MSR

References in MSR

Solubility model (Simple

Functions)

No user’s manual, but

presentation provides instruction

on spreadsheet use

Excel-based code should not

require a formal development

process

References in MSR

UMISM No user’s manual, but MSR

indicates close collaboration of

users with developer

Development procedures not

identified in MSR

References in MSR

Preliminary Initial Review Findings, 2012 May 21–23 15