WP 2.4Evaluation of NMFS Toolbox

Assessment Models on Simulated Groundfish Data Sets

Comparative Simulation Tests Overview

Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver

Nothing gives rest but the sincere search for truth.Blaise Pascal (French philosopher)

What did we do?

• Evaluated 5 NFT stock assessment models for three stocks under 4 scenarios meant to examine potential difficulties in real assessments– AIM, ASPIC, SCALE, VPA, ASAP– GB yt (retro), GB cod (domes), white hake (ageing)

• PopSim used to generate true conditions and create 100 datasets with the same random errors for all 5 models

• Evaluated Accuracy and Precision of the 100 point estimates from the models– Did not examine precision of each of the 100 runs

60 scenarios

6000 assessments

Why?

• Test hypothesis that all models are impacted similarly when presented with the same underlying problem

• A priori know that some models will not perform well under the test conditions because limiting data to VPA years

• NOT trying to declare one model “winner”• NOT trying to declare any model “bad”

PopSim Primer

Age and Length Based Population Simulator

User defines• Dimensions (Years, Ages, Plus Group Age, Lengths)• Initial NAA• Recruitment time series (or SRR)• Annual Fmult and selectivity• Biological Characteristics

– M, von B, L-W• Fishery Sampling• Surveys• Sets Template for Stock Assessment Model

Surveys vs Indices

• Surveys – Are a property of the true population– Catchability defined for all ages and years– Uncertainty added to true values at age and length

• Indices– Are a property of the model– Sum values from surveys– Can be either number or biomass based– Can be limited age range or entire age range– Can be changed between models without impacting

underlying truth

Growth

• Initial NAA distributed according to stdev1

• Growth transfer matrices created for each age based on expected von B growth for age and stdev2

• Fish not allowed to decrease in size

• Allows fishing to change distribution of length at age

10

20

30

40

50

60

70

Age

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Age

Random Fish Growth Trajectories

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

8,000,000

Num

bers

10 20 30 40 50 60 70Length

Stock Length Frequency1981

Market Sampling

• Markets declared by user

• Sampling conducted per 100 mt of landings in each market each year

Input Output

Small Large

0.0

3,000.0

6,000.0

9,000.0

12,000.0

15,000.0

18,000.0

21,000.0

1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004Years

Landings (MT)

PopSim Limitations

• PopSim is not reality

• Annual Time Steps

• Does not contain spatial components

• Does not allow gender differences

• Does not allow density dependent effects

• No integrated management– Developing MSE wrapper to use PopSim,

VPA, AgePro, and Control Rules

This Exercise

• Used utility to convert VPA run to PopSim– Gets Nyears, plus group age from VPA– Sets initial NAA and R from VPA– Sets annual Fmult from VPA– Estimates one logistic selectivity from VPA

• Length and biology stuff from user• Market stuff from user• Surveys and Indices defs from user• Tuned markets, sampling, and surveys to

represent actual assessments by lead

Farmed Out Assessments

• AIM – Rago• ASPIC – Brooks• SCALE – Nitschke• VPA – Legault, O’Brien, Sosebee• ASAP – Legault

• Used base case to get template settings reasonable

• Applied this base case to each of the test cases• Some models did additional runs with modified

templates to “fix” the problem

Results

• PopSim compares the distribution of 100 assessments with the known true values

• Exactly what is compared depends on model– E.g. VPA NAA & FAA, ASPIC B & F

• Many, many runs and scenarios– PopSim creates tables and graphs– R program to gather results and automatically

create plots

Black Line TrueCircles and Grey Line MedianRed dashed Lines 5 and 95%iles

Started by looking at direct results

Decided Bias and CV Better

General Conclusions

• Given failure of all models tested (simple & complex), we suspect other models would also be vulnerable to “retrospective agents”

• Use of age-specific indices is robust to uncertainty in survey selectivity

• If ageing is uncertain, these simulations support using models w/o age or models which allow uncertainty in catch at age

General Conclusions (cont.)

• VPA and ASAP ‘failures’ were similar in pattern

• Magnitude of bias was less for ASAP

• Precision usually somewhat better for ASAP

• Given these similarities, we suggest that ASAP may offer some advantages to VPA (esp. in terms of flexibility)