Upload
alberta-floyd
View
214
Download
0
Embed Size (px)
Citation preview
April 1, 2008TPTF
Quality Center Update
Eileen Hall
22
Quality - Summary
• Recent Successes
– CRR1 has exited FAT
– MMS3 has exited FAT–
EMS3B and EMS4.1 have exited FAT • Areas of Concern
– Lengthy defect resolution time delays test phase migration
– Late detection of detects exacerbates defect resolution time
• Mitigation Measures
– Passive data center site with live data feed will allow for earlier defect detection
– Additional measures (e.g. key vendor resource retention) in progress
Nodal Quality effort experienced recent successes but concerns remain
33
Nodal FAT Active Defects by Severity by Project
Overall, active defects are being tracked by project and by classification. Problems areas include:
Network Model Management System (NMMS) has 12 Severity 1 defects
Outage Scheduler (OS) has 6 Severity 2 defects
Congestion Revenue Rights (CRR) has 2 Severity 2
Note - Does not include closed and deferred defects
The relative low number of Severity 1 and 2 defects in FAT is encouraging
44
Nodal Active Defects by Severity Trend
Sev 3
Sev 4
Sev 1
Sev 2
• Severity 3 trend is rising due to focus on fixing higher priority severity 1 and 2 defects• Severity 2 trend indicates “defect closure rate” is slightly higher than “defect open rate”
55
Average Days to Fix Severity (1, 2, 3) Defect by Vendor
Defect turnaround time is a key indicator of vendor performance; increased emphasis being placed on vendor defect resolution as needed
66
Nodal % Reopened Defects by Vendor
Indicates what percentage of fixes delivered by a vendor failed retest.
ABB and AREVA are fixing their products with a low reopen rate, a good indication that the EMS and the MMS vendors are sufficiently testing their software defect bundles prior to releasing software to ERCOT
A high reopen rate is an indication of future vendor performance
Smaller vendors have experienced the most difficulties in terms of defect reopening
77
CRREIPEMSMMSMMS MM
Defect Summary Graph
Severity
1 D
ata
Loss
/Crit
ical
Err
or
2 Lo
ss o
f fun
ctio
nalit
y w
/o w
o..
3 Lo
ss o
f fun
ctio
nalit
y w
ith w
..
4 P
artia
l los
s of
a fe
atur
e se
..
5 C
osm
etic
/Doc
umen
tatio
n er
ror
6 E
nhan
cem
ent
Num
ber
of D
efec
ts
70
60
50
40
30
20
10
0
2 3
68
33
61
Defects found in the Early Delivery Systems (EDS)
Severity 1 – Data Loss/Critical Error
Severity 2 – Loss of Functionality w/o workaround
Severity 3 – Loss of functionality with workaround
Severity 4 – Partial Loss of a feature set
Severity 5 – Cosmetic/Documentation Error
Severity 6 – Enhancement
Severity
1
Severity
2
Severity
3
Severity
4
Severity
5
Severity
6
Focus of EDS testing effort on core functionality, therefore majority of defects detected are severity 1 - 3
Lesson learned: EMS 2 and EMS 3 were released with insufficient FAT testing, resulting in substantial number of defects showing up in the Early Delivery Systems
Mitigation measures are being put in place for earlier defect detection, e.g. live data feed in integrated test environment
Enhanced FAT testing should result in increased defect detection prior to EDS
88
1 Data Loss/Critical Error2 Loss of functionality w/o wo..3 Loss of functionality with w..
Num
ber
of D
efec
ts
30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
0
1 1
29
26
16
9
4
Notes: 1) Data only includes defects identified in EDS testing (i.e. defects that migrated into EDS from FAT are excluded)2) Data does not include closed or deferred defects 3) Release 8 (Credit, Settlement and Billing) not yet in EDS; no EDS identified defects in Release 4 at this time
EDS Active Defects Per Release by Severity
EDS active critical defects (Severity 1 and 2) are concentrated in Releases 3, 5, and 9*
* State Estimator (SE) tuning, Security Constrained Economic Dispatch (SCED) and Locational Marginal Price (LMP) posting, and Day Ahead Market (DAM), et.al. respectively
Release No.
21 53 6 97
Current testing in EDS represents approximately 25 - 30% of code to be tested in EDS
The number of defects associated with remaining EDS testing is expected to increase significantly due to the increasing complexity associated with systems’ integration
Timely fixes to the currently unknown number of defects in the code remaining to be tested in EDS represents a major project challenge