Embed Size (px)
Citation preview
Praveen Srivastava Snr Engineering Manager – Java Oracle Corps, Bangalore [email protected] https://www.linkedin.com/in/praveen-srivastava-388a9012
GB Candidate : Praveen Srivastava
Srl. Version Date Comments
1 Draft Aug 1, 2009 Project definition, SDFSS Tools used, schedule
2 1.2 Sep 24, 2009 Updated with CPM Analysis slides, P chart
3 1.3 Oct 30, 2009 Update Plan, Initial transfer function
4 1.4 Jan 08, 2010 Changed few factors for provisioning dependency
5 1.5 Mar 03, 2010 Example minitab charts
6 1.6 Mar 27, 2010 Updaintg Architectural analysis
7 1.7 Apr 04,2010 Rerun of DOE and sheet update
8 1.8 Apr 09,2010 Updated for FTR - IHLR-K10250
9 1.9 Apr 14, 2010 Updates in progress.
10 1.10 Apr 23,2010 Bulletin issued to the iDEN markets
11 1.11 June 28,2010 Value statement received from Technical Lead
12 1.12 July 22,2010 Updated with MBB review comments and success story
• Success Story
• Product (s) Overview
• SDFSS Project Overview
• SDFSS Deliverables and Tool Usage
• Project Definition (Charter)
• Requirements Phase
• Architecture Phase
• Value Statement
• Lessons Learned
SSPD Program Success Story
Networks (iDEN): HLR Provisioning Project
Green Belt: Praveen Srivastava, Bangalore Risk: High Provisioning Rate Impacting Service ( Lose Registration Request, Impacted HLR Functionality, Failed Provisioning Request, Increased High Severity NPRs)
Success Statement: First application of DFSS tools and methods for iDEN programs Increased maintenance window provisioning rate from 7 msg per second to 15 msg per
second for customers. Reduced RPN from 45 to 24 using DFMEA activities for identified critical risks Generated new feature ideas to reduce MOL effort with architectural findings. CPM and minitab-DOE helped to identify significant factors and their interactions affecting
HLR provisioning rate and establish a mathematical model which is used to predict HLR provisioning capacity. This allows customers to increase provisioning system utilization during maintenance window with 100 % improvement in performance.
No high provisioning rate issues from SR18 upgrade markets (Chicago, Peru) after revised recommendations.
HLR – Home Location Register, NPR – Number of Problems Reported, DOE – Design of Experiments, RPN – Risk Priority Number MOL – Maintenance of Line
Melody HLR
The iDEN system is an integration of traditional Push-To-Talk (PTT), half-duplex, analog radio technology and feature-rich, full-duplex digital cellular communications. This integration of mobile communication technologies provides state-of-the-art functions and benefits to mobile users while optimizing the available infrastructure resources.
The iDEN system provides both full and half-duplex operations. This melding of communications methods allows much of the voice traffic to be run in half-duplex mode, while providing full-duplex functionality when required.
- The iHLR Provisioning System is built on the framework of a Web Server. Transactions sent to the provisioning server are handled by the instantiation of provisioning servlets. These servlets are responsible for updating the subscriber database and propagating the provisioning changes to the iDEN network.
-The provisioning interface to iHLR (iPP), is based upon HTTP/TCP/IP over ethernet.
• HLR Provisioning interface, allows end users to open multiple provisioning sessions ( max 20) using http requests over tcp/ip connection. Provisioning requests are initiated by provisioning clients based on Motorola provided framework termed as iPP ( iDEN Provisioning Protocol).
• HLR handles the provisioning requests using a provisioning server and in turn update database, send message to DAP through MCP tasks and responds back to provisioning client with error or success code.
• HLR’s capability to process these requests is governed by several factors such as mobility message requests, cpu utilization, ethernet utilization, DB response etc. Provisioning craft person ( operator) is suppose to keep the provisioning rate under a certain limit to allow HLR to successfully process all the requests.
• HLR lacks the ability to dynamically determine the provisioning requests it can handle at any given time, which may vary with box performance and other conditions such as load shedding. This results in under utilization of HLR resources under non-busy hour traffic.
• This project aims to analyze the critical parameters affecting the provisioning performance. Identified critical parameter will be analyzed to optimize provisioning performance using modeling simulation and prototyping.
Phase Green Belt Deliverables DFSS Tools
Project Definition SDFSS Charter Charter
Requirements VOC, Requirements Development
and Analysis
QFD, UML Use Case,
Cognition Cockpit
Requirements Perform and document Critical
Parameter flow down and analysis,
Critical Parameter Tree,
Cognition Cockpit.
Requirements Documented risk analysis FMEA
Requirements Perform Initial transfer function. Critical Parameter Tree,
Cognition Cockpit.
Architecture Evaluate Architecture Evaluate Architecture
using Modeling &
Simulation Prototyping.
DOE
Architecture Update risk analysis FMEA
Architecture Measure non-functional Critical
Parameter
Measured Quality
Attributes.
SDFSS Requirements through Architecture
Flow Summary
D&I to optimize Provisioning Performance
SDFSS Methods Used to Optimize HLR Provisioning Performance
Raw VOC Input
Architecture
Simulation VOC HOQ
Use
Cases CPM DOE Brainstorm
Sessions
RunDuration
(D)
DB Size
(K)
MM Rate
(msg/sec)
Roamers
(%)
PT1
Rate
(msg/sec)
PT1
RTT
(ms)
PT1
msg/sec
(R1)
PT2
Rate
(msg/sec)
PT2
RTT
(ms)
PT2
msg/sec
(R2)
Prov
Response
(R1+R2))
Transactions in
prov log
(N)
( N/D )
msg/sec
MAX CPU
(%)
1 300 100 393 20% 55 34 18.08 55 35 18.06 36.14 10912 36.37 68
2 300 100 393 0% 25 30 31.46 25 28 32.73 64.19 19390 64.63 70
3 300 100 393 0% 30 30 31.63 30 28 32.94 64.57 19486 64.95 66
4 300 100 393 0% 20 30 31.55 20 28 32.86 64.41 19442 64.81 68
5 300 100 100 20% 20 25 38.01 20 25 36.33 74.34 22428 74.76 56
6 300 100 100 20% 25 26 37.35 25 25 36.55 73.9 22294 74.31 57
7 300 100 100 0% 25 24 39.6 25 23 38.96 78.56 23686 78.95 58
8 300 100 100 0% 20 24 39.62 20 23 38.72 78.34 23650 78.83 56
9 300 100 100 0% 40 24 24.87 40 22 24.85 49.72 15004 50.01 44
11 300 800 393 20% 25 42 23.07 25 42 22.49 45.56 13748 45.83 72
12 300 800 393 20% 40 41 23.6 40 41 22.9 46.5 14038 46.79 70
13 300 800 393 20% 45 36 22.1 45 35 22.1 44.2 13338 44.46 71
14 300 800 393 20% 55 35 18.08 55 34 18.08 36.16 10739 35.80 66
15 300 800 393 0% 40 29 24.82 40 28 24.87 49.69 15002 50.01 67
16 300 800 393 0% 30 31 30.87 30 30 30.98 61.85 18680 62.27 71
17 300 800 393 0% 35 30 28.39 35 30 27.98 56.37 17022 56.74 69
18 300 800 393 0% 40 27 24.86 40 26 26.87 51.73 15999 53.33 65
19 300 800 100 20% 20 24 40.5 20 25 36.31 76.81 22840 76.13 55
20 300 800 100 20% 25 34 39.69 25 25 35.75 75.44 22766 75.89 56
21 300 800 100 0% 25 24 39.72 25 24 37.35 77.07 23262 77.54 56
22 300 800 100 0% 20 24 40.72 20 25 36.47 77.19 23287 77.62 52
23 300 800 100 0% 15 25 38.43 15 25 35.62 74.05 22351 74.50 55
24 300 800 100 0% 20 25 38.4 20 25 36.4 74.8 22586 75.29 57
Verification 100 250 0% 25 27 35.01 25 27 35.01 70.02 19696 65.65 60
1 300 100 250 20% 30 34 28.93 30 33 28.49 57.42 17328 57.76 63
2 300 800 118 20% 25 24 39.66 25 26 35.6 75.26 22717 75.72 57
3 300 800 118 20% 20 25 38.51 20 26 34.89 73.4 22148 73.83 57
4 300 800 237.5 20% 20 28 34.26 20 28 32.63 66.89 20184 67.28 61
First House Of Quality for Project HLR Provisioning Project
Rating Links Legend
H (9) = High Effect
M (3) = Medium Effect
L (1) = Low Effect
0 = No Effect
Roof Links Legend
Equal Effect: =
No Effect: 0 or Blank String
Relative Effect: +, ++, -, --
Direction of Goodness Legend
Increases Customer Satisfaction: +
Decreases Customer Satisfaction: -
On Target For Customer Satisfaction: 0 or Blank String
9 H H L
4 H M
8 M H M M
10 H M M
3 H M
6 M H M
8 M H M H
8 H M M M
153 105 240 108 108 51 42 18 96 54
15.7% 10.8% 24.6% 11.1% 11.1% 5.2% 4.3% 1.8% 9.8% 5.5%
6 4 10 4 4 2 2 1 4 2
0 0 0 0 0 0 0 0 0 0
System Requirements
Direction Of Goodness
Voice of Customer, VOC's Imp
ort
an
ce
HLR
will
send I
DB
C w
arn
ing a
nd O
MC
ala
rm if
rate
is e
xceeded
HLR
will
contr
ol in
com
img p
rovis
ionin
g
flow
HLR
will
optim
ize M
M t
ask a
nd c
pu
utiliz
ation f
or
pro
vis
ionin
g r
equests
HLR
will
c
om
e o
ut
of
load c
onditio
n
as
soon a
s p
ossib
le
Faile
d p
rovis
ionin
g t
ransation w
ill b
e r
e-
trie
d.
Deta
iled p
rovis
ionin
g t
ransaction
info
rmation w
ill b
e c
olle
cte
d
and s
ore
d
IDB
C e
rror
and s
uccess r
esponses w
ill
have m
ore
info
rmation
HLR
will
support
few
er
pro
vis
ionin
g
transactions u
nder
load c
onditio
ns.
HLR
will
add a
dditio
nal lo
cal ala
rms f
or
inte
rmid
iate
conditio
ns.
Faile
d M
M m
essages w
ill b
e r
etr
ied
Minimize number of HLR provisioning related field
cases
Maximize HLR supported provisioning rate
Minimize load shedding duration in case of exceeded
provisioning rate
Minimize service impact if provisioning operator
stresses the system.
Maximize HLR provisioning transaction information
Minimize failed provisioning transactions
Minimize DVLR and HLR not in synch issues due to
high provisioning rate
Minimize time taken to root cause the issue to high
provisioning rate
Units
Scoring Totals
Relative Scores
Normalized Scores
Target Nominal Values
Start ProvisioningProvisioning
Client
Close Prov Session
Open Prov Session
Authenticate new session<<include>>
Update DB
Send MM Message
Send IDBC response back to
client
<<include>>
<<include>>
<<include>>
Check load condition<<include>>
<<include>>
Experiments run HA HLR to measure impact of factors on provisioning rate
Transfer function found by statistics analysis indicates which factor impacts the provisioning rate and on this basis identifes scenarios where max prov can be supported.
200
72.5
70.0
67.5
65.0
62.5
60.0
57.5
55.0
Roamers
Me
an
100
800
of subs
Number
Number of subs and Roamers
393100
80
70
60
50
40
MM Rate
Me
an
100
800
of subs
Number
Number of subs and MM Rate
200
80
70
60
50
40
30
Roamers
Me
an
100
393
Rate
MM
MM Rate and Roamers
393100 200
80
60
40
80
60
40
Number of subs
MM Rate
Roamers
100
800
of subs
Number
100
393
MM Rate
Number of subs, Roamers and MM Rate
House of Quality is build to identify most critical customer need.
decompose requirement into its functional pieces
Business Case •There are close to 112 HLR deployment across the globe which comprises of iDEN, Harmony and
Melody releases. Existing implementation do not have any mechanism to control or monitor the
provisioning flow. HLR provisioning performance remains same even when box resources are under
utilized or box experiences load conditions. This project addresses this issue and provides a solution
for optimizing provisioning performance to be implemented in new releases.
•Field issues due to high provisioning rate ( than recommended) are escalated to development for
analysis. The solution will monitor the rate and send alarm for high provisioning, thus reducing the
number of such field cases escalated to DART or development. In 2008, 4 field cases out of total 13,
were investigated for high provisioning rate by development
•Working on this project will reduce the field support and improve customer satisfaction.
Opportunity Statement Potential gains from this solution :
•Key focus of the project will be to identify critical parameters affecting customer’s provisioning
performance.
•Results will enable development team to implement improved provisioning performance in new
releases.
•Implemented monitoring solution in future releases will reduce number of field cases. Each of these
cases are complex in nature and consume lots of effort, this will approximately reduce support effort
by 25%.
•In some cases customer stresses the system beyond recommended rate. By completing this project
we can make the system more robust and better handle customer business.
What pain are you currently experiencing :
•Under utilized provisioning performance.
•No mechanism to control provisioning flow.
•Many HLR field cases (NPR) are root caused to higher provisioning rate resulting in other functional
issues. In 2008, 8 out of 16 and in 2009 7 out of 20 were investigated for HPR. Each of these cases are
complex in nature and consume lots of effort.
•Sometimes customer stresses the system beyond prescribed limit, it causes issues which impacts
system functioning.
Goal Statement Use DFSS tools/techniques :
•To identify critical parameters based on customer's provisioning performance needs.
•To analyze architecture, create a performance model that predicts provisioning performance.
•To identify and maximize highest provisioning rate which can be supported hence better meeting with
customer need.
•Performance Model should predict the following metrics :
1. System Capacity : Typically indicated by provisioning capacity
Project Scope The scope of the project :
•Analyze provisioning performance parameters from customer need perspective.
•Identify product risk and mitigate it.
•Enhance product robustness.
•Specify DOEs, perform analysis and make provisioning performance predictions
Project Plan Plan Start Plan End Actual End
------------------------------------------------------------------------------------
•Requirement Development
and CP Identification 07/06/2009 08/31/2009 08/31/09
•Documented Risk Analysis
and Initial transfer function 09/01/2009 10/30/2009 11/31/10
•DOE Plan, Execute
and analyze) 03/01/2010 03/31/2010 03/31/10
•Lesson Learned and
Project closure 03/31/2010 04/23/2010 04/23/10
Team Role Name Expertise
-------------------------------------------------------------------------------------------------------------------
Sponsor HLR Box Manager
Champion Engineering Manager
Team Leader Srivastava Praveen GB Candidate
Team Member HLR Architecture
Team Member Bangalore DSS support
DSS Consultants Black belt
DSS Consultants Master Black belt
Technical Lead HLR/DAP/SSC Box Manager
Phase Green Belt Deliverables DFSS Tools
Project Definition SDFSS Charter Charter
Requirements VOC, Requirements Development
and Analysis
QFD, UML Use Case,
Cognition Cockpit
Requirements Perform and document Critical
Parameter flow down and analysis,
Critical Parameter Tree,
Cognition Cockpit.
Requirements Documented risk analysis FMEA
Requirements Perform Initial transfer function. Critical Parameter Tree,
Cognition Cockpit.
HLR Provisioning requirements IR
Minimize number of HLR provisioning related field cases 9
Maximize HLR supported provisioning rate 4
Minimize load shedding duration in case of exceeded provisioning rate 8
Minimize service impact if provisioning operator stresses the system. 10
Maximize HLR provisioning transaction information 3
Minimize failed provisioning transactions 6
Minimize DVLR and HLR not in synch issues due to high provisioning rate 8
Minimize time taken to root cause the issue to high provisioning rate 8
Had multiple interaction with technical team and DART (Support Team) to arrive at the requirements and importance rating.
First House Of Quality for Project HLR Provisioning Project
Rating Links Legend
H (9) = High Effect
M (3) = Medium Effect
L (1) = Low Effect
0 = No Effect
Roof Links Legend
Equal Effect: =
No Effect: 0 or Blank String
Relative Effect: +, ++, -, --
Direction of Goodness Legend
Increases Customer Satisfaction: +
Decreases Customer Satisfaction: -
On Target For Customer Satisfaction: 0 or Blank String
9 H H L
4 H M
8 M H M M
10 H M M
3 H M
6 M H M
8 M H M H
8 H M M M
153 105 240 108 108 51 42 18 96 54
15.7% 10.8% 24.6% 11.1% 11.1% 5.2% 4.3% 1.8% 9.8% 5.5%
6 4 10 4 4 2 2 1 4 2
0 0 0 0 0 0 0 0 0 0
System Requirements
Direction Of Goodness
Voice of Customer, VOC's Impo
rtanc
e
HLR
will s
end I
DBC
warni
ng an
d OMC
alarm
if rat
e is e
xcee
ded
HLR
will c
ontro
l inco
mimg
prov
ision
ing
flow
HLR
will o
ptimi
ze M
M tas
k and
cpu
utiliz
ation
for p
rovisi
oning
requ
ests
HLR
will
come
out o
f load
cond
ition
as
soon
as po
ssibl
e
Faile
d prov
ision
ing tr
ansa
tion w
ill be
re-
tried.
Detai
led pr
ovisi
oning
tran
sacti
on
inform
ation
will
be co
llecte
d an
d sore
d
IDBC
error
and s
ucce
ss re
spon
ses w
ill
have
more
infor
matio
n
HLR
will s
uppo
rt few
er pro
vision
ing
trans
actio
ns un
der lo
ad co
nditio
ns.
HLR
will a
dd ad
dition
al loc
al ala
rms f
or
interm
idiate
cond
itions
.
Faile
d MM
mess
ages
will
be re
tried
Minimize number of HLR provisioning related field
cases
Maximize HLR supported provisioning rate
Minimize load shedding duration in case of exceeded
provisioning rate
Minimize service impact if provisioning operator
stresses the system.
Maximize HLR provisioning transaction information
Minimize failed provisioning transactions
Minimize DVLR and HLR not in synch issues due to
high provisioning rate
Minimize time taken to root cause the issue to high
provisioning rate
Units
Scoring Totals
Relative Scores
Normalized Scores
Target Nominal Values
Critical Parameter
Provisioning Rate of HLR Application is identified as critical parameter.
Provisioning operator controls the flow of provisioning rate.
Incoming provisioning messages impacts HLR functional capabilities.
• Preconditions: – HLR is up and running – Provisioning client is connected and ready to send provisioning transactions. – DAP is sending MM messages to HLR for call processing.
• Actor(s): – Provisioning client – Dispatch application processor ( DAP) – Operations and Maintenance control (OMC)
• Flow of Events: 1. Provisioning user starts a sessions by sending a login requests to HLR. 2. Provisioning server on HLR authenticates the session and allow user to login. 3. Provisioning client sends a provisioning requests over tcp/ip. 4. Provisioning server validates the request and parse it. 5. Provisioning server checks the load conditions on the box and send a IDBC messages for database
updates. Underlying database APIS are called to update the database.. 6. Provisioning server determines if requests requires an unsolicited messages to be send to DAP. Mobility
management (MM) messages is send to MM tasks for dap communication. MM task generates the unsolicited request to DAP depending upon it retry and pending queue.
7. Provisioning server receives ack from DB and MM task and send a success response to provisioning client. 8. End of use case
• Post Conditions: 1. HLR database is updated with new provisioning information. 2. DAP’S VLR is updated and synched with HLR database. 3. Provisioning client receives a success response with success code.
• Alternate Flow #1: HLR rejects the new provisioning session requests 1. HLR is already under load conditions and new provisioning session service is not supported.
2. Provisioning client is send an error response and provisioning requests can not be initiated.
3. End of Alternate Flow # 1
◦ Post Conditions Alternate Flow #1: 1. An IDBC error response is send to client.
• Alternate Flow #2: The Database updates fail.
1. A critical alarm is generated and send to HLR alarm logs.
2. Provisioning server is send an error response which in turns generate an error response and send to provisioning client.
3. No unsolicited MM message is send to DAP.
◦ Post Conditions Alternate Flow #2: 1. An IDBC error response is send to client.
2. A critical local alarm is send to HLR application alarm logs.
• Alternate Flow #3: The Unsolicited MM messages are blocked by other messages in retry/pending queue. 1. Success ack is sent to provisioning server which in turns send success IDBC ciode to provisioning client.
2. MM messages in kept in retry/pending and re tried at regular intervals through a separate procedure. – Notes. Some times unsolicited MM messages are stuck in retry/pending queue due to other system
conditions such as few blocked messages for an IGW in the market. These messages remain in retry/pending queue until the blocker messages are cleared. This some times may take even few days. This condition creates a situation where DAP’s VLR and HLR are not in synch and may results in failed subscriber registration.
◦ Post Conditions Alternate Flow #3 1. Unsolicited MM messages are tried at regular interval and till thes messages are cleared HLR and DAP’s VLR will not
be in synch.
Start ProvisioningProvisioning
Client
Close Prov Session
Open Prov Session
Authenticate new session<<include>>
Update DB
Send MM Message
Send IDBC response back to
client
<<include>>
<<include>>
<<include>>
Check load condition<<include>>
<<include>>
• HLR Rejecting subscriber registration request – If HLR is already under load condition due to busy hour
traffic, continuous flow of provisioning requests may force HCPT to lose incoming provisioning requests.
– Provisioning at a rate higher than recommended rate can fill retry/pending queue due to which subscribers of that fleet have no Service for some time.
• HLR goes to load shedding – If customer stresses the provisioning system, it can
cause HLR to load shed impacting other functionalities.
P- Diagram
CONTROL FACTORS
- Netowrk Bandwidth
- Connectivity
- MAP dialogue limitation
between DAP-HLR
External Comunication
- Database response time
- DB Application capability to
handle transactions
- Database centrice intensive
operations such as backup,
update stattistics, subscriber
reports etc.
Database Related
HLR is up and running
MM and CP messages
Provisioning Requests
SIGNAL
NOISE FACTORS
Higher Supported Provisioning Rate
Dynamic Provisioning threshold
Determination
RESPONSE
Hardware
----------------------
Number of CPUs
Available Memory
Processor
Ethernet capability
Software and Others
------------------------
MM Application
DB Application
Load levels
Number of subscribers
Webserver performance
Database performance
MM Rate
Load levels
HLR PROVISIONING PROJECT
HLR Supported Provisioning Rate
Unit - Message/Second
HLR Goes to load shedding
Can't Support higher rate
Failed Provisioning transactions
ERROR STATES
Those parameters that change with
deviations caused by process or
design.Unexpected Variation due to external
environment,internal, piece-to-piece, customer
usage, wear out)
What Errors , or failure modes are we likely to
see impact the ideal function performed as
a result of pottential "noise factor" interactions.
Here we conside the result of noise factors on
the ideal function.
The desired input signals
necessary to provide the
desired output. Causes the
system to deliver the user
intent
These are the customer intended
ideal functions. List all the positive
functions or the attributes from the
boundary diagram.System output that
determines the perceived result)
These are the design parameters whose
nominal values can be adjusted by the user
or programming modes in the software)
.
VOC related to CP
Initial Transfer Function :
Max Prov rate = f(MM_traffic, # of subscribers,roamers)
Architecture Evaluate Architecture Evaluate Architecture
using Modeling &
Simulation Prototyping.
DOE
Architecture Update risk analysis FMEA
Architecture Measure non-functional Critical
Parameter
Measured Quality
Attributes.
Provisioning Rate: 1. Evaluate Architecture : 7 Step DOE 2. Measure Performance for Critical
Parameter
• Design of Experiments conducted in Bangalore, India Lab
• Experiments carried out on Melody HLR software
• Hardware and Software – Dual Chassis ATCA (7880) blades with HLR Melody
version - SR18.00.17
– Sun based Provisioning Loader version – R13.06.00
– Sun Based Map loader – R12.00.04
– Results observed on Melody hardware are applicable to iDEN (TS40) HLR.
• Critical Parameter: – For the critical parameter understand factors affecting:
• Provisioning Rate
• To determine the factors that will be the main effects on satisfying this requirement
• To determine the relationship between the factors
• To identify the transfer function between the factors and this requirement.
• Tests were done on a Melody HLR (ATCA 7880), bi nodal setup in Bangalore iDEN lab. Provisioning messages were sent to HLR using 2 provisioning loaders. Provisioning rate is configurable from loader. Mobility and Call Processing messages were generated using map loader with standard load profile for each scenario.
• Maximum supported provisioning rate without sending HLR to loadshedding was determined using this.
Final RTT - RoundTripTime
Run MM - Mobility Message
Verification PT1 - Prov Tool 1/2
RunDuration
(D)
DB Size
(K)
MM Rate
(msg/sec)
Roamers
(%)
PT1
Rate
(msg/sec)
PT1
RTT
(ms)
PT1
msg/sec
(R1)
PT2
Rate
(msg/sec)
PT2
RTT
(ms)
PT2
msg/sec
(R2)
Prov
Response
(R1+R2))
Transactions in
prov log
(N)
( N/D )
msg/sec
MAX CPU
(%)
1 300 100 393 20% 55 34 18.08 55 35 18.06 36.14 10912 36.37 68
2 300 100 393 0% 25 30 31.46 25 28 32.73 64.19 19390 64.63 70
3 300 100 393 0% 30 30 31.63 30 28 32.94 64.57 19486 64.95 66
4 300 100 393 0% 20 30 31.55 20 28 32.86 64.41 19442 64.81 68
5 300 100 100 20% 20 25 38.01 20 25 36.33 74.34 22428 74.76 56
6 300 100 100 20% 25 26 37.35 25 25 36.55 73.9 22294 74.31 57
7 300 100 100 0% 25 24 39.6 25 23 38.96 78.56 23686 78.95 58
8 300 100 100 0% 20 24 39.62 20 23 38.72 78.34 23650 78.83 56
9 300 100 100 0% 40 24 24.87 40 22 24.85 49.72 15004 50.01 44
11 300 800 393 20% 25 42 23.07 25 42 22.49 45.56 13748 45.83 72
12 300 800 393 20% 40 41 23.6 40 41 22.9 46.5 14038 46.79 70
13 300 800 393 20% 45 36 22.1 45 35 22.1 44.2 13338 44.46 71
14 300 800 393 20% 55 35 18.08 55 34 18.08 36.16 10739 35.80 66
15 300 800 393 0% 40 29 24.82 40 28 24.87 49.69 15002 50.01 67
16 300 800 393 0% 30 31 30.87 30 30 30.98 61.85 18680 62.27 71
17 300 800 393 0% 35 30 28.39 35 30 27.98 56.37 17022 56.74 69
18 300 800 393 0% 40 27 24.86 40 26 26.87 51.73 15999 53.33 65
19 300 800 100 20% 20 24 40.5 20 25 36.31 76.81 22840 76.13 55
20 300 800 100 20% 25 34 39.69 25 25 35.75 75.44 22766 75.89 56
21 300 800 100 0% 25 24 39.72 25 24 37.35 77.07 23262 77.54 56
22 300 800 100 0% 20 24 40.72 20 25 36.47 77.19 23287 77.62 52
23 300 800 100 0% 15 25 38.43 15 25 35.62 74.05 22351 74.50 55
24 300 800 100 0% 20 25 38.4 20 25 36.4 74.8 22586 75.29 57
Verification 100 250 0% 25 27 35.01 25 27 35.01 70.02 19696 65.65 60
1 300 100 250 20% 30 34 28.93 30 33 28.49 57.42 17328 57.76 63
2 300 800 118 20% 25 24 39.66 25 26 35.6 75.26 22717 75.72 57
3 300 800 118 20% 20 25 38.51 20 26 34.89 73.4 22148 73.83 57
4 300 800 237.5 20% 20 28 34.26 20 28 32.63 66.89 20184 67.28 61
Full Factorial Design created in Minitab for 3 factors Factors: 3 Base Design: 3, 8 Runs: 8 Replicates: 2 Blocks: none Center pts (total): 3
StdOrder RunOrder CenterPt Blocks Number of subs MM Rate Roamers Provisioning Rate
1 1 1 1 100 100 0 78.34
2 14 1 1 800 100 0 77.07
3 8 1 1 100 393 0 64.57
4 2 1 1 800 393 0 51.73
5 5 1 1 100 100 20 73.9
6 15 1 1 800 100 20 75.44
7 12 1 1 100 393 20 36.14
8 3 1 1 800 393 20 36.16
9 16 1 1 100 100 0 78.08
10 10 1 1 800 100 0 77.19
11 6 1 1 100 393 0 64.81
12 13 1 1 800 393 0 49.69
13 9 1 1 100 100 20 74.34
14 11 1 1 800 100 20 76.81
15 7 1 1 100 393 20 36.14
16 4 1 1 800 393 20 36.16
Number of subs
Roamers
MM Rate
800100
200200
393100393100393100393100
80
70
60
50
40
30
Pro
vis
ion
ing
Ra
te
Box Plot of Provisioning Rate
DOE Step 2 & 3: Create Model Estimated Effects and Coefficients for Provisioning Rate (coded units)
Term Effect Coef SE Coef T P
Constant 61.66 0.1578 390.84 0
Number of subs -3.26 -1.63 0.1578 -10.33 0
MM Rate -29.47 -14.74 0.1578 -93.4 0
Roamers -12.05 -6.02 0.1578 -38.19 0
Number of subs*MM Rate -3.72 -1.86 0.1578 -11.79 0
Number of subs*Roamers 4.27 2.14 0.1578 13.54 0
MM Rate*Roamers -9.5 -4.75 0.1578 -30.11 0
Number of subs*MMRate*Roamers 2.73 1.36 0.1578 8.65 0
S = 0.631056 PRESS = 12.7434 R-Sq = 99.93% R-Sq(pred) = 99.72% R-Sq(adj) = 99.87
1.Significant factors have p-value < .05 2. Model is good because R-Sq and R-Sq (adjusted) has difference of less than 10%
ABC
A
AB
AC
BC
C
B
9080706050403020100
Te
rm
Standardized Effect
2.31
A Number of subs
B MM Rate
C Roamers
Factor Name
Pareto Chart of the Standardized Effects(response is Provisioning Rate, Alpha = .05)
MM Rate is most significant factor followed by Roamers and then followed by MM Rate and Roamer interactions.
0-25-50-75-100
99
95
90
80
70
60
50
40
30
20
10
5
1
Standardized Effect
Pe
rce
nt
A Number of subs
B MM Rate
C Roamers
Factor Name
Not Significant
Significant
Effect Type
ABC
BC
AC
AB
C
B
A
Normal Plot of the Standardized Effects(response is Provisioning Rate, Alpha = .05)
Significant effects are further away from the line in Normal Probability Plot and marked in red
Source DF Seq SS Adj SS Adj MS F P
Main Effects 3 4097.39 4097.39 1365.80 3429.65 0.000
2-Way Interactions 3 489.46 489.46 163.15 409.70 0.000
3-Way Interactions 1 29.78 29.78 29.78 74.79 0.000
Residual Error 8 3.19 3.19 0.40
Pure Error 8 3.19 3.19 0.40
Total 15 4619.82
Since p value for main effect and 2 way interaction and 3 way interaction is < 0.05. All are significant.
1.00.50.0-0.5-1.0
99
90
50
10
1
Residual
Pe
rce
nt
8070605040
1.0
0.5
0.0
-0.5
-1.0
Fitted Value
Re
sid
ua
l
1.00.50.0-0.5-1.0
8
6
4
2
0
Residual
Fre
qu
en
cy
16151413121110987654321
1.0
0.5
0.0
-0.5
-1.0
Observation Order
Re
sid
ua
l
Normal Probability Plot Versus Fits
Histogram Versus Order
Residual Plots for Provisioning Rate
Residual vs. Fitted (Predicted) value and residual vs run order plots do not show any patterns. Residuals are normally distributed (See Normal Probability Plot)
Term Coef Constant 82.3497 Number of subs 0.00474676 MM Rate -0.0398537 Roamers 0.217479 Number of subs*MM Rate -6.28961E-05 Number of subs*Roamers -
4.57326E-05 MM Rate*Roamers -0.00444015 Number of subs*MM Rate*Roamers 2.66090E-
06
Estimated Coefficients for Provisioning Rate using data in uncoded units
Y= Predicted Provisioning Rate, Number of subs = s, MM rate = m and Roamer (%) = r
Y = 82.3497 +0.00474*s -0.03895*m +0.2174*r - 0.0000628*s*m- .0000457*s*r- 0.0044*m*r + .0000266*s*m*r
Transfer Function :
800100
80
70
60
50
393100
200
80
70
60
50
Number of subs
Me
an
MM Rate
Roamers
Main Effects Plot for Provisioning RateData Means
393100 200
80
60
40
80
60
40
Number of subs
MM Rate
Roamers
100
800
of subs
Number
100
393
MM Rate
Interaction Plot for Provisioning RateData Means
• Pareto chart shows that MM Rate ( mobility & call processing message) has most significant effect on provisioning rate supported by HLR.
• The next significant factor is percentage of roamers. More the roamer percentage, less the provisioning capability.
• Interaction of MM rate rate and roamer is next is list and has significant effect on provisioning rate. A minimum of both MM rate and roamer percentage will yield maximum supported rate at any given point of time.
• Number of subscribers and in 3 way Interactions between factors do not impact the HLR provisioning capability significantly.
• However, number of mm & cp messages will be generally driven by registration requests which in turn is directly related to number of active subscribers in the HLR db, number of active subs in database does impact the provisioning capability indirectly. Inactive subscribers do not impact provisioning capability.
• Total number of mm & cp messages are driven by number of registrations and number of SRI requests. We know that at times mm and cp messages are less when compared to busy hour rate, for example during : – Maintenance window ( 41.00 registration per sec) – 7-9 AM Registration Period (83.33 registration per sec )
• During such times, HLR can support more provisioning transactions. Model predicts peak provisioning factor for each such scenario depending on the mm & cp messages and roamers present in the market.
• Provisioning Model
• Y = 82.3497 +0.00474*s -0.03895*m +0.2174*r - 0.0000628*s*m- .0000457*s*r- 0.0044*m*r + .0000266*s*m*r
• Where
– Y = Supported Provisioning Rate per second.
– m = mm and cp messages per sec
– r = percentage of roamers
– s = number of subscribers in K
• According to provisioning model, Maintenance window represented by 41.67 registration per second with 20 % roaming, will have provisioning peak factor as 2.0 and can support upto 15 msg per second. This is verified in Box test lab using maintenance window load profile.
• HA HLR supported busy hour provisioning profile is benchmarked for market conditions as 7.3 msg per sec where as lab throughput for busy hour is 36.16 mgs per sec. To get supported provisioning rate in market condition a factor of 36.14/7.3 = 4.94 is used.
• The iDEN markets are informed through a revised bulletin issued to the marked based on DOE results. This will help them to better plan their high provisioning needs during maintenance window or at a time when mm traffic is minimum
Failure Modes and Effects Analysis (FMEA):
Risk Category /
Item
Potential Failure Mode / Effects Potential Effects of Failure (What is
the impact on the customer?)
Risk
Priority
Number
(RPN)
Recommended Action
Ris
k P
rio
rity
Customer impact
due to high
provisioning rate.
High provisoning rate might cause
large number of un-solicited
messages initiated for the DAP.
For example modify subs or modify
fleet will generate ISD and IFD
HLR may "lose" in incoming
registration requests impacting
subscriber service. 45
1)Inform operator about the
HLRs capacity to support
maximum provisioning rate.
So that they do not exceed
the recommended rate.
Also intimate markets of
the window when maximum
proviosning rate can be
exceeded. Based on this
information market can
defer theit high provisioning
needs to such window. 24
(Revised)
Page: 1 of 1
Results
Software Failure Mode and Effects Analysis (FMEA)
Product:iHLR Provisioning Project FMEA Date: Dec 17, 2010
FMEA Team Members: Praveen,HLR Development Team
Risk Before RPN After RPN
Service Impact 45 24
Functionalty Impact 36 20
Double click on the sheet to see the entire sheet
• DOE results prove that HLR can supports higher provisioning rate during maintenance window and other times when mm rate is low.
• Defer High proviosning rate activities to maintenance window or other time when HLR is handling lower MM rate.
• Based on DOE analysis, future HLR releases can implement provisioning flow control and alarm mechanism to warn operator , if maximum supported rate is exceeded.
• In 2008, 16 field cases were investigated by HLR development with 12.89 SM effort, out of these, 8 were investigated from high provisioning rate angle with 5.97 SM effort.
• In 2009, 20 field cases were investigated by HLR development with 15.08 SM effort , out of these, 7 were investigated from high provisioning rate angle with 7.80 SM effort .
• So, in 2008 and 2009 alone at least 13.87 SM effort was spent on understanding high provisioning rate for field cases. Rough estimate for implementing provisioning model and recommendations like high provisioning rate warning alarm and throttling mechanism is close to 3.5 SM.
• We could have saved at least 8-10 SM in 2 years by 1) Preventing high provisioning rate field cases from occurring 2) Straightaway ruling out high provisioning rate involvement in field case analysis
• Note – MOL effort used in the analysis does NOT include CNRC/DART effort for field case investigation. These teams also investigate issue along with development and log almost similar effort. If we add dart/cnrc efforts, overall saving will be much more than this.
• Tests were run in BT environment to measure supported provisioning rate during maintenance window and 7-9 AM registration period.
• Maintenance Window Profile – Number of subs = 800 k – Registration Rate = 41.67 per second – mm and CP messages = 118 per second – Roamers = 20 % – Predicted Rate using Model = 72.93 – Prorated Provisioning Rate = 14.75 – Actual = 73.44 – Actual Prorated Provisioning Rate = 14.94
• 7 – 9 AM Registration Period – Number of subs = 800 k – Registration Rate = 83.33 per second – mm and CP messages = 237.49 per second – Roamers = 20 % – Predicted Rate using Model = 57.01 – Prorated Provisioning Rate = 11.53 – Actual = 65 – Actual Prorated Provisioning Rate = 13.53
• All the Values measured during verification Testing for supported provisioning rate are close to the Upper 95% Prediction interval. This validates the predictive provisioning model
• The iDEN markets are informed through a revised bulletin based on DOE results on April 23,2010.
From: Tewinkle Steve-EST002
Sent: Monday, June 28, 2010 8:24 PM
To: Tchon Scott-CST013
Cc: Srivastava Praveen-a23098; Kumar Taleki Prasanna-a22694; Daniels Thomas-CTD040
Subject: FW: : Green belt Project - success criterion and value statement - minutes
Scott,
Please find my value statement below for Praveen's Green Belt project:
The SDFSS Green Belt project that was done by Praveen for iHLR provisioning has identified improvements that can be made both
short and long term that will have positive impact on HLR Provisioning in terms of performance, capacity, and reduced overall
Support. Praveen's analysis was very thorough, data driven, and Customer focused. The benefits we can get from this project
are:
- This project identified the critical parameters that impact HLR provisioning as MM Rate and Roamer Percentages and based on this finding, we are able to recommend to the Customer that they can increase their maintenance window provisioning rate
from 7 to 15 msgs per second. This increased rate has already been documented in field bulletin and delivered to market in April 2010.
- Implementation of a new provisioning throttling mechanism that would regulate provisioning flow, warn the operator when maximum supported rate is being exceeded, and help reduce # of field related cases due to high provisioning rate. This will be proposed as a new iHLR feature for an upcoming system release.
- This project also will serve as a model for other iDEN teams to follow when faced with similar type of issues, etc on their box
Overall, Praveen did an outstanding job in approaching this project and his efforts will definitely result in improvements to the
provisioning process and reductions in field reported provisioning cases in the future.
Steve
– The HLR’s provisioning capacity is a function of MM rate and Roamer’s percentage.
– Predictive model’s ability to dynamically determine maximum supported provisioning rate can be used to -
• Implement throttling mechanism to regulate provisioning flow.
• Implement warning ( alarm ) mechanism, so that provisioning operator can slow down if maximum supported provisioning rate is reached.
– These two implementations alone will enhance robustness and save substantial MOL effort.
– This is proposed for SR21 as a candidate feature by the Box Manager (Steve T)
• What went well: – Efficient Simulation Model development – CPM Flow-down and engagement of stakeholders in identifying
critical parameters – Applying the results of DOE and issuance of revised bulletin to
customer on provisioning rate – Engagement of product team, field support team and box
management since start of the project.
• What could have gone better: – Do brainstorms and interviews direct with Customer
• Things I would do different in the future: – Work with champion and sponsor to ensure engagement of all
stakeholders in decision making at critical milestones
Backup Up Slide
• Service Impact (Original Text) – If provisioning transaction rates are exceeded discrepancies may occur between DLVR and iHLR
database or iHLR pending / retry queues may contain stuck transactions. Field case occurred on a TDAP that experienced registration issues due to exhaustion of internal fleet provisioning buffers.
• Service Impact (Changed) – If provisioning system is stressed and transaction rates exceed the published rate, following may
be observed: • Discrepancies may occur between DLVR and iHLR database • iHLR pending / retry queues may contain stuck transactions. • Provisioning transactions might fail due to database lock error etc. • iHLR can go to load shedding either due to high cpu utilization or filled hcpt queue • For example a field case occurred on a TDAP that experienced registration issues due to exhaustion of internal
fleet provisioning buffers.
• Note: (Original Text) – At times a higher provisioning throughput rates may be realized while the HAiHLR is not under
critical load but to eliminate possible issues it is recommended that these published rates be followed.
• Note: (Changed) – Note: It is recommended that these published rates be followed to avoid issues. However, HLR
supported provisioning rate is a function of mobility & call processing message rate and number of roamers in the market. Again, above message rate depends on registration requests to the HLR. A higher provisioning throughput may be realized while the HA HLR is not under critical load and processing less messages than compared to busy hour traffic. For example, during maintenance window with a provisioning peaking factor 2, HLR can support up to two times of following profile.
MM Rate
(per sec)
Subscriber
(K)
Roamers
(%)
Predicted rate
(per sec)
Supported Provisioning
Rate
Provisioning
Peaking
Factor
100 100 20 73.9 14.95 2.0
393 100 20 36.14 7.31 1.0
100 800 0 77.07 15.59 2.1
393 100 0 64.57 13.06 1.8
100 800 20 75.44 15.26 2.1
393 800 0 51.73 10.47 1.4
393 800 20 36.16 7.32 1.0
100 100 0 78.34 15.85 2.2
*Supported provisioning rate = Prorated provisioning rate based on benchmark under market conditions.
#This rate was benchmarked with busy hour traffic profile.
*
#
