JPMC Sec Lending

Project Code: MXC-W061

DML to SLX Integration at JPMorgan Chase

A Major Qualifying Project Report

Submitted to the faculty of Worcester Polytechnic Institute

In partial fulfillment of the requirements for the Degree of Bachelor of Science

Submitted By: _______________________ _______________________

Michael J. Kristan Megan R. Slonski

Project Center: Wall St. New York, NY B Term 2006 Sponsoring Agency:

JPMorgan Chase & Co.

Submitted To: Project Advisors: ________________________ _________________________

Michael Ciaraldi Arthur Gerstenfeld

On-Site Liaisons: Marisa Giliberti Danielle Rumore Brian Kenney

This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

i

Abstract

This MQP aimed to assist JPMorgan Chase with the successful completion of the

high priority DML to SLX Integration project. In the pre-design phase, work was carried

out to map data fields between the two systems and to analyze the anticipated effect of

this project on server performance. Concurrently, a thorough project plan was created to

be used in a critical path analysis. Struggles in creating this led to the recommendation of

a process for better managing projects.


ii

Authorship

This Major Qualifying Project is a collaborative effort of both team members. In

compiling research and writing this paper each of us dedicated a significant amount of

time and effort. There was an overall equal contribution to the executive summary,

introduction, and background section made by both team members. Megan Slonski was

mainly responsible to the sections pertaining to the critical path analysis and project

management process improvements. Michael Kristan was primarily responsible for the

sections regarding capacity planning and data mapping.


iii

Executive Summary

As the 17th largest corporation in the United States, JPMorgan Chase & Co.

provides a wide range of banking and financial services to customers worldwide. One

such service is known as securities lending. Securities lending is the market practice

whereby securities are temporarily transferred from a lender to a borrower. For the

period of the loan, the lender is protected against default through receipt of collateral

from the borrower. JPMorgan acts as the facilitator in this transaction by matching

lenders to qualified borrowers. JPMorgan also invests the collateral received on the loan

and splits the return on investment between all three parties.

In order to maintain its outstanding reputation, JPMorgan Chase & Co. must

utilize new technology, maximize efficiency of its internal processes, and continually

improve the services provided to its clients. Understanding the importance of these key

business strategies, JPMorgan executives decided that the software Securities Lending

Xpress (SLX) should be the sole front-end trading platform used to initiate all securities

lending loans globally. The reason for this decision is that SLX has checks built in that

ensure compliance to audit and government regulations. Also SLX is built on the most

up-to-date, flexible platform when compared to the other systems. This will allow future

changes to be made more quickly and at a lower cost. In this vision the other systems,

including International Securities Lending System / Dan McLune (DML), would only be

used as loan maintenance systems and would cease to perform any loan initiation activity.

The current state of the business requires that several incremental projects be completed

in order to turn this vision into a reality. One such project is to integrate DML and SLX

so that information pertaining to loan maintenance activity in DML is passed in the form


iv

of a message to SLX throughout the day and overnight using a feedback loop. Only then

can SLX use this information to accurately update lender limit and borrower credit line

utilizations.

The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan

with the successful completion of this important project. Specifically, work was done to

accomplish four main goals. The first two relate to pre-design activities that needed be

completed prior to the design and build phases of the project. Success of the pre-design

activities would contribute to the successful implementation of the feedback loop. The

first goal of this MQP was to assist in mapping and matching the data fields between the

two systems to ensure that SLX receives all of the information it needs from DML to

properly update its database. The second goal was to analyze the anticipated effects of

the feedback loop on servers and mainframes so that additional resources could be

introduced and performance would not be compromised. The third goal of this MQP was

to gather the information needed to conduct a critical path analysis that would examine

ways to expedite the delivery of the project. From this a fourth goal was established,

which was to recommend possible improvements to project management practices that

could result in shorter project durations, fewer delays, and more accurate estimates of

project cost and delivery dates for this and future projects.

The objective of the data mapping exercise was to examine the fields that DML

has defined in a loan snapshot and compare that to the data fields SLX needs to receive to

properly update borrower credit and lender limit utilizations. One of the challenges was

that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth

analysis needed to be conducted to examine how the data fields from DML should be


v

manipulated so that they are in a useable format for SLX. As part of the data mapping

exercises, sample loans were observed from the production environment. Unfortunately

a problem occurred when taking the raw loan data from DML and inputting it into a

Microsoft Access database that would allow it to be analyzed. This raw data is known as

a loan master file. There were more loan records in the loan master file than Microsoft

Access allowed to be imported at once. To get around this, a program was developed that

would split the file into multiple, smaller files that could be imported into Microsoft

Access one at a time. With this accomplished, the JPMorgan team could begin to analyze

how DML fields would map to SLX fields.

The second objective, capacity planning, was to carry out a comprehensive

analysis on the SLX servers and to gather enough information to make infrastructure

decisions moving forward. The primary deliverable was a document that gives the reader

an understanding of the resources required to run SLX in a production environment and

to analyze the impact that the DML feedback loop will have on the systems. Information

was gathered from many sources and compiled together to form a complete picture of the

resources that SLX depends on. One of the primary data sources was a feedback loop

that was formerly created as part of a project to integrate SLX with the domestic

counterpart to DML known as SLMU. The plan was to investigate the current

performance of the SLX servers with the SLMU feedback loop in use. The next step was

to compare the number of SLMU messages to the anticipated number of DML messages

to predict the effect that the DML feedback loop would have on SLX server performance.

The analysis showed that the anticipated volume of DML messages between

11AM and 1PM is so high that it will diminish the SLX server performance. The


vi

performance will be diminished to such an extent that SLX will not be able to process

messages from DML in the time deemed acceptable by business executives. This will

hinder the booking of new loans and the updating of lender and borrower utilizations

which will result in lost opportunity for JPMorgan.

The third objective of this MQP was to gather the information needed to conduct

a critical path analysis that would examine ways to expedite the delivery of the DML to

SLX Integration Project. This project was important because it was deemed a high

priority within the firm. A number of steps needed to be completed to gather the

information that was required to utilize the critical path method in the most effective way.

The basis for this analysis would be a thorough and cohesive project plan, which was

created as a product of this objective. Meetings were held with each member of the

project team to determine all project tasks, time estimates for the completion of each task,

and the network of task dependencies. Work was also done with the project delivery

manager to compile a list of all resources that would contribute to the project and to

determine the percentage of each resource’s time that would be allocated to this initiative.

All of this information was then incorporated into the project plan to be used in the future

management of the project.

The difficulty experienced in compiling the information needed for a thorough

project plan exposed the need for the fourth goal of this MQP, which was to recommend

improvements to project management processes. The improvements were aimed at

making the processes more efficient, streamlined, simple and/or error-proof. In order to

make these improvements, a thorough investigation of project management practices was

conducted surrounding the following activities: providing project cost and duration


vii

estimates to the Investment Council for project funding, creating an accurate project plan

and using it keep the project on schedule, and tracking action items that arise during team

meetings. This evaluation showed that there is a lot of room to make improvements to

how projects are managed. The biggest observation is that there is not a defined process

based on best practices in place in the Securities Lending Technology group. A different

process is used for each new project. This results in confusion, poor practices being used,

and time wasted by all team members learning a new system for each project. Therefore,

we have developed a process for managing projects that encompasses the aforementioned

aspects of project management to be used by all project managers. It is based on the

inefficiencies observed with current practices and on what was learned about the project

needs that project managers are responsible to meet. The implementation of this process

is expected to result in shorter project duration, fewer delays, and a more accurate sense

of project cost and end dates due to increased communication across team roles, greater

accountability and awareness of due dates, and simpler, repeatable processes that are less

prone to error.


viii

Acknowledgements

We would like to thank Worcester Polytechnic Institute and our sponsor

JPMorgan Chase for giving us the opportunity to complete this project with the Securities

Lending Technology group at JPMorgan. Particularly we would like to thank Danielle

Rumore, Brian Kenney, Vitoriana Morais, Mike Salamina, Reynaldo Generoso, Besley

Nelson, Peter Ennis, and our liaison Marisa Giliberti for their continuous support and

commitment to helping us make this a successful project and a meaningful learning

experience.

Furthermore, we would like to thank our advisors Professor Arthur Gerstenfeld

and Professor Mike Ciaraldi for their guidance and assistance in helping us develop our

project and write this report.

.


ix

Table of Contents

Abstract............................................................................................................................... i Authorship ......................................................................................................................... ii Executive Summary......................................................................................................... iii Acknowledgements ........................................................................................................ viii Table of Contents ............................................................................................................. ix Table of Figures................................................................................................................ xi Table of Tables ................................................................................................................ xii 1. Introduction................................................................................................................... 1 2. Background ................................................................................................................... 5

2.1. JPMorgan Chase & Co............................................................................................. 5 2.2. JPMorgan Worldwide Securities Services............................................................... 6 2.3. Securities Lending ................................................................................................... 7 2.4. JPMorgan’s Role in Securities Lending ................................................................ 10 2.5. The Technology Behind JPMorgan Securities Lending ........................................ 13

2.5.1. Custody Management Systems – COSMIC & TITAN................................... 13 2.5.2. Front-end Trading - SLX ................................................................................ 14

2.5.2.1. SLX Hardware ......................................................................................... 14 2.5.2.2. SLX Trade Views .................................................................................... 14

2.5.3. Legacy Trading Application - DML............................................................... 15 2.5.4. Legacy Trading Application - SLMU............................................................. 15

2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services ........ 16 2.7. SMLU to SLX Integration Project......................................................................... 17

3. Methodology ................................................................................................................ 20 3.1. Critical Path Analysis ............................................................................................ 20 3.2. Project Management Process Improvements ......................................................... 23 3.3. Data Mapping Exercises ........................................................................................ 23

3.3.1. Parsing the DML Loan Master ....................................................................... 24 3.3.1.1. Design ...................................................................................................... 25 3.3.1.2. Implementation ........................................................................................ 25

3.3.2. Aggregation and Summarization of Data Fields............................................. 26 3.4. Capacity Planning Analysis ................................................................................... 27

3.4.1. Server Specifications ...................................................................................... 27 3.4.2. Used Server Resources ................................................................................... 28 3.4.3. SLMU Message Statistics ............................................................................... 28

4. Results and Analysis ................................................................................................... 30 4.1. Critical Path Analysis ............................................................................................ 30 4.2. Project Management Process Improvements ......................................................... 37

4.2.1. Investment Council Estimates and Project Plan Creation............................... 38 4.2.1.1. Current Process ........................................................................................ 38 4.2.1.2. Problems .................................................................................................. 39

4.2.2. Use of Microsoft Project................................................................................. 43 4.2.3. Action Item Tracking...................................................................................... 44

4.3. Data Mapping Exercises ........................................................................................ 45 4.3.1. Parsing the DML Loan Master ....................................................................... 45


x

4.3.2. Aggregation and Summarization of Data Fields............................................. 48 4.4. Capacity Planning Analysis ................................................................................... 48

4.4.1. Processor Analysis .......................................................................................... 48 4.4.2. Memory Analysis............................................................................................ 49 4.4.3. Storage Capacity ............................................................................................. 51 4.4.4. SLMU Message Statistics ............................................................................... 52 4.4.5. DML Projections............................................................................................. 54

5. Recommendations ....................................................................................................... 56 5.1. Project Management Process Improvements ......................................................... 56

5.1.1. The Estimation Workbook.............................................................................. 58 5.1.2. Microsoft Project and the Critical Path Method ............................................. 64 5.1.3. Action Item Tracking...................................................................................... 67 5.1.4. Weekly Status Meetings ................................................................................. 70

5.2. IT Equipment Upgrades for SLX........................................................................... 71 Works Cited..................................................................................................................... 73 Appendix A : Securities Lending Technology Systems Relationships ....................... 75 Appendix B : High Level Task Document for Project Plan Creation........................ 76 Appendix C : Snapshots of the STMate Tool ............................................................... 80 Appendix D : Screenshots of the same loan in DML and SLX................................... 82 Appendix E : DML Loan Master Parser Source Code ............................................... 84 Appendix F : DML Loan Master Parser JUnit Test Case .......................................... 88 Appendix G : Microsoft Access SQL Queries .............................................................. 89

Join DML/SLX Brokers................................................................................................ 89 DML Brokers without matching SLX Records ............................................................ 89 SLX Brokers without matching DML Records ............................................................ 89 Unique LOAN-TYPE Values ....................................................................................... 89 Unique SEC-LOC Values ............................................................................................. 89 Unique SEC-TYPE Values ........................................................................................... 89

Appendix H : Invitation to Final Presentation............................................................. 90 Appendix I : Review of SLX Server Performance....................................................... 91


xi

Table of Figures

Figure 1: PDF Project Phases (Project Delivery Framework, 2006) ................................ 16 Figure 2: Current Flow of Securities Lending Information (Kenney, 2006).................... 19 Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006).......................... 25 Figure 4: Flow Chart for Business Analyst....................................................................... 31 Figure 5: Flow Chart for Quality Assurance Team .......................................................... 32 Figure 6: Flow Chart for DML Team ............................................................................... 33 Figure 7: Flow Chart for SLX Team................................................................................. 34 Figure 8: Flow Chart for SLX Mainframe Team.............................................................. 35 Figure 9: STMate Snapshot Showing Unnecessary Task List.......................................... 41 Figure 10: Class Dependency Diagram for DMLLoanMasterParser ............................... 46 Figure 11: Instability Metric for DMLLoanMasterParser (green dot).............................. 46 Figure 12: Efferent Couplings for DMLLoanMasterParser.............................................. 47 Figure 13: Class and Package Statistics for DMLLoanMasterParser............................... 47 Figure 14: Production Server CPU Usage ........................................................................ 49 Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006) .............................................. 50 Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006) .............................................. 50 Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006) .............................................. 50 Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota ............ 51 Figure 19: /app/a_slxftp on the JIP NAS is near capacity ................................................ 52 Figure 20: SLMU Messages in SLX................................................................................. 53 Figure 21: Comparing SLMU Performance Between Oct and Nov ................................. 54 Figure 22: SLMU and Anticipated DML Messages over a Trading Day......................... 55 Figure 23: Project Management Process........................................................................... 57 Figure 24: Estimation Workbook- Instructions ................................................................ 59 Figure 25: Estimation Workbook- Team Member Worksheet ......................................... 60 Figure 26: Estimation Workbook- Summary Sheet by Phase .......................................... 62 Figure 27: Estimation Workbook- Summary Sheet by Role ............................................ 63 Figure 28: Diagram of Two Layer Project........................................................................ 65 Figure 29: Action Item Tracking Sheet Instructions......................................................... 68 Figure 30: Action Item Tracking Sheet for Action Items ................................................. 69 Figure 31: Action Item Tracking Sheet for Risk Items..................................................... 69


xii

Table of Tables

Table 1: Acceptable Non-cash Collateral (Devasthali, 2006) ............................................ 7 Table 2: Key Team Members and Roles........................................................................... 22 Table 3: SLX Production Server Hardware (Ghelani, 2006)............................................ 28 Table 4: STMate Entry Based on Complexity of Repeated Tasks ................................... 42 Table 5: Code Metrics for DMLLoanMasterParser.java .................................................. 45


1

1. Introduction

As the 17th largest corporation in the United States, JPMorgan Chase & Co.

provides a wide range of banking and financial services to customers worldwide. In

order to maintain its outstanding reputation, JPMorgan Chase & Co. must utilize new

technology, maximize efficiency of its internal processes, and continually improve the

services provided to its clients. Understanding the importance of these key business

strategies, JPMorgan executives decided in 2004 that the software Securities Lending

Xpress (SLX) should be the sole front-end trading platform used to initiate all securities

lending trades globally. The reason for this decision is that SLX has checks built in that

ensure compliance to regulations and lender rules and limits. Also SLX is built on the

most up-to-date, flexible platform when compared to the other systems. This will allow

future changes to be made more quickly and at a lower cost. In this vision, the other

systems, International Securities Lending System / Dan McLune (DML), Securities

Lending Menu (SLMU), and GlobalOne, would only be used as loan maintenance

systems and would cease to perform any loan initiation activity. The current state of the

business requires that several incremental projects be completed in order to turn this

vision into a reality. One such project is to integrate DML and SLX so that information

from DML is passed to SLX throughout the day and overnight.

To understand the need for this project, one must understand the current process

and the implications it had on successful securities lending transactions. At the time,

DML was the system responsible for international loan maintenance activities. It tracked

actions such as marks, recalls and paybacks and used this information to immediately

update borrower credit line utilization and lender limit utilization. JPMorgan holds its


2

borrowers in the securities lending program to various restrictions, based on their

contract, dictating how much a particular borrower can have in outstanding loans at a

time. This is known as their credit limit. As loans are repaid and new ones are taken out,

the utilization of a borrower’s credit limit changes. DML tracks these activities and

immediately updates borrower credit line utilization so that new loans are not initiated

that will exceed the borrower’s credit limit. In a similar manner, JPMorgan custody

clients work with JPMorgan agents to determine lender limit guidelines. These limits and

restrictions are based on how much of their securities a lender wishes to have out on loan

at any time and to whom the securities are lent. As international loans are modified

throughout the day, DML tracks the activity and updates lender limit utilization so that

JPMorgan does not lend out more securities than the lender feels is too risky to have out

at one time.

The problem lies in the fact that throughout the day DML does not feed this

information to SLX, which is the most frequently used platform for initiating new loans.

Without knowledge of DML activities, and therefore without accurate intraday credit line

utilization figures in SLX, two different unacceptable situations may result. Utilization

may be understated in SLX, which can result in trades being booked that should not have

been because there is actually insufficient available credit line. If such a trade is

instructed to DML, it is rejected since DML does not have sufficient credit line available.

On the other hand, utilization could be overstated in SLX, which results in lost

opportunity since trade initiation is not permitted if there is insufficient available credit

line reflected.


3

In both instances, the Database team must be contacted to manually adjust the

available credit line in SLX. This is time-consuming for both the Database staff and the

traders and in time critical situations can still result in loss of opportunity.

Therefore, the main objectives of the DML to SLX Integration Project are:

1. SLX to have accurate international utilization at start of day

2. SLX to have accurate intraday international utilization

3. traders able to view a current picture of each open loan in SLX rather than having

to toggle between SLX and DML

4. users able to define at will which value should be used to impact utilization: either

the required collateral or the DML Calculated Value (Kenney, 2006)

At a very high level this will be accomplished by creating a feedback loop for

intraday updates and an overnight batch file. The feedback loop will create intraday

messages in response to loan activity in DML. When SLX processes one of these

messages, lender/borrower utilizations will be updated and a snapshot of the loan will be

created and visible to SLX users. A key aspect in the success of this project is to make

sure that SLX calculates risk factors and therefore fractional limits in exactly the same

way that DML does so that both systems display the same utilizations and limits.

The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan

with the successful completion of this important project. Specifically, work was done to

accomplish four main goals. The first two relate to the pre-design activities that needed

be completed prior to the design and build phases of the project. Success of the pre-

design activities would contribute to the successful implementation of the feedback loop.


4

The first goal of this MQP was to assist in mapping and matching the data fields between

the two systems to ensure that SLX receives all of the information it needs to properly

update its database. The second goal was to analyze the anticipated effects of the

feedback loop on servers and mainframes so that additional resources could be introduced

and performance would not be compromised. The third goal of this MQP was to gather

the information needed to conduct a critical path analysis that would examine ways to

expedite the delivery of the project. From this a fourth goal was established, which was

to recommend possible improvements to project management practices that could result

in shorter project durations, fewer delays, and more accurate estimates of project cost and

delivery dates for this and future projects.


5

2. Background

2.1. JPMorgan Chase & Co.

As the 17th largest U.S. corporation, JPMorgan Chase & Co. provides banking

and financial services to customers worldwide. Its annual revenues for 2005 were 79.9

billion dollars, which is an impressive 10.6 percent increase from the previous year.

According to Fortune Magazine, JPMorgan Chase trails slightly behind Citigroup and

Bank of America, who rank 8th and 12th respectively on the Fortune 500 (Fortune, 2006).

With origins dating back to the 1700’s, JPMorgan has a rich legacy of more than

200 years of banking and financial services experience. J. P. Morgan Chase & Co., is the

result of a merger between JPMorgan and Chase Manhattan in 2000, which then acquired

BankOne more recently in 2004 (History, 2006). JPMorgan Chase & Co. uses two

brands to provide financial services to its customers. According to its 2005 Annual

Report, the Chase brand provides a full range of banking and asset management services

to small businesses, corporations, and governments. Services include but are not limited

to Consumer Banking, Home Finance, Credit Cards, Auto Finance, Education Finance,

Business Credit, Equipment Leasing, and Real Estate (2006). The JPMorgan brand has

eight lines of business. Those businesses are Investment Banking, Research, Asset

Management, Private Banking, Worldwide Securities Services, Treasury Services,

Private Equity, and Private Client Services (Home, 2006). Through continued excellence

in providing these services, JPMorgan has remained a prominent member of the financial

industry and plays a major role on the international stage. JPMorgan Chase & Co is a

component of the 30 member Dow Jones Industrial Average and has over 1.2 trillion


6

dollars in assets. Both of these qualities are a reflection of the company’s long term

stability. (Annual Report 2005, 2006)

2.2. JPMorgan Worldwide Securities Services

With 12.9 trillion dollars of securities under custody, the Worldwide Securities

Services (WSS) division of JPMorgan is the leader in the security services industry.

Through its consultative approach and proven experience, JPMorgan partners with clients

to assess and attend to their individual needs. JPMorgan Worldwide Securities Services

help clients to optimize efficiency, diminish risk, and increase revenue by providing a

range of services which include: American Depositary Receipts, Broker Dealer Services,

Clearance and Collateral Management, Domestic & Global Custody, Fund Services,

Global Debt Services, Investment and Liquidity Solutions, Outsourcing Solutions,

Performance Measurement, Securities Lending, Securities Settlement and Clearing, and

Specialty Trust Services. With a presence in over 90 markets, JPMorgan Worldwide

Securities Services leverages its size and capabilities to help its global client base realize

their financial goals. The business’ client base includes asset managers, pension funds,

mutual funds, broker dealers, financial institutions and hedge funds (WSS, 2006).

JPMorgan Worldwide Securities Services is led by Executive Vice President

Michael Clark. After joining JPMorgan in 1994, Mr. Clark assumed leadership roles in

the Institutional Trust Services business in 2000 and the Investor Services business in

2005, which evolved into the Worldwide Securities Services business. Under his

leadership JPMorgan Worldwide Securities Services has continuously received top

ranking in industry surveys. ISF Magazine named it “#1 Overall Cash Provider” in 2006


7

and JPMorgan Worldwide Securities Services received the award for “Best Investor

Services” in the Fourth Annual Waters Ranking by Waters Magazine (WSS, 2006)

2.3. Securities Lending

Securities lending, a service provided by JPMorgan Worldwide Securities

Services, is the market practice whereby securities are temporarily transferred from a

lender to a borrower. For the period of the loan, the lender is protected against default

through receipt of collateral from the borrower, usually in the form of cash, although non-

cash collateral is sometimes accepted. At the end of the loan term the securities are

returned to the lender. Table 1 below displays the acceptable forms of non-cash

collateral for securities lending.

Acceptable Collateral Cash

Government Securities Equities REPO

Commercial Paper C/D

(Yankee/Domestic/Eurodollar) Time Deposits

Medium Term Notes U.S. Agencies Master Notes Bank Notes

Funding Agreement/GIC Asset Backed Securities

Money Market Funds

Table 1: Acceptable Non-cash Collateral (Devasthali, 2006)

During the period of the loan, both the title of the lent securities and the title of

the collateral securities transfer ownership between parties. Along with the titles come

the rights associated with owning the security. The borrower now has the right to sell or


8

lend the security to a third party, has voting privileges, and receives the economic

benefits of owning the security, such as dividends and coupons. Commonly, the contract

between the lender and borrower stipulates that the borrower make equivalent payments

back to the lender for money received in this manner (Devasthali, 2006). Similarly,

through contractual stipulations lenders may reserve the right to recall the lent securities

in order to resume voting privileges. Lending firms such a JPMorgan act as

intermediaries within securities lending and facilitate the lender/borrower relationship.

More information on JPMorgan’s role in securities lending is given in Section 2.4 below.

Lenders are typically JPMorgan custody clients such as insurance companies,

mutual funds, endowments, and pension funds. The term custody client refers to a

situation where the client owns a particular security, but the security is in the care of

JPMorgan. In this case JPMorgan is also responsible for maintaining the securities under

custody, performing activities such as tracking stock splits, dividend payouts, stock price

fluctuations, and mergers and acquisitions. Generally, a good lender has the following

characteristics:

• a diversified portfolio with a variety of lendable securities

• a large enough securities portfolio to make lending worthwhile because there is a

small margin of return associated with securities lending

• passive trading strategies resulting in few loan recalls when compared to actively

managed portfolios.


9

When the lender wants to sell a security in their portfolio that happens to be on

loan to a borrower, the lender must recall the loan to make the sale. Therefore, passively

managed portfolios make for better lenders (Devasthali, 2006).

Lenders benefit from securities lending because they receive incremental returns

on their securities that would otherwise lie dormant in their investment accounts. It

provides an opportunity for lenders to make money with very little risk. Lenders make

money through securities lending in a few ways. First, if the collateral received on the

loan is in the form of cash, JPMorgan invests that cash in a short-term investment. The

profit made on the investment is then split between JPMorgan, the lender, and the

borrower. If the collateral is in a non-cash form, the borrower pays a fee which is split

between JPMorgan and the lender. A lender also makes money in instances when the

lender is subject to high withholding taxes on dividends or interest, that the borrow is not

subject to. Here the security would be transferred to the borrower, who would receive the

dividend at a lower tax rate. The benefit from the difference in the tax rates is split

between the lender and the borrower. Lastly, lenders can earn money through securities

lending in the case when stock issuers offer stockholders the choice of receiving a cash

dividend or reinvesting it in additional securities at a discount price. If the lender cannot

take advantage of the discounted price because ownership of additional shares would

make their holdings larger than permitted under investment guidelines, the securities can

be sourced to a borrower who shares the economic benefit of the discounted prices with

the lender (Devasthali, 2006).

In addition to the financial benefits described above, borrowers make money

through securities lending because it allows them to cover the short position. A lot of


10

money can be made from an activity called selling short. In this case, the person borrows

shares of a stock through a securities lending transaction and then immediately sells

them. If the price of the stock falls the borrower re-buys the shares at the lower price and

repays the original lender to settle the loan. The difference in the selling price and the re-

purchase price is the profit for the borrower minus fees associated with the securities

lending transaction (Morris & Siegel, 1993). Similarly, Brian Kenney, a business analyst

for JPMorgan, describes that a borrower can make money in a situation where a broker

sells shares of a particular stock not yet owned, to a client to meet client demand. During

the float period (after the sale but before the shares are delivered to the client) the broker

borrows the shares through a securities lending transaction, hopefully at a lower price, to

give to the original client. Again the price difference less lending fees is the profit for the

broker (personal communication, October 24, 2006).

2.4. JPMorgan’s Role in Securities Lending

JPMorgan acts as a middleman between the lender and the borrower in securities

lending transactions. It facilitates the lender/borrower relationship and maintains all

transaction throughout the duration of the loan. As one of the top ranked custodians for

securities worldwide, JPMorgan primarily performs discretionary lending where it is the

custodian for the lendable assets. JPMorgan’s achievements in this area of business are

reflected in its receipt of several prestigious rankings, such as custodian of the year from

ICFA Magazine in 2005 and number one securities lender and global custodian by

Finance Asia (WSS, 2006). Globally, there are only eight major players in the securities

lending business. It takes a huge amount of capital to enter into the business and the

amount of money made on each loan is marginal. Therefore, to be a successful securities


11

lending agent one needs numerous clients each with a large portfolio of lendable

securities.

JPMorgan credits its success in the securities lending industry to its highly refined

securities lending program. JPMorgan has 390 lenders and 121 borrowers currently

involved in its program. The resulting, vast inventory of lendable securities provides

borrowers with a wide selection of securities from which to borrow. Likewise, with such

a large number of borrowers, lenders’ securities are utilized more frequently. This

maximizes lenders’ financial gain from participation in the program.

As an integral part of its securities lending program, JPMorgan works closely

with the lender to define lending limits and rules that govern how their securities can be

lent. An example is when a lender states that no more than 100 million dollars of

securities can be lent out at a time. Another example is when a lender declares that they

wish to hold back 25 percent of a certain holding from lending. These lending limits can

be broad and apply to an entire portfolio, or they can be very detailed, with specific rules

for each type of security. Furthermore, lenders have the option of restricting to whom

their securities can be lent. However, JPMorgan thoroughly screens each borrower

before allowing them to participate in the lending program. All of these lending limits

are then used by JPMorgan to match lenders to appropriate borrower loan requests

(Devasthali, 2006).

Another aspect of the JPMorgan program is that it protects its lenders and ensures

the risk associated with securities lending is marginal in three ways. First, JPMorgan

offers a broad indemnification plan that protects lenders in the case that the borrower


12

defaults on the loan or goes bankrupt. Second, JPMorgan requires borrowers to provide

at least 102 percent of the value of the lent security in collateral, in order to account for

fluctuations in the market values of the lent securities and the collateral. For example, in

lower risk loans, JPMorgan will require that the borrower furnish JPMorgan with

collateral that is equal to 102 percent of the value of the securities being borrowed.

According to Brian Kenney, an example of a lower risk loan is an established domestic

brokerage such as Merrill Lynch borrowing a sum of US stock and providing US dollars

as collateral. For higher risk loans, a rate of 105 percent may be charged. An example of

this would be a trader in Australia acting on behalf of an Israeli brokerage borrowing a

Russian stock and providing euros and a letter of credit from a bank in Spain as collateral

(personal communication, October 24, 2006). Lastly, the risk associated with the

investment of the cash collateral is the only risk that JPMorgan securities lending clients

are exposed to. To minimize this risk, JPMorgan utilizes conservative investment

strategies and develops strict investment guideline and restrictions with each of their

lending clients. This allows lenders to dictate how much risk they are exposed to.

The JPMorgan securities lending program offers unique options to its lenders and

borrowers that help to distinguish it from its competitors. JPMorgan custody clients who

agree to participate in the securities lending program are not charged a fee for their

participation. Bank of New York and JPMorgan are the only two securities lending

service providers that allow borrowers the option of using non-cash forms of collateral

against their loans. JP Morgan accepts U.S. Treasuries and Letters of Credit.

Additionally, JPMorgan recently introduced a second type of lending to its program

called non-custody lending, where JPMorgan is the lending agent but does not have


13

custody of the lendable assets. This is attractive because it opens the door for numerous

new clients to participate in securities lending with JPMorgan.

2.5. The Technology Behind JPMorgan Securities Lending

JPMorgan relies heavily on versatile and reliable technology in order to be one of

the major players in the securities lending business. To successfully facilitate securities

lending transactions, JPMorgan uses a variety of programs to manage its inventory of

securities and to initiate and maintain loans. These systems allow traders to obtain real

time data regarding all transactions and holdings in the securities lending service. This

project, along with other initiatives within JPMorgan, aims to integrate these systems in

order to optimize the uses of each and streamline securities lending processes. A better

understanding of these systems is vital in appreciating the relationships between them and

the processes currently being used by the securities lending group. A diagram detailing

the relationships between systems can be found in Appendix A.

2.5.1. Custody Management Systems – COSMIC & TITAN

JPMorgan has two systems in production that manage its inventory of lendable

securities. The inventory of international securities resides in a system called COSMIC

whereas the domestic securities in custody reside in TITAN. These repositories are

responsible for managing who owns a particular security and if it is out on loan. The

systems do not however know or care about the details of a loan. Access to COSMIC

and TITAN are indirect and are handled through front-end trading platforms.

(Devasthali, 2006)


14

2.5.2. Front-end Trading - SLX

The front-end trading platform used by the majority of traders at JPMorgan is

SLX. Currently, ninety-three percent of all US loans originated in SLX. SLX is one of

the newest applications that is currently in production and offers traders easier access to

data when compared to the legacy systems. SLX ensures compliance with auditing and

federal trade guidelines. It also is designed to ensure that all lenders have an equal

opportunity to lend their securities, which is required by government regulations. SLX is

based on an Oracle/UNIX platform and was written in a proprietary language called

Tenfold.

2.5.2.1. SLX Hardware

SLX production systems are hosted on multiprocessor UNIX based IBM AIX

nodes. The Global IT Services Group within JPMorgan offers tools and utilities that

monitor CPU, memory, disk utilization, and network traffic for each node. Real time

data is taken in five minute snapshots and is posted at midnight. Raw data is also

archived for future access. (GTI Hobbit, 2006)

2.5.2.2. SLX Trade Views

On the omni level (also known as the trade level) the borrower sees the entire loan

as one large package, regardless of the number of lenders associated with the loan. On

the component level (also known as the position or sequence level) traders can look at a

loan in more detail. If multiple lenders have securities lent in a particular loan the

sequence level view gives details about each lender and the value of its securities

associated with that loan. According to Brian Kenney, business analyst at JPMorgan


15

Worldwide Securities Services, the reason that the second level exists is because it allows

for better accounting and tracking of each loan (personal communication, October 25,

2006).

2.5.3. Legacy Trading Application - DML

At the present time, there is a legacy application that coexists with SLX and is

situated between SLX and COSMIC. This application is called DML and is used

primarily by international traders that are not based in the US or in the UK. Although

DML can perform loan activity, upon completion of this project DML will only be

responsible for loan maintenance. DML exists on a mainframe and can be accessed

through mainframe session thin clients such as RUMBA. DML is a Sunguard application

written in COBOL. DML uses DB2 and VSAM as a database platform. (Devasthali,

2006)

2.5.4. Legacy Trading Application - SLMU

SLMU is the counterpart to DML and is used for US-domestic securities lending

loan maintenance. SLMU has recently been phased out as a trading platform in favor of

SLX. A feedback loop was created between SLX and SLMU to feed SLX with real time

data related to updates in SLMU. SLMU still exists as an interface to TITAN but is no

longer used to process trades. Like DML, SLMU is a COBOL/DB2 platform application.

(Devasthali, 2006)


16

2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services

The Project Delivery Framework (PDF) is a common and integrated project life

cycle framework using the best practices from JPMorgan Chase. The framework consists

of three tiers of 8 phases each, with required and recommended deliverables and

processes per phase. The tier classification of a project, as Tier 1, Tier 2 or Tier 3,

is based on project cost, risk and complexity. It is expected that all projects sponsored

and led by teams within JPMorgan Worldwide Securities Services will adhere to the

PDF. The figure below shows the breakdown of project phases as required for all

JPMorgan Worldwide Securities Services projects.

Figure 1: PDF Project Phases (Project Delivery Framework, 2006)

As part of the PDF process the project manager and delivery manager needs to

meet with the JPMorgan Chase Investment Council (IC) to obtain approval and funding

three times for each project. One section of these presentations to the Investment Council

is an estimation of the cost and completion date of the project. At the first meeting with

the IC, the estimate, known as the E1 Estimate, needs to have a 50% confidence interval.

At the second meeting, the E2 Estimate is due and should have a confidence interval of

30%. At the final IC meeting, the E3 Estimates are presented which require a 10%

confidence interval. The accuracy of each subsequent estimate is tighter because it is


17

gathered at a later date when more information about the tasks required to complete the

project and resource constraints have been identified (Project Delivery Framework,

2006).

In the Securities Lending Technology Division a tool called STMate is used to

gather each estimate. Each key team member on the project enters anticipated tasks and

durations of each task into the worksheet. All entries are compiled in a summary sheet

and a cost and project duration estimate is then generated. Sample snapshots of the tool

with information from the E1 Estimate of this project can be viewed in Appendix B.

2.7. SMLU to SLX Integration Project

Prior to the DML to SLX Integration Project, JPMorgan undertook a very similar

project to integrate SMLU and SLX. In order for SLX to become the solitary front-end

trading platform, it must reflect accurate borrower credit line utilization. SLMU is used

for domestic loan maintenance activities such as re-pricing and returns. The problem

arises from the fact that these activities impact credit line utilization, but since SLMU did

not relay information to SLX throughout the day, these impacts were not reflected in

SLX utilization figures. Therefore, the objectives of this project became:

• The reflection of accurate borrower credit line utilization in SLX related to loans

maintained on SLMU. Utilization is to be reflected accurately in SLX both

intraday and end of day and is therefore to be based on SLX loan initiation and

cancellation as well as on information provided intraday and end of day by the

SLMU to SLX feedback loop.

• The continued reflection of approximated borrower credit line utilization related

to loans maintained on DML (all international markets except the UK) based on


18

SLX loan initiation and cancellation as well as on information provided by the

end of day DML MIS file

In order to accomplish these objectives, the project team created a feedback loop

between SLX and SLMU. The result is that each time a loan is updated in SLMU a

message is sent to SLX with this information. Borrower credit utilization and lender

limit utilization are then updated in SLX within two minutes. This ensures that only

loans that do not exceed lender/borrower limits are initiated and that clients’ participation

in loan activity is maximized. Upon completion of the SLX to SLMU Integration

Project, the flow of information between the key systems in securities lending could be

represented in Figure 2 below. Since the SLX/DML integration project occurred after

this project, the diagram also represents the status quo of information flow prior to the

completion of the DML to SLX Integration Project.


19

INT’LBORROWERCREDIT LINEUTILIZATION

Intraday:

New SLX Loans (+)SLX Cancels(-)

Nightly:

Recalc based on MISfiles

USBORROWERCREDIT LINEUTILIZATION

Intraday:

New SLX Loans (+)SLX Cancels(-)New SLMU / TitanLoans (+)SLMU Returns (-)SLMU / Titan COACAdj (+/-)

Nightly:

Recalc based on SODSnapshots from SLMU

BORROWERCREDIT LINEUTILIZATION

Intraday:

New Loans (+)Marks (+/-)Returns (-)COAC Adj (+/-)Cash Adj (+/-)

BORROWERCREDIT LINEUTILIZATION

Intraday:

New Loans (+)Marks (+/-)Returns (-)COAC Adj (+/-)Cash Adj (+/-)Cancels (-)

SLX SLMU DML

US FEEDBACK LOOPIntraday and Nightly

Input into SLX Intraday:New SLX Loans

SLX Cancels

Input into SLMU or DML Intraday:New Loans

MarksReturns

COAC AdjCancels

MIS File(used for nominees

other than DTC, FEDand Crest)

Nightly

Figure 2: Current Flow of Securities Lending Information (Kenney, 2006)


20

3. Methodology

3.1. Critical Path Analysis

As mentioned previously, this project was very important to the JPMorgan

business because it laid the foundation for several new products and services to be

developed that would bring significant amounts of new revenue to the company. Without

this project, these new products and services could not be developed. With this in mind,

there was a serious push from management to expedite the project. The critical path

method was determined to be the superior tool for analyzing ways to expedite the project.

A critical path analysis, based on a comprehensive project plan, allows managers to

prioritize activities for the effective management of project completion. Often projects

don’t finish on time because of multi-tasking and lack of prioritization. The critical path

method works to identify which tasks are critical to the on-time completion of a project.

Priority and special attention are subsequently given to these tasks because if the

completion of a critical task is delayed then the whole project is delayed. If feasible,

additional resources should be allocated to critical tasks to expedite their completion

(Internet Center for Management and Business Administration, Inc.).

A number of steps needed to be completed to gather the information that was

required to utilize the critical path method in the most effective way. The basis for this

analysis was an accurate and thorough project plan. Three key pieces of information are

required to begin constructing a project plan. The first is a listing of all tasks and

activities required to complete the project. The second is estimates regarding the time

needed to complete each task. The third is a list of predecessors for each task. A


21

preliminary project plan had been constructed prior to the start of this project. However,

as the project changed and developed, the original project plan needed to be modified

substantially.

A big challenge in obtaining these three types of information was that the project

was still far from completion, barely entering the design phase. At this point in the

project there were still many more questions than answers about how the project goals

would be accomplished. Each member in the project team, each with their own area of

expertise, had unique thoughts as to which tasks were necessary to realize the objectives

of the project. This led to disconnects in the overall picture of how the project would be

completed. These disconnects prohibited the creation of a cohesive project plan.

To obtain the information needed to create a cohesive project plan, the first step

was to create a document with the high level tasks and key deliverables of the project,

using information from the original project plan. The document also contained a list of

the four key objectives of the project and a note reminding resources that all tasks should

add value to fulfilling either a direct objective of the project or one of the documents

required by the Project Delivery Framework (PDF) process. This document was then

distributed to the project team for review. A meeting was held to discuss the document

where the agenda was to 1) brainstorm if all high level tasks were accounted for and 2)

collectively come to an agreement on a single strategy for how the project would be

completed. The document was then updated with information collected at the meeting.

The final version of the document can be seen in Appendix B.


22

At that point all team members had the same vision of the strategy that would be

used to complete the project. Subsequently, the data needed to update the project plan

could be gathered. The major team members were determined and can be seen in the

table below.

Team Member Specialty Area

Brian Kenney Business Analyst

Mike Salamina SLX Development

Peter Ennis SLX Mainframe Development

Rey Generoso DML Development

Besley Nelson Quality Assurance

Table 2: Key Team Members and Roles

Individual interviews were set up with each key team member to gather information

needed for creating the project plan and performing the critical path method. The agenda

for each interview contained the following points:

1. Document all activities that a particular individual needs to complete for the

extent of the project.

2. Estimate the number of days each activity will take to complete if the owner is

exclusively devoted to the tasks completion.

3. Identify the predecessors of each activity and the earliest possible start of each.

4. Receive suggestions for improving the STMate Tool used for making estimates

regarding project cost and duration.

The High Level Task Document and each person’s STMate spreadsheet from the E1

estimates were used to guide the interviews.


23

3.2. Project Management Process Improvements

Another objective of this project was to recommend ways in which project

management processes could be improved. The improvements were aimed at making the

processes more efficient, streamlined, simple and/or error-proof. To complete this

objective the current process for managing projects needed to be understood and

evaluated. This was partially accomplished by going through the experience of executing

many steps of the current system to develop a project plan and gather information for

estimates to present to the Investment Council. Throughout the execution of these

activities, a list of notes about inefficiencies and possible sources of error inherent in the

process was maintained. Information about current processes and ways to improve them

was also gathered in two other key ways. Informal discussions were held with the project

manager and the project team regarding their struggles and experiences with current

practices. Lastly, research was conducted using the Microsoft Project help function to

investigate proper usage of the tool and additional capabilities that were not previously

being leveraged when managing JPMorgan projects.

3.3. Data Mapping Exercises

The goal of the data mapping exercise was to examine the fields that DML has

defined in a loan snapshot and compare that to what data fields SLX needs to receive to

properly update borrower credit and lender limit utilizations. One of the challenges was

that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth

analysis needed to be conducted to examine how the data fields from DML should be

manipulated so that they are in a useable format for SLX. Once this was accomplished, to


24

verify that the manipulations are correct, the team will create a proof of concept. This

will entail creating a mock intraday message using the DML data fields as inputs that

outputs the same information in the SLX format.

To the surprise of the development team, the data was too complex to create a

one-to-one mapping such as saying the broker field in DML matched the broker field in

SLX. According to Danielle Rumore, calculations proved to be extremely difficult to

trace because of the extensive amount of COBOL logic that was built around each field

in DML over the past decade. (Personal Communication, November 15, 2006). Danielle

also explained that up-to-date documentation does not exist for DML, so very few people

know exactly what each field means. As part of the data mapping exercises, sample

loans were observed from the production environment. Screenshots of one such loan can

be found in Appendix D.

3.3.1. Parsing the DML Loan Master

The first encountered problem was taking the raw loan data from DML, known as

a Loan Master File, and putting it into a database that allowed it to be analyzed. The loan

master file contains over 220,000 records stored in a comma delimited file called a CSV

(Comma Separated Value) file. Due to software limitations in Microsoft Excel XP, only

the first 65,536 records could be displayed because row numbers are stored in a 16 bit

integer. Importing data into Microsoft Access using the Microsoft .txt Jet Engine was

also problematic because opening the text file for parsing caused a virtual memory

overflow on the workstation.


25

3.3.1.1. Design

To get around the file size limitation, a program was developed that would split

the CSV into multiple files that had less than 65,000 records. A parser was designed that

would read in the CSV file one line at a time and immediately output it to a target CSV

file. A counter would be in place to keep track of how many records were written to the

target file. Once the user-defined limit was reached, the parser would send the data out to

another file. A high level design is shown in Figure 3.

Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006)

3.3.1.2. Implementation

To implement this parser, the Java programming language was used.

Development occurred in an Eclipse 3.2 development environment using the Sun Java

JS2E SDK v1.5. A third party CSV parser package called com.csvreader.CsvReader was

used in the implementation because of its high benchmark ratings. One of the benefits of

using this package is that none of the data is buffered in memory, so it can process a CSV

file of any size.


26

The software uses nested for loops to scan the records. The outer loop reads one

line at a time. The inner loop reads each column and writes it to the target output text

file. If however, the number of columns in a row did not equal 125 (a hard coded

constant equal to the expected number of columns a DML loan), the parser would output

that entire row to a different file to be analyzed later.

Command line arguments were implemented to allow the user to specify the input

file name, the output file name, and the maximum number of records that each output file

is allowed to contain. The only known bugs associated with the program are the

following:

• Using an integer value greater than Integer.MAX_VALUE for integers will cause

problems when counting the number however that value in Java 1.5 is 2^31 – 1

which is larger than the number of integers allowed in Excel anyways.

• Using long filenames or name with spaces causes problems with the file name

string. We suggest using the MS-DOS 8.3 file naming convention when

referencing file names in the command line parameters.

The compiled .class file was then placed on a shared network drive so that any

user that has the 5.0 release of the Java Runtime Environment (JRE) could parse a loan

master. The source code for the program can be found in Appendix I.

3.3.2. Aggregation and Summarization of Data Fields

Once the data was imported into Microsoft Access, there was a need to run

various queries on the data in order to find patterns or unique values. According to

requirements and needs, the following queries were developed:


27

• Queries on the SLX/DML broker tables

• Join DML/SLX Brokers • DML Brokers without matching SLX Records • SLX Brokers without matching DML Records

• Queries on the DML Loan Master

• Unique LOAN-TYPE values • Unique SEC-LOC values • Unique SEC-TYPE values • Filter out closed loans

These queries were written in Microsoft Access SQL Query format. The queries

can been found in Appendix J. These queries along with the Microsoft Access database

were then posted to a shared folder so that any person that wanted to run these queries

could do so.

3.4. Capacity Planning Analysis

The goal of the capacity planning analysis was to carry out a comprehensive

analysis on the SLX servers and to gather enough information to make infrastructure

decisions moving forward. The primary deliverable was a document that gives the reader

an understanding of the resources required to run SLX in a production environment and

to analyze the impact that the DML feedback loop will have on the systems. Information

was gathered from many sources and compiled together to form a complete picture of the

resources that SLX depends on. The complete document can be found Appendix I.

3.4.1. Server Specifications

The first step in the capacity planning analysis was trying to understand what the

current servers had for hardware. Through documentation and utilities provided by


28

JPMorgan’s Global Technology Infrastructure (GTI Hobbit, 2006). The primary focus is

on the production system. General information about production server hardware can be

found in Table 3. All SLX servers are IBM AIX servers with PowerPC processors

running a UNIX environment. The servers can also allocate a percentage of the total

number of physical processors to a specific application (in this case SLX).

APP06 APP10 DBP01 Total Number of Processors 4 2 8

Processors assigned to SLX 4 2 3.5

Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz

Processor Type 64 bit 64 bit 64 bit

L2 Cache Size 1.5 MB 1.5 MB 1.5 MB

Physical Memory 7.75 GB 7.75 GB 62 GB

Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/paging00 : 6GB

SLX Disk Drives 9 11 6

Table 3: SLX Production Server Hardware (Ghelani, 2006)

3.4.2. Used Server Resources

The next step was to look at the current performance of the servers. There were

two areas of particular interest, current processor usage and allocated memory. Both of

these pieces of information could be obtained from a JPMorgan website that provided

real-time server statistics. (GTI Hobbit, 2006) For the analysis, data was gathered for the

same week (October 15th – October 21st) to maintain consistency. The three areas of

focus are processor usage, memory allocation, and storage quotas.

3.4.3. SLMU Message Statistics

The final step was to merge in statistics from SLMU and projected DML activity

onto the plot of production environment. SLMU data was gathered from the SLX


29

production environment query on the SLMU feedback loop. DML data was exported by

the DML system administrators and emailed to requestors.


30

4. Results and Analysis

4.1. Critical Path Analysis

The majority of the information gathered for the critical path analysis was

obtained during the interviews with key members of the project team. Each interview

focused on identifying the tasks that a particular resource would need to complete as part

of the project and what the task dependencies are. Some team members were able to

provide task durations during these meetings. Timing was such that E2 Estimates, which

are the second round estimates that should have a 30 percent confidence interval, were

due a short time after these interviews. Therefore, these meetings served a dual purpose:

to aid in the creation of a thorough project plan and to aid in filling out the STMate tool

for the E2 Estimates. Task durations for those team members that did not provide them

in the interviews were gathered from the E2 Estimate spreadsheets once they were

completed. From these meetings, flow charts were created with the tasks for each team

member to visually highlight predecessors and task dependencies. The flow charts for

each resource can be viewed in the following figures.


31

Figure 4: Flow Chart for Business Analyst


32

Figure 5: Flow Chart for Quality Assurance Team


33

Define DML fieldsanalysis for SLX

resources(Data Mapping)

DML Unit TestPlan/Strategy

Backout RecoveryPlan

DML Calculations15 Days

E2 Estimates3 Days

Brainstorm HighLevel Design w/

DML Team2 Days

Detailed Design(DD)

SOD File

DD Logical Unit ofWork DD Snapshot

DD Process LogicDesign for Loan

Master

DD Prototyping

Start Document 2

UpdateDocuments

1,3, 5

DD Document

Documents1. Traceability Matrix2. ECM3. Backout Recovery Plan4. Turnover Document5. Implementation & Rollout Document

DD Walkthrough

Intraday (I/O Module)-Create Module for

Snapshot(Build, Test, Peer Code

Review)

Intraday (I/O Module)-Create Module for Logic of

Unit Work(Build, Test, Peer Code

Review)

SOD-Programming &

Copybook(Build, Test, Peer

Code Review)

SOD- JCL Proc(Build, Test, Peer

Code Review)

SOD- CA7 SetupScheduling

(Build, Test, PeerCode Review)

Intraday- Modify ExistingModules for DML onlines(Build, Test, Peer Code

Review)

Intraday- Modify ExistingModules for Mark Process

for Japanese Assets(Build, Test, Peer Code

Review)

SIT Tests in DML

Start Document 4

Revise Documents1,2,3,5

Support QA andUAT Testing

Finalize AllDocuments

DML - ReyGeneroso

Figure 6: Flow Chart for DML Team


34

Figure 7: Flow Chart for SLX Team


35

Brainstorm HighLevel Design

SLX Mainframe-Peter Ennis

Data MappingCompletion (Informationregarding outputs from

DML and necessaryinputs to SLX)

Detailed DesignSOD Batch

Program Specs

Detailed DesignIntraday BatchProgram Specs

Detailed DesignIntraday OnlineProgram Specs

Build CopybookFile Layouts

Build CopybookCommon Logic

Module

Build JobsInfrastructure

CICS, MQ, VSAMor DB2

Build SODProgram

Build IntradayOnline Program

Enter Build Phase

Build End-of-intraday Batch

Program

Enter Testing

Write Unit TestStrategyWrite SIT Strategy

Execute SIT

Execute Unit Tests

Support QA

Figure 8: Flow Chart for SLX Mainframe Team


36

All of the information gathered from the interviews, the flow charts, and the E2

Estimates was then entered into Microsoft Project to create an initial project plan. The

next step was to identify all of the resources that would be working on the project. All

team members’ vacation schedules and bank holidays were then integrated into the

project plan to accurately portray the project schedule. The next step that we were able to

complete was to work with the delivery managers in charge of the team members to

investigate how much of each resource’s time could be allocated to working on this

particular project. The delivery manager is the supervisor to most team members and is

responsible for the thorough, timely delivery of the project. The project plan was again

updated with this information so that it would accurately represent task durations.

That was all of the information that we were able to gather during our time at

JPMorgan. At that point the project plan was handed back to the project manager to

complete the analysis. The final step in creating an accurate and thorough project plan to

use in the critical path analysis is to assign resources to each project task. After that the

project manager can use Microsoft Project’s Detailed Gantt Chart view to see the critical

path. The last step is to analyze the critical path to determine ways to accelerate the

project. The following options should be considered and analyzed:

1. Allocating extra resources to critical tasks

2. Revising task dependencies to allow for more scheduling flexibility

3. Breaking the new functionality developed by this project into individual features

that could be delivered to production separately and at different times


37


This section aims to document and evaluate current project management

processes in place in the Securities Lending Technology division of JPMorgan. The three

areas of project management practices that were evaluated in this project are: 1) the

process for gathering information for Investment Council estimates and project plan

creation, 2) the use of Microsoft Project plans to help manage the project and 3) the

tracking of action items identified in team meetings. In addition to a discussion

surrounding each of these three areas, there is also a section about general observations

regarding the culture of project teams and how projects are executed. These observations

are important in understanding how to improve the way projects are managed to result in

faster project delivery without compromising quality.

Overall, project management within JPMorgan Securities Lending Technology

does not have a defined set of best practices. The process used to manage projects differs

between project managers and even between projects by the same project manager. This

results in confusion, poor practices being used, and time wasted by team members

learning a new system for each project. For the most part deadlines are very fluid and

team members are not held accountable when deadlines are missed. Another observation

is that there is a poor sense of team within the group. Many people seem to have tunnel

vision around their specific role and seldom offer input or suggestions regarding aspects

of the project that they are not directly responsible for. This poor communication across

team roles contributes to the lack of a clear, big picture throughout the team of how the

project will be executed and the objectives satisfied.


38

4.2.1. Investment Council Estimates and Project Plan Creation

4.2.1.1. Current Process

Two important functions performed by JPMorgan Project Managers are to

provide project cost and duration estimates to the Investment Council for project funding

and to create a project plan. As stated in the methodology section of this paper, three

main types of information need to be collected from each project team member in order

to create a project plan. The first is a listing of all tasks and activities required to

complete the project. The second is estimates regarding the duration of each task.

Duration specifically means the amount of actual working time necessary to complete the

task. It does not mean the elapsed time between the start of the task and the completion

of the task. The third is a list of predecessors for each task. The first two types of

information are also necessary for providing estimates to the Investment Council.

The current process for gathering this information is to first send out the STMate

tool template to each team member. The team members fill in the STMate tool

worksheet for their job type (e.g. business analyst) with tasks and estimated durations for

each task. They also compile a list of assumptions they used when making the estimates.

Each person sends his/her STMate worksheet and list of assumptions to the project

manager. The project manager then takes the total estimated man hours and associated

cost from each worksheet and creates a separate overall project summary in Excel. The

STMate tool does have the capacity to compile a summary of all of the estimates if all

worksheets are in the same workbook, however, this function does not get used. Next the

team meets to discuss the assumptions each resource used when making their estimates.


39

The purpose is to compile an overall assumptions list to be used in the presentation to the

Investment Council.

At this point, all of the necessary information for Investment Council funding

request presentations has been obtained. However, a project plan cannot be created

without a thorough understanding of task predecessors. The project manager gathers this

information in a piecemeal fashion in a few different ways. For tasks that are assigned to

a single resource, the project manager meets with that person to discuss the order in

which the tasks need to be completed. For tasks that require a group effort, the project

manager both meets with the delivery manager to discuss possible task dependencies and

also attends brainstorming meetings where the team develops a strategy for completing a

certain group of tasks. In all instances, the project manager spends large amounts of time

gathering task predecessors. This is a very inefficient process. Due to the piece meal

fashion in which task predecessors are identified and relayed to the project manager to

enter into the project plan, several weeks can pass before a thorough, accurate project

plan can be analyzed and used to drive the project.

4.2.1.2. Problems

While the current system does accomplish the task at hand, there are many

inefficiencies and potential sources for error associated with it. The majority of the

problems lie in the STMate tool. As a brief overview, as stated in the Background

Section of this paper, the tool is an Excel workbook template that has worksheets within

it for each type of team member. The types of team members are Project Manager,

Business Analyst, Quality Assurance, Technical Manager, and Hardward Systems. Each


40

team member fills in the worksheet designed for their role with all anticipated tasks and

effort estimates for each. The tool then calculates the total man-days each resource

requires to complete their share of the project and the associated cost. This information is

displayed in a summary sheet. Some example snapshots of the tool can be seen in

Appendix C. The main problem is that the tool is unnecessarily complex with no clear

set of use instructions. There is a very comprehensive list of all possible project tasks

that could potentially be completed in each resource spreadsheet. Many team members

feel that the list is too cumbersome to read through because projects in this division vary

so greatly that the majority of the tasks on the list do not pertain to a given project. The

goal is to remind people of any tasks they may not have remembered without the list.

People should just leave the estimated time for non-applicable tasks at zero. However,

there is not a set of instructions that explains this explicitly, which leads confusion. The

figure below illustrates this point. It is a snapshot of a technical manager’s E1 estimate.

The majority of the tasks do not apply to the DML to SLX Integration Project and

therefore, have a duration of zero days entered into the columns in blue.


41

Figure 9: STMate Snapshot Showing Unnecessary Task List

Another way in which the tool is unnecessarily complex is that it is designed

around the functionality that allows a person to enter estimates based on the complexity

of a task. This function is applicable if a task involves repeating an activity several times,

where the level of difficulty is not the same for each iteration. A very basic example of

how this functionality works is that a chef is making an elaborate dinner and wants to

make a project schedule. As part of the dinner he will make six pies: 1 pumpkin, 3 apple,

and 2 strawberry rhubarb. He estimates that the pumpkin pie is easy to make and will

take 1.5 hours, the strawberry rhubarb pies are medium difficulty and will take 2 hours

each, and the apple pies are of a high difficulty and will take 4 hours each to make. The

tool has three sets of columns to calculate an overall estimate for making pies. In the


42

activity units by complexity columns he would enter the number of pies for each

complexity level. In the effort static data columns he would enter the amount of time it

takes to make a pie for each complexity level. The effort calculation columns are the

product of the two other sets of columns. The table below shows a simplistic version of

how the chef would enter this data into the STMate tool. The total estimated time to

make pies is 17.5 hours.

Activity Units By Complexity

Effort Calculation (Man Hours) Effort Static Data

Task High Medium Low High Medium Low Total High Medium LowMake Pies 3 2 1 12 4 1.5 17.5 4 2 1.5

Table 4: STMate Entry Based on Complexity of Repeated Tasks

The problem with this functionality is that in Securities Lending Technology

projects most tasks are unique and not repeated and therefore, do not require the use of

this functionality. However, the template for the tool is designed around this capability,

leaving no place to enter simple estimates for a one time task. Many times the project

manager is forced to meet with team members to discuss how they entered estimates into

the tool to make sure the way the project manager is interpreting the estimates is the way

the team member intended. The complexity of this aspect of the tool is not justifiable

when considering the rare instances in which it is useful, and the confusion and extra time

it introduces to the estimation process.

There are some other smaller issues with the tool and the way it is used. Many

cells have built-in formulas that should not be changed, but these cells are not locked. A

user can easily erase or modify these formulas without realizing it. Additionally, the

STMate tool and the way it is currently used, does nothing to ensure that the tasks one


43

team member believes are necessary to complete the project align with the tasks another

team member envisions. As mentioned previously, due to the wide range of peoples’

expertise, it is a challenge with many of these projects to have all project resources

collectively come to an agreement on a single strategy for how the project should be

completed. The process for gathering the data needed for estimates and project plan

creation should be designed to encourage all team members to have a common ideology

and subsequent high-level tasks in mind when they provide their list of tasks, estimated

durations, and predecessors. Lastly, the STMate tool is not designed to capture task

dependencies and predecessors. Likewise, there is no formal process for gathering this

information and passing it to the project manager, who can then incorporate it into the

project plan. The project managers receive this information informally little by little

through attending project meetings and holding discussions with team members about

project deliverables as they become due. However, a thorough and timely understanding

of task dependencies is vital to the creation of a useful project plan.

4.2.2. Use of Microsoft Project

Using all of the data discussed in the previous section, the project manager creates

a project plan using Microsoft Project. When new tasks arise through team meetings and

progress on the project, team members notify the project manager who updates the

project plan. The main use of the project plan for JPMorgan Securities Lending Project

Managers is to determine a completion date for the overall project and to keep track of all

deliverables and key tasks associated with the project. However, the tool can be

leveraged much more to effectively manage the project and schedule tasks to ensure a

speedy delivery. In the current system, rather than using the project plan to dictate when


44

team members should be working on specific tasks to give priority to critical tasks and

ensure proper allocation of resources, team members notify the project manager of when

they will be able to complete tasks and then the project plan is updated based on this

information. In this case it is used more as a tracking tool than a planning tool. With this

method people are not held as strictly to due dates and the project can easily be delayed.

Also in the current system the project plan is mainly used by the project manager. The

other team members do not follow the project plan and its schedule to dictate when they

work on which tasks. Therefore, team members are not aware of which activities are

critical tasks and as such do not know how to best prioritize the tasks assigned to them.

4.2.3. Action Item Tracking

Because Securities Lending projects often require a project team with widely

varied roles and backgrounds, many brainstorming sessions, working meetings, and

update meetings are held to discuss the tasks required to complete the project. New

action items are often identified at these meetings. These action items are too granular to

put into the project plan but are essential to the successful completion of key deliverables

and the overall project. Since they are not captured in the project plan, another system is

required to track action items. The current system is that after most meetings, the

meeting coordinator sends out a recap of the meeting via email. Part of this is a list of

action items and the person responsible for the completion of each. There is no

consistent method in place for checking back on the status of these action items. Also

there is no one place that houses a compiled list of action items. The project team

members are responsible for searching through their email to find action items. Since

there are so many meetings each with a separate meeting recap email and also so many


45

other emails received daily, it can become very time consuming for a team member to

keep track of all of the action items assigned to them. Likewise it is even more difficult

for the project manager to track the progress on all of the action items associated with a

project. This system is very inefficient and can cause team members to lose sight of

important tasks that are vital to a timely and successful project completion.

4.3. Data Mapping Exercises

4.3.1. Parsing the DML Loan Master

The loan master parser successfully parsed the comma separated file and did so

without causing strain on the workstation. Running time for this program was under 10

minutes. The program was also created with good object-oriented software design

practices in mind. Table 5 shows important code metrics for this program. These metrics

were generated by the Eclipse Metrics plug-in.

Cyclomatic Complexity

Lines of Code

Num of Levels

Number of Parameters

Number of Statements Method

8 126 5 1 58 main(java.lang.String[])

2 19 2 2 6 setOutputFileName(java

.lang.String, int)

2 18 2 1 6 setErrorFileName(java.l

ang.String)

Table 5: Code Metrics for DMLLoanMasterParser.java

Because this program is contained all within one class, there are very few object

dependencies. An analysis was performed on the class and package level with the use of

the CAP (Code Analysis Plugin) plugin for Eclipse. The results for the package are

shown in Figure 10, Figure 11, Figure 12, & Figure 13.


46

Figure 10: Class Dependency Diagram for DMLLoanMasterParser

Figure 11: Instability Metric for DMLLoanMasterParser (green dot)


47

Figure 12: Efferent Couplings for DMLLoanMasterParser

Figure 13: Class and Package Statistics for DMLLoanMasterParser


48

4.3.2. Aggregation and Summarization of Data Fields

Due to inefficiencies of Microsoft Access, the queries written in Appendix G took

a very long time to run. It was decided to run the queries once and then export the results

to a Microsoft Excel workbook, as the data in the referenced tables did not change after

the initial import.

4.4. Capacity Planning Analysis

The findings of the capacity planning analysis on the servers can be broken down

into three categories: processor, memory, and storage. Each category looks at current

bottlenecks and anticipated bottlenecks. The complete analysis can be found in the

Capacity Planning document at Appendix I.

4.4.1. Processor Analysis

Data was extracted from the servers showing how busy the processors were at a

given point in time during the day. Data points were recorded every five minutes. To

paint a picture of what a typical day looked like, the median value was taken (out of the

seven days) for each of the five minute intervals. The median was used instead of the

mean because weekend downtime would have skewed the plot for certain time intervals.

Each of the three production servers was then plotted over a 24 hour horizontal axis.

This plot can be found in Figure 14.


49

Median production server CPU usage (10/15/06 - 10/21/06)

0

0.2

0.4

0.6

0.8

1

0:001:00

2:003:00

4:005:00

6:007:00

8:009:00

10:0011:00

12:0013:00

14:0015:00

16:0017:00

18:0019:00

20:0021:00

22:0023:00

0:00

Time of day (Eastern Time)

CPU

usa

ge (a

s a

ratio

of 1

00)

% APP06 CPUUtilization

% DBP01 CPUUtilization


6 per. Mov. Avg. (%APP06 CPUUtilization)

6 per. Mov. Avg. (%DBP01 CPUUtilization)


Figure 14: Production Server CPU Usage

As illustrated in Figure 14, CPU usage has two areas of peaks. The first peak

starts at 9:00 PM and runs until 3:00 AM. The second peak starts at 3:00 PM and ends

around 7:00 PM. These peaks can be attributed to SLX batch processes that are running

during those periods. Surprisingly the server load is under 50% for most of the trading

day. As a result, processing time for messages is under the 2 minute service level

agreement.

4.4.2. Memory Analysis

Data was extracted from the servers showing how much physical and virtual

memory was used during the day. In the case of memory, a bigger time interval was


50

observed because memory changes less rapidly. Memory allocations for servers APP06,

APP10, and DBP01 can be seen in Figure 15, Figure 16, & Figure 17 respectively.

Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006)

Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006)

Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006)


51

Figure 15, Figure 16, & Figure 17 show memory usage on each of the servers

over a 48 day span. The plot in blue represents physical memory usage. The plot in red

represents swap space via hard disk paging files. One interesting observation is that

APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The

server then falls over to allocating and using swap space.

4.4.3. Storage Capacity

Data was extracted from the servers that showed how full each hard drive volume

was on all production servers. The goal was to find volumes that were at risk of

exceeding their quota after the implementation of the DML feedback loop. Two hard

drive volumes were identified of being at risk (volumes that are more than 80% full).

These volumes are shown in Figure 18 & Figure 19.

Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota


52

Figure 19: /app/a_slxftp on the JIP NAS is near capacity

4.4.4. SLMU Message Statistics

The next step was to observe intraday processing of SLMU messages as they hit

the SLX systems. Information was extracted from SLX queries which gave an hourly

breakdown of the number of SLMU messages that were processed during that hour. In

addition to the number of messages, the average time it took to process a single message

during that hour period was returned from the query. This information was plotted on top

of the CPU graph found in Figure 14. This allowed for correlations to be made between

how busy the processors were and how long it took to process an SLMU message. This

graph can be found in Figure 20. According to the graph, the processing time for an

SLMU message is more than two minutes between 3:30PM and 6:00PM daily. This

means that SLX is not meeting the service level agreement (SLA) that guarantees

processing time within two minutes.


53

Average SLMU messages & processing time (from 10/15/06 - 10/21/06)

0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

SLMU messageprocessing time

Processing SLA of2 minutes

6 per. Mov. Avg.(% APP06 CPUUtilization)

6 per. Mov. Avg.(% DBP01 CPUUtilization)

Figure 20: SLMU Messages in SLX

The infrastructure team has commented on this and remarked that there are no

memory limitations on TSJIPUAPP06 server. Piyush Ghelani, system administrator of

JPMorgan’s servers said, “the illusion of being out of memory comes from the fact that

users do not count file cache usage feature of the Unix kernel. Actual memory usage is

4.125 GB.” (Personal Communication, November 22, 2006) Starting after week 44 the

system administrators also limited the size of the file cache so that it would resize itself to

prevent the swap space from being used. Even with this change in place, the processing

times for SLMU messages remain unchanged as shown in Figure 21. This graph was

created using the same methods that were used in creating the October performance

assessment. The only difference is that for November statistics, the week of November

12th was used.


54

October Performance vs November Performance

0

3000

6000

9000

12000

15000

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)

Projected DMLMessages

Oct. SLMUMessages

Nov. SLMUMessages


Oct. SLMUprocessing time

Nov. SLMUprocessing time

Figure 21: Comparing SLMU Performance Between Oct and Nov

4.4.5. DML Projections

The final step was to take intraday trading information from DML and to project

when and how many messages would be sent to SLX during the day. The method of

plotting the information was to look at the sum of the loan and component updates during

the hour period and to treat each as a single message. This closely matches what SLMU

does when sending messages to SLX. The plot was then shown on the same graph as the

SLMU messages and CPU usage. In doing this, one could see if the SLMU message

peaks matched up with DML peaks. This combined graph can be seen below in Figure

22.


55

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000



6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)

Figure 22: SLMU and Anticipated DML Messages over a Trading Day

In looking at the graph, DML is anticipated to have a spike in messages between

11:00AM and 1:00PM. As a result, it is predicted that the processing time will be greater

than two minutes during this time frame and thus, the two minute SLA will not be met

between 11:00AM and 1:00PM.


56

5. Recommendations


Based on our evaluation of current project management practices and our analysis

of the areas in need of improvement, we have developed a project management process

that encompasses many of the responsibilities of JPMorgan project managers. In

particular, the process involves the three areas that have been focused on throughout this

paper: gathering project cost and duration estimates for the Investment Council, creating

a project plan and using it to manage the project, and tracking action items. The

flowchart below illustrates the process that we have developed. We recommend that

JPMorgan adopts this process. The main recommendations that summarize this process

are:

1. Use the Estimation Workbook in place of STMate to gather information for

estimates and project plan creation.

2. Take full advantage of Microsoft Project and the critical path method to schedule

tasks and better manage projects.

3. Use the Action Item Tracking Sheet to track the status of action items and project

risks.

4. Hold weekly status meetings to discuss progress and encourage communication

across project roles.

The following sections explain this process in greater detail. We also recommend that

the firm conduct a more in-depth study to identify project management best practices and

develop a common process to be used by all project managers for all projects in


57

JPMorgan Securities Lending. This effort could possibly be spearheaded by another

team on WPI students.

Figure 23: Project Management Process


58

5.1.1. The Estimation Workbook

We have developed a new tool called the Estimation Workbook that we

recommend be used in place of STMate to gather estimates for Investment Council

project funding requests. The Estimation Workbook is a simple Excel workbook that will

capture all of the necessary information from project team members, while eliminating

the unnecessary complexity and confusion associated with the STMate tool. To ensure

success in using this tool, the project team should follow the process that we have

developed for its use.

The project manager should first hold a meeting with the entire project team to

make a list of the project milestones and deliverables and their dependencies. Each team

member should then work offline to develop a list of all foreseen tasks that will be

required to accomplish the project milestones and deliverables. The team should then

come together again to compile a comprehensive list of all project tasks and task

dependencies. The team should reference the list of tasks in STMate to make sure they

account for all tasks that will need to be completed. For clarification purposes, a

milestone is a significant accomplishment or intermediate project goal. A deliverable is

any tangible outcome that is produced by the project. These can be documents, plans,

computer systems, buildings etc. Lastly, a task is any piece of work that is undertaken

that adds value to the creation of a project deliverable or fulfillment of a project

objective.

Once this list is compiled and the team agrees on it, the project manager should

set up the Estimation Workbook. A new Estimation Workbook will need to be created


59

for each project and should be based on the example that we have created for this project.

In every Estimation Workbook there will be tabs named Instructions, Assumptions,

Summary Sheet by Phase, and Summary Sheet by Role. There will also be a tab for each

key project team member labeled with their role in the project. Each team member tab

and the summary sheet by role tab should have the comprehensive list of project tasks

along the left edge. The milestones and deliverables should be in bold with the respective

subtasks listed directly beneath each. After the project manager sets up the Estimation

Workbook with all worksheets, columns and formulas, he/she should post it on a shared

drive and notify the project team that it is ready for their inputs. The figures below are

snapshots of the sample Estimation Workbook that we have developed.

Figure 24: Estimation Workbook- Instructions


60

This instruction sheet is a summary of the process that we are recommending. It

gives instructions for the project manager on how to create the Estimation Workbook and

also for team members on how to fill it in with their estimates. This instruction worksheet

should be copied into every new Estimation Workbook.

Figure 25: Estimation Workbook- Team Member Worksheet

Each member of the project team that is required to provide estimates should have

their own tab in the Estimation Workbook. The tab should be labeled with the team

member’s role such as BA for Business Analyst or QA for Quality Assurance. The tabs

should all be identical and should look like the figure above. They will each have the

comprehensive list of all project tasks in the left hand column and a list of the project


61

phases along the top edge of the worksheet. Once the project manager sends out

notification that the workbook is posted on the shared drive, each team member should

open the workbook and find the tab labeled with their role. They should then fill in their

estimates in man-days for how long they think each task will take them in each phase of

the project. These are estimates of pure effort not of elapsed duration. If a task does not

apply to a particular team member, then that team member should leave that task blank or

enter a zero. Each team member should also click on the assumption tab and review the

list of assumptions started by the project manager. If a team member wants to modify or

add to the list of assumptions, he/she should do so, making sure to note their name and

the date next to any changes. Team members should not enter any information into the

summary sheets. These sheets are linked to the team member estimate tabs and will

aggregate automatically. Once a team member finishes entering their information into

the Estimation Workbook, he/she should save the changes on the shared drive and notify

the project manager.


62

Figure 26: Estimation Workbook- Summary Sheet by Phase

The Summary Sheet by Phase provides the project manager with total time and

total cost broken down for each phase of the project. The time totals are shown in days,

hours, and months. Once all team members have filled in their worksheet, the project

manager should pull the file off the shared drive. Since this summary sheet is linked to

the data entered in the team member estimate sheets, the project manager just needs to fill

in the cost table with the cost per day of each team member. Once that is complete, the

Estimation Workbook will show the total time and cost for each phase of the project.


63

Figure 27: Estimation Workbook- Summary Sheet by Role

The summary sheet by role is also linked to the team member estimation sheets

and contains the same comprehensive list of project tasks. When the project manager

sets up the Estimation Workbook, he/she should enter the task predecessors in the right

hand column of this tab. After all team members enter their estimates, this summary

sheet shows the total time and cost for each team member along the bottom edge of the

worksheet. The time totals are again given in days, hours, and months. Also, along the

right side of the sheet, next to the task predecessors are total man-days for each task

across all team members. Between the two summary sheets, the project manager should

have total duration and cost estimates broken down in a variety of ways that will be

useful in presenting to the Investment Council.


64

We realize that this process requires a substantial amount of time upfront to

compile the comprehensive list of project tasks and dependencies; however, we feel the

benefits of gathering estimates using the Estimation Workbook outweigh this extra

upfront time. Some of the most important benefits are:

• Clearer understanding of project tasks by all team members because of the

comprehensive task list on all estimate tabs

• Cost and duration totals broken down in several ways

• Eliminates the need for the project manager to manually merge the estimates from

each team member into one estimate

• Simpler design and linked worksheets reduces the time all team members have to

spend providing estimates

5.1.2. Microsoft Project and the Critical Path Method

The next step in our proposed project management process is to create a project

plan in Microsoft Project using the comprehensive task list, task predecessors, and task

effort estimates from the Estimation Workbook. The project plan should be set up in

such a way that it has two layers. The high level layer should include the network of

project milestones and deliverables and their dependencies. Below this, there should be a

more granular layer with the network of tasks needed to complete each deliverable and

reach each milestone. A diagram illustrating this point is shown below. The pink boxes

show the network of project deliverables. The blue circles represent the network of tasks

required to accomplish each deliverable.


65

Figure 28: Diagram of Two Layer Project

The project manager and the delivery manager should also work together to

compile a list of all of the resources that will be working on the project. Then they

should look up each person’s vacation schedule and determine what percentage of each

person’s time will be allocated to that particular project. Once all of this information is

entered into Microsoft Project, it will automatically create a project schedule. The

project manager should use the Detail Gantt Chart view to see the critical path for the

project. All critical tasks are highlighted in red and should be given priority. To ensure

that resources are not over-allocated and that the schedule depicts realistic dates, the

project manager should re-schedule concurrent tasks assigned to each resource to give

priority to critical tasks and subsequently fill in non-critical tasks in the open areas of the

schedule.


66

We recommend that project managers take a Microsoft Project training class to

ensure they are leveraging as many functions of the program as possible. The main goal

is to have the most accurate project plan as possible by accounting for such things as

vacation time and resource allocation to other projects. With a solid project plan as the

foundation, the project manager can ensure on-time delivery by giving priority to critical

tasks. It may even be possible to expedite the completion of a project by analyzing

options such as allocating extra resources to critical tasks or breaking the project into key

deliverables that could be delivered separately and at different times.

Once the project manager develops a sound project plan, the project team should

use it to prioritize and manage their work loads and due dates. To help with this, we

recommend that the project manager print and post relevant sections of the project plan

on a highly visible wall. Using the project plan in this way ensures that it is used as a

planning and tracking tool, rather than simply a tracking tool as it was with the current

system. The foreseen benefits of this process are as follows:

• Substantially less time required by a project manager to create the project plan

because information is taken directly from the Estimation Workbook

• More accurate sense of project end date

• More likely on-time delivery because all team member recognize and give

priority to critical tasks

• Fewer project delays due to increased accountability and awareness of due dates


67

5.1.3. Action Item Tracking

We recommend that the JPMorgan Securities Lending Technology division begin

using the following process and tool to track the status of action items that arise through

formal and informal team meetings. As team members identify new action items, they

should send an email to the project manager with the following information:

• A description of the action item

• The milestone or deliverable from the project plan that the action item will help to

fulfill (the team should refer to the project plan for list of tasks and deliverables)

• The open date and due date of the action item

• The team member responsible for the action item

This should also be extended to tracking risks associated with the project. When

new risks are identified, the same information from above should be sent in an email to

the project manager. Additionally, the email should contain a brief risk mitigation

strategy.

The project manager should then use the Action Item Tracking Sheet that we have

developed to input this information for each action item and risk. The tool is an excel

workbook with three worksheets. The first is an instruction sheet for the project manager

and can be seen below.


68

Figure 29: Action Item Tracking Sheet Instructions

The other two sheets are where the actual action items and risks are input and

tracked. The project manager should input all information from the team emails in the

appropriate sheet. These two sheets can be seen in the figures below. The initial status

of all action items and risk items should be open. As progress and status changes are

made on action items, notification should be sent to the project manager through email.


69

Figure 30: Action Item Tracking Sheet for Action Items

Figure 31: Action Item Tracking Sheet for Risk Items


70

Using the Action Item Tracking Sheet in the manner stated above will result in

many benefits for Securities Lending Technology project teams at JPMorgan. Action

items that are essential to completing project deliverables will all be housed in one

location. This will eliminate the need to search through numerous emails and the risk

that the team loses sight of important action items. The new capability to track the status

and outcome of action items and risks ensures a more thorough completion of action

items and mitigation of risk. This will result in higher quality project deliverables.

Lastly, the due dates and updates at weekly status meetings will increase accountability

of team members and help to keep the project on schedule.

5.1.4. Weekly Status Meetings

The other key part of our process is that the delivery manager holds weekly status

meetings. At these meetings the team should have a discussion, led by the delivery

manager, to bring the team up to date on the status of project milestones, to determine a

strategy for how to reach the next project milestones, and to identify next steps.

Additionally, we recommend that the team discuss any open action items with pressing

due dates or that are pending closure. During this time, each team member should give a

brief report regarding the status of the items they own. If a team member believes an

action item or risk status should be changed to closed, the team should come to a

consensus that sufficient work has been done to close the item. After these meetings the

project manager should update the Action Item Tracking Sheet, which should be housed

in PKS for all team members to review frequently. The project manager should also

update the project plan, if applicable, after the weekly status meetings. He/she should

then print and post a new version of the project schedule on the wall.


71

These weekly status meetings will ensure that all team members are aware of the

current status of the project, even regarding areas where they are not directly involved.

We believe this will boost communication and the flow of ideas between team members

and also bring any problems or roadblocks to the forefront sooner. We also feel that the

weekly status meetings will increase accountability and adherence to deadlines because

team members will be required to report on their progress each week.

5.2. IT Equipment Upgrades for SLX

Based on the capacity planning research and analysis, we have developed a list of

recommendations to aid JPMorgan with the successful implementation of the DML to

SLX feedback loop into production, such that server performance is not compromised.

Our recommendations are as follows:

• JPMorgan should use the capacity planning document, found in Appendix I, to

estimate server loads and to upgrade hardware on the servers in order to meet

future demand.

• A member of the infrastructure team should be contacted and brought on board

for the rest of this project. This individual should be responsible for maintaining

and updating the capacity planning document.

• The infrastructure team should perform further analysis if necessary on quality

assurance, disaster recovery, and development environments to ensure that all

SLX environments have adequate resources.

• Storage space on two volumes should be increased to meet future demands from

the DML feedback loop. These storage volumes are:

o /dev/slxp1-slxhom1lv

o jipnas1:/vol/jipnas1_app0/tss_slx_prd1


72

• Add more processors in order to meet the two minute service level agreement for

intraday message processing time. Without these upgrades the service level

agreement will not be met between 11:00AM and 1:00PM and 3:30PM and

6:00PM.


73

Works Cited

[Annual Report 2005] (2006). Annual Report 2005. Retrieved November 30, 2006 from http://files.shareholder.com/downloads/ONE/62656512x0x34666/D25C186E-7D76-4BA3-AC27-445C45622401/ar05_complete.pdf

[Fortune] (2006). Fortune500 2006 Our Annual Ranking of America’s Largest Corporations. Retrieved October 23, 2006. from http://money.cnn.com/magazines/fortune/fortune500/full_list/

Ghelani, Piyush (2006). WSS Unix webdoc 1.1 for JPMChase. Retrieved November 30 2006 from http://dcfdoc.gis.chase.com

[GTI Hobbit] (2006). Hobbit Monitor 4.1.2: WSS Unix Health Status. Retrieved November 30, 2006 from http://wssunixmonitor.gis.chase.com/

[History] (2006). The History of Our Firm. Retrieved November 30, 2006 from http://www.jpmorganchase.com/cm/cs?pagename=Chase/Href&urlname=jpmc/about/history

[Home] (2006). JPMorgan Home Page. Retrieved November 20, 2006 from http://www.jpmorgan.com

Internet Center for Management and Business Administration, Inc. (2006). CPM- Critical Path Method. Retreived November 16, 2006 from http://www.netmba.com/operations/project/cpm/.

[Java Trademark] Java and the Java logo are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. or other countries.

JPMorgan Chase & Co. (2006). Securities Lending Overview. New York, NY: Amar Devasthali.

JPMorgan Chase & Co. (2006). Comprehensive Business Requirements Document: DML to SLX Integration. New York, NY: Brian Kenney.

Morris, K.M. & Siegel, A.M. (1993). The Wall Street Journal Guide to Understanding Money & Investing. New York: Lightbulb Press, Inc.

[Project Delivery Framwork] (2006). WSS Project Delivery Framework (PDF). Retrieved November 29, 2006, from http://tss-cb-cto-tech.jpmorganchase.com/dept/WSS/permits.asp


74

[WSS] (2006). About Worldwide Securities Services. Retrieved October 23, 2006, from http://www.jpmorgan.com/cm/ContentServer?c=TS_Content&pagename=jpmorgan%2Fts%2FTS_Content%2FGeneral&cid=1114735359705.


75

Appendix A: Securities Lending Technology Systems Relationships

Diagram courtesy of JPMorgan (Devasthali, 2006)

Mar

ket

Inte

rfac

es

Cle

arin

g &

Settle

ment

CO

SM

IC•

Int’l

Cus

tody

Infr

ast

ruct

ure

Dom

estic

Wo

rld

Len

d•

Non-

cust

ody

[Sun

Gar

d]

Inte

rnat

iona

lR

epo

rtin

g

WSS and Other Corp

User Interfaces Core Processing

SL

X•

Tra

de I

nitia

tion

EQ

UIL

EN

DD

alla

s M

essa

ge

Hu

b(W

MQ

I)

DM

L•

Inte

rnat

ion

al

Glo

bal

On

e•

UK

Dom

est

ic

[SunG

ard

]

Vie

ws

Po

rtfo

lio

Rep

ort

ing

TIT

AN

•U

.S.

Cus

tody

SL

X

Mai

nfr

ame

•M

essa

ge

Rou

ting

& T

ran

sfo

rma

tion

Inte

rnet

Bro

wse

r

3rd

Par

ty

Cu

sto

dia

ns

Cre

stU

DT

’s

UD

T’s

Rec

onci

liatio

ns,

Acc

rual

s, a

nd o

the

r bu

sine

ss f

unc

tions

SE

I•

B1

U.S

. C

usto

dy

Ass

et

Man

age

men

t

SL

AM

/ C

TE

•C

ash

Colla

tera

l M

gm

t

Blo

ckad

e•

LC

Col

late

ral

Allo

c/R

epo

rtin

g

SL

MU

•D

ome

stic

Lo

anet

Pir

um

DT

CF

ED

Eu

roC

lear

FT

ID

Glo

bal

-V

iew

s

MIS

Tra

de

In

itia

tion

Loa

n M

ain

ten

ance

E-m

ail


76

Appendix B: High Level Task Document for Project Plan Creation

We would like to brainstorm if these are all of the major activities and begin to determine the predecessors of each. This will facilitate the E2 estimate process and subsequently a critical path analysis to examine ways to expedite the project. Also we would like to map out and agree upon the high level design for the completion of the project. Keep in mind: **Every high level task should add value to fulfilling either a required document for the

PDF process or one of the direct objectives for the project. ** The objectives are as follows: Objective 1 – SLX to have accurate international utilization at start of day Objective 2 – SLX to have accurate intraday international utilization Objective 3 – Enable traders to view a current picture of each open loan in SLX rather than having to toggle between SLX and DML. Objective 4 – The ability to define at will which value should be used to impact utilization: either the required collateral or the DML Calculated Value. Major Milestones by Phase Requirements Phase

Tasks Documentation SLX/DML High Level Pre-design Work

• DML Calculation of limits* • Verify that DML does not

calculate limits differently for different loan types

• DML Functions that affect utilization and date sensitivity around those functions*

Detailed Requirements Document

Data Mapping Exercise • Spreadsheet mapping from DML loan

master to the loan snapshot file layout • Analysis of borrower relationships in DML vs. SLX, valid values, valid codes, etc.

• Proof of Concept Prototype in SLX to

Initiate Requirements Traceability Matrix


77

validate data mapping (To include prototyping the End User Transactions Summary and Detailed Screens)

Initiate Test Strategy/Plan E2 Estimates Performance Capacity Planning

• Examine CPU and RAM usage (currently executing this task)

• Analyze Current Performance pertaining to SLMU feedback loop to identify inaccurate/inefficient processes and determine whether we can change existing processes due to poor performance that will ameliorate the use of the future SLX/DML Feedback Loop (If changes arise because of this task, there will be some significant Testing Effort required to test changes to the current process)

• Work with GTI to determine next steps for solving anticipated problems

• Track existing issues facing production and infrastructure teams on all SLX environments

Architecture Permit to Build

Business Case Design Phase

Tasks Documentation

Update Test Strategy

High Level Design document (author Brian with input from developers) [This doc will build upon the requirements document to

show the users a picture of our High Level Design Concept]

**Implementation of a common I/O

(input/output) Module in DML [Dependent upon previous tasks listed in asterisks: DML calculation of limits and DML

functions that affect…]

Detailed Design/System Specifications Doc

Messaging Task Finalize Release Schedule End User Transactions Architecture Step 3

Data Transactions Perimeter Infrastructure Risk Assessment Form

Design in SLX Update Requirements Traceability Matrix Design in DML E3 Estimate


78

Design in SLX MF Detailed Design Walkthrough

Design SOD File Build/Unit Test

Tasks Documentation Source Code Build/Unit Test DML Update Requirements Traceability Matrix Source Code Build/Unit Test SLX Revise Requirements

Source Code Build/Unit Test SLX MF Update End User Documentation Peer Code Review Initiate Application/System Documentation

SIT Tests Technology Recovery Action Plan (TRAP) Deployment to QA Update SQA Checklist

Update Tollgate Checklist Testing

Documentation Update End User Documentation

Finalize Requirements Traceability Matrix Revise Requirements

Initiate Production Readiness Review Update SQA Checklist

Update Tollgate Checklist

QA Testing Tasks SIT Volume Performance Test

Test Setup (getting access to programs, etc.) Write Test Scripts Test Script Review

Execute Error Handling Testing Request GTI Resources

Execute Performance Testing Assisted by GTI Resources Execute Functional Requirements Testing

Analyze Test Results Regression Testing

UAT Testing Test Setup Plan UAT

Execute UAT for DML


79

Execute UAT for SLX Execute UAT for SLX MF

Analyze Test Results Training We need to brainstorm ideas for a training strategy. Implementation

Tasks Documentation

Code Deployment DML Finalize Operations Turnover Documentation

Code Deployment SLX Update Tollgate Checklist Code Deployment SLX MF Finalize Ender User Documentation

Implement Changes to Batch Schedule Finalize Application/System Documentation

Finalize ECM Calendar Entry Finalize SLA Finalize Implementation Plan

• Create Implementation Tasks • Create ECMS Record (Allow 10-15

days for Review as part of Risk for the project)

Finalize Production Readiness Review

Update SQA Checklist Project Closure

Tasks Documentation Post Production Support (2 week warranty

period) Project Closure Document

Post Implementation Survey Customer Satisfaction Survey Finalize SQA Checklist Finalize Tollgate Checklist


80

Appendix C: Snapshots of the STMate Tool


81


82

Appendix D: Screenshots of the same loan in DML and SLX

Omni view of a loan in DML

Component view of a loan in DML


83

Trade level view of a loan in SLX Production

Position level view of a loan in SLX Production


84

Appendix E: DML Loan Master Parser Source Code

/** * DML Loan Master Parser * * Written by: Michael Kristan ([email protected]) * Securities Lending Technology * JPMorgan Worldwide Securities Services * * Copyright 2006 JPMorgan Chase & Co. All Rights Reserved. * Internal use only. Not for distribution or use outside * of the firm without permission from the author or the firm. * * * About this program: * This program parses a DML Loan Master file (in .csv format) * and splits it up into smaller parts to allow for easier * processing by the Microsoft Office suite of applications. * * Command line arguments (in order): * Maximum number of records that can appear in an output file * Input .csv file name * Output .csv file name * * Example usage: * java.exe DMLLoanMasterParser 10000 C:\input.csv C:\output.csv * * In the example above, the file input.csv is read and split up * into groups of 10,000 records. Each group is output into a csv file * containing the name C:\output * * Most java virtual machines will require the full package name * of the class when being executed. In that event, use * com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser after java.exe * * Notes: * The output file is modified by the software by appending * a number to the file name. Example, the first 10000 records * are stored in output1.csv, the second 10000 records are in * output2.csv. * * There is an interesting situation where some of the DML * loan records have extra commas in them which causes errors when * trying to import into Microsoft Access. Records that do not match * the expected number of columns (defined as 125 in constant * numberOfCSVColumns) are instead written into a file with an ERR * suffix (in this example it would be outputERR.csv). * * It has been found that using MS-DOS 8.3 file naming in the * arguments helps when trying to load files that have long file names * or spaces. Example, instead of passing H:\SLX Integration\Loan Master.csv * use H:\SLXINT~1\LOANMA~1.CSV * * This program has been tested on the Sun Java 5.0 SDK & JRE * platforms. The runtime environment for this version of java can * be found at http://java.sun.com This software makes use of a * non-standard java package called csvreader. The class files for * this package can be freely obtained at http://www.csvreader.com at * the time of this writing. * * To compile this file, you need the Java SDK. The command to * compile is javac.exe DMLLoanMasterParser.java. To run this program, * you only need DMLLoanMasterParser.class and do not need the SDK. * */ package com.jpmorgan.wss.seclend.dmlslx;


85

// External classes that DMLLoanMasterParser depends on import com.csvreader.CsvReader; // Used to read CSV files import com.csvreader.CsvWriter; // Used to create CSV files /** * DMLLoanMasterParser * * This program only uses one class, DMLLoanMasterParser. All * functions in this program are static and are accessed via the * void main function. * * @author Michael Kristan */ public class DMLLoanMasterParser { /** * Main function * * This function does the parsing of the CSV file as described above * in the program description. There is no error checking to ensure * that the first parameter is an integer or that the paths in the other * parameters are valid. * * @param args A string array that contains the expected arguments as described above * @throws Exception in the event that the wrong number of command arguments are passed */ public static void main(String[] args) throws Exception { // Validate that we have enough command line arguments if(args.length != 3) throw new Exception("Not enough command line arguments"); // Extract argument parameters into integers and character strings final int maxRecordsPerFile = Integer.parseInt(args[0]); String paramInputFileName = args[1].trim(); String paramOutputFileName = args[2].trim(); // Output general information to the console screen System.out.println("DML Loan Master Parser\nJPMorgan Securities Lending\n"); System.out.println("Number of records per file : " + maxRecordsPerFile); System.out.println("Input file name : " + paramInputFileName); System.out.println("Writing contents of input file to .CSV files, please be patient..."); // Declare the CSV Reader and the CSV Writer for errors CsvReader myReader = new CsvReader(paramInputFileName); CsvWriter errWriter = new CsvWriter(setErrorFileName(paramOutputFileName)); // Setup parameters for size of documents and output file name int fileNumberCounter = 0; // Used for output file names String outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); int errNumberCounter = 0; // Used for looking at statistics // Setup parameters for the definition of the CSV file final int numberOfCSVColumns = 125; // Boolean based on the myReader.readRecord() output boolean recordsLeftToRead = true; // Create a loop that will run until the end of the input CSV file is reached while(recordsLeftToRead == true) { // Declare new output file stream and print name to console screen CsvWriter myWriter = new CsvWriter(outputFileName);


86

System.out.println("Output file name : " + outputFileName); // Loop populate an output CSV file until the maxRecordsPerFile // upper bound is reached. Then break out and have the parent // loop create a new CSV file for(int index = 0; index < maxRecordsPerFile; index++) { // Read a record and implement early exit recordsLeftToRead = myReader.readRecord(); if(recordsLeftToRead == false) break; // If we run into a situation where the the number of columns are not // equal to the defined constant (numberOfCSVColumns), output that row // to a different file so that the regular CSV file doesn't break when // importing into Microsoft Access if(myReader.getColumnCount() != numberOfCSVColumns) { for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { errWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index + ", COL_NUM_ERROR: Columns = " + myReader.getColumnCount() + " Skipping..."); errWriter.endRecord(); // Adds a CR and LF to the text file // Because we're not sending our output in this if statement to myWriter, // there is no need to update the counter that measure the numbers of // records being written to that file. Therefore as a hack, we need to decrement // the index counter by one because it will get incremented automatically by the // for loop. index--; // Increment the error number counter. errNumberCounter++; } else { // Iterate through the number of columns and output line for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { myWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index); myWriter.endRecord(); // Adds a CR and LF to the text file } } // Close the output file stream myWriter.flush(); myWriter.close(); // Increment the file number counter and set the new output file name fileNumberCounter++; outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); } // Close the file reader and error writer myReader.close(); errWriter.flush(); errWriter.close(); System.out.println("\nFile creation complete!\n\tNumber of errors : " + errNumberCounter); }


87

/** * setOutputFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @param fileNumberCounter The integer that needs to be concatentated * @return A file name with the fileNumberCounter appended to the file prior to the .csv suffix */ public static String setOutputFileName(String fileName, int fileNumberCounter) { // Parse the string so that we only have the portion before the .csv suffix // then add the filenumber and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); //System.out.println(fileName); return fileName + fileNumberCounter + ".csv"; } /** * setErrorFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @return A file name with the characters 'ERR' (without quotes) appended to the file * prior to the .csv suffix */ public static String setErrorFileName(String fileName) { // Parse the string fileName so that we only have the portion before the .csv // suffix. Then add the letters ERR and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); return fileName + "ERR.csv"; } }


88

Appendix F: DML Loan Master Parser JUnit Test Case

package com.jpmorgan.wss.seclend.dmlslx; import junit.framework.TestCase; /** * Test for DMLLoanMasterParser helper methods * * @author Michael Kristan */ public class TestDMLLoanMasterParser extends TestCase { /** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setOutputFileName(java.lang.String, int)}. */ public void testSetOutputFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\output1.csv"; String outputStringA2 = "H:\\output2.csv"; String outputStringA_99 = "H:\\output-99.csv"; assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99); inputStringA = "H:\\output"; assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99); } /** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setErrorFileName(java.lang.String)}. */ public void testSetErrorFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\outputERR.csv"; assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1); inputStringA = "H:\\output"; assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1); } }


89

Appendix G: Microsoft Access SQL Queries

JOIN DML/SLX BROKERS SELECT DISTINCTROW [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name], [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT-NO], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE], [_internal_DMLBrokers Query].[_ACCT-NO] AS DMLID FROM [_internal_DMLBrokers Query] INNER JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[_ACCT-NO]=[_internal_Remove Quotes from SLX Borrowers].DMLID;

DML BROKERS WITHOUT MATCHING SLX RECORDS SELECT [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT-NO], [_internal_DMLBrokers Query].[_ACCT-NO], [_internal_DMLBrokers Query].[SHORT-NAME], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE] FROM [_internal_DMLBrokers Query] LEFT JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[ACCT-NO] = [_internal_Remove Quotes from SLX Borrowers].DMLID WHERE ((([_internal_Remove Quotes from SLX Borrowers].DMLID) Is Null));

SLX BROKERS WITHOUT MATCHING DML RECORDS SELECT [_internal_Remove Quotes from SLX Borrowers].DMLID, [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name] FROM [_internal_Remove Quotes from SLX Borrowers] LEFT JOIN [_internal_DMLBrokers Query] ON [_internal_Remove Quotes from SLX Borrowers].DMLID = [_internal_DMLBrokers Query].[ACCT-NO] WHERE ((([_internal_DMLBrokers Query].[ACCT-NO]) Is Null));

UNIQUE LOAN-TYPE VALUES SELECT DISTINCT [Loan Master].[LOAN-TYPE] FROM [Loan Master];

UNIQUE SEC-LOC VALUES SELECT DISTINCT [Loan Master].[SEC-LOC] FROM [Loan Master];

UNIQUE SEC-TYPE VALUES SELECT DISTINCT [Loan Master].[SEC-TYPE] FROM [Loan Master];


90

Appendix H: Invitation to Final Presentation

You are cordially invited to attend a presentation hosted by Megan Slonski and Michael Kristan of Worcester Polytechnic Institute where they will discuss their contributions to the SLX-DML Integration initiative within WSS Securities Lending Technology. Date: December 14th, 2006 Time: 3:30PM – 4:30PM Location: 28th Floor Conference Center, 1 Chase Manhattan Plaza, New York, NY Dial-In: 1.866.870.8212 Participant Passcode: 66370645 Details As part of our college graduation requirements we must complete a project that ties together our four years of study. This project is referred to as a Major Qualifying Project (MQP) at the university. For our MQP, we spent time at JPMorgan Chase assisting with the SLX-DML Integration project, which was initiated by the Securities Lending Technology team within JPMorgan Worldwide Securities Services. Our contributions include a critical path analysis, recommendations for improvements to project management processes, data field mapping, and server capacity planning. This project was advised by Professors Arthur Gerstenfeld and Michael Ciaraldi of WPI. Individuals who can not physically attend are welcome to connect via the phone number listed above. Time will be reserved for questions and answers. About Worcester Polytechnic Institute (WPI) Founded in 1865 in Worcester, Mass., WPI was one of the nation’s first engineering and technology universities. WPI's 18 academic departments offer more than 50 undergraduate and graduate degree programs in science, engineering, technology, management, the social sciences, and the humanities and arts. WPI's world-class faculty work with students in a number of cutting-edge research areas, leading to breakthroughs and innovations in such fields as biotechnology, fuel cells, information security, materials processing, and nanotechnology. Students also have the opportunity to make a difference in communities and organizations around the world through the university's innovative Global Perspective Program. There are more than 20 WPI project centers throughout North America and Central America, Africa, Australia, Asia, and Europe. High level project description In 2004, a decision was made to adopt SLX as a strategic front end single trading platform, replacing the trading activity performed on the three lending platforms (SLMU, DML, Global One), leaving them in place only for loan maintenance activities. During 2005, SLX development primarily focused on enabling a viable front end for US domestic trading with the delivery of an SLMU to SLX feedback loop. The "feedback loop" design approach adopted for the US Front-End will be leveraged now for the creation of a DML to SLX loop as the first in a series of efforts that will be undertaken to enable SLX as a viable International trading front end.


91

Appendix I: Review of SLX Server Performance

The document in Appendix I provides JPMorgan Chase with a thorough analysis

of all server systems related to the DML to SLX Feedback Loop. This document is

currently in use by members of the project team as well as by the members of JPMorgan

Global Technology & Infrastructure (GTI) group.

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance

Copyright © 2006 J. P. Morgan Chase Bank, N.A. All rights reserved.

This document contains proprietary and confidential information of JPMorgan. Reproduction, disclosure, or use of any portion of this document without specific written authorization from JPMorgan is strictly prohibited. This restriction applies to the information on every page of the document. Contents may be disclosed only to authorized JPMorgan employees and consultants for the purpose of performing their job responsibilities.

Worldwide Securities Services

Securities Lending DML to SLX Integration

Review of SLX Server Performance

Version 0.5

Project Code: 71707


Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan

© 2006 J. P. Morgan Chase Bank, N.A.

All Rights Reserved. Confidential And Proprietary

For Internal Use Only - Do Not Duplicate

Owner: Securities Lending Version: 0.5 Page: 2 of 31

Table of Contents Introduction ................................................................................................................................................. 4

Purpose of Document............................................................................................................................. 4 Definitions ............................................................................................................................................... 4

Glossary of Terms & Acronyms ....................................................................................................4 SLX Environments ........................................................................................................................6

Hardware Specifications ......................................................................................................................... 6 Production server specifications ...................................................................................................6 Development server specifications ...............................................................................................7 Disaster Recovery server specifications.......................................................................................7 Quality Assurance server specifications.......................................................................................7 Cross reference of SLX hard disk volumes to UNIX mounted file systems..................................8

Analysis...................................................................................................................................................... 11 Current Database Usage...................................................................................................................... 11

CPU load on production servers.................................................................................................11 Possible zombie processes ........................................................................................................12 Memory allocation on production servers...................................................................................12 Disk space on production servers...............................................................................................14

SLX New Loan Volumes....................................................................................................................... 15 SLMU Volumes..................................................................................................................................... 17

Number of SLMU messages.......................................................................................................17 Processing times for messages..................................................................................................19 Table sizes..................................................................................................................................19 Anticipated volume increases .....................................................................................................20

DML Volumes ....................................................................................................................................... 20 Current DML intra-day load ........................................................................................................20 Table Sizes .................................................................................................................................22 Anticipated volume increases .....................................................................................................22 Requirements around timing of DML batch processes...............................................................23

Recommendations and Next Steps ......................................................................................................... 23 Recommendations................................................................................................................................ 23 Next Steps ............................................................................................................................................ 24

Reference Documents .............................................................................................................................. 25 Document Control..................................................................................................................................... 25

Document History ................................................................................................................................. 25







Document Storage................................................................................................................................ 25 Appendices................................................................................................................................................ 26

PowerPoint Slides For Capacity Planning............................................................................................ 26







INTRODUCTION PURPOSE OF DOCUMENT The purpose of this document is to give the reader a comprehensive understanding of the resources required to run SLX in a production environment and to analyze the impact that the DML feedback loop will have on the systems. Information has been gathered from many sources and compiled together to form a complete picture of the resources that SLX depends on. The goal is to provide enough information in order to make infrastructure decisions moving forward.

DEFINITIONS

Glossary of Terms & Acronyms

Term Description APD06 APD06 is short for the server name TSJIPUAPD06. APD06 is the development server for

all SLX environments (including production, DR, & QA).

APP06 APP06 is short for the server name TSJIPUAPP06. APP06 is the main SLX processing server for SLXP.

APP10 APP10 is short for the server name TSJIPUAPP10. APP10 is the user SLX server for SLXP. JPMorgan traders interface through this server.

APR05 APR05 is short for the server name TSMCCUAPR05. APR05 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.

APR06 APR06 is short for the server name TSMCCUAPR06. APR06 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.

APQ05 APQ05 is short for the server name TSMCCUAPQ05. APQ05 is a quality assurance SLX server.

APQ06 APQ06 is short for the server name TSMCCUAPQ06. APQ06 is a quality assurance SLX server.

Autoborrow Autoborrow is an intelligent, fully automated way for borrowers and lenders to transact US and non-US equity and fixed income securities. Customizable schedules and rules automate the process between the borrower and lender, making the trade execution simple. Autoborrow is part of the Equilend trading system.

Cache Cache is a collection of data duplicating original values stored elsewhere or computed earlier, where the original data is expensive (usually in terms of access time) to fetch or compute relative to reading the cache. Once the data is stored in the cache, future use can be made by accessing the cached copy rather than refetching or recomputing the original data, so that the average access time is lower.

CPU The CPU (also known as the Central Processing Unit or the processor) is the main processor on the server. The current CPU load measures how busy the server is.

DBP01 DBP01 is short for the server name TSJIPUDBP01. DBP01 is the production server that hosts the SLX Oracle Database.







Term Description DML DML (also known as SLS) is the Securities Lending platform currently used to support

trading and loan maintenance activities for international assets.

DR DR stands for disaster recovery.

Equilend Formed in 2001 by its ten original investors, EquiLend delivers standardized, centralized and global securities lending solutions to the equity and fixed income markets, while providing operational efficiencies across organizations. The EquiLend platform is designed to process equity and fixed income securities lending transactions on a global basis.

GB GB stands for gigabyte. It is a measure of size on computers. A gigabyte is equal to 1,024 MB (megabytes). Gigabytes are frequently used when measuring sizes of hard disks and sometimes used to measure memory.

GHz GHz stands for gigahertz. It is a measure of frequency and is used with computer when discussing speed. The higher the frequency, the faster the processing speed of a computer. 1 GHz is equal to 10-9 seconds.

MB MB stands for megabyte. It is a measure of size on computers. A megabyte is equal to 1,024 KB (kilobytes). Megabytes are frequently used to measure size of memory.

NAS Network-attached storage (NAS) is hard disk storage that is set up with its own network address rather than being attached to the department computer that is serving applications to a network's workstation users. By removing storage access and its management from the department server, both application programming and files can be served faster because they are not competing for the same processor resources. The network-attached storage device is attached to a local area network (typically, an Ethernet network) and assigned an IP address. File requests are mapped by the main server to the NAS file server.

Network-attached storage consists of hard disk storage, including multi-disk RAID systems, and software for configuring and mapping file locations to the network-attached device. Network-attached storage can be a step toward and included as part of a more sophisticated storage system known as a storage area network (SAN).

SAN In computing, a storage area network (SAN) is a network designed to attach computer storage devices such as disk array controllers and tape libraries to servers. As of 2005, SANs are common in enterprise storage.

SLA SLA stands for Service Level Agreement. An SLA is a contract or agreement that guarantees a certain level of performance or turnaround time usually between two applications or systems.

SLMU SLMU (also known as Securities Lending MenU) was the Securities Lending platform previously used by US domestic traders. SLMU is only used for loan maintenance now and has been integrated into SLX by a feedback loop.

SLX SLX (also known as Securities Lending eXpress) is a front-end trading platform that is strategically envisioned to replace all trading activity currently performed on any of the lending platforms with those platforms then being retained for loan maintenance type activities only.

SLXP SLXP stands for the production environment of SLX.

Swap/Page file A swap file (or swap space or, in Windows NT, a pagefile) is a space on a hard disk used as the virtual memory extension of a computer's real memory (RAM). Having a swap file







Term Description allows your computer's operating system to pretend that you have more RAM than you actually do. The least recently used files in RAM can be "swapped out" to your hard disk until they are needed later so that new files can be "swapped in" to RAM. In larger operating systems (such as IBM's OS/390), the units that are moved are called pages and the swapping is called paging.

One advantage of a swap file is that it can be organized as a single contiguous space so that fewer I/O operations are required to read or write a complete file. In general, Windows and Unix-based operating systems provide a default swap file of a certain size that the user or a system administrator can usually change.

TB TB stands for terabyte. It is a measure of size on computers. A terabyte is equal to 1,024 GB (gigabytes). Terabytes are frequently used when measuring sizes of large hard disk arrays that store log files, databases, or other data-intensive files.

QA QA stands for quality assurance. Development efforts on SLX are first tested on the QA servers and environments before being phased into production.

SLX Environments SLX Type Environment Names Related Server(s) Production SLXP TSJIPUAPP06, TSJIPUAPP10, TSJIPUDBP01

Development SLXG, SLXD, SLXI, SLXT TSJIPUAPD06

Disaster Recovery (DR) SLXP TSMCCUAPR05, TSMCCUAPR06

Quality Assurance (QA) SLXQ, SLXR TSMCCUAPQ05, TSMCCUAPQ06

HARDWARE SPECIFICATIONS Production server specifications APP06 APP10 DBP01

Total Number of Processors 4 2 8

Processors assigned to SLX 4 2 3.5

Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz

Processor Type 64 bit 64 bit 64 bit

L2 Cache Size 1.5 MB 1.5 MB 1.5 MB

Physical Memory 7.75 GB 7.75 GB 62 GB

Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB

/dev/paging00 : 6GB

SLX Disk Drives 9 11 6

Network Interfaces en8 : 10.75.107.54 en8 : 10.75.107.62 en25 : 10.75.107.22

% Allocation to SLX 100% 100% 14.29%







APP06 APP10 DBP01

Monthly Operating Cost $1,218.00 $1,218.00 $2,493.52

Development server specifications APD06

Total Number of Processors 2

Processor Speed 563.3 MHz

Processor Type 64 bit

L2 Cache Size 1.5 MB

Physical Memory 7.75 GB

Known Paging Files /dev/hd6 : 6GB

SLX Disk Drives 10

Network Interfaces en2 : 10.75.85.30

% Allocation to SLX 100%

Monthly Operating Cost $1,447.77

Disaster Recovery server specifications APR05 APR06

Total Number of Processors 2 1

Processor Speed 1.7 GHz 1.7 GHz

Processor Type 64 bit 64 bit

L2 Cache Size 1.5 MB 1.5 MB

Physical Memory 7.5 GB 7.5 GB

Known Paging Files /dev/hd6 : 4GB /dev/hd6 : 4GB

SLX Disk Drives 11 9

Network Interfaces en11 : 10.31.77.67 en11 : 10.31.77.75

en12 : 10.31.87.98

% Allocation to SLX 100% 100%

Monthly Operating Cost $1,218.00 $1,218.00

Quality Assurance server specifications APQ05 APQ06







APQ05 APQ06 Total Number of Processors 2 3

Processor Speed 1.7 GHz 1.7 GHz

Processor Type 64 bit 64 bit

L2 Cache Size 1.5 MB 1.5 MB

Physical Memory 7.5 GB 7.5 GB

Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB

SLX Disk Drives 11 10

Network Interfaces en3 : 10.31.77.65

en4 : 10.31.87.73

en3 : 10.31.77.73

en4 : 10.31.87.93

% Allocation to SLX 100% 100%

Monthly Operating Cost $1,218.00 $1,218.00

Cross reference of SLX hard disk volumes to UNIX mounted file systems Device path Mounted File System QuotaAPP06 /dev/slxp1-sftwrelv /software 640 MB

/dev/slxp1-local1lv /local 192 MB

/dev/slxp1-slxhomlv /local/app/a_slx 8.06 GB

/dev/slxp1-mqmhomlv /local/app/mqm 64 MB

/dev/slxp1-mqdatalv /local/data/mqm 64 MB

/dev/slxp1-mqmloglv /local/data/mqm/log 320 MB

/dev/slxp1-mqmerrlv /local/data/mqm/errors 64 MB

/dev/slxp1-gtamsglv /local/app/gtamsg 64 MB

/dev/slxp1-oraclelv /local/SLX/oracle 2.97 GB

APP10 /dev/slxp2-local1lv /local 192 MB

/dev/slxp2-slxhomlv /local/app/a_slx 8.06 GB



/dev/slxp2-mqmloglv /local/data/mqm/log 320 MB


/dev/slxp2-gramsglv /local/app/gtamsg 64 MB

/dev/slxp2-oraclelv /local/SLX/oracle 2.97 GB

/dev/slxp2-homedirlv /local/home 320 MB







Device path Mounted File System Quota/dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB

/dev/slxp2-slxloglv /local/app/a_slx/logs 64 MB

DBP01 /dev/dbp01-slxora1 /local/SLX/oracle 8 GB

/dev/dbp01-slxolog /local/SLX/logs/oracle 2.92 GB

/dev/dbp01-slxoaud /local/SLX/oraaudit01 1 GB

/dev/dbp01-slxwora /local/SLX/work/oradba 1 GB

/dev/dbp01-SLXobck /local/SLX/orabackup01 54.5 GB

/dev/dbp01-SLXoftp /local/SLX/oraftp 32 GB

APR05 /dev/slxp1-sftwrelv /software 2.5 GB

/dev/slxp1-local1lv /local 1.5 GB

/dev/slxp1-slxhom1lv /local/app/a_slx 8 GB





/dev/slxp1-oraclelv /local/SLX/oracle 16 GB

/dev/slxp1-mqmloglv /local/data/mqm/log 2.5 GB

/dev/slxp1-ibmhttlv /local/IBMHTTPD.8 512 MB

/dev/slxp1-slxloglv /local/app/a_slx/logs 512 MB

APR06 /dev/slxp2-local1lv /local 1.5 GB

/dev/slxp2-slxhomlv /local/app/a_slx 8 GB





/dev/slxp2-oraclelv /local/SLX/oracle 16 GB

/dev/slxp2-mqmloglv /local/data/mqm/log 2.5 GB

/dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB

APQ05 /dev/slxq1-locallv1 /local 128 MB

/dev/slxq1-slxhomlv /local/app/a_slx 12.4 GB

/dev/slxq1-slxftplv /local/app/a_slxftp 4 GB

/dev/slxq1-usrhomlv /local/home 320 MB

/dev/slxq1-oraclelv /local/SLX/oracle 2 GB







Device path Mounted File System Quota/dev/slxq1-mqdatalv /local/data/mqm 64 MB

/dev/slxq1-mqmerrlv /local/data/mqm/errors 64 MB

/dev/slxq1-mqmhomlv /local/app/mqm 64 MB

/dev/slxq1-gtamsglv /local/app/gtamsg 64 MB

/dev/slxq1-sftwrelv /software 640 MB

/dev/slxq1-ibmhttlv /local/IBMHTTPD.8 512 MB

APQ06 /dev/slxq2-slxhomlv /local 160 MB

/dev/slxq2-usrhomlv /local/app/a_slx 12.4 GB

/dev/slxq2-usrhomlv /local/home 320 MB

/dev/slxq2-oraclelv /local/SLX/oracle 2 GB

/dev/slxq2-mqdatalv /local/data/mqm 64 MB

/dev/slxq2-mqmerrlv /local/data/mqm/errors 64 MB

/dev/slxq2-mqmhomlv /local/app/mqm 64 MB

/dev/slxq2-gtamsglv /local/app/gtamsg 64 MB

/dev/slxq2-sftwrelv /software 640 MB

/dev/slxq2-ibmhttlv /local/IBMHTTPD.8 512 MB

APD06 /dev/slxd1-slxora /local/slx/oracle 2 GB

/dev/slxd1-gtamsg /local/app/gtamsg 32 MB

/dev/slxd1-mqdatalv /local/data/mqm 256 MB

/dev/slxd1-mqlogslv /local/data/mqm/logs 256 MB

/dev/slxd1-mqhomelv /local/app/mqm 256 MB

/dev/slxd1-slxhomlv /local/app/a_slx 27.7 GB

/dev/slxd1-slxoralv /local/SLX/oracle 3 GB

/dev/slxd1-slxftplv /local/app/a_slxftp 4 GB

/dev/slxd1-appesmlv /local/app/esm 256 MB

Shared Network Attached Storage (NAS) jipnas1:/vol/jipnas1_app0/tss_slx_prd1 /app/a_slxftp 4 GB

jipnas1:/vol/jipnas1_app0/tss_slx_prd2 /app/a_slx/slxp/SLXP/export 4 GB

jipnas1:/vol/jipnas1_app0/tss_slx_prd3 /app/a_slx/slxp/SLXP/out 4 GB

mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX /app/a_slxftp 4 GB

mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX2 /app/a_slx/slxq/SLXQ/export 2 GB

mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX3 /app/a_slx/slxq/SLXQ/out 2 GB

mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX4 /app/a_slx/slxr/SLXR/export 2 GB

mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX5 /app/a_slx/slxr/SLXR/out 2 GB







ANALYSIS CURRENT DATABASE USAGE

In order to do a successful capacity study on SLX, attention needs to be paid to the current state of server resources. In the following sections, an analysis was made on CPU, memory, and disk space on SLX production servers. DR, QA, and development were not included in this study because performance isn’t a priority on those servers. They also share resources with other systems that affect performance on SLX.

CPU load on production servers In Figure 1, the production server’s CPU usage was plotted on a 24 hour horizontal axis. The

values for each point were calculated by observing all 7 days of CPU values for the specified time interval and taking the median value. The median was used instead of the mean because weekend downtime would have skewed the plot for certain time intervals. A six term moving average was plotted on top of the data points to emphasize trends.

Median production server CPU usage (10/15/06 - 10/21/06)

0

0.2

0.4

0.6

0.8

1

0:001:00

2:003:00

4:005:00

6:007:00

8:009:00

10:0011:00

12:0013:00

14:0015:00

16:0017:00

18:0019:00

20:0021:00

22:0023:00

0:00


CPU

usa

ge (a

s a

ratio

of 1

00)







Figure 1 - Production server CPU usage

There are two areas of peaks. The first peak period starts around 9:00 PM and ends at 3:00 AM. The second peak period starts 3:00 PM and ends around 7:00 PM. Surprisingly, the server load is under 50% for most of the US trading day.







Possible zombie processes On APP06 there were 4 instances of the wait process that were allocating 16% of total CPU

usage (16% per process split over 4 processors). These processes were initiated on February 11th, 2006. Further investigation is needed to determine why these processes are consuming so many resources while the other wait processes are not. Table 1 shows the top processes on the server APP06.

[ps]USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMANDroot 12294 16.1 0 48 44 - A 11-Feb-06 261490:39 waitroot 20490 16.1 0 48 44 - A 11-Feb-06 260903:48 waitroot 16392 16 0 48 44 - A 11-Feb-06 260090:25 waitroot 8196 16 0 48 44 - A 11-Feb-06 259121:55 waita_slx 2150436 3.7 4 233760 233680 - A 12:49:15 170:55:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 2224178 3.1 2 123380 123328 - A 12:49:15 146:00:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1511664 3 5 296064 295984 - A 12:49:16 137:38:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1990868 2.4 4 257536 257456 - A 12:49:15 110:45:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 282740 2.3 3 163272 163192 - A 12:49:16 104:31:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 2093268 2.1 2 115688 115608 - A 12:49:15 99:22:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 532526 2 2 114376 114300 - A 12:49:16 94:39:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 815148 2 2 140768 140688 - A 12:49:16 90:49:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1810524 1.5 2 135460 135380 - A 12:49:16 68:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 528512 0.7 2 95060 94952 - A 12:49:15 31:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1228806 0 0 9128 9144 - A 12:49:14 1:33 /app/a_slx/slxp/SLXP_I/bin/tfsrvmgr SLXP_I_TSJIPUAPP06

Table 1 - Top processes on APP06

Memory allocation on production servers Figure 2, Figure 3, & Figure 4 show memory usage on each of the servers over a 48 day span.

The plot in blue represents physical memory usage. The plot in red represents swap space via hard disk paging files. One interesting observation is that APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The server then falls over to allocating and using swap space.

Figure 2 - Memory usage on APP06







Figure 3 - Memory usage on APP10

Figure 4 - Memory usage on DBP01

The infrastructure team has commented on this and remarked that there are no memory limitation or constrains on TSJIPUAPP06 server. The illusion of being out of memory comes from the fact that users do not count file cache usage feature of the Unix kernel. Actual memory usage is 4.125 GB. Starting after week 44, the system administrators also limited the size of the file cache so that it would resize itself to prevent the swap space from being used. A performance assessment was taken after week 44 to reassess the processing times. Data was collected during the week of November 12th in the same way that it was done in Figure 9. Figure 5 shows that processing times have not significantly changed after the adjustment was implemented.







October Performance vs November Performance

0

3000

6000

9000

12000

15000

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)

Projected DMLMessages

Oct. SLMUMessages

Nov. SLMUMessages


Oct. SLMUprocessing time

Nov. SLMUprocessing time

Figure 5 - Comparing October performance to November

Disk space on production servers Given the number of mounted file systems on each server, only graphs showing disk utilization greater than 80% for an SLX drive are shown. Despite the fact that these drives are mostly full, there is no evidence that short term growth will cause these volumes to become completely full.







Figure 6 - /local/app/a_slx on APP06 frequently uses more than 90% of quota

Figure 7 - /app/a_slxftp on the JIP NAS is near capacity

SLX NEW LOAN VOLUMES Table 2 shows intraday statistics on how many new loans are booked during a trading day. This data was compiled from loan statistics captured between January 2006 and June 2006. There are two different types of loans. The first are manual loans which are hand initiated by JPMorgan securities lending traders via the SLXP client. Autoborrow loans are automatically initiated loans that are processed through Equilend. A majority of loans are autoborrow loans which are handled automatically by SLX servers and do not require user intervention.

Based on the gathered data, the average manual loan is worth $8,305,877.38 and the value of an autoborrow loan $927,666.75. These numbers are strictly averages and are used for estimating financial loss due to SLX downtime.







Loans Booked - Average Hourly Breakdown From: 1-Jan-06 To: 4-Jun-06

Manual Loans Autoborrow loans Total

Time - ET # of Loans # of Loans # of Loans

Hourly Avg. 32 69 100 Daily Avg. 757 1,653 2,410

0:00 3 0 3 1:00 5 1 6 2:00 17 1 18 3:00 16 1 16 4:00 14 25 40 5:00 43 57 100 6:00 87 107 194 7:00 72 337 410 8:00 66 316 382 9:00 90 201 291

10:00 78 244 322 11:00 66 147 214 12:00 84 130 215 13:00 75 78 152 14:00 9 0 9 15:00 3 0 3 16:00 1 0 1 17:00 2 2 4 18:00 3 1 4 19:00 3 1 5 20:00 3 1 4 21:00 5 0 5 22:00 5 1 6 23:00 7 0 7

Table 2 - New loans booked in SLX

This data was then plotted against the recorded CPU usage. This plot can be found in Figure 8. An analysis of processor usage during the day shows that new loan bookings correlates very closely to CPU load. Servers APP06 and DBP01 closely follow new loan booking between the hours of 3:00AM and 2:00PM. Fortunately, because the servers have excess CPU capacity between 3:00AM and 3:00PM (nearly 60% of CPU time is idle time) there is plenty of room for growth without risking a performance hit. While it has not been proven that high processor activity on APP06 and DPB01 impact the SLX client response time that traders encounter (they communicate via the less stressed APP10), loan bookings may take slightly longer to process in the background.







New loans compared to CPU usage

0

0.2

0.4

0.6

0.8

1

0:001:00

2:003:00

4:005:00

6:007:00

8:009:00

10:0011:00

12:0013:00

14:0015:00

16:0017:00

18:0019:00

20:0021:00

22:0023:00

0:00


CPU

usa

ge (a

s a

ratio

of 1

00)

1

10

100

1,000

10,000

100,000

Num

ber o

f new

loan

s




New loan bookings




Figure 8 - New intraday loans in SLX

SLMU VOLUMES

Number of SLMU messages The total number of SLMU messages per day is 485,098*. That number is broken down into four categories:

• Message In Credit Snapshot (MICS)

• File In Credit Snapshot (FICS)

• Message In Loan Snapshot (MILS)

• File In Loan Snapshot (FILS)

*according to SLXP at 7:43PM on 11/13/2006







Credit Snapshot Loan Snapshot

Instance Type: File In 69,534 232,059

Instance Type: Message In 47,339 136,166

Table 3 - Breakdown of SLMU Messages by type

From the data in, further derivations can be made for SLMU messages as it passes through the buffer tables. These are expressed in Table 4.

Formula Value

SLXTradeSnapshotBuffer2 FICS x 2 139,068

SLXPositionSnapshotBuffer2 FILS – FICS 162,525

SLXPositionSnapshot FILS – FICS 162,525

SLXTradeSnapshot FICS 69,534

Table 4 - Calculations based on SLMU messages

In Figure 9 the CPU data from the production servers has been overlaid with SLMU messages and processing times over the course of the same sampled week. The main SLMU message peak occurs between 3:00 PM and 5:00 PM. There is a secondary peak of messages that occurs at 10:00 AM.







Average SLMU messages & processing time (from 10/15/06 - 10/21/06)

0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000



6 per. Mov. Avg.(% APP06 CPUUtilization)

6 per. Mov. Avg.(% DBP01 CPUUtilization)

Figure 9 - SLMU Messages in SLXP

Processing times for messages Processing of SLMU messages should take under 2 minutes (120 seconds) according to the service level agreement (SLA). According to Figure 9, the SLA is currently not being met between the hours of 3:00 PM and 2:00 AM. Processing time is directly related to server CPU load. During this period, both APP06 and DBP01 use more than 50% of the available processing capacity. Although the high processing time occurs outside of the US trading day, this is unacceptable from an international standpoint.

Table sizes SLX uses a series of tables to store and process SLMU intraday messages. Table 6 shows each of the tables the SLMU feedback loop uses to process along with their sizes. Most of this data is retained for one day and the tables are purged nightly. The SLMU tables are part of the main SLX table space which has a set quota. Table Q shows how much space is being used by SLX tables.

TABLESPACE TOTAL_MB FREE_MB USED_MB PCT_FREE SLX 24528 8267 16258 33 UA 10731 1804 8926 16

Table 5 - Table space statistics on SLX and UA







OWNER SEGMENT NAME SIZE IN MB SLX_DBA SLX_POSITION_SNAPSHOT 184 SLX_DBA SLX_PSTN_SNPSHT_BFFR2 184 SLX_DBA SLX_SNAPSHOT_BUFFER1 352 SLX_DBA SLX_TRADE_SNAPSHOT 72 SLX_DBA SLX_TRADE_SNAPSHOT_BUFFER2 96 SLX_DBA SLX_UPDATE_LIMIT_USED 30 SLX_DBA SLX_UPDT_BRRWR_CRDT_LN_USD 192

Table 6 - SLMU Feedback loop table sizes in SLX

SLX also gets a start of day file from SLMU that contains all snapshot information. This file is 148MB in size.

Anticipated volume increases Given that SLMU should only be used for loan maintenance activities, the volume increase should be very small.

DML VOLUMES

Current DML intra-day load Hourly observations were taken based on DML activity. This plot was added to the previous chart and shown below in Figure 10.








0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000




Figure 10 - SLMU and anticipated DML messages over a trading day

The alarming part of the DML message projection is the 11AM spike which occurs during the US trading day. A spike will potentially cause the processors to hit their 100% utilization ceiling and then have an impact on regular SLX activity (as highlighted in Figure 8). Using a loan data from SLX, if a 1 minute downtime occurs each hour between 10 and 2 due to the DML feedback loop spike, the potential loss for JPMorgan is $20million in new loans. This prediction is very conservative but does highlight the need for further investigation. A breakdown in loss can be found in Table 7.







From: To: 4-Jun-06

# of Loan

Total ValueUSD

# of Loan

Total Loan ValueUSD

Hrly Av. 32 $ 262,011,497.64 69 $ 63,876,620.12

Daily Av. 757 $ 6,288,275,943.25 1,653 $ 1,533,038,882.82 23:00 - 00:00 3 9,776,454.73$ 0 55,334.76$ 00:00 - 01:00 5 21,394,272.78$ 1 1,714,372.40$ 01:00 - 02:00 17 113,351,879.75$ 1 5,441,366.77$ 02:00 - 03:00 16 129,951,300.08$ 1 3,253,248.69$ 03:00 - 04:00 14 83,025,547.10$ 25 37,447,879.91$ 04:00 - 05:00 43 884,015,442.57$ 57 354,717,281.86$ 05:00 - 06:00 87 1,016,299,575.04$ 107 143,778,231.86$ 06:00 - 07:00 72 539,109,513.86$ 337 295,179,792.66$ 07:00 - 08:00 66 536,976,007.01$ 316 229,954,281.45$ 08:00 - 09:00 90 838,829,903.57$ 201 117,932,077.76$ 09:00 - 10:00 78 560,998,062.46$ 244 93,985,724.65$ 10:00 - 11:00 66 388,989,641.01$ 147 53,361,969.91$ 11:00 - 12:00 84 386,406,884.98$ 1 1 4,580,315$ 130 107,176,712.33$ 1 2 1,643,972$ 6,224,287$ 12:00 - 13:00 75 430,055,601.56$ 1 1 5,765,786$ 78 79,039,907.30$ 1 1 1,016,427$ 6,782,213$ 13:00 - 14:00 9 171,728,279.02$ 1 0 -$ 0 1,849,706.81$ 1 0 -$ -$ 14:00 - 15:00 3 75,867,084.33$ 0 2,719,934.33$ 15:00 - 16:00 1 5,162,395.33$ 0 69,085.51$ 16:00 - 17:00 2 4,235,037.57$ 2 1,526,454.28$ 17:00 - 18:00 3 6,633,586.62$ 1 605,583.31$ 18:00 - 19:00 3 10,135,257.59$ 1 926,060.65$ 19:00 - 20:00 3 13,884,355.94$ 1 1,421,584.56$ 20:00 - 21:00 5 14,946,578.59$ 0 234,267.84$ 21:00 - 22:00 5 23,914,216.79$ 1 596,853.58$ 22:00 - 23:00 7 22,589,064.98$ 0 51,169.65$

Total Loss : 13,006,500$

Total potential value of lost

loans

Time - ET

Loans Booked - Average Hourly BreakdownEquiLend Autoborrow LoansManual Loans

Outage Time -

Minutes

Potential # of Lost Loans

Potential Value of Lost Loans

USD

Outage Time -

Minutes

1-Jan-06

Potential # of Lost Loans

Potential Value of Lost Loans

USD

# of Requests Expired or

Not Processed

Table 7 - Financial loss due to SLX downtime

Table Sizes Because DML will use a feedback loop system similar to that of SLMU, the expected sizes of the tables should be equal to sizes in Table 5 & Table 6. Increasing the SLX table space for the new DML feedback loop should only add about 1GB of data at most and so it will not push the SLX table space over its quota. The start of day loan file from DML is roughly 220MB in size which again will not have a significant impact on free storage.

Anticipated volume increases Over a 6 month observed period, DML activity has grown at a linear rate. As shown in Figure 11, the number of loans grows by 1,990 every 90 days. The total number of loan components grows by 4,774 every 90 days. Following this trend, by July 31, 2007, DML will have 43,223 loans and 82,510 components. That is a 23% increase and a 61% increase from the previous year respectively. A table showing the observed 6 month period can be found in Table 8.

Loans Components3/31/2006 33087 472156/30/2006 35242 512609/11/2006 36703 55973

Table 8 - DML loan and component statistics







Growth of DML Activity

y = 22.115x - 825078R2 = 0.9978

y = 53.04x - 2E+06R2 = 0.9885

0

10000

20000

30000

40000

50000

60000

3/14/2006 4/28/2006 6/12/2006 7/27/2006 9/10/2006 10/25/2006

Date

Num

ber o

f loa

ns &

com

pone

nts

LoansComponentsLinear (Loans)Linear (Components)

Figure 11 - Number of DML loans and components

In Figure 11, the formula that is next to each line is a best fit linear regression generated by Microsoft Excel. The value for X is equal to the number of days after 1/0/1900 (Excel’s interpretation of dates as integers)

Requirements around timing of DML batch processes DML batch processes will generate more messages for SLX to process. Examples of batch processes include nightly recalculations and the early morning Japanese price batch process. Starting times of these batches should be observed. If they happen when the SLX server load is high, the SLA of 2 minutes will not be met.

RECOMMENDATIONS AND NEXT STEPS RECOMMENDATIONS The recommendations can be summarized into the following points:

• Review the service level agreements. The proposed service level agreement (guarantee) of a two minute maximum processing time of a transaction is not being met between the hours of 1AM and 3AM and between 3:30PM and 6:00PM. While the 1AM – 3AM time is not applicable because it is in the middle of batch processing, the SLA should be modified for 3:30PM – 6PM.







• Batch processes should be scheduled if possible to not coincide with SLMU peaks.

• There is no immediate need to increase file storage capacity to accommodate the DML feedback

loop

NEXT STEPS The next steps going forward can be summarized into the following points:

• Establish contact with a member of the infrastructure team and maintain regular contact for the

rest of the project’s duration.

• Identify times of all SLX and DML batch processes

• Determine if there is a need to increase file storage capacity on the volumes /dev/slxp1-slxhom1lv & jipnas1:/vol/jipnas1_app0/tss_slx_prd1 to allow for future growth

• Determine which volume the SLX tables are stored on (we think it’s /dev/dbp01-slxora1)

• Come up with a way to translate DML messages into storage space







REFERENCE DOCUMENTS

Reference Document Location AIX Server description webdoc http://dcfdoc.gis.chase.com

GTI Hobbit, real-time server monitoring utility http://wssunixmonitor.gis.chase.com

GTI Memory and CPU plotting macro CPU&RAM SLXv5.xls

SLMU / SLX feedback Loop documentation (multiple docs) Worldwide Securities Services PKS /

SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration /

Research / Information

SLX Production Incident Impact Calculator S:\SECLEND\Production\Production Support\SLX\SLX Production Incident Impact Calculator.xls

DOCUMENT CONTROL DOCUMENT HISTORY

Version Name Date Description 0.1 Michael Kristan 11/14/2006 Initial Draft 0.2 Michael Kristan 11/20/2006 Revised based on comments from Brian Kenney

and from Michael Salamina. 0.3 Michael Kristan 11/22/2006 Revised after Scope/Requirements Review on

11/21. Expanded section on SAN storage and updated recommendation list.

0.4 Michael Kristan 11/30/2006 Revised to include performance assessment from November (post-swapfile fix), cost analysis from Danielle Rumore.

0.5 Michael Kristan 12/3/2006 Added SLMU feedback loop table sizes and financial impact from downtime.

DOCUMENT STORAGE System Name: Worldwide Securities Services PKS Project ID: 71707 Project Name: SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration







APPENDICES POWERPOINT SLIDES FOR CAPACITY PLANNING

Capacity PlanningAn analysis of performance on SLX servers

Michael J KristanSecurities Lending Technology

JPMorgan Worldwide Securities Services

Observed data (SLMU)

• An analysis was made on the effect of SLMU messages on SLX servers during the week of October 15th.

• A report was run which took an hourly snapshot of the total number of SLMU messages processed during that hour

• SLMU hourly data was then aggregated over 7 days (including weekends)







Observed data (CPU)• CPU usage data was gathered off of three

servers for that week period (10/15-10/21)– TSJIPUDBP01– TSJIPUAPP06– TSJIPUAPP10 (not shown in these graphs)

• In order to remove any severe outliers, the median value of the 7 day samples was used for each time interval– Median was used instead of average because the

servers went offline for part of the weekend (resulting in an observed CPU usage of 0%) and it would have skewed the average

Observed data (DML)

• DML data was projected onto the graph in order to help determine when processing spikes may occur








0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000





0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000Pr

oces

sing

Tim

e (s

econ

ds)



SLMUMessages/25000

DMLMessages/25000




Response time is directly related to processor usage, not messages








0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000




Number of messages however has an influence on processor usage


0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000Pr

oces

sing

Tim

e (s

econ

ds)



SLMUMessages/25000

DMLMessages/25000




Response time is quick when both CPUs are under 50%








0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000




SLA is not being met right now


0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000Pr

oces

sing

Tim

e (s

econ

ds)



SLMUMessages/25000

DMLMessages/25000




Risk: DML messages are high from 11AM – 1PM








0

0.2

0.4

0.6

0.8

1

1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00


# of

mes

sage

s an

d C

PU u

sage

(as

a ra

tio o

f 100

)

1

10

100

1,000

10,000

100,000

Proc

essi

ng T

ime

(sec

onds

)



SLMUMessages/25000

DMLMessages/25000




Good news: SLMU peak occurs when DML messages are low

Next steps

• Take a closer look at memory usage and disk usage on servers

• Take a larger sample (2-3 weeks instead of 1)

• Identify times of all SLX batch processes• Come up with a formula to model CPU

usage based on messages