Embed Size (px)
Citation preview
OTM 6.4 Upgrade in Cisco Supply Chain
Today’s Speakers
Arul Nataraj• Technical Lead, Supply Chain IT, Cisco• 16+ Years of Experience in Oracle CRM & ERP• Responsible for Supply Chain Deliver, OTM and
Supply Chain Integration
Sree Guru Sankaran• Sr Analyst, Supply Chain IT, Cisco• 12+ Years of Experience in Supply Chain Management.• Responsible for Supply Chain Deliver and OTM solution
team
Contents
• About Cisco• Why Upgrade• Benefits• Upgrade Scope and Plan• Regression and Performance• Upgrade Schedule• Challenges• Lessons learnt
• Worldwide leader in networking
• Founded in 1984
• FY15 revenue ~ $49.2B
• ~71,500 employees
• >380 Global Sites in165+ countries
• >180 acquisitions
• >60 Manufacturing and Fulfillment locations
• #7 in Gartner’s Top25 Supply Chain Companies in 2016 report
Broad portfolio of integratedproducts, solutions and services
About Cisco
Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco’s Value Chain … Complex, Global
CVCM
CVCM IT
Manufacturing Team By the Numbers
Hybrid production model95% outsourced
IT = 69% outsourced
~9,000 employees90+ locations/32 countries
9 functional groups
$32B Division>1,000 suppliers
35,000 components
Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Supply Chain’s “Layers of Manufacturing”
TECHNOLOGYASSEMBLY
Printed Circuit Board Assembly (PCBA)
Economies of scale, majority manufactured inAsia and Mexico
COMPONENTS
Global sourcing, economies of scale and technology, open source,commercial andCisco software
LOGISTICSClose proximity to transport network, multi-function centers, licensing, andsoftware asset management
SYSTEMINTEGRATION
Direct Fulfillment (DF)
Regional, closer proximity to customers,electronic delivery
OTM Architecture – Before Upgrade
Default ClusterDefault Cluster Custom ClusterCustom Cluster
Web Tier
App Tier
DB Tier
OAMAuthC
Intranet
Apache 2.2 Tomcat 6.0
OTM Load Balancer
Intranet
Kewill Flagship 5.2
Kewill Load Balancer
SMC3 Load Balancer
FSMS 12.07
SMC Czarlite 4.18.102SMC Carrier ConnectR165
Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
End to End Process Flow
Web Services
LaneSelection
OrderReleaseCreation
Mode Selection
Review &Approveshipment
Ship Confirm Track& Trace Audit &Settle
OrderMgmt.OrderMgmt. ASCPASCP InventoryInventory Shipping
ExecutionShipping
Execution PurchasingPurchasing
• Items,Products,
• Location• Sales order• Move order• Delivery
• PlannedShipments
• Shipment Ack
• Shipconfirmation
• ShipmentEvents
Invoice
FinancialsFinancials
ShippingControl
Loadconsolidation
Oracle Transportation Management
Rate Shopping
Value Chain Partners
Bulk Plan/Instant Plan
ShipmentCreation
External Rating Engine
Fusion Transportation Intelligence
• Order• Shipment• Invoice
Rate Inquiry/Response
ETL
• RouteInquiry
• Transit Time
OTM Order Volume
Year Volume2014 3.3 M2015 5.1 M
2016 (till date) 2.9 M
Order Releases Volume
Year Volume2015 6.1 M
2016 (till date) 23.3 M
Transmissions Volume
• Continue to be in Oracle Premium Support for OTM
• New functionalities added in 6.4.1• Rate Maintenance Enhancements• User Interface Enhancements• Table structure changes - performance enhancement• Mobile App• GTM (Global Trade Management) 6.4.1
Why OTM Upgrade
Upgrade Release Plan - Proposal
Push Upgrade of 6.4.1 to Q4FY16 – May release
Pros:
Will have more time to stabilize 6.4.1 version.
Will solve the biz problem and also have stable instance.
Cons:
Downtime will be high. Right now we are seeing 85 hours ofdowntime. We are looking at possible ways to by working with Oracle.
Might get pushed to Q1 since this is a big change and Q4 is year end.
Split upgrade and move to 6.3.8 in April release and 6.4.1 in Q4 June
Pros:
Separating out the risk.
Reducing the downtime (approximately 40hrs per release)
Cons:
Multiple upgrades over the period of 2-3 quarters
We will have 2 Stabilization period and regression testing, which maypotentially impact funding.
Q1FY16 Q4FY15 -June
Q3FY16 Q3FY16April
OTM 6.3.8 OTM 6.4.1
Q1FY16 Q4FY16Q2FY16 Q3FY16
OTM 6.4.1
Option 1: Option 2:
PlannedProposed
• Planning and Execution - change in timelines : Business Continuity• Cisco had initially planned to upgrade during March 2016. However, Oracle provided upgrade script took
more than 83 hours for completion thereby putting business continuity at risk. Thus, upgrade plans hadto be deferred to May 2016 subject to Oracle providing the optimized upgrade script.
• Co-ordination and Management buy-in• OTM upgrade at Cisco involved OTM Application, DBA, Network and Infrastructure and Performance
teams. Getting 'buy-in' from management and ensuring commitment and focus from all involved teamswas challenging.
Challenges - Internal
• OTM version 6.4.1 was made generally available in December 2015 andCisco had to align their upgrade strategy accordingly.
• OTM 6.4.1 was new to the market and not many user adoption credentialswere available for decision making.
Challenges - Product
OTM Upgrade BenefitsOracle supported version for Premium Support
Latest Oracle 12c Stack – Web, App, DB
New business capabilities - Rate Maintenance
Future business capabilities – Network Routing, GTM, Landed Cost Simulator, EnhancedUser Experience, Shipment Visibility, Workbenches
Technical capabilities – Flexfields, ADF framework, Mobile Apps, 12c
FTI Enhancements – Upload Targets, New Subject Areas (Rate Maintenance)
OTM Upgrade Scope
15
OTM OTMR(archive)
OTM Apps 6.2
OS : CEL 5.5DB v 11g
OTM OTMR(archive)
OTM Apps 6.4
OS : CEL 6.4DB v 12c
OTM Apps 6.2 >6.3
Database 11g12c
OTM Apps 6.36.4
1. Server Upgrade from CEL 5.5 CEL 6.42. OTM Application Upgrade 6.2.8 6.4.13. Database Upgrade Version 11g 12c4. Storage Block Device ASM
1. Server Upgrade from CEL 5.5 CEL 6.42. OTM Application Upgrade 6.2.8 6.4.13. Database Upgrade Version 11g 12c4. Storage Block Device ASM
Upgrade Phases:
OTM Upgrade Scope
Environments:
• POC• DEV• TEST• LT• FPR• PROD
Environments:
• POC• DEV• TEST• LT• FPR• PROD
Stakeholders Engagement• A highly collaborated team effort across various stake holders
• Global Infrastructure Team• DBA, Compute, Network Design/Architecture, Performance• Network Implementation• Backup Team, Storage & Network Services• OAM, RDAC & Infosec Team
• Application Team• Architecture Team• Enterprise Release Management Team• Oracle Support Team Onsite Presence
• POC from Oracle DBA Team• POC from Oracle Support
Category Activity Time taken Downtime
OTM Application OTM Data upgrade from 6.2.8 6.3 26 hrs 26hrs
OTM Archive Application OTM Archival Data upgrade from 6.2.8 6.3 8 hrs 8hrs
Phase 1: OTM Application Data upgrade from version 6.2 > 6.3
Category Activity Time taken Downtime
OTM Database Database upgrade from 11g 12c 6 hrs 6hrs
OTM Archive Database Database (Archive) upgrade from 11g 12c 6 hrs in parallel 0
Phase 2: Database upgrade
Category Activity Time taken Downtime
OTM Application OTM Data upgrade from 6.3 6.4.1 35 hrs 35hrs
OTM Archive Application OTM Archival Data upgrade from 6.3 6.4.1 8 hrs 8hrs
Phase 3: OTM Application Data upgrade from version 6.3 > 6.4
OTM Upgrade Duration (POC)
83Hrs83Hrs
Regression and Performance
Regression Testing• Approximately 350 QA & 75 UAT test cases were executed Validate existing functionality in the upgraded instance Concurrent users testing for the existing UI interface Access Control List validation for all existing user roles
Performance Load Test to validate the 1.5x of hourly peak load volume Load distribution strategy for hourly volume by business objects Automation of Load Testing Multiple performance executions Iterative approach to evaluate and fix performance issues
Load Test Automation
• Automation included bulk order creation
• Staging orders in Oracle EBS for automated push to OTM
• Reduced manual intervention
• Increased productivity
• Frequent executions for iterative performance improvements
• Performance team engagement to monitor real time performance impact
• Post run metrics analysis
• Incorporate performance recommendations in subsequent runs
OTM Load Test ResultsLoad Run # Date Test Details Test Results
1 22-Apr Run with default DB parameters, App Properties • Baseline performance recorded withdefault parameters and properties.
2 26-Apr Run with Custom App properties as used in Prod • Performance issues observed• DB parameters updated
3 04-May Run with Custom DB parameters as used in Prod
• Updated DB parameters resolvedperformance issues
• DB parameters tuned further and packagegrants issue is fixed
4 11-May To be run with tuned DB parameters(session_cached_cursors, open_cursors, shared_pool_size)
• Cursor Leak issue encountered; SR openedwith Oracle
5 13-May DB parameters updated as per Oracle suggestions(session_cached_cursors, open_cursors, shared_pool_size) • Cursor Leak issue not observed.
6 14-May Oracle patch for DB applied on LT1OTM(SR 3-12672575911) • Cursor Leak issue not observed
7 16-May Oracle patch for Application applied on LT1OTM(SR 3-12672575911) • Cursor Leak issue not observed
OTM Upgrade Go-Live Plan
Activity Duration (Hours)
Ramp Down 1
OTM Upgrade – Version 6.2 to 6.3 18
Database 12C Upgrade 4
OTM Upgrade – Version 6.3 to 6.4 18
Application Integration 3
Application Validation/MVDC Testing 6
Code Migration Tool Setup 1
Code Deployment and IT Verification 2
Total:43Hrs
• OTMPRD DB, Application and Webservers will be available as replicas onboth 6.2.8 and 6.4.1 versions
• In case of a “No-Go”, DBA team will re-point to 6.2.8 OTM DB, Applicationand Webservers.• Re-pointing to be completed in approximately 3-4 hours.• Ramp Up and Normalization support to continue as planned on 6.2.8
version.• Communication will be sent out by Release PMO
Contingency Plan
OTM Architecture – After Upgrade
Default ClusterDefault Cluster Custom ClusterCustom Cluster
Web Tier
App Tier
DB Tier
OAMAuthC
Intranet
Kewill Flagship 5.2
Kewill Load Balancer
OTM Load Balancer
SMC3 Load Balancer
FSMS 12.07
SMC Czarlite 4.18.102SMC Carrier ConnectR165
Intranet
• Product related issues were resolved with Oracle support• Upgrade script had to be optimized by Oracle based on performance feedback from Cisco• Invalid Data connection issues - Resolved after creation of Primary JTS and Data Sources• Undefined Error with OTM UI - Resolved after changes to glog base properties file• Kewill Rating for orders releases with hazardous items – Resolved by patch• Open Cursor and DB cursor leak issues – Resolved by DB and Application patches• Invoking custom and integration utility packages - Additional grants were provided to resolve• Application server crashes - Stack size was doubled in weblogic config files• User Role flip issues - Resolved with domain grants and table set creation
Upgrade Challenges
Cisco, a large scale high-tech manufacturing org, becomes an early adopter of OTM 6.4.1.
The upgrade from 6.2 to 6.4 brought several issues related to product installation & performance. MultipleSRs and patch requests were raised with Oracle to resolve issues.
Partnership with Oracle Support & Development team help resolve all the issues encountered
Oracle Support team was actively involved during go-live has improved liaison and response times.
Exhaustive User Acceptance Testing with concurrent users will be critical to the success of upgrade andreducing post upgrade risks.
Multiple dry run were needed for validating the upgrade scripts in optimizing the overall downtime.
Collaborated team effort between various stake holders, including DBA, Network and Infrastructure, OTMApplication, Performance & Oracle support teams resulted in successful OTM Upgrade.
Learnings
Backup Slides
28
OTM Upgrade IssuesIssue Summary Issue Resolution
Error while copying Shipment Involved Party screen set from Public domainBug in OTM 6.4.1; unable to copy Shipment Involved Party screenset fromPUBLIC domain
OTMPRD one APP node is crashed due Failed to write core dump Oracle suggested doubling the stack size in weblogic conf files
When user role is changed through UI the browser gets refreshed automaticallyOracle suggested providing domain grants and deleting records from tableset detail table
ORA-04025: maximum allowed library object lock allocated for INSERT /*+ Open Cursor Leak Issue; Oracle provided a patch for DB and ApplicationOTMPRD When query for orders with search criterion, criteria is removed &blanket search executed Changes to glog base properties recommended by OracleOTMPRD Undefined Error occurred when try to perform action from OrderRelease search screen Changes to glog base properties recommended by Oracle
Kewill Rates not getting selected for order planningIssue with Kewill rating for orders with hazardous lines. Bug in OTM 6.4.1;patch released by Oracle for fixing the issue
OTMPRD Rate Record Cost Condition field is not rendered properly Usage of supported browser helped in resolving the issue
OTM Upgrade IssuesIssue Summary Issue Description
OTM 6.4 product Concerns for Cisco instance UpgradeAdded column to a custom table and ROD upgrade trigger issue wasresolved
After login into OTM application intermittently we are getting(cause.InvalidDataConnection at sun) After creation of PRIMARY_JTS the issue was resolved
POST migration from 62.8 t o 641issues in TS1OTMEntries regarding Database corrected in glog properties file as they werenot matching earlier
Integration error when executing theGLOGOWNER.PKG_INTEGRATION_UTIL.INSERT_TRANSMISSION API User had to be changed to DIR_XML_USER for Direct XML insert inbound
Unable to invoke UTC package from Custom SchemaGrants had to be provided to GLOBALREPORTUSER and custom schema tobe able to invoke custom package from custom schema
DATA_SOURCE table informationInvalid data connection and other connectivity issues were being observed;information received on Data Source from Oracle
Cisco Confidential 31© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Order Touch Points in OTM
Orac
leAp
plica
tions
–CG
1
Sell
Ship
men
tCr
eatio
n-
OTM
Ship
men
tPl
anni
ng-O
TMSh
ippi
ngCo
ntro
l-OT
MOr
der
Man
agem
ent-
OTM
Order Booked
Order ReleaseCreated in OTM
Ping OTM for RouteCode
Shipment Created(Route, Transit Time)
Route Code Updated
Delivery Created
Order ReleaseCreated in OTM
Order PlanningConstraints updated
Shipping Control Rules Ship Eligible
Order made ShippingEligible
Instant PlanEligible
Instant Plan
Shipment Created (Route, Rate,Transit Time, Carrier)
Wait for scheduled Bulk Plan
Order Approved
Create SellShipment
Update Freight CostDetails
Update Details forTrip Creation
ShipmentApproved
Delivery Interfaced to OTM forShipment Planning
No
Yes Yes
No
No
Yes
Yes
CRT Domain COB Domain
1
2
3
4
Route & Transit
Rev Control
Optimization
Track &Trace
Carrier SendingMilestones
Milestone MonitorUpdate
OTM Inbound Process Flow
