Tim Lindemulder Oracle E-Business Suite Specialist—Quote to Cash August 2005 [email protected] Case Study: Creating Price Lists in an Oracle

Tim LindemulderOracle E-Business Suite Specialist—Quote to CashAugust [email protected]

Case Study: Creating Price Lists in an Oracle ERP Dynamic Pricing Environment

Static Pricing in a Dynamic Env.August 18, 2005

Page 2

Overview

1) Overview of Oracle Environment at Agilent Technologies

2) Development of Static Price List Generator

3) Growing Pains experienced when Agilent deployed the Solution


Page 3

Agilent Overview

SemiconductorProducts

$2.0B

Life Sciencesand

Chemical Analysis

$1.3B

Test andMeasurement

$2.9B

AutomatedTest

$0.9B

Agilent Laboratories

SemiconductorProducts

$2.0B

Life Sciencesand

Chemical Analysis

$1.3B

Test andMeasurement

$2.9B

Test andMeasurement

$2.9B

AutomatedTest

$0.9B

AutomatedTest

$0.9B

Agilent Laboratories

Global diversified technology organization

FY04 Revenue = $7.4B

Employees = 28,000

Origin – dates back to the earliest days of HP in test/measurement


Page 4

Agilent’s ERP infrastructure -- Software

• Agilent has implemented Oracle E-Business Suite 11i (Oracle’s ERP product). The base version is 11.5.6 with many modules upgraded to 11.5.7 and 11.5.8.

• Oracle’s most recent release is 11.5.10 (11i10).

• Underlying database and apps tier is version 8.1.7.6.

• Major areas implemented are Quote to Cash (Order Management, Quoting, Product Configuration, Advanced Pricing), Manufacturing, and Finance.


Page 5

Agilent’s ERP Infrastructure -- Hardware

• 4 production instances running on HP Superdomes

• On-Line Transactions Processing (OLTP)

• Reporting

• Planning

• Configurator Rule Generation

• Reporting is a copy of OLTP that is refreshed every 12 hours via a DB copy from OLTP with some resident data exported before the copy and imported after the copy


Page 6

Pricing at Agilent Technologies

• Uses version 11.5.7 of Oracle Advanced Pricing

• All prices are dynamically determined by formulas using pricing attributes as they are set at the time of the pricing call

• A price formula can take between 1 and 10 seconds to calculate a price for a product.

• Agilent systems are defined by a bill of material consisting of 10 – 200 products.


Page 7

What is the limitation of dynamic pricing?

•Prior to implementing Oracle E-Business Suite, Agilent had several customer facing applications that allowed customers to access price lists online.

•The dynamic pricing formulas are too slow for those applications since they were designed to get data from static price list extracts.

•Oracle 11.5.6 doesn’t have a remote get price call.

•Solution: we developed a system of batch jobs that generates a full static price list each month into a set of custom tables.


Page 8

Batch Pricing Solution• Run the pricing formulas to generate the list prices for a particular product line, currency, end-use country, and shipping terms as of the first of each month.

• These jobs store their results in 3 custom tables. These tables are used by some Oracle-based applications for showing approximate pricing quickly (bypassing formulas).

• After the jobs run, another set of jobs goes through the tables, creates text extracts of the data, and sends it to systems that can’t read the tables directly.

• There are currently 3150 batches that generate 14 million price records.

• Batch prices are generated on the reporting instance.


Page 9

Batch Pricing Design Algorithm

• Get the list of products that are in the product line

• For each product

• Find out how many products are in its bill of materials

• Insert a blank line in the price lines table to each product in the bill

• Call the Oracle Advance Pricing API to calculate the prices with the correct formula

• Copy the output of this API from the pricing interface results to the reserved lines in the custom table

• End for loop


Page 10

Growing pains for batch pricing

• After about 18 months of production, we observed that batch price list generation for a full pricing run was taking 2 – 3 weeks.

• Batch pricing runs absorbed a significant amount of resources on our reporting instance and needed to become more lightweight.


Page 11

Improved Batch Pricing design algorithm

• Get the list of products that are in the product line

• For each product

• Find out how many products are in its bill of materials

• Call the Oracle Advance Pricing API to calculate the prices with the correct formula

• Copy the output of this API from the pricing interface results to PL/SQL memory-based tables

• End for loop

• Do a batch insert of all the data in the PL/SQL tables to the physical tables.


Page 12

Test results with improved algorithm

• The test batch run went 90% faster than comparable batch run in production.

• The amount of DML done by the test run dropped significantly.

• The increase in memory use did not affect the application as others tested on our test instance.


Page 13

Success?

• The solution went into production in March 2004.

• The March price lists (the last result of the old version of the programs) were archived.

• Note: Another related problem that was fixed was that the batch prices erased all old prices when they started running. I.e. the April batch would start on March 10 and erase the March prices. The archiving solution is independent of the batch pricing programs.

• We trained the business pricing team and sat back to enjoy the compliments that would inevitably come our way…


Page 14

Problem #1: Different setups in the reporting instance vs. everywhere elseDescription: After the migration to the reporting instance, the application failed with access errors.

Root cause: The test system setup did not match the reporting instance setup.

Solution: On the reporting instance, all applications run under user “procuser”. On the testing instance and the OLTP instance, all applications run under user “superuser”. We had to change all of our grant scripts to grant access to new objects to “procuser” as well as “superuser” and re-install the application.


Page 15

Problem #2: Each product line had only 1 product priced

Description: In the results tables after the first run for all product lines, each product line had one and only one bill of material priced, no matter how many products were in the product line.

Root cause: Our original code did multiple DML on database tables. Our improved code moved all DML to PL/SQL tables in memory and wrote the results to the database at the end. In a key portion of the loop through the product list, the counter on the memory table was reinitialized, so each product overwrote the previous one.

Solution: Fix the counter on the PL/SQL table.


Page 16

HOW DID THAT HAPPEN?!?

The batch product lines were not set up on the test instance. So, the developer picked 2 small ones to set up for testing. Unfortunately, both had only one configured item in it, so the loop was never executed until it went into production.

Solutions:

1) We asked our DBA team to add importing the batch pricing setup tables from the reporting instance to the test instance to their playbook for refreshing the test intsance.

2) We implemented more robust test scenario reviews as part of our approval for production process.


Page 17

Problem #3: All prices generated, but the extracts to external systems had no dataDescription: All the prices were in the

price lines table, but the extracts for the boundary systems had one product.

Root cause: The extracts got one field from the line details table. It is generated in the batch program and had the same PL/SQL memory table logic as the lines table, including the same reinitialization bug with the counter.

Solutions:

1) Temporarily change the extract program to do an outer join to the details table.

2) Fix the line details code.

3) Make sure the test cases include generating extracts.


Page 18

Problem #4: Reporting Instance crashDescription: Our reporting instance crashed with a memory error.

Root cause: The DBA team was warned that these tables would grow, so they added more memory extents to the EULD tablespace where the pricing tables lived. Unfortunately, they did it wrong so that when the tablespace first tried to use the new memory segment, the instance crashed.

Solution: The DBA team restored the instance and fixed the EULD tablespace setup.

Note: The users did not distinguish between failures; escalation to management began.


Page 19

Problem #5: Missing lines AND slower performance

Description: Some (not all) runs finish with fewer lines than previous runs. All runs appear to be running slower than before. The error logs said something about TEMP that the user support team did not understand.

Root cause: TEMP space not big enough to hold our new data.

Solution: Add more disk space to TEMP tablespace.


Page 20

Problem #5: Missing lines AND slower performance—First solution didn’t work!Upon further review: Each new batch job was trying to do a full table scan of the pricing headers table (which was thought to always be accessed via index). Each scan was reserving 2 GB of TEMP space of a total of 36 GB available. The TEMP errors were the system killing processes to let others run.

Root cause: Nobody had analyzed the pricing headers table in a long time. Since it had cached pricing data, its size was much bigger than table statistics show. The optimizer thought a full table scan was faster than an index scan.

Solution: Analyzed all pricing tables.


Page 21

SUCCESS!!!!!

Correct monthly run in July!!!!


Page 22

Problem #6: The reporting instance refresh is too slow.

Description: The reporting instance refresh process ran in one hour with users off of the system. It began to take more than an hour several days.

Root cause: The refresh process was to export EULD, copy the OLTP DB to the reporting instance, then re-import EULD. All the new pricing data in EULD slowed down the copy processes.

Solution: This is still a problem today. The current thought is that we need to upgrade our database to 9iR2 or 10g in order to get the tools we need to fix this.


Page 23

Problem #7: The extracts are much bigger than before.

Problem: The user community watched the line counts very carefully to see if the August results matched the March baseline. Now, instead of too few lines, they noticed too many lines in comparable extracts.

Root cause: The outer join on the line detail table enabled more records to be selected for the extracts. In other words, it found products they never knew they were missing.

Solution: Educate users that this is a good result, not a problem.


Page 24

Problem #8: The biggest product lines weren’t finishing in time.

Description: In following months, the product lines that had the most products occasionally ran slowly and had trouble finishing by the end of the month.

Root cause: The other smaller product lines ran so fast now that the bigger ones began running in parallel more often. When the same product line ran at the same time, they blocked each other in retrieving bills of material for the same item.

Solution: In the original design, this risk had been noted and recommendations for sequencing were made. They were never fully implemented by the users since they weren’t needed. By following them now, the problem was solved.


Page 25

Lessons Learned

1) Use robust test sets. Agilent’s best practice is to have at least 3 independent teams review test sets before test phase.

2) Make sure that your test environment matches your production environment. If it doesn’t, be sure to plan around the differences.

3) Find problems with your solutions before the user community does, and keep them informed about them.

4) Throwing more memory at a memory problem often does not solve the problem.


Page 26

Lessons Learned

5) Work with users to set their expectations for success and failure. Manage these expectations carefully.

6) Not every software problem has a software solution. Often the optimal solution is outside of the software.

7) Instrument your code so that you can check performance at any time on any system.


Page 27

Happy Ending

As of April 2005, the batch pricing run ran 40% faster than previously. (April was a month with an unusually large number of price changes.) All memory issues are gone. All the external systems have their pricing extracts and the pricing team is spending their time helping them decide how to present the information to their users better rather than trying to troubleshoot and run the batch processes.

Documents

Tim Lindemulder Oracle E-Business Suite Specialist—Quote to Cash August 2005 [email protected] Case Study: Creating Price Lists in an Oracle