100 Million Subscriber Performance Test Whitepaper

An Oracle White Paper

April 2011

100 Million Subscriber Performance Test Whitepaper:

Oracle Communications Billing and Revenue Management 7.4 and Oracle Exadata Database Machine X2-8

Oracle Communications BRM 7.4 and Oracle Exadata X2-8 100 Million Subscriber Performance Test White Paper

Disclaimer

The following is intended to outline our general product performance and throughput. It is intended for

information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver

any material, code, or functionality, and should not be relied upon in making purchasing decisions. The

development, release, and timing of any features or functionality described for Oracle‟s products remains at

the sole discretion of Oracle.

Oracle Communications BRM 7.4 and Oracle Exadata Database Machine X2-8 100 Million Subscriber Performance Test White Paper

Executive Overview ........................................................................... 2

Introduction ....................................................................................... 2

Project Goals ..................................................................................... 5

Oracle Exadata Database Machine X2-8 Configuration ............... 12

BRM Server Configuration ........................................................... 12

Network Configuration ................................................................. 13

Operating System Configuration .................................................. 14

BRM Schema Layout ................................................................... 14

Performance Test Results ............................................................... 15

Rating and Discounting Results ................................................... 15

Billing and Invoicing Results ........................................................ 16

CSR Results ................................................................................ 17

Conclusion ...................................................................................... 19


2

Executive Overview

Given the explosive growth in broadband, mobile and machine-to-machine communications

traffic, the need for cost-effective, high-throughput, highly-available, and scalable billing

solutions is increasingly apparent. In February 2011, the Oracle Communications Global

Business Unit (CGBU) Performance Group conducted a performance test to demonstrate the

performance and scalability of Oracle Communications Billing and Revenue Management

(BRM) with an Oracle Exadata Database Machine X2-8 for a 100 million subscriber base.

In this performance test, in which multiple workloads were tested, Oracle achieved the best

ever throughput performance for Oracle Communications Billing and Revenue Management.

One hundred million subscribers were modeled in two Oracle Real Application Clusters (RAC)

nodes. Throughput rates, normalized under a per database schema basis, proved superior to

those achieved in all prior performance tests.

Introduction

This document presents the results from a large scale Oracle Communications Billing and

Revenue Management performance test performed in February 2011 at the Oracle Solutions

Center in Menlo Park, California. This performance test was performed on Oracle

Communications BRM 7.4 – the industry’s leading convergent billing and revenue

management platform – and on the latest Oracle Exadata Database Machine X2-8.

The performance test was conducted for a 100 million subscriber base, created using a

standard Oracle Communications BRM GSM performance test price plan. The main workloads

were:

Rating and discounting

Billing with deferred taxation

Detailed invoicing

CSR activities

The throughput results reported in the document reflect either the number of operations

executed for each workload given in a fixed execution time, or the observed rate of execution


3

against a fixed number of subscribers. The 100 million subscribers were evenly distributed into

eight BRM schemas. The full rack Oracle Exadata Database Machine X2-8 was running

Oracle Enterprise Linux 5 and Oracle Database 11g Release 2. The Oracle Communications

BRM tier ran on Oracle’s Sun Fire X4270 M2 servers running Oracle Enterprise Linux 5.

Results Summary

TABLE 1. PERFORMANCE TEST RESULTS SUMMARY

RATING AND DISCOUNTING

TOTAL SUBSCRIBERS SUBSCRIBERS PER RAC NODE TOTAL THROUGHPUT

(CDRS PER SECOND)

THROUGHPUT PER SCHEMA

(CDRS PER SECOND)

100 million 50 million 50,125 6,266

BILLING


(BILLS PER SECOND)


(BILLS PER SECOND)

100 million 50 million 2,567 321

INVOICING


(INVOICES PER SECOND)


(INVOICES PER SECOND)

100 million 50 million 4,670 584

CSR ACTIVITY

OPERATIONS PER

SECOND

OPERATIONS PER SCHEMA LATENCY(MS)

41,580 5,198 10


4

Key Takeaways

Oracle Communications Billing and Revenue Management 7.4 with Oracle Exadata Database Machine X2-8 together have demonstrated unprecedented performance levels and the ability to satisfy the needs of the most demanding Tier 1 service providers.

The performance test results demonstrate that:

Oracle Communications BRM 7.4 with Oracle Exadata Database Machine X2-8 is able to process more than 50,000 operations/sec; operators using this technology can process 2 billion CDRs in less than 12 hours.

Billing achieved processing speeds of 2,500 subscribers/sec; operators using this technology can complete a bill run for 100 million subscribers in less than 12 hours.

Each Oracle Exadata Database Machine RAC node is able to support at least 50 million subscribers. This reflects an approximate 10x reduction in database server CPU requirements.

Oracle Communications BRM with Exadata can greatly simplify production deployments at a fraction of the cost of traditional technology. Within the confines of this test, one Oracle Exadata Database Machine X2-8 replaced 48 conventional x86 servers and 10TB of SAN storage.


5

Project Goals

The primary goal of the performance test was to measure the performance of the industry-leading

convergent billing and revenue management platform, Oracle Communications BRM 7.4, on the latest

Oracle Exadata Database Machine X2-8 in a multi-schema configuration. The BRM Rated Event

Loader (REL) feature was run to measure the rating and loading performance of batched CDRs for

GSM types of services. The main performance goals of this performance test were to measure batch

rating and discounting, billing and invoicing results under realistic workloads. Another goal was to

determine Oracle Communications BRM, Oracle Exadata Database Machine X2-8 and Oracle RAC

scalability over a multi-node DB configuration. The results of this performance test provide the

technical data necessary to transfer performance capacity planning information to the Oracle field

(professional services and sales organizations), system integrators, and partners for Oracle‟s Sun

platform.


6

Application and Hardware Overview

The following section provides a brief description of the applications and hardware systems that

comprised this performance test.

Oracle Communications Billing and Revenue Management

The Oracle Communications Billing and Revenue Management architecture provides the foundation

for the end-to-end Oracle Communications Billing and Revenue Management solution. Functionality is

segregated into layers using well-defined interfaces, enabling each to be modified and enhanced

without disruption of functionality at the platform level.

At the same time, this design enables the platform to evolve without adverse effects on functional

capabilities. As a result, Oracle is able to develop and advance Oracle Communications Billing and

Revenue Management‟s capabilities rapidly while enabling service providers to easily extend the system

to meet their unique business requirements.

Figure 1. Oracle Communications Billing and Revenue Management architecture

The Oracle Communications Billing and Revenue Management platform utilizes a modern n-tier

architecture, which is typically deployed with the following three-tier configuration:


7

The Applications Tier

The applications tier consists of programs and processes that use the object-oriented API as an

interface to the Oracle Communications Billing and Revenue Management system. This tier includes

the Oracle Communications Billing and Revenue Management client applications such as Customer

Center, Pricing Center, and other native as well as third-party applications developed by customers and

partners using the same documented API. All enterprise as well as network applications can integrate

and interact with the Oracle Communications Billing and Revenue Management system through the

applications tier.

BRM Server Tier

Oracle Communications Billing and Revenue Management server tier consists of three components:

the Real-Time Revenue Management Server, the Batch Processing Server, and the Object Server.

The Real-Time Engine enforces business policies. It consists of Connection Managers (CMs) and

discount, rerating, zoning, and facilities modules. The CMs manage connections between the

application tier and the functional modules, process data collected by Oracle Communications Billing

and Revenue Management client applications, and enforce the business rules. This architecture allows

easy customization of the Oracle Communications Billing and Revenue Management system to meet

unique business requirements.

The Batch Processing Engine is optimized to handle large volumes of transactions (e.g. hundreds of

millions of transactions per day) in terms of preprocessing, enrichment, duplicate checking,

aggregation, and rating - among other functions.

The Object Server provides an abstraction of the stored data. This layer consists of Data Managers

(DMs) that translate requests from CMs into a language recognized by the Oracle Communications

Billing and Revenue Management database or other data access systems. The Oracle Communications

Billing and Revenue Management system provides separate DMs for each supported database. In

addition, there are DMs for other external systems, such as the credit card processing service provided

by the Payment Managers.

The Data Tier

The data tier consists of Oracle Database and other data access systems. The Oracle Communications

Billing and Revenue Management application currently supports Oracle Database Enterprise Edition as

well as Oracle RAC as the primary database system. Oracle Communications Billing and Revenue

Management is also pre-integrated with other data access systems in this tier; for example, credit card

processing, LDAP and taxation systems.

Scalability

The Oracle Communications Billing and Revenue Management application achieves high scalability in

several ways. First, its multi-tier architecture is designed to run all Billing and Revenue Management

processes on the same computer or distributed among several computers. Distributed processing

allows for maximum flexibility and optimal load distribution for configuring an Oracle


8

Communications Billing and Revenue Management system as the number of users expands. As a

result, service providers can add as many servers as required in the application, Oracle

Communications Billing and Revenue Management server, and data tiers.

Figure 2. Oracle Communications Billing and Revenue Management scalability architecture

A second factor that contributes to the high scalability of Oracle Communications Billing and Revenue

Management is the combination of transactional real-time and near real-time batch processing that

utilizes a multi-threaded, pipelined architecture and in-memory processing. Integrated support for both

types of rating makes extremely high performance possible.

To further improve scalability, Oracle Communications Billing and Revenue Management lets

providers easily balance load processing by reconfiguring client and database connections. This enables

providers to smooth out usage spikes at the front-end and avoid bottlenecks at the database level.

Finally, transaction management functions have been built into the object layer, enabling Oracle

Communications Billing and Revenue Management to fully scale and take advantage of the underlying

hardware. Without this capability, load balancing would be limited beyond a certain transaction level.

Oracle Exadata Database Machine X2-8

Oracle Exadata Database Machine provides an optimal solution for all database workloads, ranging

from scan-intensive data warehouse applications to highly concurrent online transaction processing

(OLTP) applications. With its combination of smart Oracle Exadata Storage Server Software, complete

and intelligent Oracle Database software, and the latest industry-standard hardware components,

Oracle Exadata Database Machine delivers extreme performance in a highly-available, highly-secure

environment.

Extreme Performance for Online Transaction Processing, Data Warehousing and Consolidating Mixed Workloads


9

The Oracle Exadata Database Machine is an easy-to-deploy, out-of-the-box solution for hosting the

Oracle Database. By engineering its hardware and software together, Oracle has eliminated much of

the integration effort, cost and effort of database deployment.

The system can be deployed for OLTP, data warehousing, or mixed application workloads, lets service

providers consolidate multiple computing environments in the data center, and delivers unparalleled

performance.

The unique technology driving the performance advantages of the Exadata Database Machine is the

Oracle Exadata Storage Server. By pushing SQL processing to the Exadata Storage Server all the disks

can operate in parallel, reducing database server CPU consumption while using much less bandwidth

to move data between storage and database servers. As data volumes continue to grow exponentially,

conventional storage arrays struggle to efficiently process terabytes of data, and push that data through

storage networks to achieve the performance necessary for demanding database applications. Exadata

Storage Servers provide a high-bandwidth, massively parallel solution delivering up to 75 GB per

second of raw I/O bandwidth and up to 1,500,000 I/O operations per second (IOPS). Much of these

performance gains come from the incorporation of Exadata Smart Flash Cache in each Exadata

Storage Server and the Oracle Databases‟ storage hierarchy. With 14 Exadata Storage Servers in a 42U

Rack, 5.3 TB of Exadata Smart Flash Cache is integrated into theExadata Database Machine X2-8.

In addition, Exadata Database Machine is the world's most secure database system. Building on the

high security capabilities in every Oracle Database, Exadata Database Machine provides the ability to

query fully encrypted databases with virtually no overhead at hundreds of gigabytes per second. This is

done by moving decryption processing from software into the Exadata Storage Server hardware.

Figure 3. The Oracle Exadata Database Machine X2-8


10

Extreme Scalability

The Exadata Database Machine X2-8 is a full rack system with 2 database servers and 14 Exadata

Storage Servers. Each database server comes with 64 Intel CPU cores (8 x 8-core Intel® Xeon®

X7560 processors) and 1 TB of memory. It is available with either 600 GB High Performance

SAS disks or 2 TB High Capacity SAS disks.

While an Exadata Database Machine X2-8 rack is an extremely powerful system, a building-block

approach is used that allows Exadata Database Machine X2-8 to scale to almost any size. Exadata

Database Machine X2-8 racks can be connected using the integrated InfiniBand fabric. As new

racks of Exadata Database Machines are incrementally added to a system, the storage capacity and

performance of the system grow. A system composed of two Exadata Database Machines X2-8

racks is simply twice as powerful as a single rack system providing double the I/O throughput and

double the storage capacity. It can be run in single system image mode or logically partitioned for

consolidation of multiple databases. Scaling out is easy with Exadata Database Machine. Oracle

RAC can dynamically add more processing power and Automatic Storage Management (ASM) can

dynamically rebalance the data across Exadata Storage Servers to fully utilize all the hardware in

each configuration.


11

Performance Testing Environment

This section describes the system architecture and configuration environment for the Oracle

Communications Billing and Revenue Management and Oracle Exadata Database Machine X2-8

performance test.

Hardware Configuration

The system layout is displayed below:

Figure 4. System layout for BRM and Exadata X2-8 performance test

BRM Servers


12

The system layout consisted of:

1 x full rack Oracle Exadata Database Machine X2-8, used as Oracle database and storage servers

17 x Sun Fire X4270 M2 servers used as BRM application servers

2 x Cisco Gigabit Switches

Oracle Exadata Database Machine X2-8 Configuration

The full rack Oracle Exadata Database Machine X2-8 contained the Exadata Database Server, the

Exadata Storage Server, and the InfiniBand Network.

The Exadata Database Server consisted of two Sun Fire X4800 servers to form a 2-node Oracle RAC;

each server was configured with:

8 x 8-core Intel X7560 CPUs @ 2.27GHz, total 64 cores, 128 threads

1 TB RAM

6 x Gigabit Ethernet ports bonded together, connected to the public/performance test network

The Exadata Storage Server consisted of 14 Sun Fire X4270 M2 storage servers; each server was configured with:

2 x 6-core Intel L5640 CPUs @ 2.27GHz, total 12 cores, 24 threads

24 GB RAM

12 x 600 GB HP (High Performance) disks.

386 GB flash storage

There were a total of 168 disks from the 14 storage servers. Three logical devices were carved out of

each disk, one for the DBFS ASM Disk Group, one for the DATA ASM Disk Group, one for the

RECO ASM Disk Group. This way every disk group was spread across all disks. All three disk groups

were created with normal redundancy (one mirror copy). The BRM DB was initially created on the

DATA Disk Group.

There was a total of 5.3 TB of flash storage across 14 storage servers. A 2 TB Flash Disk Group was

created with normal redundancy, leaving 1.3 TB of flash for regular I/O to disks. The Flash Disk

Group was used to offload the intensive I/O to BRM DB.

BRM Server Configuration

Seventeen Sun Fire X4270 M2s were used for the BRM applications. Each server consisted of:

2 x 6-core Intel X5680 CPUs @ 3.33GHz; totaling 12 cores, 24 threads


13

96 GB RAM

4 x Gigabit Ethernet ports

6 x 300 GB 15K RPM SAS disk drives

The 4 x Gigabit Ethernet ports were configured as:

2 x Gigabit Ethernet ports bonded together, connected to the public/performance test network

2 x Gigabit Ethernet ports bonded together, connected to the BRM private network for batch workloads

The 6 x 300 GB 15K RPM SAS disk drives were configured as:

1 boot disk

2 disks striped together to hold the BRM software and batch REL interim directory

3 disks striped together to hold the Batch Rating pipeline input/output directory

Even though the network and disks on the 17 servers were configured symmetrically, as the servers

served different purposes, not all the components were used on each server, e.g., the Batch Rating

pipeline input/output directory was used only on the Batch Router server and the eight servers running

Batch Rating + REL.

The 17 Sun Fire X4270 M2s were configured in the following manner for the BRM workloads - batch,

billing, and invoicing:

Server 1: Batch Router, Test Driver

Server 2-9: 8 x (Batch Rating + REL), 8 x BRM Server

Server 10-17: 8 x BRM Server

Each BRM workload was evaluated in isolation and the systems were configured accordingly. One

instance of BRM Server is defined as 2 CMs, 2 RTPs, 1 DM Oracle, 1 EAI JS, 1 DM IFW SYNC

processes.

Servers 2-9 each had 1 NFS export from the Batch Rating pipeline input directory. The Batch Router

process on Server 1 used these eight NFS mounts to distribute files for the eight Batch Rating

instances to process. The NFS ran on the BRM private network for Batch workloads, to minimize

network contention and overhead.

Network Configuration

The 2 RAC nodes from Oracle Exadata Database Machine X2-8 and all 17 app servers were connected

via a Gigabit switch to form the public/performance test network. Communication among Batch

Router and Batch Rating instances was separated to a private Gigabit network. Jumbo Frame was


14

enabled on all NICs and switches. NIC bonding (EtherChannel) was used on the Gibabit NICs to

increase network bandwidth.

Operating System Configuration

All 17 BRM servers were installed with Oracle Enterprise Linux 5

The two RAC nodes were running Oracle Enterprise Linux 5

The 14 storage servers were running Oracle Enterprise Linux 5

BRM Schema Layout

The 100 million subscriber base was divided into 8 schemas, 12.5M per schema. The schema size was

determined based on physical memory available for one Batch Pipeline instance, startup time of such

instance and size of BRM table/index partitions.

Since the Oracle Exadata Database Machine X2-8 had two large RAC nodes, each node hosted four

schemas. For better connection isolation, eight services were created for the eight schemas so there

was a one-to-one mapping.


15

Performance Test Results

Plan Type

The mobile price plan used in this 100 million subscriber performance test is a standard configuration

used by Oracle Communications performance engineering. The performance test comprised a mobile

price plan that is representative of real world customer cases.

The plan comprised two services, a GSM telephony service and an SMS service (internal plan name is

BenchTelcoGSMD). The telephony service was a standard voice service with caller ID, call waiting

and voice mail. The price plan utilized two resources, Euros and free minutes. It charged 50 Euros per

month for telephony and SMS, with 10,000 free minutes for telephony. The GSM tariff model used 15

different impact categories and had 21 rate plan configuration entries. Each call detailed record (CDR)

created three balance impacts: two in Euros and one in free minutes.

During billing, every account that was billed had two cycle fees and one fold was generated. [A fold is a

special event that usually occurs at the end of the accounting cycle. A fold can be configured to either

change a resource balance (currency or non-currency) to zero or convert the resource balance into

other resources].

In the GSM batch rating tests, performance was measured for and „end-to-end‟ system. Each step of

the „end-to-end‟ system included CDR file routing, rating and discounting, and database updates using

REL.

The results are expressed in terms of operations per second. For all test cases, runs are only reported

when there were no error messages in any log files (operating system, database system, BRM Server)

during the performance test run. Billing had the added requirement that the execution against a fixed

number of accounts must have successfully completed.

Rating and Discounting Results The rating and discounting workload performs evaluations of service usage and associated rating and discounting. This involves processing of CDR files and updating of subscriber balances i.e., “end-to-end” CDR file processing.

Every run processed 50 files of 800,000 CDRs, for a total of 40 million CDRs. The more schemas the bigger input files for Router.


16

Components Overall

CDRs/sec

Batch Rating

CDRs/sec

REL CDRs/sec

App1 % Util

DB % Util

Memory

(KB)

Router, 8 Batch Rating + REL, 8 schemas

Router 62208 16/0 40497096

Batch Rating 1 6775 7763 7528 23/0/0 44/6/0 44504184

Batch Rating 2 6858 7788 7620 23/0/0 43/6/0 44554324

Batch Rating 3 6747 7763 7497 23/0/0 44/6/0 44439200

Batch Rating 4 6839 7751 7599 23/0/0 43/6/0 44662928

Batch Rating 5 6784 7763 7538 23/0/0 44/6/0 44658364

Batch Rating 6 6858 7763 7620 23/0/0 43/6/0 44623732

Batch Rating 7 6775 7751 7528 23/0/0 44/6/0 44515148

Batch Rating 8 6811 7763 7568 23/0/0 43/6/0 44298860

End-to-End 50125

Table 2. Rating and Discounting Results

Billing and Invoicing Results

Oracle Communications Billing and Revenue Management billing is used to generate a bill for a given

subscriber account on a cyclical or on-demand basis. The generation of the bill is dependent on the

billing cycle of the account—monthly, bi-monthly, quarterly, semi-annually, or annually. To generate a

bill for a given customer account, a setup of applications is run that finalizes the bill for a given period.

Billing was evaluated as a key workload during this performance test analysis. Following are the billing

results:

42 backends per DM_Oracle, 2 DM_Oracle per BRM schema.

88 children for billing, 88 children for detailed invoicing per schema.

100,000 accounts billed/invoiced per schema.


17

Workload ops/sec App1

% Util App2 % Util

DB1 % Util

# events /account

Billing w/ Deferred Taxation 1

319.49 40/11 40/11 32/11/0 100


321.54 39/10 40/11 31/10/0 100


319.49 39/10 40/11 32/11/0 100


320.51 40/11 40/11 31/10/0 100


320.51 40/11 40/11 32/11/0 100


321.54 39/10 40/11 31/10/0 100


322.58 39/10 39/10 32/11/0 100


321.54 40/11 40/11 31/10/0 100

Billing w/ Deferred Taxation Summary

2567.2

Invoice_DETAILED 574.71 18/4 19/5 17/6/0 100








Invoice_DETAILED Summary

4670.4

Table 3 Detailed Billing and Invoicing Results

CSR Results

In addition to the rating and discounting, billing and invoicing tests, a standard test call CSR-select was

also run to measure the real-time performance of a typical mix of customer service representative

(CSR) tasks.

Actual BRM administrative tools were traced while performing typical operations of a CSR, and sets of

operations were grouped together to represent typical interactions with customers. Detailed

distributions of the operations are based on real-world data from our large customers.


18

CSR-Select was run with 640 driver threads

Test Ops/sec Resp. time DB1 % Util DB2 % Util APP % Util

CUSTOMER SERVICE

CSR-select 41579.7 15ms 19/22/0 11/6/0 12/9

Table 4. CSR Activity Results


19

Conclusion

The results achieved with Oracle Communications Billing and Revenue Management 7.4 and the

Oracle Exadata Database Machine X2-8 performing a 100-million-subscriber workload demonstrate

unprecedented performance, scalability and low total cost of ownership. The results demonstrate

Oracle Communications Billing and Revenue Management‟s ability to handle the most stringent

workload requirements of Tier 1 service providers while substantially reducing the costs associated

with typical deployments.

Results demonstrate:

Performance of complex rating and discounting of over 50,000 operations/sec, resulting in the

ability to process over 2 billion CDRs in a single 12 hour business day.

The ability to perform complex billing with taxation for over 2,500 subscribers/sec, the equivalent

to processing a bill run for 100 million subscribers in just over 11 hours.

An approximate 10x capacity increase in supported subscribers per Oracle RAC node (up to 50

million subscribers) with Oracle Exadata Database Machine.

A significant reduction in hardware requirements for Oracle Communications BRM production

deployments, as one Oracle Exadata Database Machine replaces the equivalent of 48 conventional

x86 servers and 10 TB of external SAN storage (as compared to previous Oracle Communications

BRM performance tests).

Oracle Communications Billing and Revenue

Management 7.4 and Oracle Exadata X2-8 100

Million Subscriber Performance Performance

Test Whitepaper

April 2011

Author: CGBU Performance Group

Oracle Corporation

World Headquarters

500 Oracle Parkway

Redwood Shores, CA 94065

U.S.A.

Worldwide Inquiries:

Phone: +1.650.506.7000

Fax: +1.650.506.7200

oracle.com

Copyright © 2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other

warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or

fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are

formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any

means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license

and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open

Company, Ltd. 1010

Documents

100 Million Subscriber Performance Test Whitepaper