BPM – Speed Dialing the Enterprise

October 9th, 2014

BPM Forum

Agenda

Business Case Scenario

Extraction Enhanced XML

Transformation Big Memory

Loading Batching

Monitoring Performance

Profiling

ETL Tool

Ability to load large amounts of data from XML, flatfiles, CSV

Ability to transform large amounts of data in-memory

Ability to write transformation logic in the tool

Ability to add out-of-box data validity checks and profiling

Easy administration

Agenda

Business Case Scenario

Extraction Enhanced XML

Transformation Big Memory

Loading Batching

Monitoring Performance

Profiling

Big XML Processing

Large XML (> 1 GB) can be parsed in-memory by caching portions on Ehcache

Multiple files can be processed simultaneously

Ability to optionally use BigMemory to keep parsed portions on disk instead of Heap

Simultaneously process multiple large XML documents

Ehcache

BigMemory

Integration Server

Large XML files

Agenda

Business Case Scenario

Extraction Enhanced XML

Transformation Big Memory

Loading Batching

Monitoring Performance

Profiling

Big MemoryCapabilities

BigMemoryTerabytes of data in memory, 1000x faster than disk Scale up on a single machine and out across your machinesFull fault-tolerance – 99.999% availability

Benefits

BigMemory

In-memory real-time access to

information

Predictable latency at extreme

scale

Simple to use on commodity

hardware

Simple and Predictable Scaling

Simple & predictable scaling

Universal Access

Continuous UptimeCore Platform Enhancements

BigMemory

Ability to scale to TB of memory with

improved fault tolerance

Support for extended and hybrid storage

Leverage SSD & Flash technologies

Operational flexibility

Universal Access

Simple & predictable scaling

Universal Access

Continuous UptimeCore Platform Enhancements

BigMemory

Broadening the applicability of Terracotta platform

Support for other application development environments/languages Support .NET/C# and C++ clients

BigMemory SQL Support integration via lingua

franca of data access world - SQL

Continous Uptime

Simple & predictable scaling

Universal Access

Continuous UptimeCore Platform Enhancements

BigMemory

Providing continuous availability of data with zero downtime across different deployment topologies

Purpose-built support for data replication and recovery across WAN

Ensuring Business Continuity across (multi) data centers

Core Platform Enhancements

Simple & predictable scaling

Universal Access

Continuous UptimeCore Platform Enhancements

BigMemory

Platform improvements to support increased performance and operational efficiency

Accessed through APIs

Search Performance –Optimized Pagination

Full Data lifecycle support when no clients connected to BigMemory Server

Agenda

Business Case Scenario

Extraction Enhanced XML

Transformation Big Memory

Loading Batching

Monitoring Performance

Profiling

Fast / Bulk writes for large sets

Batch adapters

Transaction management

Connection pooling

Database procedure calls

Table listeners

Agenda

Business Case Scenario

Extraction Enhanced XML

Transformation Big Memory

Loading Batching

Monitoring Performance

Profiling

Monitoring – Process Visibilty

Triggering and scheduling

On Demand

Scheduled

Event driven

• HTTP(S)• FTP(S)• SOAP• JMS• Scheduler• SMTP• BPM Event• Java Client• C/C++ Client• .NET Client• Adapter• FLOW

Performance

Read file from disk 2 ms

Iterate through 2 MB XML file 1 ms

Iterate through 96 MB flat file 2 ms

Iterate through 1-2 MB flat file 1 ms

Transformations less than 1ms - 2ms

Acces to Big Memory 1 ms

Batch inserts in 5 tables, 2000 rows in total 600 - 800 ms

Batch insert in 2 tables, 6000 rows in total 150 ms

Process 1 XML for for AT end-to-end 10 - 25 sec

Process 240 XML files for AT end-to-end on 1 server 2-4 min

Process 90 MB flatfile, 270.000 lines for DE end-to-end on 1 server, 540.000 rows 10 min

Process splitted 90 MB flatfile (200 files), 270.000 lines for DE end-to-end on 1 server, 540.000 rows 3 - 4 min