SAP Sybase Event Streaming Processing

High Performance CEP: The SAP Sybase Event Stream Processor

Technical Product Overview

Speaker’s Name/Department (delete if not needed)

Month 00, 2011

© 2012 SAP AG. All rights reserved. 2

Insight from fast-moving data: Continuous Intelligence

Analyze events as they occur

• Continuous insight

• Immediate response

Rapid application development

• Reduce/eliminate dependence on specialist

programming skills

• Cut implementation/deployment time

• Improve business agility

Non-intrusive deployment

• Event-driven architecture

• Adapt to existing data models


The SAP Sybase Event Stream Processor

• Unlimited number of input

streams

• Incoming data is processed as it

arrives, according to the

business logic defined using high

level authoring tools

• Send output to apps, dashboards

• Capture data in SAP HANA, Sybase IQ,

Sybase RAP for historical analysis,

compliance

?

INPUT

STREAMS

Sensor Data

Transactions

Events

Reporting

Tools

Applications

Studio

(Authoring)

Reference

Data

SAP®

Sybase®

ESP SAP HANA, Sybase IQ

Dashboard

Message

Bus


Sample Architecture of a CEP-based Application:

Real-time P&L

Historical Analytics,

Ad-hoc Analysis,

Compliance

Mark

et

Data

Mid

dle

ware

Reuters

Bloomberg

Nasdaq

…

Sybase

IQ

Order Management Systems

Market Prices

Trading Systems

Live Dashboard

Alerts


Two approaches to CEP

Rules Continuous

Queries


Key Concepts

Input Streams Events arrive on input streams

Derived Streams,

Windows Apply continuous query

operators to one or more

input streams to produce a

new stream

Database Analogy • Streams are like tables

• Events are rows in the table

• Fields in an event message are

columns in the table

Windows can Have State • Retention rules define how many or how

long events are kept

• Opcodes in events can indicate

insert/update/delete and can be

automatically applied to the window


ESP Studio: Both visual and textual authoring

• ANALYST-LEVEL SKILLS (EXCEL, VBA)

• EASY TO UNDERSTAND COMPLEX MODELS

• NO NEED TO LEARN LANGUAGE SYNTAX

• RAPID PROGRAMMING

• EASY TO USE LANGUAGE (CCL)

• MODULAR, PROJECT BASED APPROACH

Visual Dataflow Authoring Language-based Authoring

ESP Studio supports both Visual and Textual authoring

• Visual diagram produces CCL (code-behind approach)

• User can switch back and forth between diagram and language

• Eclipse-based


CCL: Continuous Computation Language SQL-based Event Processing Language

Leverage familiarity and simplicity of SQL

Instead of “snapshot” queries, continuous queries

Filter (WHERE), Aggregate (GROUP BY), Join, Compute

Extensions for event streams

Windows, Patterns

Example CCL Query

Calculation of 10

minute VWAP

CREATE LOCAL WINDOW VWAP10

PRIMARY KEY DEDUCED

AS

SELECT

TICKER.Symbol AS Symbol ,

Sum ( TICKER.Price * TICKER.Size) / Sum ( TICKER.Size)

AS VWAP

FROM TICKER KEEP 10 MINUTES

GROUP BY TICKER.Symbol ;


Key CCL Language Features

• Stateless and stateful elements.

– STREAMS (stateless without keys)

– WINDOWS (stateful with keys)

– Windows push updates and automatically process

opcodes.

• Retention policies on Windows

– Named windows (windows created with “CREATE

WINDOW..”)

– Unnamed windows (FROM A KEEP 5 ROWS)

– Time and Count based retention policies

– Custom windows with EventCache

• Parameters, Variables

– Variables can maintain state

• Full support for JOINS

– ANSI JOIN Syntax

– Stream-Window Joins and Window-Window joins

• Filter

• Aggregate

– Group By, Group Filter, Group Order

• Pattern Matching

– Watch for specific sequence of events within time

interval

– Within or across streams

– Pattern can include missing events

• Transform/Project

– Change event schema, compute new fields


SPLASH scripting adds extensibility to CCL

SPLASH (Stream Processing LAnguage SHell) was introduced in Aleri to provide

extensibility and overcome the limitations of SQL

It complements CCL, providing the ability to write custom functions and stream/window

operators using a simple, procedural language that is optimized for stream processing

Syntax is similar to C and Java, it’s spirit is closer to little languages like AWK or Perl

ESP 5.0 has full support for SPLASH

For more info see the SPLASH Tutorial on sybase.com (in the Aleri Product Documentation/Manuals)

if (temperature < 0) {

display := 'below zero';

} else if (temperature = 0) {

display := 'zero';

} else {

display := 'above zero';

}

Control Structures: if, while, switch

Data Structures:

- Variables - Vectors - Dictionaries - Event Cache

Flex Streams:

- Custom event “handlers” written in SPLASH


Modularity

Benefits:

Facilitates re-use

Simplifies team development

Improves manageability of

large projects

CREATE MODULE • Defines a module consisting of

windows and streams

• Define inputs, outputs,

parameters

IMPORT • Equiv. to #include – include

external .ccl file(s)

• Beyond modules, re-use

declarations, etc

LOAD MODULE • At load point, define bindings

and parameters

• Same module can be loaded

multiple times


Support for External function libraries (UDF)

• Call out to external function libraries

• Support for C and Java

• Can be used to supplement built-in functions

• Can also be used to “trigger” external actions


ESP Studio: RUN/TEST Tools

Single click to Run a project

Rich Suite of Testing, Tuning and Debugging tools

Stream viewer

Record/playback

Manual event input

Performance Monitor

Event Tracer

Debugger with Breakpoints

Multi-server control

View/monitor projects running on multiple servers and clusters


ESP Advanced Cluster Architecture

Controller

Manager

Manager

Manager

Manager

Controller Controller

Private

Cloud

Project Project

Project Project Project

Project Project


Project Project



ESP Cluster Architecture Key Features

Manager • Equi-peer management of cluster.

• Program deployment and lifecycle control.

• Program load balancing at deployment.

• Launcher failure detection.

• Container failure detection.

• Program failover.

• Publishes lifecycle updates for Managers, Controllers and Programs.

• Name resolution.

Controller • Acts as a daemon for Managers to launch Container processes.

• Allows for launching of any configured Program type.

Container • Process that hosts a Program (i.e. – esp_server).


Choice of high availability configurations

Configurable to your level of

availability and flexibility needs

Instant failover to "hot" spare

• highest degree of availability

• recovery in seconds; no data loss

Shared resources "warm" spares

• flexible configurations

• recovery in minutes

• parallel & pipeline configurations

All managed by fault tolerant

cluster manager

FT Cluster

Manager

Parallel & Pipelined

Shared Warm Spares

Shared

State

Synchronized

Hot Spare


ADAPTERS

Standard

(Built-in)

• JMS, MQ, TIBCO

• Sockets

• Databases

• Files

• MS Excel

• Email (out)

Optional

(Add-on)

• Market Data • Reuters, NYSE Tech

• FIX

• Rep Server

Custom

(SDK)

• C/C++

• Java

• .NET


Integration/Connectivity

In-Process Adapters

• Entirely under the control of the ESP server.

• Performance advantages

• No hot-standby at the adapter level

• C/C++ only

External Adapters

• Independent of ESP Server

• Uses ESP pub/sub API

• TCP socket connectivity to ESP Server

• Auto-failover in hot-standby configurations

• C/C++, Java, .NET

Application with Native ESP

Support

• Embed ESP pub/sub API directly in producer or consumer applications


USING Sybase Replication Server for Change Capture Real-time change capture: turn database transactions into event streams

Replication Agent

Replication Server

Database

SAP Sybase ESP

Transactions (insert, update,

delete) Real-time Event

Stream


ESP Project

High Level ESP Architecture

Query Engine

Launcher

Cluster Manager

Se

cu

rity

(K

erb

ero

s, P

AM

, S

SL

)

Data Source/Destination

Client

Log File

Security

Distributed Cache SDK Interface URI Resolution

Lifecycle/Status

SD

K -

.N

ET

\ JA

VA

\C+

+

Adapter

Studio

Ga

tew

ay

Window Stores

Adapter Framework

Load Balancing Failover

Co

mm

an

d &

Co

ntr

ol

Compiler

Tools SQ

L Q

ue

ry

Internal

Adapter

s

SDK

External

Adapter

s


What sets SAP Sybase ESP apart

• Performance

− Scalable for extreme throughput

− Consistent low-latency

− Only CEP vendor to submit to STAC

benchmarking

• SPLASH scripting

− Overcome limitations of SQL

− Flexibility and productivity

• Authoring flexibility

− Language or UI/IDE

• “Private Cloud” Architecture

− Scalable, Dynamic

• State Management

− Unique ability to automatically apply

incoming events as

inserts/updates/deletes to a table

− Simplifies modeling

− Extremely efficient

• Dynamic

− Add continuous queries to live system

• High Availability

− Hot-Hot with Auto-Failover

• Advanced Subscriptions

− Initial state followed by updates

− Subscriptions with predicates


Source: STAC Research Sep 08 – conducted against Aleri 3.1; 5.0 performance is

similar

The only CEP vendor to publish STAC benchmarks

High Performance

STAC Certified measurement of throughput and latency

• Linear scaling with addition of more cores:

180,000 messages/sec on 16 cores

300,000 messages/sec on 24 cores

• Consistent/predicable low latency.

Only minor increase in latency from 20,000 to 300,000 messages/sec:

Mean: 1.296 - 1.568 ms

Medium: 1.177 - 1.502 ms

Std Dev: 0.567 - 1.104 ms

Min: 0.091 – 0.115 ms

99th percentile: 2.479 – 3.165 ms

PANOPTICON EX REAL-TIME DATA VISUALIZATION


Visually Rich Real-time Dashboards with

Historical Drill-down and Analysis


Panopticon EX Features

Visual Data Analysis

Real-time streaming from Sybase ESP

Historic Time Series data from Sybase RAP or IQ

Drag-and-Drop Designer

Rich palette of charting objects

Simple configuration

No coding required

Multi-dimensional Analysis (in-memory OLAP)

Delivered via Desktop (thick client) or Web (browser)

SDK provides APIs for ability to customize, embed


Panopticon EX for ESP

Native integration with Sybase

ESP, RAP, and IQ

True real-time dashboards,

subscribing to ESP streaming

output

Seamless drill-down into

historical time series data in

Sybase RAP

Panopticon EX

Reduces Insight

Latency

It’s Not Just the Time it

Takes to Get the Data

It’s also the time it

takes to understand the

data

Rich Palette of

Visualization Tools

Designed to Make it

Easier to Understand the

Data

Thank you

Contact information:

F name MI. L name

Title

Address

Phone number

Technology

SAP Sybase Event Streaming Processing