Upload
microsoft-private-cloud
View
2.499
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
SQL Server 2008 R2 StreamInsight Overview
NameTitle
Microsoft Corporation
Information Platform Vision
Empowered ITPervasive Insight
Dynamic
DevelopmentMISSION CRITICAL PLATFORM
Desktop & Mobile CloudServer & Datacenter
Scalable relational database platform
Consistent, familiar model & tools
Self-managed, highly available cloud services
MPP support for 10s to 100s TB DW
Highly scalable appliances
Seamless integration with Microsoft BI
A Comprehensive Platform
Managed Self-Service BI
Multi-server management
Virtualization & Live Migration
Enterprise-level security, scalability
High-scale, complex event processing
Data consistency across heterogeneous systems
MPP support for 10s-100s TB DW
Highly scalable appliancesSeamless BI Integration
Scalable relational database platformConsistent, familiar model & tools
Self-managed, highly available cloud service
TRUSTED, SCALABLE PLATFORM
IT & DEVELOPER EFFICIENCY
MANAGED SELF-SERVICE BI
Multi-server management
Virtualization & Live Migration
Accelerated development& deployment
Self-service analytics
Self-service reporting
Streamlined collaboration& management
COMPREHENSIVE PLATFORM FOR IT VALUE
What is CEP?
6
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec Tens of thousands of events/sec or more
request
response
Eventoutput streaminput
stream
Relational Database Applications
Financial trading Applications
Scenarios for Event Processing
7
Aggregate Data Rate (Events/sec.)
Latency
0 10 100 1000 10000 100000 ~1million
Months
Days
hours
Minutes
Seconds
100 ms
< 1ms
Operational Analytics Applications, e.g., Logistics, etc.
Manufacturing ApplicationsMonitoring Applications
CEP Target Scenarios
Data Warehousing Applications
Web Analytics Applications
Example CEP Scenarios
8
Dat
a St
ream
Stream Data Store & Archive
Event Processing Engine
Dat
a St
ream
Asset Specs & Parameters
Power, Utilities:• Energy consumption• Outages• Smart grids• 100,000 events/sec
Visual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentation Algorithmic tradingProactive condition-based maintenance
Web Analytics:• Click-stream data• Online customer
behavior• Page layout• 100,000 events /sec
Manufacturing:• Sensor on plant floor• React through device
controllers• Aggregated data • 10,000 events/sec
• Threshold queries• Event correlation from
multiple sources• Pattern queries
Lookup
Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
Financial Services:• Stock & news feeds• Algorithmic trading• Patterns over time• Super-low latency• 100,000 events /sec
Industry trends• Data acquisition
costs are negligible
• Raw storage costs are small and continue to decrease
• Processing costs are non-negligible
• Data loading costs continue to be significant
Manage business via KPI-triggered
actions
Mine historical dataDevise new KPIs
Monitor KPIsRecord raw
data (history)
Virtuous Cycle: Monitor, Manage, Mine
9
CEP advantage• Process data
incrementally, i.e., while it is in flight
• Avoid loading while still doing the processing you want
• Seamless querying for monitoring, managing and mining
Microsoft’s CEP Solution
10
Data Sources, Operations, Assets, Feeds, Sensors, Devices
Monitor &
Record
Operational Data Store & Archive
CEP Engine
f(x) g(y)
CEP Enginef(x) f'(x)
g(y) h(x,y)
History
Deploy
Resultsf'(x) h(x,y)
Manage&
Benefit
Mine&
Design
Input Data Streams
Input Data Streams
OutputData Streams
CEP Deployment Alternatives
Data Sources
Aggregation & Correlation
11
CEP
CEP
CEP
CEP for lightweight processing and filtering
CEP for aggregation and correlation of in-flight events
CEP for complex analytics including historical data
Event processing engines are deployed at multiple places on different scales• At the edge – close to the data source• In the mid-tier – consolidate related data sources, • In the data center – historical archive, mining, large scale correlation
Devices
SensorsWeb servers
Feeds
CEP
CEP CEP
CEP
CEP CEP
CEP CEP CEP
Complex Analytics &Mining
MSFT CEP Engine
Reference data
APPLICATION
Input A
dapte
rs
EVENT SOURCES
Devices, Sensors
Web servers
Event stores & Databases
Stock tickers & News feeds
Outp
ut A
dapte
rs
EVENT TARGETS
Event stores & Databases
Pagers & Monitoring
devices
KPI Dashboards, SharePoint UI
Trading stations
Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern
specifications with low latency
KEY SOLUTION REQUIREMENTS
• Event driven• Low latency processing (< milliseconds)• High volume (>100,000 events /second)• Correlated events from multiple sources• Time interval oriented standing queries
Overview: CEP Platform from Microsoft
Bloomberg.com
PLATFORM FOR KEY MARKET OPPORTUNITIES• Manufacturing: Process monitoring i.e. scheduling, control, inventory • Financial: algorithmic trading, compliance monitoring, fraud detection• Web: Click stream analysis ,ad-delivery model• Utility: Grid monitoring • Healthcare: Patient Monitoring• Networking: traffic monitoring ,security intrusion detection• Telecom: Network monitoring• Oil and Gas: Geological data analysis, process control
Events• Events expose different temporal characteristics
−Point in time events−Interval events with fixed duration−Interval events with initially unknown duration• Rich payloads capture all properties of an event
t1 t4t3t2 t5Time
Payl
oad/
val
ue
ab
c de
Event Types
• Events in Microsoft’s CEP platform use the .NET type system• Events are structured and can have multiple fields• Fields are typed using the .NET framework types • CEP engine provisioned timestamp fields capture all the
different temporal event characteristics• Event sources populate time stamp fields
Timestamps/Metadata
LongpumpID
StringType
StringLocation
Doubleflow
Doublepressure
… … … … … …
Event Streams
• A stream is a possibly infinite sequence of events−Insertions of new events−Changes to event durations−Examples: time series sensor data, log entries
• Stream characteristics:−Event/data arrival patterns
‐ Steady rate with end-of-stream indication Examples: files, tables
‐ Intermittent, random, or in burstsExample: retail scanners, web, weather telemetry
−Out of order events‐ Order of arrival of events does not match the order of their
application timestamps
• Takeaway: CEP engine does the heavy lifting for you when dealing with out-of-order data
15
Event Stream Adapters
• Design time specification of an adapter‐ for a particular event type and particular source/sink‐ methods to handle event and stream behavior, and‐ properties to indicate adapter features to the engine• Adapter features
16
Event properties Adapter template for point in time and other event types
Stream properties Event arrival properties for out-of-order, arrival rate etc.
Payload specification .NET scalar types for payload fields
Typed vs. generic Adapter templates for single type vs. generic adapters
Support for optimization
Predicate/project pushdown property and supporting code
Problem Feature
Core CEP Query Engine
• Hosts “standing queries”− Operators consume and
produce streams
− Queries are composable
− Query results are computed incrementally
17
• Query instance management:− Submit, start, stop
− Runtime statistics
• Takeaway: CEP engine does the heavy lifting for you when processing temporal event data
CEP Engine
Output Adapters
Input Adapters
Event
Standing Queries
Event
Event
Event
Event
Event
Event
Event
Event
Typical CEP Queries
• Typical CEP queries require combination of functionality− Complex type describes event properties− Calculations introduce additional event properties− Grouping by one or more event properties− Aggregation for each event group over a pre-defined
period of time, typically a window− Multiple event groups monitored by the same query− Correlate event streams − Check for absence of activity with a data source− Enrich events with reference data− Collection of assets may change over time
• We want to make writing and maintaining those queries easy or even effortless
CEP Query Features
• Operators over streams− Calculations (PROJECT)− Correlation of streams from different data sources (JOIN)− Check for absence of activity with a data source (EXISTS)− Selection of events from streams (FILTER)− Stream partitioning (GROUP & APPLY)− Aggregation (SUM, COUNT, …) − Ranking and heavy hitters (TOP-K)− Temporal operations: hopping window, sliding window
• Extensibility – to add new domain-specific operators
• Queries are written over specific event types− They can be evaluated on all data sources with the same
event type
• Support for streaming data, reference data (lookup), and historical data (replay)
LINQ Query Examples
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3group e3 by e3.i into SubStreamsfrom s4 in SubStreamsfrom e4 in s4.SlidingWindow(FiveMinutes,ThreeSeconds)select new {
pl = new MyNewPayload(e4.i, e4.f)};
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1join e2 in MyStream2 e1.ID equals e2.IDwhere e1.f2 = “foo”select new { e1.f1, e2.f4 };
Join
FilterProject
Grouping
Window
Extensibility
• Built-in operators do not cover all functionality−Need for domain-specific extensions−Integrate with functionality available in existing libraries• Support for extensions in the CEP platform:
−User-defined operators, functions, aggregates−Code written in .NET, deployed as .NET assembly−Query operators can refer to functionality of the assembly−LINQ queries can easily refer to user-defined functionality• Temporal snap-shot operator framework
−Interface to implement user-defined operators−Manages operator state and snapshot changes−Framework does the heavy lifting to deal with intricate
temporal behavior such as out-of-order events
Developer Experience with the CEP Platform• Goals
− Easy to use and highly productive
− Most comprehensive toolset− Supports various deployment
models for CEP applications‐ Embedded applications‐ Server applications
− Compelling story for domain-specific extensions
• CEP development environment− Builds on top of .NET− Microsoft Visual Studio as IDE− Applications written in C#− Queries written in LINQ
22
LINQ:
var queryFilter = from c in TestEventStreamwhere c.Field1 > 1select c;
Managing CEP Data Sources & Queries• Typical CEP Application
‒ Multiple different data sources and adapters‒ Several queries• Intellectual property is in the queries• Protect investment in queries over time
‒ Re-use existing queries as newer version of an asset, device or data source become available
‒ Re-use existing queries for new data source types‒ Seamlessly switch between live data feeds and historical
data
Query Binding
• Binding a query with data sources yields a query instance• Configure multiple query instances for a query• Similar to data independence: the same query can be bound to
different data sources as long as they deliver the required event type
Data Sources Analytics & Queries
QT1
CEP EngineO
utput Adapters
Input Adapters
Event
Standing Queries
Event
Event
Event
Event
Event
AT2
AT1
Q1
Q1’
AT3Q1’’
CEP Query Configuration
• Query binding:−Coupling adapters with a query−Event types required by the
query need to be matched by types delivered by adapters
• Query can be re-used for all data sources of the same type• No changes to query
necessary for re-use
QueryTemplate
Output Adapter
Join
Proj.
Type Type
Type
InputAdapter
Type
Type
InputAdapter
Type
Query
Recap: CEP Platform from Microsoft
26
CEP Engine
Output Adapters
Input Adapters
Event
Standing Queries
Event sources Event targets
`
Event
Event
Event
Event
Event
Event
C_ID
C_NAME
C_ZIP
Event
Event
Static reference data
CEP Application Development
Development experience with .NET, C#, LINQ and Visual Studio 2008
CEP platform from Microsoft to build event-driven applications
Event-driven applications are fundamentally different from
traditional database applications: queries are
continuous, consume and produce streams, and compute
results incrementally
Flexible adapter SDK with high performance to
connect to different event sources and sinks
The CEP platform does the heavy lifting for you to deal
with temporal characteristics of event stream data
CEP Deployment Scenarios
CEP Engine
Reference data
Custom CEP Application
Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application
ISV Application with CEP Engine
CEP Engine
Reference data
Scenario 4: Operational Intelligence w/ CEP
MadisonETL Pipeline with CEP engine
CEP EngineCEP CEP
CEP CEP
KPIs KPI mining
Scenario 3: CEP Enabled Device
Device with Embedded CEP Engine
CEP Engine
.NET, C#, LINQ
CEP Platform Roadmap
• Focus on −Custom development platform for CEP applications−CEP platform for Microsoft partners
• TAP (Technology Adopter Program) will launch in July 2009• Product availability targeted for 2010
timeJuly 2009
CTP2TAP program
2010
General Availability
CTP3TAP continues
Late 2009
A not-for-profit, user-run association to enable the SQL Server community to connect, share, and learn
Get ongoing SQL Server learning from MVPs, top experts, and real-world users:
• Training webcasts, technical education, worldwide event info, and moreSign up for your FREE PASS membership today: www.sqlpass.org
• Go to local PASS Chapter meetings Visit the PASS Chapter booth to learn about upcoming local events, or visit www.sqlpass.org/PASSChapters
• No local Chapter? Start a Chapter with support and resources from PASS
www.sqlpass.org/getchapterinfo
• Attend PASS Summit 2010World's largest and best conference for SQL Server and BI technical education www.sqlpass.org/na2010 Follow / Join us online: Twitter @sqlpass #sqlpass, LinkedIn group, Facebook
group
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.