Upload
amazon-web-services
View
1.149
Download
3
Embed Size (px)
DESCRIPTION
Companies are dealing with increasingly large data sets and looking for ways to significantly improve the scale and cost of Big Data analysis with AWS. This hands-on session shows you how you can achieve that. With hundreds of pre-built connectors, you will learn how to get your on-premise and cloud data into Redshift in minutes, not days, and at a significantly reduced costs using Informatica Cloud Integration. With fully certified support for large scale RDS deployments and Informatica’s Vibe Data Stream solution for automated streaming data collection for Kinesis, Informatica offers a comprehensive cloud integration solution for Big Data analytics with AWS. The ability to seamlessly migrate Informatica’s PowerCenter to Amazon Cloud (EC2) offers customers a Cloud migration path, with even higher performance and lower costs.
Citation preview
Power Big Data Analytics with Informatica Cloud Integration
Ron Lunasin, Informatica Cloud Product Management
Ajay Gandhi, Informatica Cloud Product Marketing
Alan Lundberg, Informatica Vibe Data Stream Marketing
March 2014
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
2
Leader in Cloud and Hybrid IT Integration
3
Gartner MQ for Integration Platform as a Service, Jan 2014
The Forrester Wave: Hybrid2
Integration, Q1 2014
Hybrid IT Architecture is the New Normal
Informatica Cloud PlatformCloud Integration and Data Management
Cloud Data
Quality
Cloud Master
Data
Management
Cloud
Process
Automation
Cleanse and
De-Dupe
Visualize
RelationshipsImprove User
Experience
Leverage
Existing Systems
Cloud
Integration
Secure
Sandbox
Cloud Test
Data
Management
You Need More Than Just Integration
Hundreds of Connectors
• 100% Cloud
• Developer and App User
Collaboration
• Productivity for Advanced
Integration Use Cases
• Vibe Integration Packages
Visual Productivity for Advanced Cloud and Hybrid IntegrationInformatica Cloud Designer
Introducing Vibe Integration Packages (VIP’s)Redefining Hybrid IT Integration Agility
VIPs=Pre-built parameterized integration workflows
Built by developers for app users and other developers/partners
App users configure VIPs using wizards to build custom integrations
VIPs work with Cloud and PowerCenter
VIPs can be distributed via Informatica Marketplace
VIPs are easily embedded into 3rd-party apps via APIs
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
9
Common Customer Use Cases
• Reduce costs by extending
DW rather than adding HW
• Migrate completely from
existing DW systems
• Respond faster to
business; provision in
minutes
• Improve performance by an
order of magnitude
• Make more data available
for analysis
• Access business data via
standard reporting tools
• Add analytic functionality to
applications
• Scale DW capacity as
demand grows
• Reduce HW & SW costs by
an order of magnitude
Traditional Enterprise DW Companies with Big Data SaaS Companies
Using the Cloud Isn’t an “All or Nothing” Choice
11
Integrating AWS With Existing On-Premises IT
12
Use Cloud To Make On-Premises Apps Better
13
Backup
Analytics
Cloud Apps That Integrate With On-Premises Apps
14
AWS serves
application
content & data
Integration to data
centers for
financial
transactions
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
15
Support for AWS Database and Big Data Services
16
Pre-built Cloud & PowerCenter
Connectors for RDS and Redshift
Vibe Data Streaming
for Kinesis
InformaticaCloud.com/Amazon-Redshift
2
Informatica Cloud Architecture Overview- Redshift
4Secure
Agent
Your Company or VPC
Amazon
Redshift
31
Amazon
RDS
Map Once. Deploy Anywhere.
ON PREMISE HADOOP 3rd PARTY
APPLICATIONS
CLOUD
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
19
©2013 Informatica. Proprietary and Confidential 20
Redshift Upsert – Manual Coding Way
Extract the data from source1
Put into flat files and compress2
3 Transfer Compressed Files To S3
4 Wait for S3 Consistency
5Copy Data From S3 Into Staging Table6Inner Join With Target Table To Delete Rows To Be Updated
Insert Updated Rows From Staging Table
7
Delete Staging Table
8
9
Delete Files From S3
Create Staging Table in Redshift
10
Or, Do It In 3 Simple Steps…
Redshift Upsert – Informatica Cloud Way
1
2
3
Choose Upsert Operation
Map Your Fields
Run Or Schedule!
Informatica Cloud Amazon Redshift Architecture
Firewall
Informatica Cloud Secure Agent
Metadata Mappings
Build mapping and execute job
1
1Retrieve Account Data2
2
3 Put Account Data into Flat File
4 Transfer compressed Flat File to S3
5 Initiate copy from S3
6 Load data into Amazon Redshift
6
3
54
DEMO!
23
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
24
2626
Data / Sensor Diversity…
27
How to make sense of it all…
Streaming Collection: Vibe Data Stream (VDS)
28
Vib
e D
ata
Str
ea
m B
us
Pu
blis
h /
Su
bscrib
e
Leverage High Performance
Messaging Infrastructure.
Publish with Ultra Messaging
for global distribution without
additional staging or landing.
Cloudera,
Pivotal,
Hortonworks,
MapR
Targets
Web Servers,
Operations
Monitors, rsyslog,
SLF4J, etc.
Handhelds, Smart
Meters, etc.
Discrete Data
Messages
Sources
VDS
Node
VDS
Node
VDS
Node
VDS
Node
VDS
Node
Management
and Monitoring
Internet of Things,
Sensor Data
VDS
Node
Real Time
Analysis, Stream
Processing
No SQL
Databases:
HBASE,
Cassandara,
Riak, MongoDB
29
Transactions,
OLTP, OLAP
Social Media, Web Logs
Machine Device, Scientific
Documents and Emails
Vibe Data
Stream
Vibe Data
Stream
Vibe Data
Stream
AWS Kinesis + Informatica – Framework for Deeper Insight
30
Level 2Reduce time-to-information & time-to-decision
Operational pattern matching, alerts, Real-time analytics
Level 3 Create Visibility & Insight to Understand the Business Impact
Operational KPIs, Alignment of IT & Business, Drill down
Service Delivery Applications
OSS / BSS Applications
Network Applications
Level 1Instrument for Problems & Opportunities
Detection, response, correlation & extrapolation of trends
Vibe Data
Stream
Use Cases – Solving the Difficult Problems
31
Detect
Patterns
Exception
Monitoring
Process
Monitoring
• Deviations from norm
(Monitoring, Fraud, Error)
• Trending up/down to exceed
a threshold
• SLA monitoring
• 3 events within 5 milliseconds
• A then B then C occurs
• Geospatial processing
• Are process workflows
operating properly?
• Are manual processes
completed on time?
• Detect Missing Work and
Queued Work
Informatica in “Lambda Architectures”
Adapted from “Runaway Complexity in Big Data”, Nathan Marz, Sept. 25/2012
Transactions, OLTP, OLAP
Social Media, Web Logs
Machine Device, Scientific
Documents and Emails
Batch LayerBatch
View
Big Data Analytics + Real Time Streams
Speed LayerReal Time
View
Serving LayerMerged
View
• Stream Processing
• Filter / Classify
• Correlate
• Solution approach that complements and augments
traditional BI and reporting
• Combines approaches and techniques from various
technology areas, including:
• End-to-end and comprehensive data Integration
• Event processing and event-driven architectures
• Rapid data provisioning via a common data access layer
• Access to LIVE data in operational systems
• Access to all types of data including unstructured data
AWS Kinesis + Informatica = “Real Time Operational Intelligence”
Sense
Reason
Respond
Visualize
Architectural Implications
Batch processing
Data structured, homogenous High Volume and variety
Distributed SystemsCentralized Database-centric
Client Server Systems
Prioritize Modeling
events as enterprise
objects / assets
Real Time
Yesterday Today
Events treated as
2nd class citizens
Informatica User Interactions
Developers / OEMs
Developer IDE
TemplatesAnalyst
MyRulePoint Portal
SDKs
User Tool
Business User
36
Streaming Collection: Vibe Data Stream Dev Benefits
• Central Monitoring Console
for Deployment
• Fault Tolerant
• High Availability
• Vertical &
Horizontal
Scaling
• Ease of Configuration
Streaming Collection: Topology View
37
Informatica on Amazon Kinesis
• Solving tough infrastructure problems..
• .. So you stay focused on solving tough business problems..
• Coming soon on Amazon..
• Stay tuned..
• Drop an email with your use cases, needs to me at
38
Today’s Agenda
• Informatica Cloud Integration and Data Management
• Common Customer Use Cases
• Informatica for AWS DB and Big Data Services
• Informatica Cloud + Redshift Demonstration
• Informatica Vibe Data Stream for Kinesis
• Next Steps and Q&A
39
Next Steps
• Visit us at Booth# 107 to see more demos!
• Get started with Informatica Cloud
• InformaticaCloud.com
• Learn more about our Redshift Connector
• InformaticaCloud.com/Amazon-Redshift
40
Q & A
41
• Ron Lunasin, Informatica Cloud Product Management
• Ajay Gandhi, Informatica Cloud Product Marketing
• Alan Lundberg, Informatica Vibe Data Stream Marketing
@infacloudInformaticaCloud.com