Upload
aymanmabdelsalam
View
212
Download
0
Embed Size (px)
Citation preview
8/11/2019 WP-Leveraging a Staging Server
1/12
Best Practices in Leveraging a StagingArea for SaaS-to-Enterprise Integration
David S. Linthicum
Introduction
SaaS-to-enterprise integration requires that a number of architectural calls are made around
the process of integration, including the frequency of data movement, the mechanism
to maintain data integrity, and the underlying integration technology and location of thattechnology. When approaching SaaS-to-enterprise integration, there are many architectural
options. These options include the direct migration of information from source to target, or
the use of a staging area to process information more thoroughly from source to target.
A staging area is a relational database that exists within the firewall. Staging areas are a place
to store temporary data for import, and thus allow you to check and alter this data before
committing it to the final target database.
Clearly, the use of a staging area as an approach to integrating SaaS-based systems to
enterprise-based systems provides a number of advantages, including the ability to better
maintain data integrity and data quality, as well as providing better control over the data.
However, a staging area approach to integration requires that the data integration architect
understand both the core notions and best practices of SaaS-to-enterprise integration.
Moreover, the architect must leverage the right technology.
In this paper well look at the notion of leveraging a staging area for SaaS-to-enterprise
integration, which includes suggesting a methodology that leverages best practices. Well
examine the value of this approach, as well as provide information around the proper
enabling technology.
Integration Patterns
Those who deal with SaaS-to-enterprise integration need to deal with a number of issues,
including semantic mediation, data quality, interface mediation, routing, filtering, and other
logical operations on the data moving from SaaS-to-enterprise, or enterprise-to-SaaS. The
idea is to make the data as useful as possible when it reaches the target system, including
any business intelligence operations that need to occur and/or any business operations on
that data, on-premise or SaaS-resident.
WHITE PAPER
8/11/2019 WP-Leveraging a Staging Server
2/12
Thus, in order to figure out the best data integration approach, you first need to understand
your own integration requirements and your existing application and data infrastructure. This
includes the usage of the data when placed on the target system, and how that maps from
the source. To that end, there are two basic patterns emerging around data integration for
SaaS-to-Enterprises. They include Direct integration and Staging integration.
Direct
Direct integration, as depicted in Figure 1, is really about moving information from one
data source and data schema to another, and translating the differences in semantics from
the source to the target system. While we depict databases here, this could as easily be
data bound to enterprise applications, such as ERP or CRM, or data bound to SaaS-based
applications, such as Salesforce.com.
The idea is to extract the information found in the source system, such as customer
information, and place it in the target system using whatever native schema the target system
uses, and translate the differences in schema and content, as needed and on the fly. Thisallows information to flow from one system to another without having to change either the
source or the target system. The Direct integration approach accounts for the differences in
schema using a translation and routing engine that exists between two systems. This is more
along the lines of traditional integration, as has been defined for years.
[ 2 ]
Figure 1: Direct integration simply moves information from source to target (e.g., SaaS-to-Enterprise), translat-
ing and transforming the data on the fly. While this is okay for simple data integration needs, it falls short
when the integration requirements are more complex.
8/11/2019 WP-Leveraging a Staging Server
3/12
Direct integration is typically leveraged when quick and dirty information integration needs
to occur on operational-type data, usually from one system to another. For instance, updating
the on-demand CRM system with customer data entered in the sales order entry system,
which is on-premise.
Direct integration is characterized by:
Limitations in supporting complex data operations, generally doing very simple data
operations such as transformation and routing.
More ne grained, meaning the data sets are usually small and repeat.
Operational focused, or when data needs to ow between systems to support ongoing
light transactions.
Staging
In contrast to direct integration, a staging approaching to integration supports more complex
and valuable data integration operations, including support for many large data sets and data
operations that are more complex and of higher value. Using a staging area, or a temporary
location where the data from the source system or systems is replicated, provides a logical
location to perform complex operations on the data that would be difficult if not impossible
to do when using direct integration approaches.
An example would be extracting data from multiple enterprise data stores, manipulating the
data as to structure and content, performing complex operations such as data replication,
data aggregation, and data cleansing, and then posting the data to a SaaS-based application
such as Salesforce.com. Or, perhaps doing the same type of data integration in reverse order,
from SaaS to the enterprise.
Leveraging a staging area for integration is character ized by:
The ability to perform more complex operations on data, including complete
transformation of semantics and the data content using any number of dimensions since,
in essence, you operate on an intermediary database that you control completely.
The ability to leverage more coarse grained and complex data sets that may not
always repeat.
Informational focused, supporting valuable information externalization approaches,
including business intelligence.
More exibility around business cycles, data processing cycles, widely disbursed systems,
and hardware and network limitations, where it may not be feasible to extract all
operational databases at the same time.
The ability to better support complex database functions, including replication, cleansing,
and aggregation.
[ 3
8/11/2019 WP-Leveraging a Staging Server
4/12
As Figure 2 depicts, the use of a staging area is really about the gathering of information from
many different sources that leverage different structures, data content, and operations on
that data. For instance, within some enterprise systems you maybe able to extract sales data
on a daily basis. However, when considering financial data, daily extracts are not suitable or
not possible, since financial data typically requires a month-end reconciliation process. Youmay find similar things in geographically disbursed data, such as extracting sales data from
London and New York at the same time, when the data is posted at different times of the
day due to time zone differences.
[ 4 ]
Figure 2: The use of a staging area is really about gathering the information from many different sources and
creating a single, unified view of the data for integration to a SaaS-based systems , such as Salesforce.com.
8/11/2019 WP-Leveraging a Staging Server
5/12
Defining the Value
Considering the advantages of leveraging staging for SaaS-to-enterprise integration, there are
many business advantages over more traditional integration approaches, including:
The cost advantage of keeping volatility within a single domain.
The value of agility, as related to the ability for the integration patterns to change as
the needs of the business change.
The value of customized complex views for specic systems and users.
The value of leveraging clean, reliable data.
The cost advantage of keeping volatility within a single domain refers to the fact that, since
a staging area is where all of the data from different sources is temporary housed, using
different formats and different time sensitivity, then any changes made to the source and/or
target systems are handled within the staging area. This is especially useful when considering
enterprise-to-SaaS integration, since the dynamic nature of the information on both the
enterprise and SaaS sources and targets constantly changes, in terms of structure, content,and the interfaces. All of these changes can be abstracted within the staging area which is
able to adjust using a configuration mechanism, rather than force expensive redevelopment.
The value of agility is an outcome of keeping volatility in the single domain of the staging
area, since the more a company changes, and thus requires changes to the enterprise
and SaaS-delivered systems, the more value agility br ings. Remember, its agility that allows
enterprises to adjust to take advantage of new market trends, or quickly reposition the
business to avoid the negative and take advantage of the positive. Thus, the value of agility is
substantial, considering that the ability to quickly move into a new market could make the
company millions of dollars in the forthcoming year, or provide the ability to move quickly
away from an unprofitable business process to avoid huge losses.The value of customized, complex views for specific systems and users refers to the fact that,
when you use a staging area, youre able to provide the information thats needed to the
system that needs it, customized for users specific needs. An example would be the ability
to extract customer sales information from a SaaS-based system and credit information
from a credit reporting system. Through the use of a staging area where the information
is combined, it is then updated using a specific format into a sales reporting system that is
leveraged by the executives within the company. This allows the information to be combined
and presented to the target system, in this case, the target customer repor ting systems, using
views of the information that are specific to that system.
Finally, the value of leveraging clean, reliable data that goes through a cleansing and validationprocess insures there is no missing or invalid information. The value of this is obvious.
[ 5
8/11/2019 WP-Leveraging a Staging Server
6/12
Leveraging a Staging Area Best Practices
In order to do effective staging for enterprise-to-SaaS problem domains, we need to follow a
few basic steps. These steps include:
Understand the source and target data.
Dene the data staging approach.
Design the data staging operations.
Implementation and testing (see Figure 3).
[ 6 ]
Figure 3: For SaaS-to-Enterprise data staging integration, you need to carefully consider the approach and the
design, after understanding your own requirements and existing infrastructure. Testing needs to occur as well, in
order to validate the solution.
8/11/2019 WP-Leveraging a Staging Server
7/12
[ 7
Understand the source and target datarefers to the process of understanding all
source and target systems at a semantic level, meaning what the data means in the larger
context of the enterprise architecture. This is important since integration requires that we
have an understanding of the source information in order to perform the operations that
will be completed within the staging area, including aggregation, replication, cleansing, andtransformation.
Define the data staging approachrefers to the process of creating a high-level approach
to the data staging for your particular problem domain, including definition of the core
purpose of the data staging operations, and a description of the transformation, replication,
cleansing, and aggregation operations that need to occur. This describes the What needs to
be done step.
Design the data staging operationsrefers to what happens to the data as its fed from
the operational data sources, including consumption of the information into a common
format, and the manipulation of the data for use by the target system, as defined by the
approach we created in the previous step. This describes the How something is done step.
Implementation and testingrefers to implementing the staging area and making sure that
it works as designed, and is able to maintain integrity of the data as its manipulated within
the staging area. This typically requires the creation of a test plan, test data, and acceptance
testing before the staging area is placed into production.
8/11/2019 WP-Leveraging a Staging Server
8/12
Leveraging Informatica PowerExchange for Salesforce CRM
Informatica PowerExchange for Salesforce CRM lets you use Informatica PowerCenter to
migrate, replicate, cleanse, and synchronize your Salesforce CRM or Force.com application
data, with your on-premise enterprise business applications and databases (see Figure 4).
PowerExchange takes advantage of the staging approaching to integration, as defined in this
paper.
[ 8 ]
Figure 4: Informatica PowerExchange for Salesforce CRM
8/11/2019 WP-Leveraging a Staging Server
9/12
Benefits of using PowerExchange for Salesforce CRM include the reduction of development
costs due to the configuration and visual approach that this technology provides. Thus, there
is an increase in IT productivity since it reduces or eliminates coding to the Salesforce.com
API. Because you capitalize on the value of data integration through the use of staging, you
take full advantage of your investment in Salesforce.com.
Other features of PowerCenter include:
Execute bidirectional Salesforce integration without requiring code.
Bring Salesforce data within your corporate rewall.
Use one tool to transform and cleanse data before delivering it to your on-premise
enterprise software and databases.
Pass Salesforce data directly from source to target without intermediate staging in
message queues.
Simplify the tasks of migrating, replicating, cleansing, and synchronizing Salesforce data
across the Internet with business analytics applications or master data hubs. Distill new and updated Salesforce data from unchanged background data, and send
only the changes for further processing.
Receive Salesforce data in a batch on any chosen schedule, or as a real-time stream
of updates, all with a common architecture, and all without coding or scripting.
Support any enterprise data integration initiative involving Salesforce data.
Scale to support Web services in SOAs, event-driven architectures, and traditional
data integration techniques on multiple platforms.
Choose your preferred technology and evolve gradually as conditions change.
[ 9
8/11/2019 WP-Leveraging a Staging Server
10/12
[ 10 ]
Leveraging Informatica On-Demand
When it comes to populating the staging area, you should consider Informatica On-Demand.
Informatica On-Demand allows you to leverage a staging area, either to or from the SaaS-
delivered applications, such as Salesforce.com. The facilities of Informatica On-Demand allow
you to use this staging area without a significant amount of data transformation, and it iseasily leveraged by the application administrator.
Moreover, by leveraging Informatica On-Demand, the application administrator is able to
focus on the staging area, and not the transformation. The transformation is carried out by
mechanisms introduced by the developer, leveraging tools such as PowerCenter (discussed
previously). This allows for the implementation of str ict validation rules and logic required by
the target systems.
Therefore, the use of both Informatica On-Demand and PowerCenter frees IT from the
requirements of understanding the natives of a par ticular application, and thus changes that
are requested by the end user (see Figure 5). Informatica is the only vendor that provides
the technology to serves both needs, and thus is one-stop-shopping for those looking to do
data integration, from SaaS-to-enterprise, leveraging a staging area.
Figure 5: The use of both Informatica On-Demand and PowerCenter frees IT from the requirements of under-
standing the natives of a particular application, and changes that are requested by the end user. Informatica
is the only vendor that solves both of those problems.
8/11/2019 WP-Leveraging a Staging Server
11/12
[ 1
Data Staging Taking Center Stage
As we learned in this paper, the need for data integration to suppor t integration from and
to SaaS-based applications, such as Salesforce.com to on-premise information systems, is
a core need as the interest in SaaS and cloud computing grows. However, those who look
to address this integration using more traditional, tactically-oriented tools are doomed toreplicate improperly processed information to the target system that receives it.
The use of staging as an approach to data integration provides the data or enterprise
architect with the ability to perform complex processing on an intermediate form of the
data, which in turn provides a location to gather the data when its ready to be gathered,
aggregates the data so it has better business meaning, and cleans the data where needed to
insure data quality. In essence, staging provides a location where value can be added to the
data as it moves from on-premise to SaaS, and from SaaS to on-premise.
8/11/2019 WP-Leveraging a Staging Server
12/12
[ 12 ]
2009 David S. Linthicum, LLC 6986 (06/04/2009)
About the Author
David Linthicum (Dave) is an internationally known Enterprise Application Integration (EAI),
Service Oriented Architecture (SOA), and cloud computing exper t. In his career, Dave has
formed or enhanced many of the ideas behind modern distr ibuted computing including EAI,
B2B Application Integration, and SOA, approaches and technologies in wide use today.Currently, Dave is the founder of David S. Linthicum, LLC, a consulting organization dedicated
to excellence in SOA product development, SOA implementation, corporate SOA strategy,
and leveraging the cloud computing. Dave is the former CEO of BRIDGEWERX, former
CTO of Mercator Software, and has held key technology management roles with a number
of organizations including CTO of SAGA Software, Mobil Oil, EDS, AT&T, and Ernst and
Young. Dave is on the board of directors ser ving Bondmart.com, and provides advisory
services for several venture capital organizations and key technology companies.
In addition, Dave was an associate professor of computer science for eight years, and
continues to lecture at major technical colleges and universities including the University of
Virginia, Arizona State University, and the University of Wisconsin. Dave keynotes at many
leading technology conferences on application integration, SOA, Web 2.0, cloud computing,and enterprise architecture, and has appeared on a number of TV and radio shows as a
computing expert.