Upload
ravi405
View
214
Download
0
Embed Size (px)
Citation preview
8/11/2019 Improving Performance of Informatica Lookups
1/5
Boost Performance of Informatica Lookups
Introduction
Lookups are expensive in terms of resources and time.
A set of tips about how to setup lookup transformations would dramatically improve the main
constrains such as time and performance.
In this article you will learn about the following topics:
- Lookup cache
- Persistent lookup cache
- Unconnected lookup
- Order by clause within SQL
Lookup Cache
Problem:
For non-cached lookups, Informatica hits the database and bring the entire set of rows for each record
coming from the source. There is an impact in terms of time and resources. If there are 2 Million rows
from the source qualifier, Informatica hits 2 Million times the database for the same query.
Solution:
When a lookup is cached: Informatica queries the database, brings the whole set of rows to the
Informatica server and stores in a cache file. When this lookup is called next time, Informatica uses the
file cached. As a result, Informatica saves the time and the resources to hit the database again.
When to cache a lookup?
As a general rule, we will use lookup cache when the following condition is satisfied:
N>>M
N is the number of records from the source
M is the number of records retrieved from the lookup
Note: Remember to implement database index on the columns used in the lookup condition to provide
better performance in non-cached lookups.
http://www.clearpeaks.com/blog/etl/boost-performance-of-informatica-lookupshttp://www.clearpeaks.com/blog/etl/boost-performance-of-informatica-lookupshttp://www.clearpeaks.com/blog/etl/boost-performance-of-informatica-lookups8/11/2019 Improving Performance of Informatica Lookups
2/5
Persistent Lookup Cache
Issue:
Informatica by default caches the lookup.
Lets consider the following example: A lookup table is used many times in different mappings. In each
Lookup transformation, Informatica builds the same lookup cache table again and again.
But is there a need to build the lookup cache every time for each lookup?
Solution:
We can build the cache file once instead of creating the same cache file many times by using the
persistent cache option that Informatica provides to save resources and time.
We need to set the following parameters in the Look up transformation to use Persistent cache feature:
- Lookup caching enabled
- Lookup cache persistent
Fig 1: Persistent Cache is enabled
8/11/2019 Improving Performance of Informatica Lookups
3/5
Once enabled, the same cache file will be used in all the consecutive runs, saving time in building the
cache file. However, if the lookup data changes, then the cache must be refreshed by either deleting the
cache file or checking the option Re-cache from lookup source.
Fig 2:Re-cache from Lookup Source Enabled
In case of using a lookup reusable in multiple mappings we will have 1 mapping with Re-cache option
enabled while others will remain with the Re-cache option disabled. Whenever the cacheneeds to be
refreshed we just need to run the first mapping.
Unconnected lookup
Problem:
Imagine the following mapping with 1,000,000 records retrieved from the Source Qualifier:
Figure 3: Connected Lookup Transformation
Suppose out of a million records, the condition is satisfied 10% of the amount of records. In case of
connected lookup, the lookup will be called 900,000 times even there isnt any match.
Solution:
It is possible calling the Lookup transformation only when the condition is satisfied. As a result, in our
scenario the transformation will be called and executed only 100,000 of times out of 1M. The solution is
using an Expression transformation that calls the lookup transformation that is not connected to the
dataflow:
8/11/2019 Improving Performance of Informatica Lookups
4/5
8/11/2019 Improving Performance of Informatica Lookups
5/5
Figure 5: To Avoid ORDER BY in SQL Override
To sum up, it is possible to enhance Informatica lookups by using different set of configurations in order
to increase performance as well as save resources and time. However, before applying any of the
mentioned features, an analysis of the tables and the SQL queries involved needs to be done.