Lookuptransformation with Dynamic Cache Scenario
Unconnected Lookup Transformation with Role Playing Date Dimension Scenario
Unconnected Lookup Transformation with Role Playing Date Dimension Scenario
Lookup transformation is a passive or active transformation that looks up data in a flat file, logical data object, reference table, or relational table. The Lookup transformation can return one row or multiple rows from a lookup. The Integration Service queries the lookup source based on the lookup ports in the transformation. It compares Lookup transformation port values to lookup source column values based on the lookup condition. It passes the result of the lookup to other transformations or a target.
Connected Lookups
In Connected lookup
transformation, we connect input and output ports to other transformation in the mapping.
- For each input row, the Integration Service queries the lookup source or cache based on the lookup ports and the condition in the transformation.
- If the transformation is uncached or uses a static cache, the Integration Service returns values from the lookup query.
- If the transformation uses a dynamic cache, the Integration Service inserts the row into the cache when it does not find the row in the cache. When the Integration Service finds the row in the cache, it updates the row in the cache or leaves it unchanged. It flags the row as insert, update, or no change.
- The Integration Service passes return values from the query to the next transformation. If the transformation uses a dynamic cache, you can pass rows to a Filter or Router transformation to filter new rows to the target.
- If there is no match for the lookup condition, the Integration Service returns the default value for all output ports.
Unconnected Lookups
Unconnected Lookup
transformation is not connected to other
transformations in the mapping.
- It receives input from the result of a :LKP expression in a transformation such as an Expression transformation or Aggregator transformation.
- The Integration Service queries the lookup source or cache based on the lookup ports and condition in the transformation.
- The :LKP expression passes parameters to the Lookup transformation and receives a result back from the lookup.
- If there is no match for the lookup condition, the Integration Service returns NULL.
- You can call the Lookup transformation more than once in a mapping.
Lookup Query
The Integration
Service queries the lookup based on the ports and properties you configure in
the Lookup transformation. The Integration Service runs a default lookup query
when the first row enters the Lookup transformation. If you use a lookup
against a relational table, you can override the lookup query.
Lookup Filter
You can configure a
Lookup source filter for a relational Lookup transformation that has caching
enabled. If you configure a SQL override
and a filter on the lookup query, the Integration Service ignores the filter.
Run-time Properties
Lookup Caching Enabled When It is enabled, the
Integration Service queries the lookup source once, caches the values, and
looks up values in the cache. When you disable caching, each time a row passes
into the transformation, the Integration Service issues a select statement to
the lookup source for lookup values. The Integration Service always caches flat
file lookups.
Lookup Data Cache Size Maximum size the
Integration Service allocates to the data cache in memory. Default is Auto.
Lookup Index Cache Size Maximum size the
Integration Service allocates to the index in memory. Default is Auto.
Cache File Name Prefix Prefix for the cache
file. You can specify the cache file name prefix for a persistent lookup cache.
Pre-build Lookup Cache Allows the Integration
Service to build the lookup cache before the Lookup transformation receives the
data.
Lookup Cache Directory Name Directory used to
build the lookup cache files when you configure the Lookup transformation to
cache the lookup source.
Re-cache from Lookup Source Rebuilds the lookup
cache to synchronize the persistent cache with the lookup table.
Advanced Properties
Lookup Cache Persistent Indicates whether the
Integration Service uses a persistent lookup cache.
Case Sensitive String Comparison The
Integration Service uses case-sensitive string comparisons when performing
lookups on string columns when the lookup source is a flatfile.
Null Ordering You can choose to sort null
values high or low. By default, the Integration Service sorts null values high
for flatfile. For relational lookups, null ordering is based on the database
default value.
Tracing Level Sets the amount of detail that
appears in the log for this transformation.
Update Else Insert Applies to dynamic lookup
cache only. The Integration Service updates the row in the cache if the row
type entering the Lookup transformation is update, the row exists in the index
cache, and the cache data is different than the existing row. The Integration
Service inserts the row in the cache if it is new.
Insert Else Update Applies to dynamic lookup cache only. The
Integration Service inserts the row in the cache if the row type entering the
Lookup transformation is insert and it is new. If the row exists in the index
cache but the data cache is different than the current row, the Integration
Service updates the row in the data cache.
Output Old Value on Update The Integration
Service outputs the value that existed in the cache before it updated the row.
Otherwise, the Integration Service outputs the updated value that it writes in
the cache.
Update Dynamic Cache Condition Applies to
dynamic lookup cache only. An expression that indicates whether to update the
dynamic cache. The Integration Service updates the cache when the condition is
true and the data exists in the cache. Default is true.
Connection Connection to the relational
database that contains the relational lookup source.
Sorted Input Indicates that input data is
presorted by groups.
Datetime Format Define a datetime format and
field width.
Thousand Separator Value is None. This field is
read-only.
Decimal Separator Value is a period. This field
is read-only.
*The last four
properties are applicable only for flatfiles.
Nice
ReplyDeleteThe lookup tranformation can be tuned using cached or uncached lookups. See the below mathematical approach to determine the best way to do so.
ReplyDeleteN = Number of records coming from source (For the purpose of simplicity, let’s assume N also equals to the number of times the lookup will be called)
M = Number of records retrieved from the Lookup query
t = time required to retrieve a single record from database based on the Lookup query
T = time required to retrieve all the records from database based on the Lookup query = M * t
L = Time required to build the lookup index cache for cached lookup = f(N)
C = Network Time required to do one round trip for data fetch from database
In case of an uncached lookup, total time required will be:
T uncached = N * t + N * C
In case of a cached lookup, total time required to build the cache will be:
Tcached = T + L + C
In the above equation we can put L = f(N) and since C is generally very small number, we can ignore C and rewrite the above equation as –
Tcached = T + f(N)
Now if we assume that the database table has INDEX defined upon it and the index is being used while retrieving records from database then, we can further assume –
T = M * t
Replacing the above value in our earlier equation for Tcached we get –
Tcached= M * t + f(N)
We should use uncached lookup if –
Tcached > Tuncached
=> M * t + f(N) > N * t + N * C
Dividing all sides by N, we get –
C + t < (M/N)*t + f(N)/N
For a given Informatica and database setup, C, t & f(N) are fixed.
So the above equation is more likely to be true if (M/N) ratio is on higher side, i.e. N << M – that is – number of records coming from source is considerably less than number of records present in the lookup table (Refer back to point no. 1 in the discussion above).
Similarly, if N >> M – that is number of records coming from source is considerably high than the number of records present in lookup table, then we should use cached lookup.
Informatica Training in Bangalore,Informatica Training,Informatica Online Training.
ReplyDeletehttp://www.tekclasses.com/courses/etl/informatica-training-in-bangalore/
nice blog..
ReplyDeleteInformatica Training in Bangalore,Informatica Training,Informatica Online Training.
http://www.tekclasses.com/courses/etl/informatica-training-in-bangalore/
How to populate the data in to the target, if the lookup returns 3 matching values, how to insert all these data into the target? PHP Training in Chennai
ReplyDeleteFor this while creating lookup transformation itself, u need to select the checkbox, which says "return multiple rows when lookup". This checkbox comes when it asks for creating lookup using table/file. If you create a lookup with this check box then multiple values can be returned from lookup for a single condition.
Deletecan any one explain me the significance of "Output Old Value on Update" option in dynamic lookup transformation.?
ReplyDeleteI really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Informatica Powercenter 9.X Developer training, kindly contact us http://www.maxmunus.com/contact
ReplyDeleteMaxMunus Offer World Class Virtual Instructor led training on Informatica Powercenter 9.X Developer training. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Nitesh Kumar
MaxMunus
E-mail: nitesh@maxmunus.com
Skype id: nitesh_maxmunus
Ph:(+91) 8553912023
http://www.maxmunus.com/