Transform a source file to target file
There is a requirement to filter duplicates from a source file and to generate a target file. The first approach we made is to use foreach property in Data Mapper to filter duplicates s shown
$Input_bhi_productmaster_schema/Root/Record[not( preceding:: Material_ID = Material_ID )] ------> but it took 2 hours for processing 45000 records as it is taking more time ,we made other design that at first we are moving the source file to database table putting constraint as a primary key to filter duplicates ,now it is taking considerable time but there is a extra overhead in moving the source to database table and then taking again it as a input stream for generating target file.
In Brief , Is there is a way to transform a source file to target file by filtering duplicates(based on one column or combination of columns) using only Data Mapper with effective performance .
-
There are couple of alternate ways that you can follow as they can be quick and splitting can also be applied in second approach. Below are the approaches that you can follow:
-
Use a Context Variable/Map/File in which you will store the value of each record before processing it and Use Lookup on this variable/map in mapping that will check if the value is present in it or not, if it is not present then it will be inserted in the map/variable and will be sent to target. In other case if the value is found in this map then that record will not be sent to Target.
-
Another approach would be using a CustomPlugin before the Data mapper in which you can write a Java Code for removing the duplicate entries of records. Distinct records will be passed to the Data mapping so splitting can also be used in Data Mapper. Please refer to the below link that will help you creating this Custom Plugin in Adeptia:
http://xquare.ow2.org/fusion/API%20Tutorial.html
0 -
Please sign in to leave a comment.
Comments
1 comment