Transform a source file to target file

There is a requirement to filter duplicates from a source file and to generate a target file. The first approach we made is to use foreach property in Data Mapper to filter duplicates s shown

$Input_bhi_productmaster_schema/Root/Record[not( preceding:: Material_ID = Material_ID )] ------> but it took 2 hours for processing 45000 records as it is taking more time ,we made other design that at first we are moving the source file to database table putting constraint as a primary key to filter duplicates ,now it is taking considerable time but there is a extra overhead in moving the source to database table and then taking again it as a input stream for generating target file.

In Brief , Is there is a way to transform a source file to target file by filtering duplicates(based on one column or combination of columns) using only Data Mapper with effective performance .

1 comment

Please sign in to leave a comment.