Grouping Large Data

Situation:

I have a large data set (20MB-50MB) in xml; generated through other data transformations. I need to group this data set up by a single element and output the results of that single element.

I used the xpath expression from the article, http://support.adeptia.com/entries/22255876-how-to-use-duplicate-rule-in-mapping, which works great on small data sets, however, on a large data set, the xpath expression is very slow.

I then applied the splitting feature to the data set. This was fast, very nice, however, the results were a merged data set of the individual chunks grouped instead of the total of all chunks grouped.

Any ideas of a better way to group up an element of a large data set?

Solution:

1. Create a key. Data Mapper->Global->Key Functions

2. Apply record filter: Data Mapper->(Select Destination Record)->Node->Properties->ForEach

3. Apply Mapping Rules: Data Mapper->(Select Destination Element)->Node->Textual Rules

Generated XSL