List and map metadata fields are part of CloverETL’s effort to enhance its support for both rich data structures and unstructured data. This enhancement eases the previously rigid metadata model, which required each processed data record to be exactly described to the last item. With the new version, data coming from systems where the structure of record varies from record to record (as are CRMs, social networking systems) can now be easily processed in CloverETL by utilizing the variable length lists – or even more flexible maps.

This new feature called container types was added in 3.3.0 M2. Any field can now operate in three modes:

1. single value: normal field, stores one value of the selected type
2. list: the field can hold a list of values of the selected type
3. map: the field is a map from strings to the selected type

Container fields can be read or written to in CTL, just like with any other list or map. In addition, some components can read or write data into container fields or use them in transformations.

XMLReader and XMLWriter Examples

For example, XMLReader can map multiple occurrences of a sub-element to a list field. That means that the XML snippet below:

<?xml version="1.0" encoding="ISO-8859-1"?>
<root>
    <ParentElement>
        <Number>-62.13571388346442</Number>
        <Number>13.465953975177856</Number>
        <Number>-188.61795040912824</Number>
        <String>lists</String>
        <String>in</String>
        <String>metadata</String>
        <String>work</String>
        <String>now</String>
    </ParentElement>
    <ParentElement>
        <Number>106.86947999682504</Number>
        <Number>78.20039253465931</Number>
        <Number>-40.87408457269909</Number>
        <String>string list</String>
        <String>element</String>
    </ParentElement>
</root>

with the following mapping:

<Context xpath="/root/ParentElement" outPort="0">
    <Mapping xpath="Number" cloverField="Numbers"/>
    <Mapping xpath="String" cloverField="Strings"/>            
</Context>

will produce the following output:

Similarly, the lists can be written out using XMLWriter.

Container fields can also be processed in CTL:

function integer transform() {
    $out.0.concatenatedStrings = join("+", $in.0.Strings);

    return OK;
}

Thus the map and list metadata fields give you the needed flexibility to process more complex and variable input than before.

Download a CloverETL project with this example