Container Types: List and Map Metadata Fields

Share this article:

List and map metadata fields are part of CloverETL's effort to enhance its support for both rich data structures and unstructured data. This enhancement eases the previously rigid metadata model, which required each processed data record to be exactly described to the last item. With the new version, data coming from systems where the structure of record varies from record to record (as are CRMs, social networking systems) can now be easily processed in CloverETL by utilizing the variable length lists – or even more flexible maps.

This new feature called container types was added in 3.3.0 M2. Any field can now operate in three modes:

  1. single value: normal field, stores one value of the selected type
  2. list: the field can hold a list of values of the selected type
  3. map: the field is a map from strings to the selected type

Container fields can be read or written to in CTL, just like with any other list or map. In addition, some components can read or write data into container fields or use them in transformations.

Metadata Fields

XMLReader and XMLWriter Examples

For example, XMLReader can map multiple occurrences of a sub-element to a list field. That means that the XML snippet below:

<?xml version="1.0" encoding="ISO-8859-1"?>
        <String>string list</String>

with the following mapping:

<Context xpath="/root/ParentElement" outPort="0">
    <Mapping xpath="Number" cloverField="Numbers"/>
    <Mapping xpath="String" cloverField="Strings"/>            

will produce the following output:

Metadata Fields

Similarly, the lists can be written out using XMLWriter.

Container fields can also be processed in CTL:

function integer transform() {
    $out.0.concatenatedStrings = join("+", $in.0.Strings);

    return OK;

Metadata Fields

Thus the map and list metadata fields give you the needed flexibility to process more complex and variable input than before.

Download a CloverETL project with this example

Share this article:
Contact Us

Further questions? Contact us.


Talk to peers on our forum.

Want to keep in touch?

Follow our social media.