Creating Custom Component in CloverETL Step By Step

Share this article:

CloverETL is designed to be an extensible system. Although it already contains a lot of components, there can be cases where you need to create your own – be it a specific and often-used task, use of an external library, your own connector, etc. The full guide for creating custom component is described in the documentation, but for those of you who want a quick look at the process without the lengthy details, I’ve prepared some brief instructions. If you follow this guide step by step, you’ll be able to create custom components in no time.

Hello World

Let’s create a component called “HelloWorld” that works just like SimpleCopy, but additionally adds a “Hello” message to every string field. Then we’ll deploy that component in the CloverETL Designer so it can be used in a transformation graph.

We’ll start with a plain CloverETL Designer installation, then add the required tools, implement the component in Java, bundle it, and finally, deploy.

Complete instructions for creating custom component

Taking the fast track...

  • Start with steps 1-3 below
  • Download these pre-built projects: CloverETL-Custom-Component-Template.zip
  • Import projects from the template archive into your workspace.
    File → Import... and select General / Existing Projects into Workspace.
    New Java Project wizard New Java Project wizard
  • Finish with step 21 below. Done.

If you want to make changes...

  • If you want to change the component behavior, edit HelloWorldComponent.java.
    Remember, in this case you will need to perform steps 17 through 21
  • Component descriptor for Engine: com.cloveretl.customcomponent\plugin.xml.
    Remember, if you change this file you will need to perform steps 17 through 21
  • Component descriptor for Designer: com.cloveretl.designer.customcomponent\customcomponent.xml.
    Remember, if you change only this file you will just need to perform step 21

Step By Step

Install the Designer and plugins

  1. Download, install and run CloverETL Designer (http://www.cloveretl.com/download)
  2. Install Eclipse Plug-in Development Environment:
    1. Go to Help → Install new software.
    2. Select Indigo – http://download.eclipse.org/releases/indigo update site
    3. Search for General Purpose Tools / Eclipse Plug-in Development Environment.
  3. Switch to Java perspective. You are now ready to deploy Eclipse plugins.

Implement the component Java class

  1. Create new Java project com.cloveretl.customcomponent in your current workspace. (File → New → Other > Java/Java Project)
    New Java Project wizard New Java Project wizard step 2
  2. Hit Next and add CloverETL Engine to the project build path.
    Add Library - CloverETL Engine
  3. Create a new java class
    • Name: HelloWorldComponent
    • Package: com.cloveretl.customcomponent

    You don't need to set anything else here – you will paste the full source code in the next step anyway.

    Create new class
  4. Copy & paste the following full class implementation. Later, you can modify this to suit your needs. This one class implements all the functionality of the component.

    package com.cloveretl.customcomponent;

    import org.jetel.data.DataRecord;
    import org.jetel.data.DataRecordFactory;
    import org.jetel.exception.AttributeNotFoundException;
    import org.jetel.exception.ConfigurationStatus;
    import org.jetel.exception.XMLConfigurationException;
    import org.jetel.graph.InputPort;
    import org.jetel.graph.Node;
    import org.jetel.graph.Result;
    import org.jetel.graph.TransformationGraph;
    import org.jetel.metadata.DataFieldType;
    import org.jetel.util.SynchronizeUtils;
    import org.jetel.util.property.ComponentXMLAttributes;
    import org.w3c.dom.Element;

    /**
    * A sample custom component
    *
    * Copies records from input to output, adding a "Hello" message to every string field
    *
    */
    public class HelloWorldComponent extends Node {

    /**
    * Component internal type ID, see getType()
    */
    public final static String COMPONENT_TYPE = "HELLO_WORLD";

    /**
    * The message to say.
    * Can be overridden by setting "Message to say" attribute of the component.
    *
    * @see customcomponent.xml and "defaultHint" attribute - the default value here MUST be the same as in defaultHint
    */
    private String message = "Hello";

    /**
    * Name of the "Message to say" attribute in the XML
    *
    * @see fromXML() and customcomponent.xml
    */
    public final static String XML_MESSAGE_ATTRIBUTE = "message";

    /**
    * Standard constructor
    *
    * @param id
    * Component ID in the graph
    */
    public HelloWorldComponent(String id) {
    super(id);
    }

    /**
    * Our constructor with custom message
    *
    * @param id
    * Component ID in the graph
    *
    * @param message
    * Custom message to say
    */
    public HelloWorldComponent(String id, String message) {
    super(id);

    if (message != null && message.trim().length() > 0) {
    this.message = message;
    }
    }

    @Override
    public Result execute() throws Exception {

    /**
    * Initialize input port and a bucket input record that will be field with records coming to input port
    */
    InputPort inPort = (InputPort) getInputPort(0); // ports are numbered from 0
    DataRecord inRecord = DataRecordFactory.newRecord(inPort.getMetadata()); // always use DataRecordFactory to create record instances
    int numFields = inRecord.getNumFields();
    inRecord.init(); // initialize records only once
    inRecord.reset(); // just to make sure all field values are reset

    /**
    * Initialize output and "bucket" output records that will be filled as "copy-by-name" from the input with additional message
    */
    int numOutputPorts=getOutPorts().size();
    DataRecord[] outRecord = new DataRecord[numOutputPorts];
    for (int i = 0; i < numOutputPorts; i++) {
    outRecord[i] = DataRecordFactory.newRecord(getOutputPort(i).getMetadata());
    outRecord[i].init();
    outRecord[i].reset();
    }

    /**
    * MAIN LOOP
    */
    while (inPort.readRecord(inRecord) != null && runIt) { // always check internal "runIt" state

    /**
    * Add "message" to every field in the input which has "string" data type
    */
    for(int fieldNo = 0; fieldNo < numFields; fieldNo++) {
    if (inRecord.getField(fieldNo).getMetadata().getDataType().equals(DataFieldType.STRING)) {
    inRecord.getField(fieldNo).setValue(message + " " + String.valueOf(inRecord.getField(fieldNo).getValue()));
    }
    }

    /**
    * Send output to all out ports
    * For each output port, copy the modified input record (by name) and then send it out
    */
    for (int i=0; i < numOutputPorts; i++) {
    outRecord[i].reset(); // to make sure we have clean record before copying
    outRecord[i].copyFieldsByName(inRecord);
    writeRecord(i, outRecord[i]); // sends out record to the output edge
    }

    /**
    * Mandatory yield after every record
    */
    SynchronizeUtils.cloverYield();
    }

    return runIt ? Result.FINISHED_OK : Result.ABORTED;
    }

    /**
    * Factory method that creates the component from transformation graph source XML
    */
    public static Node fromXML(TransformationGraph graph, Element xmlElement) throws XMLConfigurationException, AttributeNotFoundException {

    ComponentXMLAttributes xattribs = new ComponentXMLAttributes(xmlElement, graph);

    String id = xattribs.getString(XML_ID_ATTRIBUTE);
    String message = xattribs.exists(XML_MESSAGE_ATTRIBUTE) ? xattribs.getString(XML_MESSAGE_ATTRIBUTE) : null;

    return new HelloWorldComponent(id, message);
    }

    /**
    * checkConfig checks configuration of the component for Designer and Engine.
    * Here, you would check connected input and output ports, required metadata etc.
    */
    @Override
    public ConfigurationStatus checkConfig(ConfigurationStatus status) {
    super.checkConfig(status);

    /**
    * Check that we have one input port and at least one output
    */
    if(!checkInputPorts(status, 1, 1) || !checkOutputPorts(status, 1, Integer.MAX_VALUE, false)) {
    return status;
    }

    return status;
    }

    @Override
    public String getType(){
    return COMPONENT_TYPE;
    }

    }

  5. Create a blank file plugin.xml in the project root.
    Create plugin.xml
  6. Put this XML content into plugin.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <?xml version="1.0" encoding="UTF-8"?>
    <plugin id="custom.component" version="1.0.0" provider-name="Company">
    <runtime><library path="bin/"/></runtime>
    <requires engine-version="3.3.0"></requires>
    <extension point-id="component">
    <parameter
    id="className"
    value="com.cloveretl.customcomponent.HelloWorldComponent"/>
    <parameter id="type" value="HELLO_WORLD"/>
    </extension>
    </plugin>

Create CloverETL Designer plugin

  1. Create new Plug-in project com.cloveretl.designer.customcomponent in your current workspace. (File > New > Plug-in Development > Plug-in Project)
    Create Plugin project A
  2. On the second page of the new project wizard:
    • uncheck Generate an activator, a Java class that controls the plug-in's lifecycle
    • uncheck This plug-in will make contributions to the UI
    Create Plugin project B Create Plugin project C
  3. Create a blank file customcomponent.xml in the project root
    Create customcomponent.xml
  4. Put this content into customcomponent.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <ETLComponentList>
    <ETLComponent category="Custom" name="Hello World"
    type="HELLO_WORLD" iconPath="platform:/plugin/com.cloveretl.gui/icons/transformers/SimpleCopy"
    passThrough="true">
    <shortDescription>Hello World</shortDescription>
    <description>A sample custom component. Greets the world.</description>
    <inputPorts>
    <singlePort name="0" required="true" />
    </inputPorts>
    <outputPorts>
    <multiplePort required="true" />
    </outputPorts>
    <properties>
    <property category="basic" displayName="Message to say" modifiable="true" name="message" nullable="true" defaultHint="Hello">
    <singleType name="string" />
    </property>
    </properties>
    </ETLComponent>
    </ETLComponentList>
  5. Create a new class
    • Class name: CLProvider
    • Package: com.cloveretl.designer.customcomponent
    • Superclass: com.cloveretl.gui.plugin.engine.AbstractClassLoaderProvider

    Leave the class body empty. (AbstractClassLoaderProvider cannot be resolved yet, but you can ignore it)

    Create class CLProvider
  6. Open META-INF/MANIFEST.MF
    1. Go to Dependencies tab
    2. Add com.cloveretl.gui plugin to the list of Required Plug-ins
      Add com.cloveretl.gui to Required plug-ins
    3. Move to Extensions tab
    4. Add com.cloveretl.gui.component extension
    5. Set the following:
      • file to customcomponent.xml
      Add com.cloveretl.gui.component to Extensions
    6. Add com.cloveretl.gui.enginePlugin extension
    7. Set the following:
      • directory to plugins
      • ClassLoaderProvider to com.cloveretl.designer.customcomponent.CLProvider
      Add com.cloveretl.gui.enginePlugin to Extensions
  7. Create a new folder plugins in com.cloveretl.designer.customcomponent project.
    Create plugins folder
  8. Right-click plugins folder and select Import..., General → File system.
    Import to plugins
  9. Use com.cloveretl.customcomponent project’s location in From directory field.
    • Check plugin.xml
    • Check bin
    Creating Custom Component
  10. Click Finish and verify that files have been copied to plugins folder.
    Verify plugins import
  11. Open plugin.xml in the com.cloveretl.designer.customcomponent project
    1. Switch to Build tab:
    2. Make sure that the following items are checked:
      • META-INF
      • customcomponents.xml
      • plugin.xml
      • plugins
      Creating Custom Component
  12. Right click com.cloveretl.designer.customcomponent and select Export... from the context menu.
    1. Select Plug-in development → Deployable plug-ins and fragments
      Creating Custom Component
    2. Select Destination / Install into host (no need to change anything else).
      Creating Custom Component
  13. Click Finish and restart Eclipse when asked to finish the installation of the new plugin.
  14. Congratulations! From here on out, you should be able to use the new component in your graphs.

(UPDATE) Deploying to CloverETL Server

If you are interested in seeing how this component can work on CloverETL Server, continue reading with our dedicated post.

Final notes

The steps above should guide you through the whole process of creating a simple component. You can create a regular CloverETL project in your workspace and use the component in a transformation to test it.

PLEASE NOTE: We have observed a few issues with deploying plugins into Eclipse.
If your component is not working after steps above, please check again the settings of the “Build” tab when editing plugin file (Step 18). This seems to be an Eclipse bug.

For more information, please check our documentation.

Share this article:
Contact Us

Further questions? Contact us.

Forum

Talk to peers on our forum.

Want to keep in touch?

Follow our social media.

Topics

see all