Custom CTL Functions in CloverETL

Share this article:

A few weeks ago I published a blog post about how to create a custom CloverETL component. This is great if you have special logic that operates on whole records. But what if you have specific needs when it comes to processing single values, or more precisely, field values? This is where custom CTL functions come into play. In this post, I’ll cover how to extend CTL2, the embedded data transformation language inside CloverETL.


The instructions will be very similar to building a custom component so I recommend skimming through it. However, in the following step-by-step guide, I will provide references whenever needed.

As an example, we’ll implement a “Lorem Ipsum” function that will produce English-like bogus text for testing. For this purpose, we’ll use an external loremipsum Java library by Sven Jacobs. Eventually we will create two CTL functions – one without arguments and another with the word count as a single integer argument.

Brief description

This is what we’ll do.

  1. First, we’ll implement a CloverETL Engine plugin:
    • A Java class extending org.jetel.ctl.extensions.TLFunctionLibrary and implementing the desired CTL functions
    • A plugin.xml with ctlfunction extension definition
    • We’ll use org.jetel.ctl.extensions.TLFunctionAnnotation annotation to mark methods that serve as implementation of our CTL functions
  2. Second, we’ll create a CloverETL Designer plugin:
    • Put plugin.xml into it with the engineplugin extension definition
    • Import the engine plugin to a directory specified by the engineplugin extension definition

Once finished, the new function will be available in Designer dialogs (like Source tab in Transform Editor of a Reformat) and also it will work properly in runtime, including linking the extra library. So let’s get to it!

Step-by-step Creation of Custom CTL Functions

The prerequisites are identical to building a custom component (see Complete instructions)

  1. Creating the Engine Plugin:
    • Start a new Java project com.cloveretl.custom.loremipsum
    • Put the external library to lib/loremipsum-1.0.jar and make sure it’s on classpath
    • Create a new java class LoremIpsumCTLLibrary extending org.jetel.ctl.extensions.TLFunctionLibrary
      package com.cloveretl.customctl;
       
      import org.jetel.ctl.Stack;
      import org.jetel.ctl.extensions.TLFunctionAnnotation;
      import org.jetel.ctl.extensions.TLFunctionCallContext;
      import org.jetel.ctl.extensions.TLFunctionLibrary;
      import org.jetel.ctl.extensions.TLFunctionPrototype;
      import de.svenjacobs.loremipsum.LoremIpsum;
       
      public class LoremIpsumCTLLibrary extends TLFunctionLibrary {
       
          LoremIpsum loremIpsum = new LoremIpsum();
       
          @Override
          public TLFunctionPrototype getExecutable(String functionName)
                  throws IllegalArgumentException {
              TLFunctionPrototype ret =
                      "getWords".equals(functionName) ? new GetWordsFunction() : null;
              return ret;
          }
       
          @Override
          public String getName() {
              return "LoremIpsumCTLLibrary";
          }
       
          @TLFunctionAnnotation("Generates Lorem Ipsum text")
          public String getWords(TLFunctionCallContext context) {
              return this.loremIpsum.getWords();
          }
       
          @TLFunctionAnnotation("Generates Lorem Ipsum text")
          public String getWords(TLFunctionCallContext context, Integer count) {
              if(count==null) {
                  return this.getWords(context);
              }
              return this.loremIpsum.getWords(count);
          }
       
          class GetWordsFunction implements TLFunctionPrototype {
              @Override
              public void execute(Stack stack, TLFunctionCallContext context) {
                  if (context.getParams().length > 0) {
                      stack.push(getWords(context, stack.popInt()));
                  }else{
                      stack.push(getWords(context));
                  }
              }
              @Override
              public void init(TLFunctionCallContext context) { }
          }
      }
      
    • Create plugin.xml in root of the engine plugin project:
      <?xml version="1.0"?>
      <plugin
         id="com.cloveretl.custom.loremipsum"
         version="0.0.0.devel"
         provider-name="CloverETL">
       
          <runtime>
              <library path="bin/"/>
              <library path="lib/loremipsum-1.0.jar"/>
          </runtime>
       
          <extension point-id="ctlfunction">
              <parameter id="libraryName" value="LoremIpsumCTLLibrary"/>
              <parameter id="className"
      value="com.cloveretl.customctl.LoremIpsumCTLLibrary"/>
          </extension>
      </plugin>
      
    • Note one thing: If you export the engine plugin to a .jar outside of Eclipse, the compiled class files will not be in the bin folder. Instead, they’ll be placed directly in the root of the .jar. If that’s the case change <library path="bin/"/> to <library path="./"/>.


  2. Designer plugin:
    • Again, the instructions are basically identical to creating a Designer plugin for a custom component – please follow the referenced steps in the other blog post:
    • Step 10 – Plugin Project Creation
    • Step 11 – Configuring the New Plugin Project
    • Step 14 – Create a Class Loader Provider Class
    • Step 15 – Only 1, 2, 6 and 7 – i.e. only those related to com.cloveretl.gui.enginePlugin
    • Steps 16 through 23 – Identical
    • For step 16 – It’s a best practice to put engine plugins into a special folder named by plugin ID. So in our case, you would create a directory com.cloveretl.custom.loremipsum in the plugins directory and perform filesystem import into it.
    • When editing plugin.xml in Step 20, set the classpath to the following three folders: the Designer plugin folder (“.”), engine plugin classes and libraries to classpath (in our case: ., plugins/com.cloveretl.custom.loremipsum/bin/, plugins/com.cloveretl.custom.loremipsum/lib/loremipsum-1.0.jar) Custom CTL Functions

The final structure of the Designer plugin with provided engine plugin should look like this:

Custom CTL Functions

The engine plugin classes are located in plugins/com.cloveretl.custom.loremipsum/bin. This is mainly for simplicity. When you develop complex plugins, we recommend packing the engine classes into a .jar file and putting it directly to the plugin (to the same level as engine plugin.xml file).

After the steps described in this tutorial, you should be able to use your own CTL functions. You can check the attached example projects, which contain the sources and exported plugin .jar, as well as an example graph.

Download CloverETL Custom CTL Function Example



Share this article:
Contact Us

Further questions? Contact us.

Forum

Talk to peers on our forum.

Want to keep in touch?

Follow our social media.

Topics

see all

Recent Posts