We are pleased to announce our new CloverETL 4.1 Milestone Release (release notes). CloverETL 4.1 expands on the concepts of subgraphs and metadata propagation introduced in CloverETL 4.0 and adds a couple of new features. In this blog we'll introduce new features and improvements in CloverETL 4.1.
Subgraphs were a major feature introduced in CloverETL 4.0. Based on user feedback we keep adding new features and improvements. Here are the most important ones:
The introduction of optional subgraph ports is a true benefit when designing generic, reusable subgraphs. Previously all ports had to have an edge connected, but you can now define some ports as optional. If an edge is missing or connected to a disabled component, CloverETL will now run the graph without complaints.
This allows you to design truly generic subgraphs where users can choose to use only what they need – e.g. they can choose between alternative outputs, connect optional auxiliary inputs, etc.
Dynamically disabled components
Imagine a subgraph with an optional input – naturally the components processing this port would need to be turned on or off based on whether an outside edge is connected. Now you can do just that - we call this dynamically disabled components. You can now set components to disable and cascade that change dynamically based on whether an edge is missing. This is, of course, in addition to the old ways – manually enabling/disabling a component or using parameters.
Public parameters with custom editors
Another addition to 4.1 subgraphs is that you now can create typed public parameters in addition to plain strings introduced in 4.0. Formerly you could either export a specific component attribute or create a new parameter but that was limited to strings only.
Now any parameter can have a type which means much friendlier parameter editors for users configuring your subgraphs!
Generic Java Components
CloverETL appeals to both programmers and business analysts alike due to its elegant simplicity on one hand and access to complex code (e.g. graph XML source, CTL transformations, etc.) underneath the hood on the other. Introduction of Generic Java components is an example of the latter.
Generic components allow programmers to turn their own Java code into CloverETL components – be it custom readers, writers or any transformation. A rather deep working knowledge of Java is needed, but it gives you or your developers a tremendous power to have the flexibility you sometimes need for embedding Java algorithms you might already have elsewhere.
It is especially powerful when coupled with subgraphs that can hide a lot of complexity (including proprietary custom code) from user's sight and offer a nice and friendly UI instead. This significantly broadens the reach of CloverETL and allows you to connect data and sources that are not supported natively.
However, keep in mind that although tempting for some (programmers at heart), it is not advisable to overuse Java components in your designs. A simple rule of thumb is that whenever there's a Clover component available for the task you need, use it instead of implementing your own variation. Your transformations will be cleaner and more readable (think of sharing with a colleague who doesn't know Java), optimized (some components have evolved over years of fine tuning) and future-proof (we keep components up-to-date but you are responsible for your code - the less you put in there, the less maintenance you have).
Support for Amazon Web Services (AWS) and HP Vertica
A huge trend in data management and integration is distributed cloud storage. This is rapidly becoming a preferred choice for many and we keep adding and improving CloverETL's in that direction. In 4.1 we focused on HP Vertica and Amazon Web Services.
As we have written previously, it is easy to integrate CloverETL with AWS. In CloverETL 4.1, we've added native support of the S3 protocol into file operations and improved the directory management in AWS.
For HP Vertica, we're bundling a driver and we've improved support for HP Vertica's field types. Also, there's a subgraph available for parallel bulk loading! You might want to check out HP Vertica on their website.
More details in Tooltips and other UI improvements
With the introduction of the metadata propagation, we felt the importance of providing clear and concise metadata information within tooltips. Check out need shortcut links and navigation around your graph, edges and metadata from within the new active tooltips.
There are many more UI enhancements we will continue sharing with you in helpful videos, blogs and social media connections.
It is important to note that milestone releases, such as 4.1M1, are meant for test driving new features in preparation of the next production release and not for production.
CloverETL Milestone Release 4.1M1 is available for download.