For example:

• Each file has a time of creation, its size, name and the path where it is located, and (in case of an XLS file) the name of the sheet that is read.
• When a data source is being read, the reader starts to work at an explicit time; each record is also read at an explicit time.
• Records can be numbered in the order in which they are read.
• Information about errors may be available (in DBExecute and DBOutputTable).

All this information can easily enrich the read data in CloverETL by using the auto-filling functionality.

Auto-filling is a feature that is available on the metadata definition level. When you open the Metadata Editor and select any field of the metadata there is an Autofilling property (under the Advance tab). You can select one of the following values:

Remember that any of these functions can be applied to a field not contained in a file, database table, generated data, or JMS message.

Use case

Imagine that you have an export of customers from a database in an Excel file. The Excel file is organized into many sheets; each sheet is named by a state abbreviation and contains only customers from one state. See figure bellow.

Now you want to merge all customers to one CSV file but for each customer you want to also store the state in separate column. It looks very easy . For CloverETL it is, not necessarily so for other ETL tools.

The final graph is very simple, there are only two components.

The most important part of this graph is hidden in the definition of the metadata on the edge. We have to enrich the metadata, which we created from the Excel file by using the proper wizard in CloverETL Designer, with a new field “state” by setting the Autofilling property to sheet_name as you can see below. And that’s all.

The Autofilling field can be placed on any position in a metadata definition. It’s conveniant that we place the autofilling fields at the end of the metadata definition, after the field with the record delimiter. But this conveniance does not apply when the same metadata are also used for writing to a flat file (in our case). Thus we moved the autofilling fields to the position before the field with a record delimiter.

When you run the graph, you will get the following results:

You can use multiple autofilling fields with different functions at the same time to get more useful information from the file. For example, you can get the file name and the sheet names at the same time. Or you can get the number the records, etc. For example, when the following auto-filling functions are used (sheet_name, source_size, row_timestamp, global_row_count), the result will look like this:

Remember that only the edges connected to the output port(s) of the following reader components can use the auto-filling functionality:

• UniversalDataReader
• CloverDataReader
• XLSDataReader
• DBFDataReader
• MultiLevelReader
• XMLExtract
• XMLXPathReader
• DBInputTable
• DataGenerator
• JMSReader

DBOutputTable and DBExecute can use only two error auto-filling functions – ErrCode and ErrText.

New Features and Components:
Infobright Data Writer
In response to customer requests, this component writes data into Infobright software, a column-oriented relational database. Infobright is a provider of solutions designed to deliver a scalable data warehouse optimized for analytic queries.

Web Services component
The new component makes communication with Web Services easier than ever. It provides user friendly graphical interface for mapping your data into Web Service fields, automatically generates requests and process responses. It offers faster, easier and more comfortable way to interact with remote Web Services.

Reading formatted values from XLS
Additionally to reading plain data from Microsoft^TM Excel^TM sheets, the Excel component is now also capable of reading user-formatted values such as currencies, dates or numbers.

New tracking option
Customers can now see all absolute speed rates for finished data transformations, facilitating comparative analysis in pursuit of process improvements.

New Aspell Lookup table
Brand new implementation of this component brings better performance, improved configuration and better customization.

Improved treatment of empty (NULL) values
Developers can now specify special strings that should be treated as empty (NULL) when data is being parsed. This feature simplifies processing of typical application export files which often contain values insignificant for ETL processing. Additionally it may lead to improved processing throughput and lower memory consumption of data transformation.

More user friendly File URL dialog and improved LDAP functionality.

Customers can evaluate these new features along with CloverETL’s other leading capabilities with a free 30-day trial of the CloverETL Designer Pro evaluation, which is available at www.cloveretl.com Information management professionals can also evaluate the enterprise integration features of CloverETL Server via an online demo, which is also available at www.cloveretl.com.

Customers often tend to have obscure requirements. In a recent project we faced an interesting issue. Output records had to be split into unknown number of excel files according to their category. In addition, records within each file should have been written in datasheets according to their subcategory. The number of subcategories varied from 1 to 1024, so the whole solution seemed to me quite impractical.

Fortunately, we could solve the customer’s requirement very easily using CloverETL. For example, there are (among others) two fields – category and subcategory – in your metadata coming into XLSWriter. Then, if you set File URL in the form of filename_#.xls, Data sheet set as $subcategory and Partition key as category, the writer will split records into files according to the categories and into datasheets according to the subcategories.

Finally, the customer came to a conclusion that one file with many records is better than dozens datasheets within dozens files with very few records.

Settings of XLS_WRITER