Create File with Incremental Number on each run of job

Create File with Incremental Number on each run of job
5 (100%) 5 votes

This post is based on  below comment from Mahesh who is facing problem, and wanted help from us.

Here is the comment for reference.

 

Hi Umesh,
the above post is very useful.
Could you please help me out on this ?
My requirement is :-
The file which gets generated after running etl is “XYZ_20150603_1.csv”

where XYZ is hardcoded value,
20150603 is the system date
and ‘1’ indicates number of times the etl has run in a single day.
Suppose if etl has ran for 2 times then the value needs to be ‘2’
My question is how to generate a counter like 1, 2, 3, …… based on the number of times an etl has run in a single day.

After reading this comment, I created solution but it was difficult to warp all the things as a replay so writing here.

Just follow these steps it will show you how to create file name as per mentioned scenario.

Step 1. List down the existing file name for same day.

  • Add tFileList component and configure as follows.
    • Case Sensitive=No
    • Use Global… =checked.
    • Files

"XYZ_"+TalendDate.getDate("yyyyMMdd")+"*"

In first step, it will list down all the files from provided directory. File mask will be used to search for a file name start with “XYZ_” and has today’s date in “yyyyMMdd” format, and ends with “.csv”.

See the Image below for more details.

Create File Name with Incremental Number
Create File Name with Incremental Number

Step 2. Extract incremental number from file name.

  • Add tJavaFlex next to tFilelist and connect with Iterator connection.
  • Now create context variable “Counter” with Integer type.
  • Add below code to the begin part of tJavaFlex.

String fileName=((String)globalMap.get("tFileList_1_CURRENT_FILE")); String counter =fileName.substring(fileName.lastIndexOf("_")+1).replace(".csv", "");

Above code will extract incremental number from file and store in “counter” variable which local to tJavaFlex.

  • Add below code to the Main part of tJavaFlex.

First line will parse the local counter variable value to Integer and store in “count” variable.

On second line we are incrementing extracted value with +1 and store in context variable “Counter”.

See the image for more details.

 

Create File Name with Incremental Number tJavaFlex code
Create File Name with Incremental Number tJavaFlex code

Step 3. Create File name with Incremented number.

  • Add tRowGenerator and create sample schema to generate sample data,
  • Add tFileOutputDelimited and add below code to the file name after path.

"D:/Sample/XYZ_"+TalendDate.getDate("yyyyMMdd")+"_"+context.Counter+".csv"

This will take last incremented number and create file name with provided date pattern along with incremented number.

Here is the complete setting.

Create File Name with Incremental Number tFileInputDelimited Setting
Create File Name with Incremental Number tFileInputDelimited Setting

If you execute this job first time it will create first file with incremented number 0+1= 1 like this==> “XYZ_20150609_1.csv”.

See the Files created by each run of job when I execute it from Talend.

Create File Name with Incremental Number Output
Create File Name with Incremental Number Output

 

 

About dwetl

3 comments on “Create File with Incremental Number on each run of job

  1. Tried the below code but file name is being generated with null value….plz guide where I am missing.
    Sample file name that generted after executing the code is : out_20160605_null.csv.

    1. I tested this code and working fine, I think you are missing any step, I suggest you to re validate all the steps and code once again, if you still face the problem then post here.

      Thank You
      Umesh

Leave a Reply

Your email address will not be published. Required fields are marked *