This post is based on below comment from Mahesh
Here is the comment for reference.
the above post is very useful.
Could you please help me out on this ?
My requirement is :-
The file which gets generated after running etl is “XYZ_20150603_1.csv”
where XYZ is hardcoded value,
20150603 is the system date
and ‘1’ indicates number of times the etl has run in a single day.
Suppose if etl has ran for 2 times then the value needs to be ‘2’
My question is how to generate a counter like 1, 2, 3, …… based on the number of times an etl has run in a single day.
After reading this comment, I created solution but it was difficult to warp all the things as a replay so writing here.
Just follow these steps it will show you how to create file name as per mentioned scenario.
Step 1. List down the existing file name for same day.
- Add tFileList component and configure as follows.
- Case Sensitive=No
- Use Global… =checked.
In first step, it will list down all the files from provided directory. File mask will be used to search for a file name start with “XYZ_” and has today’s date in “yyyyMMdd” format, and ends with “.csv”.
See the Image below for more details.
Step 2. Extract incremental number from file name.
- Add tJavaFlex next to tFilelist and connect with Iterator connection.
- Now create context variable “Counter” with Integer type.
- Add below code to the begin part of tJavaFlex.
String fileName=((String)globalMap.get(“tFileList_1_CURRENT_FILE”)); String counter =fileName.substring(fileName.lastIndexOf(“_”)+1).replace(“.csv”, “”);
Above code will extract incremental number from file and store in “counter” variable which local to tJavaFlex.
- Add below code to the Main part of tJavaFlex.
int count= Integer.valueOf(counter); context.Counter=count+1; System.out.println(count);
First line will parse the local counter variable value to Integer and store in “count” variable.
On second line we are incrementing extracted value with +1 and store in context variable “Counter”.
See the image for more details.
Step 3. Create File name with Incremented number.
- Add tRowGenerator and create sample schema to generate sample data,
- Add tFileOutputDelimited and add below code to the file name after path.
This will take last incremented number and create file name with provided date pattern along with incremented number.
Here is the complete setting.
If you execute this job first time it will create first file with incremented number 0+1= 1 like this==> “XYZ_20150609_1.csv”.
See the Files created by each run of job when I execute it from Talend.