Delete files more than N number of days old

Delete files more than N number of days old
4.4 (88%) 10 votes

Scenario: Delete files more than ‘N’ number of days older, using Talend.

Step 1: Create Job and add tFileList on canvas and configure as follows.

  • Set Directory path as your files location.
  • Select FileList Type as “Files”.
  • Case Sensitive=”No”.
  • Add File mask as “*.txt”. ( you can add per your file type).

Step 2: Add tFileProperties set path from tFileList_1 global variable. that could be ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

Step 3: Add tMap and connect with tFileProperties using main flow.

Step 4: Create context variable “Ndays” as Integer with default value =1, and “FilePathWithName” as String .

Step 5: Open tMap_1, add tMap variable “FileDate” with “Date” type, add another variable “OldDate” with “Date” Type see the picture for more details.

tMap Settings for Delete older Files
tMap Settings for Delete older Files

 

tMap Settings for Delete older Files

Step 5: Add below code to the respective variables.

You have file created date in “FileDate” variable, and the date from which you want to delete all the files, stored in “OldDate” variable.

Step 6: Create output link in tMap, and name as “DeleteFlow”. then drag and drop all the source columns to this output. use below code into the abs_path column.

context.FilePathWithName=row1.abs_path

We are doing assigning file name and path to the context variable FilePathWithName.

Step 7: Click on “Filter Expression” button on output link, and add below code in it.

Var.FileDate.before(Var.OldDate)==true

this code will check file created whether it is older than OldDate variable or not if yes then it will be pass to the next component for delete.

Final tMap_1 Setting looks like below image.

tMap Settings for Delete older Files
tMap Settings for Delete older Files

Step 8: Add tFileDelete component and connect with tMap using “deleteFlow”, then set the file Name = context.FilePathWithName context variable.

Step 9: Go to the context, click on "Ndays" variable and assign default value=30 (that is 30 days). save the job and execute.

You will see only one file deleted because today`s date is "13-03-2015" if we minus -30 days from today`s date we will get "11-2-2015" so we have only one file which is older than "11-2-2015".

you can add N number of days from day 1 to 360 or more. depends on your criteria how older files you want to delete.

output.

Before and After file delete output
Before and After file delete output

 

About Umesh

I am Software consultant with approx 7 years of experience mainly in Business Intelligence and data warehousing assignments using Talend. Writing is not my passion but i am doing it to help others. if you have any special case where you want me to demonstrate then please post me.

Leave a Reply

Your email address will not be published. Required fields are marked *