LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-08-2021, 05:10 PM   #1
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Rep: Reputation: 0
Adding the date to the output file name of an awk script


Part 1)
I have a script the scans numerous csv's in a directory. The script selects various entries from those csv's and outputs the results to another csv in a sub directory.
I want to, via the script, include the date in the name of the output file.
Currently I have the date typed into the script, in the file name. Obviously I have to change that date every differing date that I run the script.
The current script looks like

gawk -F, '{ do various things } ' *.csv > output/filename_01-11-2021.csv

So, if I ran the script today I would have to manually change the "01-11-2021" to "08-11-2021" etc.

From what I have found so far it seems the automation would be via the use of the system-variable?? "date" but I can not figure out how to use it.

Question, if "date" is what I need to use, is the date recorded/set when the script is started or when it finishes or what? The script takes a while to run and has in the past straddled midnight.



Part 2)
In addition. I want to then process the above output file and out put the results to a different csv. Again I have the script to do this but the dates have to be changed every day that I run the script.
Using the above output file as the input for the second script the second script currently looks like

cd /output
gawk '{ do other various things } ' filename_01-11-2021.csv > filename2_01-11-2021.csv

As it tends to be late when I run the first script, the date may have changed whilst the first script was running. It might be better if the second script took the latest csv in the directory "output" as the input rather than looking for a name containing the value of "date".

I am on a windows computer at the moment but am wondering if something like the following would work

cd /output
ls -tr | tail -n1 |gawk '{ do other various things } ' > filename2_????.csv

where ???? represents the date as obtained in Part 1).

I would be grateful if someone could tell me how to bring about part 1), using, if feasible, the given script as the staring point.

Thank you.

Thanks

Last edited by sean mckinney; 11-08-2021 at 05:13 PM.
 
Old 11-08-2021, 05:34 PM   #2
michaelk
Moderator
 
Registered: Aug 2002
Posts: 22,081

Rep: Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441Reputation: 4441
Something like...
Code:
file1=$(date "+filename_%Y%m%d.csv")
file2=$(date "+filename2_%Y%m%d.csv")
gawk -F, '{ do various things } ' *.csv > /output/$file1
cd /output
gawk '{ do other various things } ' $file1 > $file2
 
2 members found this post helpful.
Old 11-08-2021, 05:34 PM   #3
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,037

Rep: Reputation: Disabled
See description of the date command in the GNU Coreutils Manual. You may experiment a bit to get a hang of it. E.g. try from the command line
Code:
date +%d-%m-%Y
 
1 members found this post helpful.
Old 11-08-2021, 07:25 PM   #4
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Original Poster
Rep: Reputation: 0
I will give those a go when I am next on the Linux machine. Thanks
 
Old 11-09-2021, 12:58 AM   #5
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Original Poster
Rep: Reputation: 0
Perfect. MANY thanks.
 
Old 11-09-2021, 02:36 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,971
Blog Entries: 3

Rep: Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102
Quote:
Originally Posted by michaelk View Post
Something like...
Code:
file1=$(date "+filename_%Y%m%d.csv")
file2=$(date "+filename2_%Y%m%d.csv")
gawk -F, '{ do various things } ' *.csv > /output/$file1
cd /output
gawk '{ do other various things } ' $file1 > $file2
Or if you want to be really paranoid about avoiding race conditions:

Code:
d=$(date)
file1=$(date -d "$d" "+filename_%Y%m%d.csv")
file2=$(date -d "$d" "+filename2_%Y%m%d.csv")
gawk -F, '{ do various things } ' *.csv > /output/$file1
cd /output
gawk '{ do other various things } ' $file1 > $file2
As in these examples, it is best to stay with the ISO 8601 date format whenever possible.
 
1 members found this post helpful.
Old 11-09-2021, 10:05 AM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,852

Rep: Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111Reputation: 3111
You could also do it all in gawk as it has commands for time and you can redirect inside the script
 
Old 11-09-2021, 11:07 AM   #8
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,224
Blog Entries: 1

Rep: Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585Reputation: 1585
> Or if you want to be really paranoid about avoiding race conditions...

Very good point, it is unlikely to be midnight between the two `date` command, but still possible

Last edited by NevemTeve; 11-09-2021 at 11:08 AM.
 
Old 11-10-2021, 02:02 PM   #9
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Original Poster
Rep: Reputation: 0
What I ended up doing .....
I added the second script into the end of the 'program' that runs the first script, so it became.......

file1=$(date "+filename1_%Y-%m-%d.csv")
gawk -F, '{ do various things } ' *.csv > /output/$file1

cd /output

file2=$(date "+filename1_%Y-%m-%d_addendum.csv")
gawk '{ do other various things } ' $file1 > $file2

It appears to work beautifully.
Many thanks

Last edited by sean mckinney; 11-10-2021 at 02:24 PM.
 
Old 11-11-2021, 04:15 AM   #10
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,856

Rep: Reputation: 827Reputation: 827Reputation: 827Reputation: 827Reputation: 827Reputation: 827Reputation: 827
Calling the date command twice bears the risk that the date have changed in between.
If you want to ensure an identical date then call date once and store its output in a variable:
Code:
today=$( date "+%Y-%m-%d" )
That is the common part.
Now it's easy to form the filenames; just append the appropriate strings:
Code:
file1="filename_${today}.csv"
file2="filename2_${today}.csv"
 
2 members found this post helpful.
Old 11-11-2021, 01:08 PM   #11
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Original Poster
Rep: Reputation: 0
That is a good point. If I were to run the program late at night on two consecutive days the results could be confusing.
I will think on this.
 
Old 11-12-2021, 12:17 AM   #12
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,925

Rep: Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631
I would definitely have gone with madeingermany's post ( Post #10) idea immediately; I was just curious to see what else was being said.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Adding a files creation date to it's file name? msmith88 Linux - Newbie 11 09-28-2018 04:48 PM
How to determine xrandr output name from sysfs output name? thinkpadboi Linux - Software 4 07-11-2018 09:29 PM
[SOLVED] sed inside awk or awk inside awk maddyfreaks Linux - Newbie 4 06-29-2016 02:10 PM
[SOLVED] Once again... awk.. awk... awk shivaa Linux - Newbie 13 12-31-2012 05:56 AM
[SOLVED] Use date with touch to create new file with date based name craigjl Linux - Newbie 5 03-12-2010 10:46 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:49 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration