LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 06-25-2019, 02:32 PM   #1
JugaruGabi
LQ Newbie
 
Registered: Jun 2019
Posts: 7

Rep: Reputation: Disabled
CAT multiple files with filename pattern


Hello,

I have the following scenario:
an application is creating each day a file that is having some records in plain text. The files are being saved with the following name:

Code:
auto_test_01-Jan-2019.txt
auto_test_02-Jan-2019.txt
auto_test_03-Jan-2019.txt
auto_test_04-Jan-2019.txt
..
auto_test_31-Jan-2019.txt
auto_test_01-Feb-2019.txt
I need to build up a bash file that is requesting a starting date, an ending date and, based on the starting date and ending date, it needs to CAT the files on the server between these dates (including the starting and ending date) into a big file which will be processed later using an awk command that is currently stored on a different server.

Can someone help me build up this bash in order for the requirement to be achieved?

I have tried some ideas but I could not get it to work as expected (either it shows the output for the first file, either is showing the output for the last file multiple time).

My knowledge in bash scripting is at the beginner level, so please bare with me.

Thanks!
 
Old 06-25-2019, 03:19 PM   #2
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,153

Rep: Reputation: 424Reputation: 424Reputation: 424Reputation: 424Reputation: 424
Quote:
I need to build up a bash file that is requesting a starting date, an ending date
Get the dates as variables.
Code:
read -p "Enter starting date" s_date
read -p "Enter ending date" e_date
echo "$s_date"
echo "$e_date"
Quote:
it needs to CAT the files on the server between these dates
Examples:
Code:
a=2
b=20

eval "c=({$a..$b})"; echo ${c[*]}

for ((i=a; i<=b; ++i)); do echo "$i"; done
Quote:
into a big file which will be processed later
Example:
Code:
list=(auto_test_01-Jan-2019.txt
auto_test_02-Jan-2019.txt
auto_test_03-Jan-2019.txt
auto_test_04-Jan-2019.txt
)

for i in "${list[@]}"; do
    cat "$i" >> BigFile.txt
done
 
Old 06-25-2019, 03:21 PM   #3
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware 14.2 current / ArcoLinux / Void Linux
Posts: 8,815

Rep: Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837
my draft
Code:
#!/bin/bash
#set -x

testfiles=(
auto_test_01-Feb-2019.txt
auto_test_01-Jan-2019.txt
auto_test_02-Jan-2019.txt
auto_test_03-Jan-2019.txt
auto_test_04-Jan-2019.txt
auto_test_31-Jan-2019.txt
)

wd="$HOME/testing"
wd2=$HOME/filess

mkdir -p "$wd"
mkdir -p "$wd2"

##this loop was just for creating the test files
for i in "${testfiles[@]}"
do
	touch $HOME/testing/$i
done

#populate files in dir into an array
mapfile -t workingArray < <(find "$wd" -type f -name "*.txt" )

#sort the files by dd-mm-yy out into another array
 
arr=($( sort -t- -k 3.1n,3.2 -k 2.1M,2.3 -k 1.1n,1.2  < <(printf '%s\n' "${workingArray[@]}")))
#found that part up there on line and put it together. sort and add to another array. 


#here we finalize the process to a main file
#cat contents of files by dd-mm-yy order into a main file with path/file 
#to tell where it came from , then contents, then new line for next file
for i in "${arr[@]}"
do
#adds in file name of file and path
	printf "%s\n" "$i" >> $wd2/logfile 
#contents of file
	cat $i >> $wd2/logfile
#new line for next file
	printf "%s\n" >>  $wd2/logfile
done

Last edited by BW-userx; 06-25-2019 at 04:52 PM.
 
Old 06-26-2019, 12:57 AM   #4
JugaruGabi
LQ Newbie
 
Registered: Jun 2019
Posts: 7

Original Poster
Rep: Reputation: Disabled
Unsure if my initial post was too clear:

The files that are in dir1 are:
Code:
auto_test_01-Jan-2019.txt
auto_test_02-Jan-2019.txt
auto_test_03-Jan-2019.txt
auto_test_04-Jan-2019.txt
.
.
.
.
.
.
.
auto_test_15-Jan-2019.txt
The dots from this count represent the files from days up until 31st January.

Based on the string parameter inserted in:
Code:
read -p "Enter starting date" s_date
read -p "Enter ending date" e_date
where
Code:
s_date = 01-Jan-2019
e_date = 15-Jan-2019
the bash must execute the following cat command automatically:
Code:
cat auto_test_01-Jan-2019.txt auto_test_02-Jan-2019.txt auto_test_03-Jan-2019.txt auto_test_04-Jan-2019.txt auto_test_05-Jan-2019.txt auto_test_06-Jan-2019.txt auto_test_07-Jan-2019.txt auto_test_08-Jan-2019.txt auto_test_09-Jan-2019.txt auto_test_10-Jan-2019.txt auto_test_11-Jan-2019.txt auto_test_12-Jan-2019.txt auto_test_13-Jan-2019.txt auto_test_14-Jan-2019.txt auto_test_15-Jan-2019.txt > Period:auto_test$s_date-$e_date.txt
That means, the concatenation command should concatenate automatically the files having the filename pattern auto_test_<date>.txt stored in the period from 1st of January up until 15th of January 2019.

Also, I have seen that both of you have used the list option; isn't there a way to skip this part in the scripting area?
If I am creating the bash file, it is mandatory to create the list with the all 365 days that are present in the calendar?
Isn't there an elegant option to pass the information based on the starting date, ending date and the pattern of the filename?

Most likely, the report will be run with the following parameters as dates:
Code:
s_date=15-Jan-2019
e_date=15-Feb-2019
and it must cat the all of the files in the interval between these 2 dates.
 
Old 06-26-2019, 01:32 AM   #5
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,203

Rep: Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859Reputation: 1859
Can be done with one line in bash. Keep it in a file and edit as appropriate.
Code:
 cat auto_test_{15..31}-Jan-2019 auto_test_{01..15}-Feb-2019 >> Period

Last edited by allend; 06-26-2019 at 01:34 AM.
 
Old 06-26-2019, 01:44 AM   #6
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and CentOS
Posts: 5,879

Rep: Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362
Hi,

make use of the date command.
Code:
#!/bin/bash

s_date=15-Jan-2019
e_date=15-Feb-2019

if [ "$1" != "" ] ; then
  s_date=$1
fi
if [ "$2" != "" ] ; then
  e_date=$2
fi

fs=""
i=0
while true ; do
    date=$(date +'%d-%b-%Y' --date "$s_date +$i days")
    ((i+=1))
    f="auto_test_${date}.txt"
    if [ -f $f ] ; then # Check the file exists.. maybe some days are missing?
      fs="$fs $f"
    fi
    if [ "$date" == "$e_date" ] ; then
	break
    fi
done
cat $fs > auto_test$s_date-$e_date.txt
Modify the final line as desired.

Evo2.

Last edited by evo2; 06-26-2019 at 01:56 AM. Reason: Take dates as arguments and cleanup!
 
2 members found this post helpful.
Old 06-26-2019, 03:14 AM   #7
JugaruGabi
LQ Newbie
 
Registered: Jun 2019
Posts: 7

Original Poster
Rep: Reputation: Disabled
Code:

Code:
#!/bin/bash

s_date=15-Jan-2019
e_date=15-Feb-2019

if [ "$1" != "" ] ; then
  s_date=$1
fi
if [ "$2" != "" ] ; then
  e_date=$2
fi

fs=""
i=0
while true ; do
    date=$(date +'%d-%b-%Y' --date "$s_date +$i days")
    ((i+=1))
    f="auto_test_${date}.txt"
    if [ -f $f ] ; then # Check the file exists.. maybe some days are missing?
      fs="$fs $f"
    fi
    if [ "$date" == "$e_date" ] ; then
	break
    fi
done
cat $fs > auto_test$s_date-$e_date.txt
Made the trick work.
Thanks.

One last question though.
In the meantime, business has decided that in the future they will be switching the pattern to:
auto_test_2019-Jun-15.txt for the files.

1. So, if I am changing the following line:
Code:
date=$(date +'%d-%b-%Y' --date "$s_date +$i days")
to
Code:
date=$(date +'%Y-%b-%d' --date "$s_date +$i days")
the bash hangs and nothing else happens.

2. If am switching the s_date and e_date to:
Code:
s_date=2019-Jan-15
e_date=2019-Feb-15
and
Code:
date=$(date +'%d-%b-%Y' --date "$s_date +$i days")
to
Code:
date=$(date +'%Y-%b-%d' --date "$s_date +$i days")
the bash results in errors like:
Code:
date: invalid date ‘2019-Jan-15 +1199 days’
date: invalid date ‘2019-Jan-15 +1200 days’
date: invalid date ‘2019-Jan-15 +1201 days’
date: invalid date ‘2019-Jan-15 +1202 days’
@evo2 What am I doing wrong?

Last edited by JugaruGabi; 06-26-2019 at 03:22 AM.
 
Old 06-26-2019, 03:32 AM   #8
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and CentOS
Posts: 5,879

Rep: Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362
Hi,

it seems that '%Y-%b-%d' is not a format the date automatically recognises. So, you'll need to use a different format for loop control to what is used in the file names.

Eg. Make the following change in the script:
Code:
((i+=1))
f="auto_test_${date}.txt"
becomes
Code:
f="auto_test_$(date +'%Y-%b-%d' --date "$s_date +$i days").txt"
((i+=1))
And keep the rest of the script as is.

Evo2.
 
1 members found this post helpful.
Old 06-26-2019, 05:24 AM   #9
JugaruGabi
LQ Newbie
 
Registered: Jun 2019
Posts: 7

Original Poster
Rep: Reputation: Disabled
Yep, this worked.
Thanks a lot.

This can be closed now.
 
Old 06-26-2019, 08:23 AM   #10
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware 14.2 current / ArcoLinux / Void Linux
Posts: 8,815

Rep: Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837Reputation: 1837
Just taking what works, and adding more error checking, ease of use to it, and a log.
Code:
#!/bin/bash

#set -x

#So script does not have to 
#be ran in the same dir as 
#the files that need to be
#processed are in.
#working dir where files
#are kept
wd="$HOME/dayfiles"
#Where final process file
#is being kept
pf="$HOME/Processed"

#make sure that
#directory is present.
mkdir "$pf"

#Log file of files
#that have been processed
LogFileDir=$HOME/logfile
LogFileName="processedFiles"

mkdir $LogFileDir

#for date header for logfile
FLAG=T

#for ease of changing  specs
#in usage message
dateFormat="dd-M-yy"
example=" 03-Feb-2018 15-Mar-2018"

#default/test dates ??
s_date=15-Jan-2019
e_date=15-Feb-2019
#error checking
#if less than 2 arguments
#print error / usage message

if [[ $# -lt '2' ]] ; then
	printf "
	Needs two arguments
	begining date, ending date
	date format $dateFormat
	example: 
	$0 $example\n"
	exit
else
	e_date=$2 ; s_date=$1
fi

fs=""
i=0
while true ; do
    date=$(date +'%d-%b-%Y' --date "$s_date +$i days")
    ((i++))
    #f="auto_test_${date}.txt"
    #add path to files/file
    
    f="$wd"/"auto_test_${date}.txt"
  #############################################
  #to create some test files to ensure this
  #works, Can be removed
    if [[ "$MAKE_TEST" = "YES" ]] ; then
	touch "$f"
        echo "$f" > "$f"
    fi
    #################
    if [ -f $f ] ; then # Check the file exists.. maybe some days are missing?
      fs="$fs $f"
    fi
    if [ "$date" == "$e_date" ] ; then
	break
    fi
   
done
cat $fs > "$pf"/auto_test$s_date-$e_date.txt

#adds time stamp to logfile for each run
[[ $FLAG = T ]] && { echo $(date) >> $LogFileDir/$LogFileName ;
	FLAG=F ; }
#add file name of the file that was just completed being
#processed to a Log file for assurance of its 
#completion in case of some type of faliure
#one needs to know where they left off
printf "%s\n" $fs >> $LogFileDir/$LogFileName

Last edited by BW-userx; 06-26-2019 at 09:01 AM.
 
Old 06-26-2019, 01:17 PM   #11
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,122

Rep: Reputation: 504Reputation: 504Reputation: 504Reputation: 504Reputation: 504Reputation: 504
If there is a file each day, you can use simple start and stop matches, if the files are sorted in sequence.
Code:
# example:
s_date=15-Jan-2019
e_date=15-Feb-2019
ls | sort -t- -k 3,3n -k 2,2M -k 1,1 |
  sed -n "/$s_date/,/$e_date/p"
Or, relying on the correct time stamp
Code:
ls -tr |
  sed -n "/$s_date/,/$e_date/p"
If okay, add a pipe to
Code:
  xargs cat

Last edited by MadeInGermany; 06-26-2019 at 01:20 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
vi search multiple pattern and remove only exact matching pattern amateurscripter Linux - Newbie 4 05-07-2018 01:19 PM
[SOLVED] How to remove certain pattern files except another certain pattern files from a list? Mike_Brown Linux - Newbie 4 04-23-2016 12:30 AM
Removing filename~ files from ls just like .filename files are hidden nkoplm Linux - General 3 10-04-2011 10:31 AM
change uploaded files from filename.avi to filename.avi.html like www.rapidshare.de latheesan Linux - Newbie 3 06-16-2005 04:33 AM
filename- and filename~ files? slinky2004 Linux - Newbie 5 10-17-2004 10:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 01:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration