[SOLVED] merge files in a directory

sysmicuser · 08-25-2016, 02:03 AM

Hey Guys,

I want to merge all files (Apache Tomcat Access Logs) for a particular date say "Aug 24" to be merged into a single file.

Is there any quick hack for that ?

Code:

[tomcat@localhost logs]$ ls -alrth access_log2016-08-*|grep "Aug 24"
-rw-rw-r--. 1 tomcat        tomcat          16M Aug 24 00:00 access_log2016-08-23.23.log
-rw-rw-r--. 1 tomcat        tomcat         6.8M Aug 24 01:00 access_log2016-08-24.00.log
-rw-rw-r--. 1 tomcat        tomcat         4.7M Aug 24 02:00 access_log2016-08-24.01.log
-rw-rw-r--. 1 tomcat        tomcat          14M Aug 24 03:00 access_log2016-08-24.02.log
-rw-rw-r--. 1 tomcat        tomcat          18M Aug 24 04:00 access_log2016-08-24.03.log
-rw-rw-r--. 1 tomcat        tomcat          15M Aug 24 05:00 access_log2016-08-24.04.log
-rw-rw-r--. 1 tomcat        tomcat         5.6M Aug 24 06:00 access_log2016-08-24.05.log
-rw-rw-r--. 1 tomcat        tomcat         8.9M Aug 24 07:00 access_log2016-08-24.06.log
-rw-rw-r--. 1 tomcat        tomcat          19M Aug 24 08:00 access_log2016-08-24.07.log
-rw-rw-r--. 1 tomcat        tomcat          32M Aug 24 09:00 access_log2016-08-24.08.log
-rw-rw-r--. 1 tomcat        tomcat          45M Aug 24 10:00 access_log2016-08-24.09.log
-rw-rw-r--. 1 tomcat        tomcat          44M Aug 24 11:00 access_log2016-08-24.10.log
-rw-rw-r--. 1 tomcat        tomcat          49M Aug 24 12:00 access_log2016-08-24.11.log
-rw-rw-r--. 1 tomcat        tomcat          51M Aug 24 13:00 access_log2016-08-24.12.log
-rw-rw-r--. 1 tomcat        tomcat          53M Aug 24 14:00 access_log2016-08-24.13.log
-rw-rw-r--. 1 tomcat        tomcat          52M Aug 24 15:00 access_log2016-08-24.14.log
-rw-rw-r--. 1 tomcat        tomcat          84M Aug 24 16:00 access_log2016-08-24.15.log
-rw-rw-r--. 1 tomcat        tomcat          57M Aug 24 17:00 access_log2016-08-24.16.log
-rw-rw-r--. 1 tomcat        tomcat          48M Aug 24 18:00 access_log2016-08-24.17.log
-rw-rw-r--. 1 tomcat        tomcat          37M Aug 24 19:00 access_log2016-08-24.18.log
-rw-rw-r--. 1 tomcat        tomcat          38M Aug 24 20:00 access_log2016-08-24.19.log
-rw-rw-r--. 1 tomcat        tomcat          40M Aug 24 21:00 access_log2016-08-24.20.log
-rw-rw-r--. 1 tomcat        tomcat          37M Aug 24 22:00 access_log2016-08-24.21.log
-rw-rw-r--. 1 tomcat        tomcat          26M Aug 24 23:00 access_log2016-08-24.22.log
[tomcat@localhost logs]$

They should be merged into single file as per date time in ascending order so access_log2016-08-23.23.log then access_log2016-08-24.00.log so on and so forth.

We can do manually like cat individual file but want to explore more smarter way. I heard from my colleague a command called plus to concatenate into single file ,but not 100% sure..

Please assist.

chrism01 · 08-25-2016, 04:36 AM

Code:

#my files
 ls
f1  f2  f3 

# contents
cat *
f1-data

f2-data

f3-data

# concat in date order (f1, f2, f3)
for file in $(ls -rt)
do 
cat $file>>f4
done

cat f4
f1-data

f2-data

f3-data

malekmustaq · 08-25-2016, 04:48 AM

Code:

cat file 1 >> file 2

contents of file 1 is appended to file 2.
Check it

Code:

cat file 2

If there are plenty of files you can FOR IN WHILE and DO script it.

grail · 08-25-2016, 10:23 AM

I would use your existing glob and test the file date information with stat and then cat those desired. You could also try feeding a while loop with find assuming you know how long ago you want to concatenate.

schneidz · 08-25-2016, 10:30 AM

Code:

cat access_log2016-08-24* > sysmicuser.log

?

keefaz · 08-25-2016, 11:55 AM

Code:

files=($(ls -alrth access_log2016-08-*|grep "Aug 24" | awk '{print $9}'))
cat "${files[@]}" > access_log2016-08_Aug24_merged.log

edit, I like schneidz simple suggestion though

grail · 08-25-2016, 12:49 PM

schneidz example is simple but in the example there is a file with 23 at the top which still has a 24th date on it.

Rinndalir · 08-25-2016, 12:56 PM

@schneidz is the correct answer.

keefaz · 08-25-2016, 02:44 PM

Quote:

Originally Posted by grail

schneidz example is simple but in the example there is a file with 23 at the top which still has a 24th date on it.

Yes with 00:00 modification time, personally I would go away counting it in the previous set (2016-08-23)