LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-25-2016, 03:03 AM   #1
sysmicuser
Member
 
Registered: Mar 2010
Posts: 423

Rep: Reputation: 0
merge files in a directory


Hey Guys,

I want to merge all files (Apache Tomcat Access Logs) for a particular date say "Aug 24" to be merged into a single file.

Is there any quick hack for that ?
Code:
[tomcat@localhost logs]$ ls -alrth access_log2016-08-*|grep "Aug 24"
-rw-rw-r--. 1 tomcat        tomcat          16M Aug 24 00:00 access_log2016-08-23.23.log
-rw-rw-r--. 1 tomcat        tomcat         6.8M Aug 24 01:00 access_log2016-08-24.00.log
-rw-rw-r--. 1 tomcat        tomcat         4.7M Aug 24 02:00 access_log2016-08-24.01.log
-rw-rw-r--. 1 tomcat        tomcat          14M Aug 24 03:00 access_log2016-08-24.02.log
-rw-rw-r--. 1 tomcat        tomcat          18M Aug 24 04:00 access_log2016-08-24.03.log
-rw-rw-r--. 1 tomcat        tomcat          15M Aug 24 05:00 access_log2016-08-24.04.log
-rw-rw-r--. 1 tomcat        tomcat         5.6M Aug 24 06:00 access_log2016-08-24.05.log
-rw-rw-r--. 1 tomcat        tomcat         8.9M Aug 24 07:00 access_log2016-08-24.06.log
-rw-rw-r--. 1 tomcat        tomcat          19M Aug 24 08:00 access_log2016-08-24.07.log
-rw-rw-r--. 1 tomcat        tomcat          32M Aug 24 09:00 access_log2016-08-24.08.log
-rw-rw-r--. 1 tomcat        tomcat          45M Aug 24 10:00 access_log2016-08-24.09.log
-rw-rw-r--. 1 tomcat        tomcat          44M Aug 24 11:00 access_log2016-08-24.10.log
-rw-rw-r--. 1 tomcat        tomcat          49M Aug 24 12:00 access_log2016-08-24.11.log
-rw-rw-r--. 1 tomcat        tomcat          51M Aug 24 13:00 access_log2016-08-24.12.log
-rw-rw-r--. 1 tomcat        tomcat          53M Aug 24 14:00 access_log2016-08-24.13.log
-rw-rw-r--. 1 tomcat        tomcat          52M Aug 24 15:00 access_log2016-08-24.14.log
-rw-rw-r--. 1 tomcat        tomcat          84M Aug 24 16:00 access_log2016-08-24.15.log
-rw-rw-r--. 1 tomcat        tomcat          57M Aug 24 17:00 access_log2016-08-24.16.log
-rw-rw-r--. 1 tomcat        tomcat          48M Aug 24 18:00 access_log2016-08-24.17.log
-rw-rw-r--. 1 tomcat        tomcat          37M Aug 24 19:00 access_log2016-08-24.18.log
-rw-rw-r--. 1 tomcat        tomcat          38M Aug 24 20:00 access_log2016-08-24.19.log
-rw-rw-r--. 1 tomcat        tomcat          40M Aug 24 21:00 access_log2016-08-24.20.log
-rw-rw-r--. 1 tomcat        tomcat          37M Aug 24 22:00 access_log2016-08-24.21.log
-rw-rw-r--. 1 tomcat        tomcat          26M Aug 24 23:00 access_log2016-08-24.22.log
[tomcat@localhost logs]$
They should be merged into single file as per date time in ascending order so access_log2016-08-23.23.log then access_log2016-08-24.00.log so on and so forth.

We can do manually like cat individual file but want to explore more smarter way. I heard from my colleague a command called plus to concatenate into single file ,but not 100% sure..

Please assist.
 
Old 08-25-2016, 05:36 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,707

Rep: Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497Reputation: 2497
Code:
#my files
 ls
f1  f2  f3 

# contents
cat *
f1-data

f2-data

f3-data

# concat in date order (f1, f2, f3)
for file in $(ls -rt)
do 
cat $file>>f4
done

cat f4
f1-data

f2-data

f3-data
 
Old 08-25-2016, 05:48 AM   #3
malekmustaq
Senior Member
 
Registered: Dec 2008
Location: root
Distribution: Slackware & BSD
Posts: 1,667

Rep: Reputation: 494Reputation: 494Reputation: 494Reputation: 494Reputation: 494
Code:
cat file 1 >> file 2
contents of file 1 is appended to file 2.
Check it
Code:
cat file 2
If there are plenty of files you can FOR IN WHILE and DO script it.
 
Old 08-25-2016, 11:23 AM   #4
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,769

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
I would use your existing glob and test the file date information with stat and then cat those desired. You could also try feeding a while loop with find assuming you know how long ago you want to concatenate.
 
Old 08-25-2016, 11:30 AM   #5
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,281

Rep: Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916
Code:
cat access_log2016-08-24* > sysmicuser.log
?
 
Old 08-25-2016, 12:55 PM   #6
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,230

Rep: Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724
Code:
files=($(ls -alrth access_log2016-08-*|grep "Aug 24" | awk '{print $9}'))
cat "${files[@]}" > access_log2016-08_Aug24_merged.log
edit, I like schneidz simple suggestion though

Last edited by keefaz; 08-25-2016 at 12:57 PM.
 
1 members found this post helpful.
Old 08-25-2016, 01:49 PM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,769

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
schneidz example is simple but in the example there is a file with 23 at the top which still has a 24th date on it.
 
Old 08-25-2016, 01:56 PM   #8
Rinndalir
Member
 
Registered: Sep 2015
Posts: 733

Rep: Reputation: Disabled
@schneidz is the correct answer.
 
Old 08-25-2016, 03:44 PM   #9
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,230

Rep: Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724
Quote:
Originally Posted by grail View Post
schneidz example is simple but in the example there is a file with 23 at the top which still has a 24th date on it.
Yes with 00:00 modification time, personally I would go away counting it in the previous set (2016-08-23)
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
merge columns from multiple files in a directory based on match of two columns prasanthi yanamala Linux - Newbie 2 11-12-2015 11:11 AM
Best way to merge two directory trees with replace? legacyprog Linux - General 5 12-24-2011 09:15 PM
Merge tar files ust Linux - Newbie 6 06-20-2011 06:02 PM
Merge directory trees mrwall-e Linux - Software 5 02-20-2011 09:37 AM
compare two files and merge nelex Programming 5 02-25-2009 09:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:09 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration