LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-04-2015, 06:13 PM   #1
rmalkmus
LQ Newbie
 
Registered: Mar 2015
Location: Denver
Distribution: Ubuntu
Posts: 6

Rep: Reputation: Disabled
exporting log data to a file that matches stdout


hey guys,

Let's say I want to find out which log files have related ntp information in them. I use cat and grep to search through the files in /var/log and then export that to a file. this is the command...

# cat /var/log/* | grep ntp > /home/log.txt

The file created from this command will not include the directories the log entries are apart of. Why not? For example, if you do this same command without exporting to the /home/log.txt file it will show you in stdout which directory each log entry is in. Hope I'm making sense here. My question is, is there a clever way to export to a file in a way so that the file created is structured exactly like the stdout of the command below?

# cat /var/log/* | grep ntp
 
Old 06-04-2015, 06:28 PM   #2
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 979
Blog Entries: 2

Rep: Reputation: 235Reputation: 235Reputation: 235
grep what-you-want /var/log/* > where-you-want-it

Doing cat first and "| grep" is a mistake as the grep will see stdin as the source of the data and know nothing about filenames.

Just in case there had been only one file matching (there won't in this case) you could include /dev/null in the list.

Because /var/log/ usually contains compressed logs you probably want to use zgrep .
 
1 members found this post helpful.
Old 06-04-2015, 06:29 PM   #3
Habitual
LQ Addict
 
Registered: Jan 2011
Location: Youngstown, Ohio
Distribution: LM17.1/Xfce4.11.8
Posts: 7,189
Blog Entries: 10

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Code:
sudo grep ntp /var/log/ -l > $HOME/log.txt
for that directory file match/listing.
Code:
sudo grep ntp /var/log/ -Rl > $HOME/log.txt
for sub-directory file match/listing.

Example output:
Code:
sudo grep ntp /var/log/ -Rl 
/var/log/bootstrap.log
/var/log/dmesg
/var/log/kern.log.1
/var/log/atop/atop_20150602
/var/log/atop/atop_20150527
/var/log/auth.log
/var/log/dmesg.0
/var/log/installer/partman
/var/log/installer/syslog
/var/log/boot.log
cat log.txt:
Code:
/var/log/bootstrap.log
/var/log/dmesg
/var/log/kern.log.1
/var/log/atop/atop_20150602
/var/log/atop/atop_20150527
/var/log/auth.log
/var/log/dmesg.0
/var/log/installer/partman
/var/log/installer/syslog
/var/log/boot.log
Hope that helps.

Last edited by Habitual; 06-04-2015 at 06:32 PM.
 
2 members found this post helpful.
Old 06-04-2015, 06:46 PM   #4
rmalkmus
LQ Newbie
 
Registered: Mar 2015
Location: Denver
Distribution: Ubuntu
Posts: 6

Original Poster
Rep: Reputation: Disabled
Oh cool! Okay that makes sense. Thanks guys, both replies were helpful.
 
Old 06-04-2015, 06:49 PM   #5
rmalkmus
LQ Newbie
 
Registered: Mar 2015
Location: Denver
Distribution: Ubuntu
Posts: 6

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by linosaurusroot View Post

Just in case there had been only one file matching (there won't in this case) you could include /dev/null in the list.
What do you mean by this exactly linosaurusroot? This is the only thing i'm a little fuzzy on. I could include /dev/null in the list? What is the list?
 
Old 06-04-2015, 07:02 PM   #6
joec@home
Member
 
Registered: Sep 2009
Location: Galveston Tx
Posts: 291

Rep: Reputation: 70
Quote:
Originally Posted by Habitual View Post
Code:
sudo grep ntp /var/log/ -l > $HOME/log.txt
for that directory file match/listing.
Code:
sudo grep ntp /var/log/ -Rl > $HOME/log.txt
for sub-directory file match/listing.

Example output:
Code:
sudo grep ntp /var/log/ -Rl 
/var/log/bootstrap.log
/var/log/dmesg
/var/log/kern.log.1
/var/log/atop/atop_20150602
/var/log/atop/atop_20150527
/var/log/auth.log
/var/log/dmesg.0
/var/log/installer/partman
/var/log/installer/syslog
/var/log/boot.log
cat log.txt:
Code:
/var/log/bootstrap.log
/var/log/dmesg
/var/log/kern.log.1
/var/log/atop/atop_20150602
/var/log/atop/atop_20150527
/var/log/auth.log
/var/log/dmesg.0
/var/log/installer/partman
/var/log/installer/syslog
/var/log/boot.log
Hope that helps.
Excellent post this one was, but on busier systems you can seriously bog down a system unless you sort out files that are not ASCII legible, I.E. binary log files.Some tools I developed to better manage this:

Code:
LOGS_INDEX(){
echo "Indexing log files found in /var/log/ tree. This may take a moment."
LOG[1]=`find /var/log/. |
xargs file |
awk '{ if ( $2 == "ASCII" ) print $1 }' |
cut -d: -f1`
echo ""
} ;\

LOGS_SEARCH_A(){
echo
echo "Search found the following information"
echo "###########################################################################"
echo "${LOG[@]}" |
xargs egrep $FUZZY_TYPE "$SEARCH"
echo "###########################################################################"
echo ""
} ;\

LOGS_SEARCH_B(){
echo
echo "Search found the following information"
echo "###########################################################################"
echo "${LOG[@]}" |
xargs grep -h $FUZZY_TYPE "$SEARCH" |
sort
echo "###########################################################################"
echo ""
} ;\
Example of using the tools, first load the libraries:

Code:
[root@localhost ~]# LOGS_INDEX(){
> echo "Indexing log files found in /var/log/ tree. This may take a moment."
> LOG[1]=`find /var/log/. |
> xargs file |
> awk '{ if ( $2 == "ASCII" ) print $1 }' |
> cut -d: -f1`
> echo ""
> } ;\
>
[root@localhost ~]# LOGS_SEARCH_A(){
> echo
> echo "Search found the following information"
> echo "###########################################################################"
> echo "${LOG[@]}" |
> xargs egrep $FUZZY_TYPE "$SEARCH"
> echo "###########################################################################"
> echo ""
> } ;\
>
[root@localhost ~]# LOGS_SEARCH_B(){
> echo
> echo "Search found the following information"
> echo "###########################################################################"
> echo "${LOG[@]}" |
> xargs grep -h $FUZZY_TYPE "$SEARCH" |
> sort
> echo "###########################################################################"
> echo ""
> } ;\
>
Then call them to run the search

Code:
[root@localhost ~]# SEARCH='What I am searching for'
[root@localhost ~]# LOGS_SEARCH_A > /root/loginfo.txt
FUZZY_TYPE can be switches such as "-ab5", and you only need to run LOGS_INDEX once per session. LOGS_SEARCH_A is egrep and LOGS_SEARCH_B is plain grep. Basically I am using a cheat in that I am storing the routines as a local hash variable of the session shell. This way I do not need to save them as files to run, but would work the same as a script if needed for cron jobs and the like. However for cron jobs the shell changes and you would need to run LOGS_INDEX each time.
 
Old 06-04-2015, 08:40 PM   #7
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 979
Blog Entries: 2

Rep: Reputation: 235Reputation: 235Reputation: 235
Quote:
Originally Posted by rmalkmus View Post
What do you mean by this exactly linosaurusroot? This is the only thing i'm a little fuzzy on. I could include /dev/null in the list? What is the list?

If the only file in /var/log/ is PENGUIN then /var/log/* means /var/log/PENGUIN and "grep ntp /var/log/PENGUIN" doesn't show what file it comes from.

But "grep ntp /var/log/* /dev/null" does because now grep is searching both /var/log/PENGUIN and /dev/null .
 
1 members found this post helpful.
Old 06-08-2015, 09:55 AM   #8
rmalkmus
LQ Newbie
 
Registered: Mar 2015
Location: Denver
Distribution: Ubuntu
Posts: 6

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by linosaurusroot View Post
If the only file in /var/log/ is PENGUIN then /var/log/* means /var/log/PENGUIN and "grep ntp /var/log/PENGUIN" doesn't show what file it comes from.

But "grep ntp /var/log/* /dev/null" does because now grep is searching both /var/log/PENGUIN and /dev/null .

I see what you mean now. Thank you
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
tar - write to stdout & create log files ( from stdout and stderr ) [solved] paziulek Linux - General 2 02-23-2014 01:26 PM
[SOLVED] stderr stdout to a log file troubles metallica1973 Linux - Software 3 11-12-2011 05:28 PM
Is there any way to log startup messages without making stdout >> file ? harryhaller Slackware 8 06-25-2011 08:32 PM
Dump all uploaded data to stdout or file (DSL, PPPoE) hansschmucker Linux - Networking 0 10-21-2004 07:43 PM
Monitoring a Specific Port and Exporting to a log file chrisfirestar Linux - General 0 10-27-2003 05:17 AM


All times are GMT -5. The time now is 12:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration