LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-11-2012, 08:47 AM   #1
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Rep: Reputation: Disabled
substitute characters in a continuous log file


Hi,

I was trying to pipe the output from a logfile through sed to make the contents more readable.
The 'cat' command gives me the results I expect, but with a 'tail -f' it is not working.
How should I use this command for a 'tail -f' situation ?
Code:
cat kermit_11-10-12.log | sed 's/ \{2,\}/\n/g' | grep -v Idle
 
Old 10-12-2012, 08:42 AM   #2
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
115 views and no reply yet ?
This can't be such a hard question for command-gurus, can it ?
Maybe I should have chosen another title, something like
'using sed in combination with tail -f'

Last edited by oneindelijk; 10-12-2012 at 08:43 AM.
 
Old 10-12-2012, 10:26 AM   #3
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,061

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
This probably has to with buffering, as described in this thread, more info here.
 
2 members found this post helpful.
Old 10-12-2012, 12:18 PM   #4
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Log files are generally large, so better open the file with more filter, instead of cat, and redirected i.e. save the edited stream of data to some temporary file, as follow:
more kermit_11-10-12.log | sed 's/ \{2,\}/\n/g' | grep -v Idle > /tmp/logdata.txt
then do:
tail -f /tmp/logdata.txt
Hope it will help.
 
1 members found this post helpful.
Old 10-12-2012, 03:52 PM   #5
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
Hi, thanks for your reply,
but suddenly
tail -f kermit_11-10-12.log | sed 's/ \{2,\}/\n/g'
works just as expected !
It doesn't for my collegue, who had an exact copy of my system,
I must have installed some library that made it work, I think.

A question about your code...
Would the 'more' command be watching the logfile for changes ?
Because I'm reading the logfile as it is being created by kermit...

Last edited by oneindelijk; 10-12-2012 at 03:52 PM. Reason: wrote 'cat' instead of 'tail -f'
 
1 members found this post helpful.
Old 10-12-2012, 04:15 PM   #6
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
Thumbs up

Quote:
Originally Posted by ntubski View Post
This probably has to with buffering, as described in this thread, more info here.
Sorry, I didn't notice your reply until after I had posted something again.
This is probably the solution of the problem ! Thanks !
 
Old 10-13-2012, 08:20 AM   #7
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,632
Blog Entries: 3

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
Quote:
Originally Posted by oneindelijk View Post
tail -f kermit_11-10-12.log | sed 's/ \{2,\}/\n/g'
works just as expected !
Does that still work for you if you stick the 'grep -v' on the end? For me, it worked with either command but not both. However, following the advice in ntubski's first thread, it works if I run

Code:
tail -f <file> | sed -u 's/foo/bah/g' | grep -v 'baz'
EDIT: oops, you marked it as solved. Well, anyway, the solution is now obvious for anyone else with said problem...
 
Old 10-13-2012, 10:58 AM   #8
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Quote:
Originally Posted by oneindelijk View Post
Hi, thanks for your reply,
but suddenly
tail -f kermit_11-10-12.log | sed 's/ \{2,\}/\n/g'
works just as expected !
It doesn't for my collegue, who had an exact copy of my system,
I must have installed some library that made it work, I think.

A question about your code...
Would the 'more' command be watching the logfile for changes ?
Because I'm reading the logfile as it is being created by kermit...
No. more command will not watch for any changes, but it's a simple filter used to read large files in a page wise manner. You simply cannot open large files with cat command.
 
Old 10-15-2012, 11:27 AM   #9
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
That's weird. Suddenly it doesn't work anymore (after a reboot this morning).
It tried this:
tail -f test.log | stdbuf -o0 sed 's/y/ff/g'
and this:
tail -f test.log | sed -u 's/y/ff/g'
Nothing shows up.
When I do
cat test.log | sed -u 's/y/ff/g' (or without the '-u')
I get what I expect.
Any ideas why this suddenly stopped working? Could it be a global setting for buffering?
 
Old 10-16-2012, 04:31 AM   #10
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
Hi,

We've spent so much time on something that should be so easy !
(So it is still not working)
I tried tail -f /var/log/messages | sed (anything)
which works, so I suppose it has to do with the format the logfile is in.
I've been trying to change the options in Kermit from binary to text, but none of these allow
a combination of tail -f and sed to give any output.
What could be the reason of this ?
(And what could be the reason that I had it working at one point
(I shouldn't have taken days off for the weekend )

Thanks for any help
(maybe I should open a new topic about C-Kermit ?)
 
Old 10-16-2012, 05:02 AM   #11
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,061

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
Quote:
Originally Posted by oneindelijk View Post
I tried tail -f /var/log/messages | sed (anything)
which works, so I suppose it has to do with the format the logfile is in.
Can you figure what special format the Kermit log uses then? What's the output from file log? Can you get a small example log that doesn't work (small enough that you could reasonably post the output from od -tax1 log)?

Quote:
And what could be the reason that I had it working at one point
Possibly the logs get emptied on reboot, and didn't contain the problematic character(s)?
 
Old 10-16-2012, 05:26 AM   #12
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ntubski View Post
Can you figure what special format the Kermit log uses then? What's the output from file log? Can you get a small example log that doesn't work (small enough that you could reasonably post the output from od -tax1 log)?


Possibly the logs get emptied on reboot, and didn't contain the problematic character(s)?
Here's a part of od -tax1 krmt
Code:
0000000   1   0   :   5   4   :   2   3   .   7   7   1   :  sp   s   m
         31  30  3a  35  34  3a  32  33  2e  37  37  31  3a  20  73  6d
0000020   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0000040  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0000060  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0000100   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0000120  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0000140   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0000160   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0000200   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0000220  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0000240  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0000260   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0000300  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0000320   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0000340   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0000360   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0000400  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0000420  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0000440   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0000460  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0000500   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0000520   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0000540   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0000560  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0000600  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0000620   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0000640  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0000660   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0000700   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0000720   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0000740  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0000760  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0001000   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0001020  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0001040   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0001060   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0001100   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
0001120  sp   O   V   E   R   F   L   O   W  sp  sp  sp  sp  sp  sp  sp
         20  4f  56  45  52  46  4c  4f  57  20  20  20  20  20  20  20
0001140  sp  sp  sp  sp  sp  sp   *   *   *  sp   E   R   R   O   R  sp
         20  20  20  20  20  20  2a  2a  2a  20  45  52  52  4f  52  20
0001160   ^   E  sp   *   *   *   s   m   x  sp   S   I   G   N   A   L
         5e  45  20  2a  2a  2a  73  6d  78  20  53  49  47  4e  41  4c
0001200  sp   C   O   U   N   T   E   R  sp   O   V   E   R   F   L   O
         20  43  4f  55  4e  54  45  52  20  4f  56  45  52  46  4c  4f
0001220   W  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   *   *
         57  20  20  20  20  20  20  20  20  20  20  20  20  20  2a  2a
0001240   *  sp   E   R   R   O   R  sp   ^   E  sp   *   *   *   s   m
         2a  20  45  52  52  4f  52  20  5e  45  20  2a  2a  2a  73  6d
0001260   x  sp   S   I   G   N   A   L  sp   C   O   U   N   T   E   R
         78  20  53  49  47  4e  41  4c  20  43  4f  55  4e  54  45  52
And here is the original log (As you see there's just one timestamp at the beginning of the file because it contains just one line)
Code:
10:54:23.771: smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smx SIGNAL COUNTER OVERFLOW             *** ERROR ^E ***smxNS Init                              Ninit: initialize network data          Ninit: this computer is `DymoPnP`       Dymo NBB FW Ver: 2.1 (Built Oct 15 2012 10:30:42) Esc to ExitNBB info: SPI comm task started         Net: configuring interface 0           Portinit called for wifinet             WiFi2Ethernet init                      NBB init: network initialization ok    WiFi: enabling interface               WiFi:looking for USB dongle            WiFi: interface not open yet           WiFi: enabling interface               WiFi:looking for USB dongle
When I use in C-Kermit the option set session-log binary and I open the logfile with geany or so, I can see a lot of Escape codes being used (for coloring the output and such), but even than tail -f file | sed 's/ /x/g' gives no output (while the same command but tail -f replaced by cat does)

Tell me if you need any other information (I can attach a log file in the binary format)...
 
Old 10-16-2012, 06:06 AM   #13
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,061

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
Quote:
Originally Posted by oneindelijk View Post
And here is the original log (As you see there's just one timestamp at the beginning of the file because it contains just one line)
I can reproduce your problem if there is no newline at the end of the file. I'm not sure how to fix it though...
 
Old 10-16-2012, 07:18 AM   #14
oneindelijk
Member
 
Registered: Oct 2012
Location: Belgium
Distribution: Xubuntu
Posts: 30

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ntubski View Post
I can reproduce your problem if there is no newline at the end of the file. I'm not sure how to fix it though...
And if you use tail -f file | sed -u 's/ /Xx/g' ?
 
Old 10-16-2012, 07:51 AM   #15
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,632
Blog Entries: 3

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
I can also reproduce if there isn't a newline, and the '-u' option to sed makes no difference. I assume this is because 'tail' is waiting for a newline before passing anything through the pipe, but can't think of any way to fix this without 'rolling your own' version of tail...
 
  


Reply

Tags
buffer accumulation, buffer flush, stdbuf, tail grep buffer


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] bash command history log - how to write it continuous? Skaperen Linux - Software 12 04-15-2011 03:56 PM
I never claimed to be that sharp... Python - log scanned barcode characters to file dogfuel Programming 7 10-09-2009 11:12 AM
Continuous ACPI errors in message log wufo Slackware 1 08-06-2009 01:15 AM
bash script to find out more than 1 continuous special characters in a file. kkpal Linux - Newbie 1 06-02-2008 05:56 AM
Regarding replacing characters in log file rajaniyer123 Linux - General 4 04-12-2008 11:32 AM


All times are GMT -5. The time now is 09:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration