LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 09-28-2020, 10:56 AM   #1
ychaouche
Member
 
Registered: Mar 2017
Distribution: Mint, Debian, Q4OS, Mageia, KDE Neon
Posts: 109

Rep: Reputation: 6
[BASH] Multiplex the output of two or more commands to standard output


Dear #LQ,

I would like to get any errors from 3 different logs in real time on the terminal.

Separately, the three commands that would do that, if they were run in sperate terminals/consoles, would be :

1/ tail -f /var/log/mail.warn
2/ tail -f /var/log/dovecot.log | egrep -i '(warn|critic|error)'
3/ tail -f /var/log/fail2ban.log | egrep -i '(warn|critic|error)'

How can I combine these three commands into one such that the output from all three would be shown at the same time continuously ?

I know that tail -f can take more than one file, but what about the tail -f | grep commands ?
 
Old 09-28-2020, 11:25 AM   #2
boughtonp
Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 966

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738

I haven't checked, but process substitution should work here?

i.e replace "filename" with "<( command that acts on filename )" for each grep filter.

so...
Code:
tail -f /var/log/dovecot.log | egrep -i '(warn|critic|error)'
becomes
Code:
tail -f <( egrep -i '(warn|critic|error)' /var/log/dovecot.log )
...and if that works, you should be able to put all three after the single tail.

 
Old 09-28-2020, 12:58 PM   #3
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,883

Rep: Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250
if I understand well you need to redirect these outputs to the same terminal. Or just execute them in the same terminal in the background

Last edited by pan64; 09-28-2020 at 01:00 PM.
 
Old 09-28-2020, 01:33 PM   #4
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,963

Rep: Reputation: 708Reputation: 708Reputation: 708Reputation: 708Reputation: 708Reputation: 708Reputation: 708
Quote:
Originally Posted by ychaouche View Post
1/ tail -f /var/log/mail.warn
2/ tail -f /var/log/dovecot.log | egrep -i '(warn|critic|error)'
3/ tail -f /var/log/fail2ban.log | egrep -i '(warn|critic|error)'

How can I combine these three commands into one such that the output from all three would be shown at the same time continuously ?

I know that tail -f can take more than one file, but what about the tail -f | grep commands ?
Does this work for you
Code:
tail -f file_a file_b ... | egrep -i '(warn|critic|error)'
If not then post some representative sample data and the output you expect.

Code:
tail -f <( egrep -i '(warn|critic|error)' /var/log/dovecot.log ... )
@boughtonp Unfortunately this does not work because egrep does not keep reading the files continuously.

Last edited by crts; 09-28-2020 at 01:35 PM.
 
Old 09-28-2020, 02:39 PM   #5
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,513
Blog Entries: 11

Rep: Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473Reputation: 3473
Create a named pipe (man mkfifo), tail your logs into the pipe and tail the pipe into your terminal.

Maybe something like this (untested):

Code:
mkfifo mypipe

tail -f /var/log/mail.warn >>mypipe &
tail -f /var/log/dovecot.log | egrep -i '(warn|critic|error)' >>mypipe &
tail -f /var/log/fail2ban.log | egrep -i '(warn|critic|error)' >>mypipe &

tail -f mypipe

Last edited by astrogeek; 09-28-2020 at 02:59 PM.
 
3 members found this post helpful.
Old 09-30-2020, 05:00 AM   #6
ychaouche
Member
 
Registered: Mar 2017
Distribution: Mint, Debian, Q4OS, Mageia, KDE Neon
Posts: 109

Original Poster
Rep: Reputation: 6
Thanks guys. What do you think of

Quote:
(tail -f /var/log/dovecot.log | grep login --line-buffered) | tee >( tail -f /var/log/apache2/roundcube.access)
?

One downside of this is that I have to do a double ^C to stop it. The first ^C will give me back the command line prompt, but lines from the second command keep coming, as if it is still running in the background. I need another ^C to stop the second command.
 
Old 09-30-2020, 05:06 AM   #7
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,883

Rep: Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250
that is wrong. But I don't know what did you want to achieve with that.
( ) are not required
piping/redirecting the output of tee into tail is pointless.

Last edited by pan64; 09-30-2020 at 05:22 AM.
 
Old 09-30-2020, 06:10 AM   #8
ychaouche
Member
 
Registered: Mar 2017
Distribution: Mint, Debian, Q4OS, Mageia, KDE Neon
Posts: 109

Original Poster
Rep: Reputation: 6
Dear Pan64,

Thanks for pointing out the uselessness of the parentheses, which I have removed. The command is now :

Quote:
tail -f /var/log/dovecot.log | grep login --line-buffered | tee >( tail -f /var/log/apache2/roundcube.access)
Which works quite well

Quote:
root@messagerie[10.10.10.19] ~ # tail -f /var/log/dovecot.log | grep login --line-buffered | tee >( tail -f /var/log/apache2/roundcube.access)
Sep 30 11:02:59 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
Sep 30 11:03:00 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
Sep 30 11:03:00 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
Sep 30 11:03:03 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
Sep 30 11:03:03 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
10.10.10.32 - - [30/Sep/2020:11:02:56 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4791 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
192.168.100.68 - - [30/Sep/2020:11:02:56 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4587 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0"
192.168.100.68 - - [30/Sep/2020:11:02:57 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1088 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
192.168.100.240 - - [30/Sep/2020:11:02:58 +0100] "GET / HTTP/1.1" 200 10799 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
192.168.90.122 - - [30/Sep/2020:11:02:58 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4626 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; rv:81.0) Gecko/20100101 Firefox/81.0"
105.96.32.132 - - [30/Sep/2020:11:02:59 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 917 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0"
41.102.18.235 - - [30/Sep/2020:11:03:00 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1115 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
192.168.211.135 - - [30/Sep/2020:11:03:00 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 920 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
10.10.10.32 - - [30/Sep/2020:11:03:03 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1144 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
105.105.128.94 - - [30/Sep/2020:11:03:03 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4783 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
Sep 30 11:03:04 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
192.168.90.66 - - [30/Sep/2020:11:03:04 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4558 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; rv:81.0) Gecko/20100101 Firefox/81.0"
Sep 30 11:03:05 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
10.10.10.32 - - [30/Sep/2020:11:03:05 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1096 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
Sep 30 11:03:06 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
105.96.42.9 - - [30/Sep/2020:11:03:06 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1027 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
Sep 30 11:03:06 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
192.168.211.81 - - [30/Sep/2020:11:03:06 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 920 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
Sep 30 11:03:07 imap-login: Info: Login: user=<xxx@mydomain.tld>, method=PLAIN, rip=10.10.10.19, lip=10.10.10.19, lport=143, service=imap
192.168.211.60 - - [30/Sep/2020:11:03:07 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1095 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0"
Sep 30 11:03:08 imap-login: Info: Aborted login (no auth attempts in 0 secs): user=<>, rip=192.168.100.240, lip=10.10.10.19, lport=143, service=imap
^C
root@messagerie[10.10.10.19] ~ # 10.10.10.32 - - [30/Sep/2020:11:03:08 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 919 "https://messagerie.mydomain.tld/?_task=mail&_caps=pdf%3D1%2Cflash%3D0%2Ctif%3D0&_uid=6200&_mbox=INBOX&_action=show" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
41.110.64.109 - - [30/Sep/2020:11:03:09 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1104 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"

^C
root@messagerie[10.10.10.19] ~ #

But I don't know how to get the same result without tee. I know tail can handle more than one file, so I've given it two files : the roundcube.access file and the standard input (-), the -- is mandatory to tell the shell that the '-' is for the standard input file and not a missing option

Quote:
root@messagerie[10.10.10.19] ~ # tail -f /var/log/dovecot.log | grep login --line-buffered | tail -f /var/log/apache2/roundcube.access -- -
==> /var/log/apache2/roundcube.access <==
192.168.90.122 - - [30/Sep/2020:11:06:50 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4770 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"
197.201.15.42 - - [30/Sep/2020:11:06:52 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1111 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
197.201.15.42 - - [30/Sep/2020:11:06:52 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4772 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
105.96.27.45 - - [30/Sep/2020:11:06:53 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4588 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX.PODCAST" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.63"
105.110.100.223 - - [30/Sep/2020:11:06:55 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1117 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36 OPR/70.0.3728.189"
10.10.10.32 - - [30/Sep/2020:11:06:56 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4792 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
41.102.18.235 - - [30/Sep/2020:11:07:00 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 1115 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
105.96.32.132 - - [30/Sep/2020:11:07:00 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 919 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0"
192.168.211.135 - - [30/Sep/2020:11:07:00 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 921 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
192.168.90.122 - - [30/Sep/2020:11:07:01 +0100] "POST /?_task=mail&_action=refresh HTTP/1.1" 200 4627 "https://messagerie.mydomain.tld/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 6.1; rv:81.0) Gecko/20100101 Firefox/81.0"

==> standard input <==
^C
root@messagerie[10.10.10.19] ~ #
The problem is that this command prints about 10 lines from roundcube.access and no lines from standard input. Even if I let the command run longer, there is no more lines even from roundcube.access.

Any tips ?
 
Old 09-30-2020, 06:20 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,883

Rep: Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250
did you try post #5?
 
Old 09-30-2020, 09:41 AM   #10
ychaouche
Member
 
Registered: Mar 2017
Distribution: Mint, Debian, Q4OS, Mageia, KDE Neon
Posts: 109

Original Poster
Rep: Reputation: 6
I didn't try it yet as I was experimenting with regular files (not pipes), it seems to work the same with regular files, and then I was dragged by other ideas as well (job management in bash being one of them). I have questions about that solution :
1/ Can I do the same with regular files ? what is the benefit of using a pipe ?
2/ Should I kill the 3 tails when I'm finished ? should I use killall for that or carefully pick the pids with ps then issue a kill with the three pids ?
 
Old 09-30-2020, 10:02 AM   #11
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,883

Rep: Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250
0. tee >(tail -f file) is still wrong. You are lucky because it looks like it works, but it is not the right way.
1. I don't know what do you mean by "can I do the same"
2. yes, if you want to make it properly you need to stop all the child processes. https://stackoverflow.com/questions/...esses-of-shell
3. also you would need to implement some interrupt handling to catch Ctrl-C and kill those processes https://raymii.org/s/snippets/Bash_B...C_SIGTERM.html
 
Old 09-30-2020, 10:56 AM   #12
ychaouche
Member
 
Registered: Mar 2017
Distribution: Mint, Debian, Q4OS, Mageia, KDE Neon
Posts: 109

Original Poster
Rep: Reputation: 6
Quote:
1. I don't know what do you mean by "can I do the same"
I meant to type all the commands without the first (mkfifo), like so :

Quote:
tail -f /var/log/mail.warn >>mypipe &
tail -f /var/log/dovecot.log | egrep -i '(warn|critic|error)' >>mypipe &
tail -f /var/log/fail2ban.log | egrep -i '(warn|critic|error)' >>mypipe &

tail -f mypipe
I have tried it and it seems to work, except I need to kill the other tails when I'm done.
 
Old 09-30-2020, 11:07 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,883

Rep: Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250Reputation: 5250
with mkfifo you will have a named pipe or socket (name is mypipe) without mkfifo mypipe will be a regular file.
Any way you need to remove it at the end.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
multiplex I/O (select with signal handlers) tanalam Programming 1 08-03-2008 02:18 PM
Bash Loops multiplex A/V lpn1160 Linux - General 1 02-23-2008 07:01 AM
multiplex internet connections in linux sailu_mvn Linux - Software 1 03-16-2007 07:19 AM
multiplex communication channels: selectors kpachopoulos Linux - Networking 2 02-09-2007 05:11 AM
multiplex(channel?) soundcard issue jmr71769 Linux - Hardware 0 12-11-2003 11:44 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 07:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration