[SOLVED] piping email to a bash script for processing

mark1967 · 09-05-2011, 12:48 PM

Hi all
I want to turn the following command into a bash script:

cat file | ssh user@system 'script'
ssh is setup so that I can do this without a password for this specific command.

However I'm not getting far.
The only way I can find to get a bash script to read a file is using 'read', which is one line at a time, and this doesn't work here.

Is there any way I can get all of the input file and pipe it into an ssh command?
I feel like I'm missing something fairly basic but I've been at this for a few hours now and I'm not getting too far.

Thanks

Tinkster · 09-05-2011, 01:44 PM

You could try making the 'script' a forced command; I used that in
a previous employment for a certain application - not w/ emails,
but with lengthy textual output; and not into bash, but into perl.
Shouldn't make a difference for the use-case, though.

Cheers,
Tink

ta0kira · 09-05-2011, 02:07 PM

Quote:

Originally Posted by mark1967

However I'm not getting far.
The only way I can find to get a bash script to read a file is using 'read', which is one line at a time, and this doesn't work here.

Can you post a short version of the script? In what way does the script "read" the file data? Are you wanting to pass the data as a filename to something else, e.g.

Code:

#!/bin/bash

./myprogram-needs-a-file <(cat)

I seem to recall a circumstance where read wouldn't work directly so I had to cat | read.
Kevin Barry

KenJackson · 09-05-2011, 02:43 PM

You could cat the data to a local temporary file and then work on the temp.
I just ran this little experiment:

File abc on the local host:

Code:

abcd
xx example yy
efgh

File ~/bin/testabc on the remote host (~/bin is in PATH):

Code:

#!/bin/sh
FILE=/tmp/mail$$
cat > $FILE
trap "/bin/rm -f $FILE" 0 1 2 3 13 15

# Now process $FILE, e.g.:

grep example $FILE

Command on the local host:

Code:

cat abc | ssh remote testabc
xx example yy

mark1967 · 09-06-2011, 10:09 AM

Tinkster - thanks for the input - I think Perl is the answer, it's fairly easy just using Net::SSH but I thought that bash should be easy too.

KenJackson - that's a neat idea thanks. the problem is that postfix executes milters (mail filters) directly piping the mail to the script, and it's going to be much more robust if I don't have to break it up in this way. I think it would be better to revert to perl.

ta0kira - hopefully the above answers your question. Postfix pipes the mail into the script directly. If the script was local, that would be fine, but the script is remote, so efectively postfix will
cat mail_file | localscript.
Inside "localscript" I want something that reads mail_file and pipes that to "ssh user@remote_system milter_script"
I hope that makes sense.

I have to say I don't really have any example to post yet. i found read fairly quickly but then couldn't see how to read all of the mail into something that I could pipe to ssh.

ta0kira · 09-06-2011, 10:33 AM

Quote:

Originally Posted by mark1967

ta0kira - hopefully the above answers your question. Postfix pipes the mail into the script directly. If the script was local, that would be fine, but the script is remote, so efectively postfix will
cat mail_file | localscript.
Inside "localscript" I want something that reads mail_file and pipes that to "ssh user@remote_system milter_script"
I hope that makes sense.

So having a stand-alone line ssh user@remote_system milter_script doesn't work? You shouldn't need to pipe to ssh if the data is already being piped to the script. Are you trying to protect standard input from being read by something besides ssh?

Code:

#!/bin/bash

exec 3<&0 < /dev/null

read stuff #<-- nothing is read here

ssh user@remote_system milter_script <&3 #<-- only ssh can read standard input

Kevin Barry

Tinkster · 09-06-2011, 12:09 PM

Quote:

Originally Posted by mark1967

Tinkster - thanks for the input - I think Perl is the answer, it's fairly easy just using Net::SSH but I thought that bash should be easy too.

I didn't think of perl as "the answer" up there, but of the forced
command. Have a read about them: they work fairly well. Of course
I have to admit I never tried bash on the receiving end.

Cheers,
Tink

KenJackson · 09-06-2011, 05:52 PM

Quote:

Originally Posted by mark1967

KenJackson - that's a neat idea thanks. the problem is that postfix executes milters (mail filters) directly piping the mail to the script, and it's going to be much more robust if I don't have to break it up in this way. I think it would be better to revert to perl.

Do what you must.
But I changed my experiment script on the remote server to this:

Code:

#!/bin/sh
DATA="$(< /dev/stdin)"

# Now process the text, e.g.:

echo "$DATA" | grep example

And it works the same way at the local host:

Code:

cat abc | ssh remote testabc
xx example yy

ta0kira · 09-06-2011, 06:42 PM

Quote:

Originally Posted by KenJackson

But I changed my experiment script on the remote server to this:

Code:

#!/bin/sh
DATA="$(< /dev/stdin)"

# Now process the text, e.g.:

echo "$DATA" | grep example

This has a slight functional difference in that any null characters will be discarded. I don't know enough about the problem to know if that's an issue (i.e. if binary data will ever be sent), however.
Kevin Barry

Tinkster · 09-06-2011, 07:06 PM

Quote:

Originally Posted by Tinkster

I didn't think of perl as "the answer" up there, but of the forced
command. Have a read about them: they work fairly well. Of course
I have to admit I never tried bash on the receiving end.

Cheers,
Tink

And with these words:

Code:

on machine 2
user@machine2$ echo command="/home/p652900/test",no-port-forwarding,no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAoGgEV84faPDa9hsUmGUi2ZixGqAe03jQQ0Fkaz1llgZ9RCex/EJZ/Lny771t+oXyuYxaOtOIlya8oZDfBvaaYhWqxUPq81eF9PL43+ytI/C6m9nNhv4f4Tw12usd/Xi7qPUmaAT9fuWQCYHQniOoCLgY6fDcPTb97EDEIl9rIk4C8U+mij/ywq+UhgFLCmOtQM0RBw+q2bPoyjRm53RifAys1snjWlvGKxATGntI2TFE+sk933Id9MtOz/bB2yULWnRa9MPFXmYvVsJ1Hx0e/Ds4iSUy8k+K0HnQOvneAEGZG98bOzCmOnCvchOk8wqtwzSfPOkrMQ3QzvaqcmKOLQ== user@machine1 >> ~/.ssh/authorized_keys

vim /home/user/test
#!/bin/bash
count=1
while read line
do
echo $count $line 
count=$(($count + 1))
done


on machine 1
cat /etc/shells | ssh -t user@machine2
1 /bin/bash
2 /bin/ash

Cheers,
Tink

mark1967 · 09-07-2011, 02:14 AM

Wow - that's all exactly what I need - thanks all

Tinkster · 09-07-2011, 02:02 PM

Out of curiosity: which route did you take in the end?