sed -n question

NNP · 11-21-2006, 03:56 PM

Hey,
I have a script that does something similar to the following

Code:

while [ 1 ]; do echo 'something'; sleep 1; done | sed -n 's/something/somethingElse/p' > outputFile

For some reason though the resulting file is empty. If i remove the redirect e.g

Code:

while [ 1 ]; do echo 'something'; sleep 1; done | sed -n 's/something/somethingElse/p'

(I have trivialised the example here, I have a valid reason for using -n but the above example better illustrates the issue)

It outputs fine to the terminal but piping the result to another script reading on stdin or to a file doesn't work. Is there something I'm missing about sed -n or is it something other problem?

I am using the bash shell btw.

Thanks,
nnp

kaz2100 · 11-21-2006, 04:19 PM

Hi,

Code:

~$ while [ 1 ]; do echo 'something'; sleep 1; done | sed -n 's/something/somethingElse/p'
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse
somethingElse

This is what I got on my penguin.

I think this is what you want.

Check your sed ???

Happy Penguins!!

matthewg42 · 11-21-2006, 04:29 PM

With output is sent to the terminal, the IO is line buffered, but when output is re-directed to a file, the IO is buffered differently - nothing will be written to disk until some buffer is filled. For me it appears to be 4096 bytes.

I don't know exactly where this number comes from - perhaps the shell itself, or perhaps the default for the C streams library. If someone can give the definitive answer, I'd like to know.

Anyhow, with the output string "somethingElse\n", and a sleep time of 1 second per output, it'll take almost 5 minutes before you get any output.

NNP · 11-21-2006, 04:38 PM

Thank you both for your answers.
@kaz2100: hehe, yup, thats the code fragment that works. Its the other one that doesnt

The main reason I am doing this is so I can pipe the results of the bash commands into a python script for parsing.

e.g to test my network

Code:

nmap -sS -p $PORTS $IPRANGE | grep -B 2 " open " | sed -n 's/\(Interesting ports on \)\(\([0-9]*\.\)\{3\}[0-9]\{1,3\}\):/\2/p' | ./somePythonScript.py

All the above should do is feed IP addresses to the python script as it scans. (Has to be done concurrently)

The python script would look somewhat similar to the following

Code:

#!/usr/bin/env python

import os, sys
ipaddress = sys.stdin.readline()

while ipaddress and ipaddress != '':
    print '[+] ip ' + ipaddress
# Do some stuff with the IP address here
    ipaddress = sys.stdin.readline()

But the python script doesn't appear to recieve the result from the sed command until the nmap scan is finished. Is this a result of the same buffering you mentioned earlier? I would have thought piping the output would avoid this kind of buffering. If not, is there any way to avoid the buffering or to implement what I'm trying to do?

edit: ok, curiously enough, there again seems to be some sort of buffering on the pipe. If I leave the above command run long enough e.g ensure nmap generates a lot of data it eventually works. I guess the buffering is something I'm going to have to live with.

matthewg42 · 11-21-2006, 04:48 PM

use the -u option on the sed command, this activates unbuffered mode.

kaz2100 · 11-21-2006, 04:59 PM

Hi,

Aiya....

I was careless......

NNP · 11-21-2006, 05:04 PM

Thanks guys.