Shell Redirection Delay

pymehta · 04-06-2005, 10:37 AM

Quoted anonymous :
"When doing `./executable > file.txt`,
the file is not continuously written to; ie. the program output goes to some sort of buffer which is transferred to the file in chunks after quite a long time or when the execution of the file stops. Is there any way to make the updation of the file more frequent?"

bigearsbilly · 04-07-2005, 08:01 AM

hemlockz · 07-21-2009, 04:07 PM

BUMP the old post. I have the exact same question , this came up in the search.

AlucardZero · 07-21-2009, 04:47 PM

hemlockz · 07-21-2009, 05:14 PM

I am trying to make a script to capture output from hcidump, use a sed substitution to filter out extra data from hcidump, and I tee that to a file on a SSD, and to stdout on the screen. I want to see realtime data on the stdout since this application is detecting new devices all the time, and visual feedback is necessary for good antenna placement in the field. Without the tee command I see the filtered output from sed on stdout in real time. As soon as I add a tee or try and redirect it to a file I start seeing a delay.

hemlockz · 07-21-2009, 06:08 PM

Okay just tell me a better way to script it than. I'm not an expert.

sudo hcidump -Vt | sed -n 'N;s_<PATTERN>_<REPLACE>_p'|tee -a $FILENAME &

Kenhelm · 07-21-2009, 08:47 PM

The sed 'w' command appends the contents of the pattern space to a file.
http://www.gnu.org/software/sed/manu...Other-Commands
I don't have 'hcidump', however this method works using 'free -k -s 1' instead of 'hcidump'

Code:

sudo hcidump -Vt | sed -n 'N;s_<PATTERN>_<REPLACE>_p'"w $FILENAME"

GaijinPunch · 07-21-2009, 09:02 PM

If I'm not mistaken this is on a per-program basis. tail will dump ever 1 second, for example, while most other programs will not until the buffer is full.

chrism01 · 07-22-2009, 12:50 AM

No, I've seen people confused by 'tail -f blah' not show any output. The system usually uses some buffering by default.

If its your own code you can disable buffering.

Here's a good explanation: http://perl.plover.com/FAQs/Buffering.html.

hemlockz · 07-22-2009, 12:25 PM

This is great! This works perfect because it writes unbuffered to the file, and still displays the output on stdout. Thank you guru.

One minor follow-up question... I have to put the literal path of the output file. When I use

Code:

sed -n 'N;s_<PATTERN>_<REPLACE>_p'"w ~/Desktop/$FILENAME"

I get a "could't open file" error. If I replace the ~ with /home/user it works well. Is there a way to make ~? its not a big deal but would be nice for portability.

Thank you also for the perl solution. That seems like it would work well too. I may end up converting this sh script to perl if it gets any more complicated, so that was useful info.

Quote:

Originally Posted by Kenhelm

The sed 'w' command appends the contents of the pattern space to a file.
http://www.gnu.org/software/sed/manu...Other-Commands
I don't have 'hcidump', however this method works using 'free -k -s 1' instead of 'hcidump'

Code:

sudo hcidump -Vt | sed -n 'N;s_<PATTERN>_<REPLACE>_p'"w $FILENAME"

Kenhelm · 07-22-2009, 07:31 PM

Code:

# The Bash shell variable $HOME can be used instead of ~
sed -n 'N;s_<PATTERN>_<REPLACE>_p'"w $HOME/Desktop/$FILENAME"

# This is a way of getting ~ to work
sed -n 'N;s_<PATTERN>_<REPLACE>_p'"w $(echo ~)/Desktop/$FILENAME"