LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 04-03-2010, 01:23 AM   #1
genmaicha
Member
 
Registered: Apr 2009
Posts: 38

Rep: Reputation: 15
pipe buffering/blocked write caching question


I understand that the linux pipe is a buffer and that any data written to it will stay there until it is read, and if the max capacity of the buffer is reached, any additional writes will block (by default).

HOWEVER, the behavior of the pipeline below suggest that the write operations are buffered/cached before ever being written to the pipe on the client side

here is write.sh, which creates 1000 byte string and writes it 100 times to stdout... the idea being that it'll block as soon as the 64kb linux pipe size is reached:
Code:
# create a 1kb buffer named chunk
chunk=""
for i in `seq 1 100`
do
  chunk="${chunk}0123456789"
done

# write it to stdout 100 times
for i in `seq 1 100`
do 
  echo writing chunk $i >&2
  echo $chunk
done
here is read.sh, which reads the 1000 byte chunks one at a time but has a sleep statement in order to 'force' filling the pipe:
Code:
a=1
while read line
do
  echo `date`: read chunk $a
  a=$((a+1))
  sleep .5
done
here is (part of) the output of "./write.sh | ./read.sh":
Code:
writing chunk 1
writing chunk 2
writing chunk 3

... <omitted> ...

writing chunk 55
writing chunk 56
writing chunk 57
writing chunk 58
Fri Apr 2 23:01:49 PDT 2010: read chunk 1
writing chunk 59
writing chunk 60
writing chunk 61
writing chunk 62
writing chunk 63
writing chunk 64
writing chunk 65
Fri Apr 2 23:01:50 PDT 2010: read chunk 2
Fri Apr 2 23:01:51 PDT 2010: read chunk 3
writing chunk 66
writing chunk 67
writing chunk 68
writing chunk 69
Fri Apr 2 23:01:51 PDT 2010: read chunk 4
Fri Apr 2 23:01:52 PDT 2010: read chunk 5
Fri Apr 2 23:01:52 PDT 2010: read chunk 6
Fri Apr 2 23:01:53 PDT 2010: read chunk 7
writing chunk 70
writing chunk 71
writing chunk 54

... <omitted> ...
This is not what I was expecting: I was expected that once the capacity was reached, any reads would be followed immediately by a write to take advantage of the freed space. Instead, the blocked write operation seems to wait for some random amount of time/space to free until it unblocks and writes.

What exactly is going on, what's it called, and how can I control this behavior?

Last edited by genmaicha; 04-03-2010 at 03:41 AM.
 
Old 04-04-2010, 12:45 PM   #2
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 452Reputation: 452Reputation: 452Reputation: 452Reputation: 452
Quote:
Originally Posted by genmaicha View Post
I understand that the linux pipe is a buffer and that any data written to it will stay there until it is read, and if the max capacity of the buffer is reached, any additional writes will block (by default).

HOWEVER, the behavior of the pipeline below suggest that the write operations are buffered/cached before ever being written to the pipe on the client side

here is write.sh, which creates 1000 byte string and writes it 100 times to stdout... the idea being that it'll block as soon as the 64kb linux pipe size is reached:
Code:
# create a 1kb buffer named chunk
chunk=""
for i in `seq 1 100`
do
  chunk="${chunk}0123456789"
done

# write it to stdout 100 times
for i in `seq 1 100`
do 
  echo writing chunk $i >&2
  echo $chunk
done
here is read.sh, which reads the 1000 byte chunks one at a time but has a sleep statement in order to 'force' filling the pipe:
Code:
a=1
while read line
do
  echo `date`: read chunk $a
  a=$((a+1))
  sleep .5
done
here is (part of) the output of "./write.sh | ./read.sh":
Code:
writing chunk 1
writing chunk 2
writing chunk 3

... <omitted> ...

writing chunk 55
writing chunk 56
writing chunk 57
writing chunk 58
Fri Apr 2 23:01:49 PDT 2010: read chunk 1
writing chunk 59
writing chunk 60
writing chunk 61
writing chunk 62
writing chunk 63
writing chunk 64
writing chunk 65
Fri Apr 2 23:01:50 PDT 2010: read chunk 2
Fri Apr 2 23:01:51 PDT 2010: read chunk 3
writing chunk 66
writing chunk 67
writing chunk 68
writing chunk 69
Fri Apr 2 23:01:51 PDT 2010: read chunk 4
Fri Apr 2 23:01:52 PDT 2010: read chunk 5
Fri Apr 2 23:01:52 PDT 2010: read chunk 6
Fri Apr 2 23:01:53 PDT 2010: read chunk 7
writing chunk 70
writing chunk 71
writing chunk 54

... <omitted> ...
This is not what I was expecting: I was expected that once the capacity was reached, any reads would be followed immediately by a write to take advantage of the freed space. Instead, the blocked write operation seems to wait for some random amount of time/space to free until it unblocks and writes.

What exactly is going on, what's it called, and how can I control this behavior?
There are a lot of factors involved - process switching, stdout latency, etc.
 
Old 04-04-2010, 05:51 PM   #3
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Wheezy (Fluxbox WM)
Posts: 1,363
Blog Entries: 52

Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Quote:
Instead, the blocked write operation seems to wait for some random amount of time/space to free until it unblocks and writes.
I think you will find that it is not random; the threshold appears to be 4096 empty bytes before the sending process can proceed. Even if you can change this threshold (I'm not sure it is possible, I'm guessing it is the pipe block size), your system logic shouldn't depend on it.

You cannot expect this streaming to be unbuffered when you are using buffered (high level) reads and writes.

Last edited by neonsignal; 04-04-2010 at 05:58 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
write caching, sync, commit, noflushd monohouse Linux - General 0 11-01-2008 07:23 PM
Mod recent blocked related question (netfilter). WHO IS BLOCKED CarLost Linux - Security 6 07-29-2008 03:53 PM
Should I enable write caching on my raid array? abefroman Linux - Hardware 1 09-26-2005 04:38 PM
hdparm write caching setting nouse66 Linux - Hardware 3 03-10-2003 10:13 PM


All times are GMT -5. The time now is 05:00 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration