LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 05-29-2009, 09:56 AM   #1
Gerbillaz
LQ Newbie
 
Registered: May 2009
Posts: 4

Rep: Reputation: 0
Question "more" and "less" piped commands : the mystery of the hidden stdin ! - Solved !


Hello everyone

I've searched the web for 2 hours now, but I can't find any answer to this question :

when you type "cat file.txt | less", how comes you're able to send commands to the display you get, by pressing keyboard touches ?

There , the "|" redirects the stdin file descriptor of the less command, so that it points to the output of "cat file.txt". The normal stdin that "less" shoudl have received, i.e the keybord input, is discarded, unknown to the "less" command.
So how comes "less" knows when you press "enter" or "space" ?
I crossed some months ago someone who told that, from a "strace" he had made, he though the less command uses its stderr as a secondary input... still I don't understand how this coudl happen - I don't know any syscall or c++ function able to do such things.

Any linux guru around, please save me from my total incomprehension ^^

Last edited by Gerbillaz; 06-01-2009 at 08:11 AM.
 
Old 05-29-2009, 10:11 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

I do believe you are turning things around:

cat file | less

The first thing that the shell does is set up the pipe between cat file and less,
The the cat command is executed and the output is given to less thru the pipe (that's a bit simplified: less can take chunks of the text instead of all the text if the text is big). less is the active program, not cat, hence you can use the commands as described in the less manpage.

BTW: Even though it is a good thing to understand how things work, the example given is not how you should use less (or a lot of other commands for that matter): less <file> is. This is a way of doing things one sees a lot (cat file | grep something vs grep something file).
 
Old 05-29-2009, 10:28 AM   #3
Gerbillaz
LQ Newbie
 
Registered: May 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Hi

Quote:
Originally Posted by druuna View Post
cat file | less

less is the active program, not cat, hence you can use the commands as described in the less manpage.
What do you mean by "active program" ?
When you catch user input through functions like fgetc() (in C) or "read" (in bash scripts), it seems the characters are taken from stdin, which in the case of a redirection is the file/pipe to which we connected the "less" command.
If "less" isn't connected to its parent terminal via its stdin, how could it be notified when new characters arrive ? There must be some trick there, for less to gain input from both a pipe and the terminal :?

Quote:
Originally Posted by druuna View Post
BTW: Even though it is a good thing to understand how things work, the example given is not how you should use less (or a lot of other commands for that matter): less <file> is. This is a way of doing things one sees a lot (cat file | grep something vs grep something file).
Agreed, the awkward code above is just for the matter of dissecting ^^

Last edited by Gerbillaz; 05-29-2009 at 10:31 AM.
 
Old 05-29-2009, 10:36 AM   #4
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi again,

Piping is not redirecting!

cat file | more -> this is piping

ls -l *.txt 1>file -> this is redirecting

And you can, ofcourse, combine them (1 pipe and 2 redirects): cat infile | grep something 1>outfile 2>&1

Maybe this will clear things up:
Advanced Bash-Scripting Guide / Chapter 19. I/O Redirection

If things still are unclear, just ask.
 
Old 05-29-2009, 12:47 PM   #5
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
I think what the original poster rightfully wonders is how the standard input stream used by more/less can seemingly access and process two sources of input. The answer is that while it's stdin file descriptor indexes one stream, it is still able to open and read the console device, which it reads directly. Normally, the standard input stream is the console device, and more/less is able to detect when it is not, and deliberately open it for reading and writing. It then uses the console device to read keystrokes and to send prompts and carriage control sequences to.
--- rod.
 
Old 05-29-2009, 12:58 PM   #6
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
@theNbomr: Isn't that, more or less (no pun intended), the same as my #2 reply?
 
Old 05-29-2009, 03:08 PM   #7
Gerbillaz
LQ Newbie
 
Registered: May 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks for all the answers and links B-)

@druuna - I don't understand this distinction between piping and redirecting. From what I've read there, piping is mainly a way of saying "redirect the stdout of command 1 to the stdin of command 2", i.e making two redirections in one shot. So the stdin of "grep" is redirected to receive characters not from the console, but, via the pipe, from the "cat" command.


@theNbomr - well, it'd explain everything ; do you have some ideas about how this can be achieved ? I know facilities like (in python) os.isatty() to test whether a file descriptor is a terminal or not, but if it's not, how can we connect back to our controlling terminal and force it to send data character per character (I've made attempts with /dev/tty, but it only prints data when we make a carriage return) ?

Last edited by Gerbillaz; 05-29-2009 at 03:11 PM.
 
Old 05-29-2009, 03:41 PM   #8
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
@Gerbillaz: There is a big difference, piping involves a lot more then just sending output to stdout and/or stderr.
Maybe this article will shed some light on pipes: Introduction to Named Pipes.

The way you described it ("redirect the stdout of command 1 to the stdin of command 2") is roughly correct, but hasn't that much to do with the actual stdin/stdout/stderr. It is more complex.

This: cat infile > more will create a file called more, filled with the content of infile.
This: cat infile | more will give more the content of infile so that more can do its thing.

You asked about the active command before. What I meant by that is this:

After typing the command and pressing enter the shell will evaluate all you have typed, sees that a pipe is requested between 2 commands and sets this up (bash being the active program atm, cat and more still do 'nothing'). After the pipe is set up, cat will echo the infile to the pipe (cat [and the pipe] being the active program, more is doing 'nothing'). If cat is finished, more starts reading from the pipe and is the active program. This is a bit simplified but basically this is how it works.
 
Old 05-29-2009, 05:11 PM   #9
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
druuna: Most or all of what you are saying is true, however I do not think it answers the question asked. As I understood the question, and Gerbillaz's reply seems to confirm my understanding, he wants an explanation of the apparent paradox demonstrated by the behavior of more/less, when its standard input has been attached to a pipe that is supplying the data which it is displaying. The apparent paradox is that it simultaneously reads input from the console, which is normally the source for applications' standard input stream, as well as it's actual standard input stream. I think you are overstating the complexity of IO redirection. Beej's Guide to IPC explains the processes (pun intended) and gives example code to demonstrate the method.
Gerbillaz seems to have a grasp on the concept, and has touched on the essential issue with a bit of Pythonese. I do not know of a good resource that explains the concept, but I cannot imagine that the underpinnings of the concept would be very hard to find in the less source code.
--- rod.
 
Old 05-29-2009, 06:03 PM   #10
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by Gerbillaz View Post
@theNbomr - well, it'd explain everything ; do you have some ideas about how this can be achieved ? I know facilities like (in python) os.isatty() to test whether a file descriptor is a terminal or not, but if it's not, how can we connect back to our controlling terminal and force it to send data character per character (I've made attempts with /dev/tty, but it only prints data when we make a carriage return) ?
I think you can use stty in a shell, or from a system() call, maybe, to invoke raw mode. I suppose this implies that you can use tcsetattr() to do the same thing. See Serial Programming HOWTO for details. Look at the section 'Non-Canonical Input Processing'.
--- rod.
 
Old 05-29-2009, 06:34 PM   #11
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,125

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
Quote:
Originally Posted by Gerbillaz View Post
when you type "cat file.txt | less", how comes you're able to send commands to the display you get, by pressing keyboard touches ?
Good question and you got me curious as well and I don't think the answers you've gotten have been very focused nor helpful.

Quote:
I crossed some months ago someone who told that, from a "strace" he had made, he though the less command uses its stderr as a secondary input...
Great idea. I think easy to test, and wrong!

I tried
cat file.txt | less 2> trash.tmp
redirecting stderr made no apparent difference in the behavior of less.
So I tried
cat file.txt | less > trash.tmp
Now I've redirected stdout, so of course I don't see the output. But more importantly, less no longer waits for input.

So I have seen that less is using stdout to figure out whether/where to get its input.

Quote:
still I don't understand how this coudl happen - I don't know any syscall or c++ function able to do such things.
I don't know those syscalls either. But I assume they must exist. less apparently can get significant information about the device that is open on stdout and apparently both detect that it is an interactive device and reopen it on another handle to get input from it.

If I was curious enough or less lazy I'd download a copy of the source code for less and find out.

Last edited by johnsfine; 05-29-2009 at 06:37 PM.
 
Old 06-01-2009, 08:10 AM   #12
Gerbillaz
LQ Newbie
 
Registered: May 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Cool

@druuna - maybe we're not speaking about the same pipes. The C/bash pipelines I know are 100% concurrent, i.e they run at the same time, only sleeping when they can't pull/push data from their i/o pipes. Thus, with small buffered pipes linking them, you can process gigabytes of data through a complex pipeline, without taking more than some kilobytes of memory.

Wikipedia quote : "In most Unix-like systems, all processes of a pipeline are started at the same time, with their streams appropriately connected, and managed by the scheduler together with all other processes running on the machine. An important aspect of this, setting Unix pipes apart from other pipe implementations, is the concept of buffering: a sending program may produce 5000 bytes per second, and a receiving program may only be able to accept 100 bytes per second, but no data are lost. Instead, the output of the sending program is held in a buffer, or queue. When the receiving program is ready to read data, the operating system sends it data from the buffer, then removes that data from the buffer. If the buffer fills up, the sending program is suspended (blocked) until the receiving program has had a chance to read some data and make room in the buffer. In Linux, the size of the buffer is 65536 bytes."


@everyone - thanks for the links and hints, I didn't know about those cool terminal manipulation functions. I've eventually browsed the sources of "less" to be sure. It was well commented, so I found that :

Code:
#if HAVE_DUP
	/*
	 * Force standard input to be the user's terminal
	 * (the normal standard input), even if less's standard input 
	 * is coming from a pipe.
	 */
	inp = dup(0);
	close(0);
#if OS2
	/* The __open() system call translates "/dev/tty" to "con". */
	if (__open("/dev/tty", OPEN_READ) < 0)
#else
	if (open("/dev/tty", OPEN_READ) < 0)
#endif
		dup(inp);
#endif
Obviously, that's the way "less" gets its controlling terminal, and after some more calls to parameterizing functions (raw_mode(), setmode()...), it seems all is in place for it to process inputs, from both file descriptors "inp" (which contains the stdin the shell gave it) and "0" (which has come back to the controlling terminal).
So there it is... 'less', 'more' and other "complex" commands are forced to do slight bits of hacking to retrieve all the information they need. Good to know B-)
 
Old 06-01-2009, 08:23 AM   #13
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
@Gerbillaz: I was looking at this problem from a command line point of view, even though you did mention syscall/c++ (people tend to come up with answers to their own questions, often just a wild guess). It is also the reason I backed off after post #8 (I do know a bit of c/c++ but not enough to help).

I also did take a look at the more code, and could not find any reference to 'forcing stdin to user's terminal', but more is part of the util-linux (ng) package and the code I (we) are looking for is probably buried somewhere.

For those that are looking for the code Gerbillaz mentioned: Take a look at lsystem.c.

Anyway, glad so read that you found an answer to your question.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Getting bash to login as root and recognize commands like "vi" and "locate" ajaygoeslinux Linux - Software 4 05-04-2009 11:51 PM
Standard commands give "-bash: open: command not found" even in "su -" and "su root" mibo12 Linux - General 4 11-11-2007 11:18 PM
Shell and batch operations on hidden files but not on ".." & "." danielsbrewer Linux - Software 5 09-12-2007 09:06 AM


All times are GMT -5. The time now is 04:02 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration