Trying to understand how Pipe works in this situation
Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Trying to understand how Pipe works in this situation
I'm using ffmpeg to pull in an input rtmp stream and then push that out to a destination. In case the main input fails, I would like my ffmpeg process to switch to a backup input.
Unfortunately, ffmpeg CLI does not seem to have this ability built in.
So I came across a solution which uses pipes to manage this.
As for exec used for redirections, see this chapter in the Advanced Bash-Scripting Guide.
I should add that using redirections in this way is rarely needed in Bash, because it has process substitution. But when writing a POSIX-conform shell script they often come very handy.
I'm trying to understand if this pipe file is an actual file or just a temporary construct.
Also, if it is an actual file, will it's size keep increasing as the stream continues.
Thanks
No, it's not a file. It has an entry in a directory somewhere and this takes you to a normal inode on disk, so it has an owner, permissions, access dates and so on. But there are no data blocks. Instead, when a process accesses the "file", the kernel directs the data between the newly opened channel in that program and one in the program at the other end of the pipe.
As for exec used for redirections, see this chapter in the Advanced Bash-Scripting Guide.
I should add that using redirections in this way is rarely needed in Bash, because it has process substitution. But when writing a POSIX-conform shell script they often come very handy.
No, it's not a file. It has an entry in a directory somewhere and this takes you to a normal inode on disk, so it has an owner, permissions, access dates and so on. But there are no data blocks. Instead, when a process accesses the "file", the kernel directs the data between the newly opened channel in that program and one in the program at the other end of the pipe.
So technically this would show when I do an ls?
And does it exist once the source starts "inputting" into the pipe, or only after the "sink" starts reading out of the pipe?
Yes, it behaves just like any other file. For example, if you have a traditional sysvinit startup, there will be a fifo called initctl in the /dev directory and you can list it with ls. init listens on that fifo and programs like shutdown use it to give instructions to init.
Quote:
And does it exist once the source starts "inputting" into the pipe, or only after the "sink" starts reading out of the pipe?
The inode exists from the moment you create the fifo. I don't know at what point the kernel creates the channel. It can certainly exist without the listening process reading from it, because it is possible up to a point to write into a blocked pipe. But as soon as the pipe is full, the write operation halts until the other program reads what was written. My guess is that there is a kernel buffer which actually receives and passes on the text. Maybe a kernel maven can tell us that.
it is possible up to a point to write into a blocked pipe. But as soon as the pipe is full, the write operation halts until the other program reads what was written.
Thanks for the info. My concern is that in my case, the input stream would be coming into the server and therefore the pipe regardless of whether I am reading out of the pipe. So when I finally do read out of the pipe, I don't want a situation where the pipe feeds "older" chunks of the stream to the next stage.
you can imagine that pipe as a "real" pipe (or tube). From one side you put data into it and on the other side you will take that out.
the pipe [may] exists without any data pushed/pulled.
fifo means First In First Out, so the order of the data cannot be changed inside the pipe. https://www.softprayog.in/programmin...fifos-in-linux
I've been trying to picture this and I think I know now how pipes originated. They're just a small wrinkle on normal disk I/O.
The kernel maintains huge buffers whenever it has enough free memory for them. When it has to read from a disk, it reads much more than was requested and stores the rest in a buffer. That way it can satisfy subsequent read commands from the process without having to do any further mechanical disk access. And similarly output from a process goes into a buffer and gets written to disk at a convenient time, when the cpu is less busy. A sync function synchronises the disk with the buffers when required.
When a process wants to do I/O, the kernel assigns a file descriptor (just a number) and links it to the appropriate internal buffer. Now to create a pipe, all the kernel has to do is assign two file descriptors to such a buffer, one in read mode and one in write mode. That way the buffer needn't be linked to the disk surface in any way. Instead one program simply reads out of the buffer what the other program has written into it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.