I need to fetch one file from an rsync server. This file is too large to store on disk space due to the system running has booted from a small USB memory stick. The file is actually a CPIO file being used to populate the hard drive of the target system, so that hard drive is also not available for staging the file.

Currently, I am using an HTTP server to do this. But there are multiple issues with that. If this one problem with rsync could be resolved, then that could clear up the other issues short and sweet.

I tried to get rsync to write to /dev/stdout, but it would not do so. I searched for an option for rsync to just write directly to stdout, but found none.

why rsync? Can you not just cat it over ssh or something like that? rsync is generally run over SSH by any security minded sysadmin these days anyway.

RSYNC is actually an easier server to set up than FTP or HTTP. If it had a means to select one file by name, and deliver it to stdout, then I can use it in a script. That script will run on other machines in the network to get one file by name (the name varies and the script even constructs the name from pieces of data). I can do this now with tools like "lynx" or "wget". But I have to set up FTP or HTTP servers to support those, and that's a more complex task to do so and keep secure (mostly because of legacy demands on them to do so many other complex things). RSYNC is easier in part because I've already scripted an on-the-fly rsync server (can be launched from cron with no config setup). I already use that for many other purposes. But in this case I ran into the issue of this involving large files AND no space to stage the file (hence the need to write it to stdout).

For security reasons, every machine on this LAN is unable to ssh directly into any other machine on the LAN. I have tools for doing data transfers manually between machines, which are secure even over the internet. I ssh from "home base" into source and destination machine, and run a script on each. On the source machine the files to send are named. On the destination machine no files are named. The script figures out the data direction that way. On the machine that does listening (can be either source or destination), no address/hostname is given. On the machine that initiates connection, the other machine's address/hostname is given. I give the port number to both, as well as the encryption passphrase ("none" leaves out the encryption overhead, which boosts speed on gigabit secure LANs). It's a nifty script, but it doesn't serve this one case.

I've tried stuff with rsync like using --inplace and --append in conjunction with /dev/stdout, but that doesn't work (and luckily I tried it non-root, or maybe it would have replaced /dev/stdout with a file). It just won't open the existing target for writing. I just can't find anything in rsync to do that.

So at this point I guess I have to invent my own thing or use HTTP (FTP is harder to set up so I won't even bother with that). I don't want the issues that TFTP would have.

That still doesn't answer acid_kewpie (I think?)
In fact, what about http://linux.die.net/man/1/nc or http://linux.die.net/man/1/ncat?
In fact the latter has an option to fake an http proxy if you want, but they both can send to stdout...

The host accessing the data is outside the "trust enough to login via ssh" realm. Basically, ssh isn't an available option.

As for using something like nc, I already have played around with socat (the man page for it on linux.die.net is messed up and needs to be regenerated). But nonw of these offer a safe way to let a name selection be used without adding code to make sure tricks are not used to access things not allowed.

File servers like rsyncd can be made secure. It's just the client can't deliver the file in a way I need in this case (though I use rsync for a lot of other things where files are written to disk and that works well).

If programs like links, lynx, nc, ncat, wget, and such, were to implement rsync as a protocol (some implement http and ftp), then we might have a winner, since these do normally do output to stdout for many things. One possibility is to set up an HTTP server and use "lynx -source http://the.server/name". But I'm trying to avoid the headaches of having a running HTTP server for this. An rsyncd server is much less difficult.