Edit and direct continuously output from program (rsstail)

iwtbf · 06-26-2015, 06:54 AM

I'm using rsstail and I want to edit the output.

rss gives continuously output on the form

Code:

$ » rsstail -l -n 4 -u https://news.ycombinator.com/rss
Title: Why does Windows not recognize my USB device if I plug it into a different port?
Link: http://blogs.msdn.com/b/oldnewthing/archive/2004/11/10/255047.aspx
Title: LLVM merges SafeStack
Link: https://github.com/llvm-mirror/llvm/commit/7ffec838a2b72e6841d9fb993b5fe6a45f3b2a90
Title: Life paint
Link: http://www.volvolifepaint.com/
Title: Uber's Original Blog and First Posts
Link: http://uberexpansion.com/flashback-original-uber-blog/

My goal is to get a text file consisting of only the links
http://blogs.msdn.com/b/oldnewthing/...10/255047.aspx
https://github.com/llvm-mirror/llvm/...5fe6a45f3b2a90
http://www.volvolifepaint.com/
... etc, for later use.

That is, for every output something like

Code:

echo -e "Title: The man with 1,000 Klein Bottles UNDER his house - Numberphile\nLink: http://www.youtube.com/watch?v=-k3mVnRlQLU" | awk NR\>1 | cut -c7-

But since the output of rsstail is continuous and I think the ">>"-operator is char-by-char I'm not sure how to accomplish this.

I guess an alternative would be to just

Code:

rsstail -l -n 4 -u https://news.ycombinator.com/rss >> output.txt

and then edit the output.txt file with a cronjob or something but it would be nicer to do it on the fly.

Any suggestions or pointers on how to do this?
Any help would be highly appreciated!

rknichols · 06-27-2015, 05:16 PM

Just piping the output through sed should do the job.

Code:

rsstail -l -n 4 -u https://news.ycombinator.com/rss | sed -n 's/^Link: //p'

Translation: Do not print anything by default (-n). On all lines that begin with "Link: ", delete those characters and print what remains.

But, there is still a question of how rapidly you want to see the output. When rsstail sees that its stdout is not a terminal, it will likely switch from line-buffered to block-buffered output and wait until it has a full 4K block to write. sed will do the same thing. Some programs have an option to force line-buffering or non-buffering (e.g., the "--line-buffered" option in grep, or the "-u" (--unbuffered) option in sed). If you need the output to appear immediately, you might check whether rsstail has an option like that. Lacking that, there are programs like unbuffer that will present a pseudo-tty device that you can use as the target of the pipe.

If your output is going to a file, you probably don't need to worry about any of that, but if you are testing with output going to a terminal, you need to be aware of the buffering involved.

(unbuffer is typically distributed as part of the expect package.)

ondoho · 06-27-2015, 06:56 PM

i've used rsstail for a while and i think it's not very efficient; especially if you have some needs beyond just spitting out plain text.
you might want to hack yourself a script with xmllint or try some html-xml-utils.
once you get the hang of it, it's much easier than fighting with a limited utility.

chrism01 · 06-29-2015, 05:56 AM

You could use this maybe http://search.cpan.org/~mgrabnar/File-Tail-1.2/Tail.pm