[SOLVED] Problem using piped data as input

NirvanaII · 11-06-2010, 12:55 PM

Neither of the following commands are working and I don't know why. Any ideas, anybody?

Code:

stuart@stuart:~/$cat showthread.php.html | grep -o "http://www.megaupload.com/[^\"< ]*" | uniq | head -n3 | plowdown -
cannot stat '-': No such file or directory

Code:

stuart@stuart:~/$ awk NR==$x Musiclist.txt | cut -d/ -f6 | basename $1 .mp3
.mp3

Here's some system info on the bash i'm using:

Code:

stuart@stuart:~$ apt-cache showpkg bash
Package: bash
Versions:
4.1-3 (/var/lib/dpkg/status)
Description Language:
                 File: /var/lib/dpkg/status
                  MD5: 82836500b5a514df81f56f5dc639231c

3.2-4 (/var/lib/apt/lists/ftp.us.debian.org_debian_dists_stable_main_binary-i386_Packages)
Description Language:
                 File: /var/lib/apt/lists/ftp.us.debian.org_debian_dists_stable_main_binary-i386_Packages
                  MD5: 82836500b5a514df81f56f5dc639231c


Reverse Depends:
  remastersys,bash
  plowshare,bash 3.2
  firehol,bash 2.04
  bash-completion,bash 3.1dfsg-9
  bash-completion,bash 3.1dfsg-9
  foomatic-filters,bash 2.05
  linux-patch-xenomai,bash 2.0
  kernel-patch-wrr,bash 2.0
  txt2regex,bash 2.04
  rtai-source,bash 2.0
  rdup,bash 3.0.0
  podget,bash 3.0
  netscript-2.4,bash 2.03
  mason,bash 2.04-1
  linux-patch-tuxonice,bash 2.0
  linux-patch-grsecurity2,bash 2.0
  linux-patch-debianlogo,bash 2.0
  kernel-patch-viewos,bash 2.0
  kernel-patch-nfs-ngroups,bash 2.0
  kernel-patch-kdb,bash 2.0
  kernel-patch-badram,bash 2.0
  gt5,bash 2.5
  foomatic-filters,bash 2.05
  foomatic-db-engine,bash 2.05
  firehol,bash 2.04
  fiaif,bash 2.04
  exim4-config,bash 2.05
  emacs-goodies-el,bash 2.05a
  devscripts-el,bash 2.05a
  dpatch,bash 3.0-6
  diffmon,bash 2.0
  common-lisp-controller,bash 2.04-9
  linux-patch-tomoyo,bash 2.0
  biabam,bash 2
  beagle,bash 3.0
  bashdb,bash 3.1
  bash-doc,bash 3.2-1
  bash-builtins,bash 3.2-4
  bash-completion,bash 3.1dfsg-9
  bash-completion,bash 3.1dfsg-9
  backupninja,bash 2.05b-26
  linux-patch-aufs,bash 2.0
  kernel-patch-atopcnt,bash 2.0
  kernel-patch-atopacct,bash 2.0
  armagetronad-dedicated,bash 3
  webmin,bash
  webmin,bash
Dependencies:
4.1-3 - base-files (2 2.1.12) debianutils (2 2.15) dash (2 0.5.5.1-2.2) libc6 (2 2.4) libncurses5 (2 5.7+20100313) bash-doc (0 (null)) bash-completion (2 20060301-0) bash-completion (3 20060301-0) bash-completion (3 20060301-0) bash-doc (1 2.05-1)
3.2-4 - base-files (2 2.1.12) debianutils (2 2.15) libc6 (2 2.7-1) libncurses5 (2 5.6+20071006-3) bash-doc (0 (null)) bash-completion (2 20060301-0) bash-completion (3 20060301-0) bash-completion (3 20060301-0) bash-doc (1 2.05-1)
Provides:
4.1-3 -
3.2-4 -
Reverse Provides:

Is there a bash repository? It has been suggested to me by the author of the second command that bash may be buggy.

aluser · 11-06-2010, 01:12 PM

The second command looks wrong, in a couple ways. Post some sample lines from Musiclist.txt, and explain what you want the command to do, and we can help you

For the first one, I don't know what "plowdown" is but I suspect it doesn't understand "-" to mean standard input. Try redirecting the output from the rest of the pipeline into a file, then running plowdown on that. i.e.,

Code:

long | list | of | stuff | head -n 3 > /tmp/somefile.txt
plowdown /tmp/somefile.txt

NirvanaII · 11-06-2010, 01:35 PM

Thanks for the quick reply.

Plowdown is an app that invokes wget to download files from filehosting sites like Rapidshare, and the like.
http://www.webupd8.org/2009/09/downl...egaupload.html

In both examples my primary concern is to solve the problem of using pp's which I cannot get to work.

The following command - though I don't like creating files where piping will do - I had to break out of with ^C.

Code:

cat showthread.php.html | grep -o "http://www.megaupload.com/[^\"< ]*" | uniq | head -n3 > downloads.txt; plowdown downloads.txt

With the second I am trying to get a standard output of the file without the extension. Here is the first line of Musiclist.txt:

Code:

/mnt/sda1/Music/Belly/06 - The Bees.mp3

So that's "06 - The Bees" as an output, not .mp3.

GrapefruiTgirl · 11-06-2010, 01:53 PM

For your second command (though I'm not exactly sure what you were doing with the awk command you had), try this:

Code:

 awk '{gsub(".*/|\\.mp3$","");print}' file

it will read a file full of lines like:

Code:

/mnt/sda1/Music/Belly/06 - The Bees.mp3

and get rid of the bold parts.

Also note that I'm not sure how in your first post, the two commands you're running are supposed to be tied together... I'm thinking you want this awk command (in a pipeline, without the filename) to come after the `head -n3` but before the `plowdown`, so my apology if this isn't right - and if not, please explain some more.

aluser · 11-06-2010, 02:12 PM

The $1 isn't doing what you want because it needs to run from inside a script. Normally you'd use it when you're putting your bash into a separate file with a "#!/bin/bash" line at the top, but if you really want to use it on the command line you can do things like this:

Code:

$ bash -c 'echo $2' unused a b c
b

GrapefruiTgirl's awk script is a better way to do it, rather than rely on having exactly 6 slashes. However, you could make the original work like this:

Code:

awk NR==$x Musiclist.txt | cut -d/ -f6 | basename "`cat`" .mp3

GrapefruiTgirl: The awk command was to get a specific line from the file. It's equivalent to:

Code:

awk "NR==$x { print }" Musiclist.txt

NirvanaII · 11-06-2010, 02:15 PM

GrapefruiTgirl

The two commands are linked in my mind, not because i wish to use them in conjunction with one another, but because in both cases I want use the piped output on the last command in the chain.

So:

...head -n3 | plowdown {piped data}
...cut -d/ -f6 | basename {piped data} .mp3

It's not a case of getting the commands to work by any means, but in this particular way. I hope this makes sense.

GrapefruiTgirl · 11-06-2010, 02:23 PM

OK, I'm still a bit fuddled, but maybe `xargs` can help?

Code:

...head -n3 | xargs -I{} plowdown {}

As for the second command, maybe:

Code:

...cut -d/ -f6 | xargs -I{} basename '{}' .mp3

NirvanaII · 11-06-2010, 02:34 PM

Quote:

Originally Posted by aluser

The $1 isn't doing what you want because it needs to run from inside a script. Normally you'd use it when you're putting your bash into a separate file with a "#!/bin/bash" line at the top, but if you really want to use it on the command line you can do things like this:

Code:

$ bash -c 'echo $2' unused a b c
b

Ah! This explains things somewhat. This is why pp's have worked for me within functions and xargs only - this is the answer to my question. So I could rework the commands with a 'bash -c'prefix, or there is some better way...

Is this plain erroneous then?

Code:

curl http://some-website.com/page.html | \
grep -o "http://www.megaupload.com/[^\"< ]*" | uniq | head -n10 | plowdown -

This code is copied this from the link above.

And yes, GrapefruiTgirl, you're solution is far more elegant than mine!

NirvanaII · 11-06-2010, 02:38 PM

Quote:

Originally Posted by GrapefruiTgirl

OK, I'm still a bit fuddled, but maybe `xargs` can help?

Code:

...head -n3 | xargs -I{} plowdown {}

As for the second command, maybe:

Code:

...cut -d/ -f6 | xargs -I{} basename '{}' .mp3

xargs throws a fit with punctuation so I gave up on using it in situations like this (song titles have apostrophes etc; links, question marks)

GrapefruiTgirl · 11-06-2010, 02:58 PM

Using the -0 option to xargs can solve some (or all - it works for me) of those punctuation problems, but maybe not extreme cases unless the input is "cleaned", and you don't want to do that because then the filenames may be wrong. Here's an example of "still works for me":

Code:

root@reactor: echo "/file/path/he'-l?<href:\!lo.mp3" | xargs -0 -I{} basename "{}" .mp3
he'-l?<href:\!lo.mp3

root@reactor:

Well, without xargs, then just pipe the stuff into a while loop instead. Process each line in the while loop, and run plowdown on each item.

It seems you have all the necessary pieces here to do what you need, it's a matter of putting them together the right way.

At this point in time, I don't know precisely what command(s) you are running or what exact problem(s) remain, i.e. have you come up with a new command based on some of the suggestions in here, or are you still wondering what to do next? Show us the current command you are using, and explain what problem(s) are still occurring. Basically, bring us up to speed again on the current state of the problem: Have you modified your commands any, or are we still at post #1 and looking for a solution?

NirvanaII · 11-06-2010, 04:04 PM

Quote:

Originally Posted by GrapefruiTgirl

Have you modified your commands any, or are we still at post #1 and looking for a solution?

Ha! I've yet to think through a solution, but aluser has provided me with the theoretical answer. I'll set my mind to it...

Using xargs as you do maybe worth exploring as a means to circumventing aluser's solution, which calls for further inventiveness. I find that the -0 option fails in other instances, however -see:

Code:

stuart@stuart:~$ cat Musiclist.txt | xargs -0 -I{} basename "{}" .mp3
03 - Dead River.mp3

stuart@stuart:~$ ls mp3/mark_lanegan/the_winding_sheet/ | xargs -0 -I{} basename "{}" .mp3
01-mockingbirds.mp3
02-museum.mp3
03-undertow.mp3
04-ugly_sunday.mp3
05-down_in_the_dark.mp3
06-wild_flowers.mp3
07-eyes_of_a_child.mp3
08-the_winding_sheet.mp3
09-woe.mp3
10-ten_feet_tall.mp3
11-where_did_you_sleep_last_night.mp3
12-juarez.mp3
13-i_love_you_little_girl.mp3

stuart@stuart:~$

But anyway, i'll do some thinking and come back if i'm still stumped for solutions to the initial two problems.

Dark_Helmet · 11-06-2010, 07:35 PM

I don't have a solution for either question, per se. The reason I'm posting though is to let you know that the plowdown program is a straight Perl script.

I followed the link provided, downloaded the tarball and examined the source code. I wanted to verify whether plowdown could read download URLs directly from stdin. I'm not a Perl coder (unless I have to be), but it does not appear to support that.

Though, I'm sure someone with Perl experience (or someone who wants to learn) could add that ability. The plowdown command is actually just the "download.sh" file located in the src directory of the tarball.

So, in short, the "advanced" command given as an example on the linked page above is "too advanced" for plowdown--unless someone adds that capability.

GrapefruiTgirl · 11-06-2010, 08:03 PM

Thanks Dark_Helmet for that information. If the program does not support input on stdin, OP needs to make either a loop, and run plowdown on each file, OR string together a list (a sequence) of files and give them all at once on the plowdown commandline.

I'd also like to suggest further:

based on what understanding of the thread so far, this is not an issue of positional parameters, but more of an issue of some data not coming out the end of a pipeline in the usable format we want; I'm still not 100% clear on how the two commands in the OP actually relate to each other (but no matter), but maybe a thread title (click Edit -> Advanced on post #1) edit would be appropriate to better describe the issues. If I were to change the thread title, even though there seem to be two similar but distinct (and related) problems here, I'd use maybe something like:

"problem piping grepped URLs into plowdown (a perl downloader)"

Plus, here's a take on this command:

Code:

curl http://some-website.com/page.html | \
grep -o "http://www.megaupload.com/[^\"< ]*" | uniq | head -n10 | while read line; do
    plowdown "$line"
done

Anyhow, looking forward to OP's next post w/an update on things..

NirvanaII · 11-07-2010, 07:03 AM

I realised that plowdown made use of wget, but not that was a perl script, so thankyou Dark_Helmet.

I have got both commands to work now, using various solutions posted here. I quite like aluser's suggestion of using the cat command ("`cat`"), as it deals with my ostensible issue of parsing piped data to a command. And yes GrapefruiTgirl, I can see that using two examples, one using pp's, the other not, confused this one problem.

These solutions worked fine:

Code:

stuart@stuart:~/$ awk NR==$x Musiclist.txt | cut -d/ -f6 | basename "`cat`" .mp3
06 - The Bees
stuart@stuart:~/$ cat showthread.php.html | grep -o "http://rapidshare.com/[^\"< ]*" | uniq | tail -n2 | while read line; do plowdown "$line"; done
Waiting 31 seconds... 31s left^C

Using xargs instead of 'while read line' worked too.

I do have another issue but that's for another post. Thankyou to all.