[SOLVED] Need help with downloading from http site (Robby's packages) with file globbing

damgar · 09-15-2010, 08:10 AM

I've never figured out a good program to use with file globbing from http sites. Wget works with ftp sites and file globbing and for mirroring I use lftp, but I would really like to download just the files that start with "xf" from Robby's site. Any suggestions?

XavierP · 09-15-2010, 09:09 AM

Have you looked at wget? It's installed with Slackware and it's man page describes how to use globbing with it.

damgar · 09-15-2010, 09:18 AM

Quote:

Originally Posted by XavierP

Have you looked at wget? It's installed with Slackware and it's man page describes how to use globbing with it.

Wget globbing doesn't work for http, only ftp. I wound up using wget after I figured out that Robby has a mirror that has ftp access, but I would still like a way to do file globbing downloads from http sites.

Lufbery · 09-15-2010, 10:11 AM

lftp seems to work pretty well with file globbing.

I downloaded the XFCE stuff from Robby's site with lftp. The command was something like:

Code:

lftp -c "open http://rlworkman.net/pkgs/13.1/ ; mirror -e -n -I xfce* x86_64"

I'm not at home so I can't test it yet, but it should work.

Run it in the directory you want to download into, and it will create a local mirror of Robby's packages in the x86_64 directory.

Obviously, you'll want to change it to what you need.

Edit: I now recall that I ran it again for the Thunnar plugins using -I thunnar* in place of -I xfce*.

Regards,

damgar · 09-15-2010, 10:23 AM

Quote:

Originally Posted by Lufbery

lftp seems to work pretty well with file globbing.

Code:

lftp -c "open http://rlworkman.net/pkgs/13.1/ ; mirror -e -n -I xfce* x86_64"

Fantastic! I didn't realize that the lftp commands have individual switches. It was the -I switch that I was overlooking. Thank you.

Code:

lftp -c "open http://rlworkman.net/pkgs/13.1/ ; mirror -I xf* x86_64"

For completeness of record: the -I switch tells the mirror command to include only files matching the following glob-string when it mirrors the target directory.

Lufbery · 09-15-2010, 11:37 AM

Quote:

Originally Posted by damgar

Fantastic! I didn't realize that the lftp commands have individual switches. It was the -I switch that I was overlooking. Thank you.

Code:

lftp -c "open http://rlworkman.net/pkgs/13.1/ ; mirror -I xf* x86_64"

For completeness of record: the -I switch tells the mirror command to include only files matching the following glob-string when it mirrors the target directory.

Yeah, LFTP is pretty cool. the -X switch is helpful too, because it excludes only files matching the glob string when mirroring. So if you wanted everything in Robby's directory except the XFCE stuff, you could use -X xf*.

Regards,

XavierP · 09-15-2010, 02:53 PM

Quote:

Originally Posted by damgar

Wget globbing doesn't work for http, only ftp. I wound up using wget after I figured out that Robby has a mirror that has ftp access, but I would still like a way to do file globbing downloads from http sites.

Apologies. Tiredness made me skip the http and focus on the ftp part of your post. That, and I usually get packages via ftp from Robby and Alien_Bob.

damgar · 09-15-2010, 03:27 PM

Quote:

Originally Posted by XavierP

Apologies. Tiredness made me skip the http and focus on the ftp part of your post. That, and I usually get packages via ftp from Robby and Alien_Bob.

No problem at all. Thanks for the reply.