Simple WGET Question

corepuncher · 11-01-2012, 05:10 PM

I have a url http://www.url.com/dir1/dir2/dir3/dir4

I want to download all the files in that dir with wget. All csv files. *.csv does not work.

I tried google search but everyone wants to give me all sorts of options that do other things. Shouldn't this be really easy? Thanks.

acid_kewpie · 11-01-2012, 05:26 PM

You don't necessarily know what files are there to download, you can't literally just request a wildcard.

you can use the -r option to download recursively, try a full command like "wget -r -l1 --no-parent -A.csv domain.com/a/b/c"

alternatively, curl is useful for downloading patterns of files like [000-999].csv would cover those 1000 files explicitly.

theNbomr · 11-01-2012, 05:43 PM

Don't forget that wget is just like a web browser (technically, is exactly like, except it doesn't render anything except to stdout). Whatever you could get by typing a URL into a GUI web browser, you can get with wget. As acid_kewpie mentions, wget can work recursively, meaning it can find and follow the links on a page you initially load.

--- rod.

corepuncher · 11-01-2012, 08:10 PM

Thanks for your response.

I tried this command but it created a huge dir structure along with it. I don't want all the directories copied, just the files from the very last directory.

Quote:

Originally Posted by acid_kewpie

You don't necessarily know what files are there to download, you can't literally just request a wildcard.

you can use the -r option to download recursively, try a full command like "wget -r -l1 --no-parent -A.csv domain.com/a/b/c"

alternatively, curl is useful for downloading patterns of files like [000-999].csv would cover those 1000 files explicitly.

theNbomr · 11-02-2012, 08:02 AM

You must have either not transcribed the command correctly, or your wget does not support the --limit (-l; lower-case 'ell') option that would have limited the depth of recursion to one directory. Consult your man page for wget.

--- rod