LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   [Mocha] - I would like to retrieve multiple files at one go. (https://www.linuxquestions.org/questions/linux-newbie-8/%5Bmocha%5D-i-would-like-to-retrieve-multiple-files-at-one-go-892282/)

haojam 07-18-2011 05:22 AM

[Mocha] - I would like to retrieve multiple files at one go.
 
Dear Sir,
I would like to retrieve the lists of multiple files at one go. Each file is big size so wget command is too slow to download all 14 files. Could you please tell any other option to do it.

ftp://ftp-trace.ncbi.nlm.nih.gov/sra...X017/SRX017837

Regards,
Mocha
Seoul National University College of Medicine

b0uncer 07-18-2011 05:40 AM

Quote:

Originally Posted by haojam (Post 4417927)
Each file is big size so wget command is too slow to download all 14 files.

Could you be a little more specific about this? I don't think wget "is slow", it should download as fast as the connection allows (theoretically, up to the point when disk I/O and such things start to slow things down--but this will probably not be a problem). If you mean that you have a very fast network connection, but that the specific connection through which you download a file from the server is slow (e.g., the bottleneck is that single connections to the server are limited to some slower speed than what you are capable of), then I guess you could try to download the files in parallel, e.g., several at once. This would speed things up if each of the connections could work at the same speed as one connection, e.g., the "speed limit" was on a per-connection basis, not over all the connections to the server. However, if it is not absolutely necessary, I recommend not doing this: there is probably a reason for the speed limit, and one should always respect the limits set on remote servers.

If you need an example on how you can use wget (other programs would be fine too) to do parallel downloads, see for example this. Basically it fetches a list of what to download, then pipes that to (e)grep and xargs which are used to pick up files and build (argument) lists for multiple wget commands, which then start (simultaneously) the downloads.

If I misunderstood your question, please ignore my answer (unless it's helpful in some way) and perhaps re-phrase your post to get better/correct answers.


All times are GMT -5. The time now is 07:55 PM.