wget quit with memory exhausted while download large file recursively

TruongAn · 09-25-2014, 12:16 PM

I want to download a whole directory listing by apache on my server. I use "wget -r" for that. The directory contain some very large files so I specified -c switch to resume in case download process is interupted mid way. The exact option I used was:

Code:

wget  -c -r -np -nH -R 'index*' --cut-dirs=1

And then come the problem, when I resume downloading, if wget encounter some fully retrieved file, it will hang up and took a lot of memory before proceed to the next file. The larger the file, the more memory it took. And if the file was too large (8GiB in my case) wget stop with a memory exhausted messange. The last output it show me (in --debug mod) was

Code:

---response end---
416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

Disabling further reuse of socket 3.
Closed fd 3

The server was just an apache web server with default settings. I don't suspect the server was the cause and really has no more ideas where to look at. Please help

business_kid · 09-27-2014, 04:11 AM

If you ran out of memory, it is the kernel that would have shut your connection down, not apache.

That said, wget making an like that is strange. The code dates, afaik, from the days where memory wasn't as plentiful as it is today.

Edit: The compare function could be passed out to another utility (e.g. diff) which may be memory hungry.

TruongAn · 09-29-2014, 11:48 AM

Quote:

Originally Posted by business_kid

If you ran out of memory, it is the kernel that would have shut your connection down, not apache.

That said, wget making an like that is strange. The code dates, afaik, from the days where memory wasn't as plentiful as it is today.

Edit: The compare function could be passed out to another utility (e.g. diff) which may be memory hungry.

I too thought it was not apache problem . Is there anything I could do on the wget side?

business_kid · 09-30-2014, 09:04 AM

Yes - figure out the meaning of this message

Code:

   The file is already fully retrieved; nothing to do.

To me it seems to be saying that your wgat was successful. If wget is hitting the max memory while using the -c option, stop using it.

TruongAn · 10-01-2014, 10:42 AM

Quote:

Originally Posted by business_kid

Yes - figure out the meaning of this message

Code:

   The file is already fully retrieved; nothing to do.

To me it seems to be saying that your wgat was successful. If wget is hitting the max memory while using the -c option, stop using it.

How can I resume an interrupted downloading process without -c option then

business_kid · 10-01-2014, 02:50 PM

You should have the directory tree there. I would give it 'wget -c url/file' with no other options and process it when you have it down.