wget problem with mirroring

Tinku · 01-15-2006, 06:42 AM

I am trying to mirror a website.So,I am using the following command line.

Code:

 wget -r -l0 http://en.wikipedia.org/

I even tried the mirror argument.

This is what I get.

Code:

tinku@localhost:~/docs/books$ wget -r -l0 http://en.wikipedia.org/
--18:19:24--  http://en.wikipedia.org/
           => `en.wikipedia.org/index.html'
Resolving proxy.esi.edu... 168.1.1.192
Connecting to proxy.esi.edu [168.1.1.192]:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://en.wikipedia.org/wiki/Main_Page [following]
--18:19:24--  http://en.wikipedia.org/wiki/Main_Page
           => `en.wikipedia.org/wiki/Main_Page'
Connecting to proxy.esi.edu[168.1.1.192]:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Length: unspecified [text/html]

    [     <=>                             ] 42,008        29.06K/s             

18:19:26 (29.05 KB/s) - `en.wikipedia.org/wiki/Main_Page' saved [42008]

Loading robots.txt; please ignore errors.
--18:19:26--  http://en.wikipedia.org/robots.txt
           => `en.wikipedia.org/robots.txt'
Connecting to proxy.esi.edu[168.1.1.192]:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 3,793 [text/plain]

100%[====================================>] 3,793         --.--K/s             

18:19:26 (1.15 MB/s) - `en.wikipedia.org/robots.txt' saved [3793/3793]


FINISHED --18:19:26--
Downloaded: 45,801 bytes in 2 files

And thats it.Only 2 files downloaded

Any help will be appreciated
tia
tinku

kwacka · 01-15-2006, 07:37 AM

The option l gives the levels to which you want wget to go down into the recursive (r) directories.

Making it l 0 means that it only gets that level, l 1 the next level, l 2 next and so on.

Take care when using this - nless you want wikipedia and all the sites linked to it. 8-)

Tinku · 01-15-2006, 07:41 AM

No change.I tried changing 0 to other levels,but only the same 2 files are being downloaded.

trickykid · 01-15-2006, 08:18 AM

I think it's the fact that by default, wget will look for .html files, your dealing with a wiki, which most of the internal links do not end with any type of extension.

You'll probably have to play around with the -E or --html-extensions option as I don't really see any other way around it. The only other problem I see is that it would cause pages that are not .html files to have the .html extension.

man wget for more options and details.