Wednesday, May 27, 2009

Best spidering tool for Windows is ..

wget period!

For example:

wget -rkp -np http://site-to-spider

-r: recursive
Turn on recursive retrieving

-k: convert-links
After the download is complete, convert the links in
the document to make them suitable for local viewing.

-p: page-requisites
This option causes Wget to download all the files that
are necessary to properly display a given HTML page.

-np: no-parent
Do not ever ascend to the parent directory when
retrieving recursively.

No comments: