Generate a sitemap for Google. Easily create xml sitemaps. Crawl as many webpages as you want, up to 10,000 with this free google sitemap generator.
This script crawls a web site from a given starting URL and generates a Sitemap file in the format that is accepted by Google. The crawler does not follow links to other web sites. It also respects the 'nofollow' tags and will not crawl into directories disallowed in the robots.txt file.
Comments