Terence.Tech | Coding projects of Terence Chisholm

The SEO Javascript Website Crawler

You can crawl sites and capture information from all the pages of your site or multiple sites. All in one go. Unlike other site crawlers, that use libraries like PhantomJS and other things you have to be an expert to understand... this crawler is proper ole school Javascript, with a few bits of jQuery too.

That means, you can view and download the source, and tweak it... with basic JavaScript knowledge. Awesome.

How to use it:


After it's done, you can copy and paste a spreadsheet with SEO Data and other page data into Excel or Numbers (for Mac). You can also do fancy stuff like:

You can set advanced configurations on your crawl, by going to http://terence.tech/crawler/#config then click the (X) to close the controller layer, and type in the configurations settings you desire, then click the link to save those as a shareable URL. Then press CRAWL and start your crawl.