shelob

https://github.com/bmnick/shelob

A giant spider that starts on a given page, finds all links on the page, ensure they resolve, and recurses if the link is underneath the starting url