Does anyone out there use a particular crawler or sitemapping tool, which they are happy with, just to crawl their own site? Idea is to look for broken links, weird response codes, and such like. In this particular case I don't have access to the server backend file system directly, so crawler would need to crawl via http.

Looking for something simple and FOSS/Linux friendly. Found some "free" online sites, but there are limitations unless I pay. Found some scripts here and there but they seemed kind of shoddy or hard to use.

I would use Selenium for this.