Bash script to scan URLs
Let's say there is a site www.myexamplesite.local
It has the /~users/ directory, which is not allowed for listing. I know the userdir naming policy: the directory must begin with an underline, a tilde, or an exclamation mark, followed by latin letters and numerics, 2 to 24 symbols per name in total.
To get the userdir list, I want to use bash and wget, to scan that array for anything that doesn't return 403 or 404 and list. I don't want to flood the server with requests and I don't care much for how long will it take to make a complete list of user directories valid for view, so a random pause (say, 0.1 to 3 seconds) between requests is needed, as well. And I certainly don't want to make it look suspicious to that server's administrator ('cause we all know how paranoid one needs to be an admin, anything that doesn't look perfectly normal, legit or not, will be dealt with by banhammer, by default), so I need to randomly switch user-agents from a pre-existing list.
Where and how should I look for howtos and examples to build that script? URLs, keywords, please? |
