Share your knowledge at the LQ Wiki.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


Closed Thread
  Search this Thread
Old 03-09-2011, 03:07 AM   #1
LQ Newbie
Registered: Sep 2008
Posts: 4

Rep: Reputation: 0
Post Bash script to scan URLs

Let's say there is a site www.myexamplesite.local
It has the /~users/ directory, which is not allowed for listing. I know the userdir naming policy: the directory must begin with an underline, a tilde, or an exclamation mark, followed by latin letters and numerics, 2 to 24 symbols per name in total.
To get the userdir list, I want to use bash and wget, to scan that array for anything that doesn't return 403 or 404 and list. I don't want to flood the server with requests and I don't care much for how long will it take to make a complete list of user directories valid for view, so a random pause (say, 0.1 to 3 seconds) between requests is needed, as well. And I certainly don't want to make it look suspicious to that server's administrator ('cause we all know how paranoid one needs to be an admin, anything that doesn't look perfectly normal, legit or not, will be dealt with by banhammer, by default), so I need to randomly switch user-agents from a pre-existing list.

Where and how should I look for howtos and examples to build that script? URLs, keywords, please?
Old 03-09-2011, 04:58 AM   #2
LQ 5k Club
Registered: May 2003
Location: London, UK
Distribution: Debian "Testing"
Posts: 6,096

Rep: Reputation: 408Reputation: 408Reputation: 408Reputation: 408Reputation: 408
Please read the LQ Rules

We are not here to help you hack someone's server.
Old 03-09-2011, 05:15 AM   #3
LQ Guru
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
I agree with tredegar: if the user directory is not allowed for listing why do you want to circumvent this restriction? Eventually you can try to contact the site administrator or the user itself and ask information, especially if it is a local web site. However if a restriction is applied, they have had their good reasons. Said that, thread closed. Please, contact me if you disagree or simply want to explain your reasons.

Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Command Line Scan in Bash LXer Syndicated Linux News 0 01-17-2011 08:20 PM
SSH connection from BASH script stops further BASH script commands tardis1 Linux - Newbie 3 12-06-2010 09:56 AM
Bash - Read user input: URLs without the enter key stroke make Programming 2 09-23-2010 05:29 PM
LXer: Bash scripts to scan and monitor network LXer Syndicated Linux News 0 05-13-2010 01:00 AM
how to check urls and stop internet urls in network gface Linux - Networking 5 03-24-2005 10:48 PM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:12 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration