LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 04-09-2008, 03:39 AM   #1
Phyrexicaid
Member
 
Registered: Aug 2004
Location: JHB - South Africa
Distribution: Kubuntu 7.04; openSUSE 10.1
Posts: 176

Rep: Reputation: 30
Automating database web search


I run weekly reports on an online database (I don't have full access to the database) where I need to generate states based on a certain field. Currently the only way to do this is to select each option from a drop down list, perform the search and copy the number I need from the search results. I then hit back, select the next option from the drop down list and repeat.

Quite tedious and time consuming. The database maintainers won't add the search option I need.

I was thinking that if I could see what query is being sent, I could then write a script that automatically sends the query, picks out the string I need from the result and save that.
What I need therefore is:
  • Ability to log into the site
  • Create query
  • Strip info I need out of result
  • Save result
  • Rinse and repeat

Any suggestions /help would be appreciated!
 
Old 04-09-2008, 04:00 AM   #2
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 4&5, Fedora 10, CentOS 5.4, IPCop
Posts: 569

Rep: Reputation: 55
It all depends on the interface to the database. If the web-address that displays the search results contains the search parameters - e.g. in the way a google search does - then you can copy this address and use "wget" to repeat that search from the command line or a script.

wget can be instructed to save the results to a file and you can then use grep/sed/whatever to interrogate the search results (which is now an HTML file on your disk) for the data you need.

If you need to login to the database, wget can do this too.
 
Old 04-09-2008, 04:22 AM   #3
Phyrexicaid
Member
 
Registered: Aug 2004
Location: JHB - South Africa
Distribution: Kubuntu 7.04; openSUSE 10.1
Posts: 176

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by blacky_5251 View Post
It all depends on the interface to the database. If the web-address that displays the search results contains the search parameters - e.g. in the way a google search does - then you can copy this address and use "wget" to repeat that search from the command line or a script.

wget can be instructed to save the results to a file and you can then use grep/sed/whatever to interrogate the search results (which is now an HTML file on your disk) for the data you need.

If you need to login to the database, wget can do this too.
No, that's the problem, I opened the frame that the search is being performed in to see what the web-address is and all it gives is:
Code:
www.some.site/admin_customers.asp?mode=search&ID=
No matter what search is being performed.
 
Old 04-09-2008, 04:25 AM   #4
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 4&5, Fedora 10, CentOS 5.4, IPCop
Posts: 569

Rep: Reputation: 55
Can you get an address by holding the mouse over the "search" button?
 
Old 04-09-2008, 04:30 AM   #5
Phyrexicaid
Member
 
Registered: Aug 2004
Location: JHB - South Africa
Distribution: Kubuntu 7.04; openSUSE 10.1
Posts: 176

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by blacky_5251 View Post
Can you get an address by holding the mouse over the "search" button?
No, it just gives the address posted above. I'm looking at the page source and it seems the variables are hidden:
Code:
<form  action="admin_customers.asp?mode=search&ID="
			
			method="post" id="searchForm" name="searchForm" onsubmit="return validateForm()">
		

	<input type="hidden" id="mode" name="mode" value="search">
	<input type="hidden" id="ID" name="ID" value="">			
		
	<input type="hidden" id="startpos" name="startpos" value="">					
	<input type="hidden" id="sort" name="sort" value="">
	<input type="hidden" id="sortorder" name="sortorder" value="">
	<input type="hidden" id="checkedItems" name="checkedItems">
	<input type="hidden" id="uncheckedItems" name="uncheckedItems">
	<input type="hidden" id="selectAll" name="selectAll">
	<input type="hidden" id="deSelectAll" name="deSelectAll">
	<input type="hidden" id="recordselected" name="recordselected">
	<input type="hidden" id="resultsPerPage" name="resultsPerPage">
	<input type="hidden" id="candidateActivity" name="candidateActivity">
	<input type="hidden" id="queryID" name="queryID">
	<input type="hidden" id="AdminQueryID" name="AdminQueryID" value="" />
	<input type="hidden" id="AdminQueryName" name="AdminQueryName" value="" />
 
Old 04-09-2008, 04:44 AM   #6
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 4&5, Fedora 10, CentOS 5.4, IPCop
Posts: 569

Rep: Reputation: 55
All I can suggest is that you try to "guess" the address by creating the URL yourself using info from the hidden fields you found. Assign them values, separate them properly, and see how you get on.
 
Old 04-09-2008, 07:17 AM   #7
Phyrexicaid
Member
 
Registered: Aug 2004
Location: JHB - South Africa
Distribution: Kubuntu 7.04; openSUSE 10.1
Posts: 176

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by blacky_5251 View Post
All I can suggest is that you try to "guess" the address by creating the URL yourself using info from the hidden fields you found. Assign them values, separate them properly, and see how you get on.
I've found an easier way to go about it. I'm using iMacros for Firefox, so now I can automate the process.

What I'm doing at the moment is letting the script run and writing down the numbers I need on each page (only one number, so not too bad), but what I'd like to do is save each page using:
Code:
SAVEAS TYPE=MHT FOLDER=* FILE=EasternCape
But then the file with the info I need is saved in the EasternCape_files folder. What I would need to do is write a script to descend into that folder, grep the portion of the file I need and dump the info into a text file so it has
Code:
EasternCape, info
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Database Structure of Search Engine pijush General 1 05-21-2007 07:17 AM
LXer: Automating Web site and WordPress upgrades with Subversion, mirrorlinks and bash LXer Syndicated Linux News 0 11-11-2006 11:33 AM
Search in vast MySQL database - how? vargadanis Programming 8 10-13-2006 01:30 PM
Php search mysql database jamesjoy Programming 4 12-02-2005 08:28 AM
Automating a full database backup dominant Linux - Newbie 2 07-10-2004 08:59 AM


All times are GMT -5. The time now is 05:31 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration