LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 10-23-2006, 11:17 AM   #1
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Rep: Reputation: 15
Using Linux to Automate Online Data Entry


Hello, I have a tricky question that I want to leave as open-ended as possible. At my work, I frequently need to do a LOT of repetitive data entry through a web-browser (usually firefox). It usually takes days and days, and is the perfect thing for a computer to do for a human, but I can't figure out a way for my computer to do this for me.

Does anybody out there have any theories about how to accomplish online data entry from a database to a website?

I am competent with scripts, and could probably come up with something if there was a way to apply scripts to a browser. Would lynx be a solution? I'm baffled. Help?

The other thing that's on my mind is that I can do all of the data entry using only the keyboard, so is there a way to automate keyboard entry? Hmmmm...
 
Old 10-23-2006, 12:13 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965
i really don't think you can be as open ended as possible... there are plenty of ways to post data, but without some sample conetxt to apply it to we can't suggest a suitable way to do it. you can easily use curl or wget to automatically perform GET's and POST's to deliver forms to a remote site, but sometimes that's not enough... if you're just filling in a page then i expect they will do your job fine.
 
Old 10-23-2006, 12:17 PM   #3
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
Hummmm...now you've got me. I don't know those utilities. I can give more specifics...Imagine a webpage that has a series of check boxes, radio buttons and form fields. I need to go through a database by hand and, using the info within it, enter the correct thing into each of these fields. Once the fields are all entered correctly, I need to click the enter button...These utilities can do that? If so, can you give me an example?
 
Old 10-23-2006, 12:21 PM   #4
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965
yeah that's possible, it's not somethign i've done myself though. google will have plenty of exampels for posting forms. This could of course be used for fairly undesirable ends. hope that's not you...

http://curl.netmirror.org/docs/httpscripting.html
 
Old 10-23-2006, 12:25 PM   #5
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
No, certianly not, though that thought did occur to me when I first thought about asking this question. This is for work, we have massive databases of information that the government regularly wants for audits and such, and doing it by hand is just not so good on the hands/wrists.

Posting forms is the name of the game though, huh? That's a big help. Can you give any specific examples to get me started?

Thanks again.

Last edited by mlissner; 10-23-2006 at 12:36 PM.
 
Old 10-23-2006, 06:26 PM   #6
metalx1000
Member
 
Registered: Jun 2006
Distribution: Debian
Posts: 109

Rep: Reputation: 16
I know this is a Linux Forum, but I do know how to do what you are trying to do with Visual Basic. I don't know if you know anything about Visual Basic or if you even have access to it, but here is a link that may help.

http://www.vbforums.com/printthread.php?t=384076
 
Old 10-23-2006, 06:35 PM   #7
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
Nah, not my cup of tea. I'm more of a shell scripter than a VB guy. Any other thoughts out there?
 
Old 10-24-2006, 06:51 AM   #8
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965
well there are plenty of examples in the above link i gave you.
 
Old 10-24-2006, 04:06 PM   #9
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
Ah. I thought it was part of your signature and missed it the first time.
 
Old 10-27-2006, 07:30 PM   #10
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
Follow-Up

As a follow-up, I've been playing with cURL for the past few days, but I can't figure out how to use it. I'm familiar with HTML, so understanding the source code works, but it seems almost like just doing the post command doesn't really work.

I'm doing something like this:

Code:
curl -u mlissner:password -F username="newuser" -F password="password" http://www.example.com
which seems to me like it would send the values newuser and password to the username and password fields of www.example.com. It doesn't seem to work though for whatever reason. It kind of seems like the information isn't getting handed to the right thing.

Like on my website, I have a contact form. The fields in the form are handed to a bit of php, which then emails me. I can't get this to work through cURL either, and I'm not sure why. Here's the code for what I used to try to communicate with my contact form on my site (much to no success).

Code:
curl -F name="Michael Lissner" -F email="mlissner@hotmail.com" -F message="Hello, testing..." http://www.michaeljaylissner.com/contact.php
Does anybody know what's going wrong here, or have a more in depth tutorial they can recommend. Scouring the Internet has been frustrating at best. Even wikipedia has the same article as everywhere else when it comes to this utility...

Thanks in advance.
 
Old 11-06-2006, 06:50 PM   #11
mlissner
Member
 
Registered: Jun 2006
Location: Berkeley, CA
Distribution: FC5
Posts: 37

Original Poster
Rep: Reputation: 15
One more note on this subject. I'm still having quite an issue doing this, so I wonder if anybody can provide an example that actually works so I know what I'm working with? I'm mostly just trying to fill in forms, but ultimately I'd like to do it on websites that require me to log-in.

This seems like a normal enough application of cURL, but perhaps I'm asking too much because to do this somehow involves logging into websites, which seems to be too much of a task for cURL.

An example of how I'd like to use this might be to send an email through gmail using only cURL to post the correct information to the correct places, or to post a message here without using a browser? Can these kinds of things be accomplished if the sites have unique URL's?

Am I missing something, or what? Any examples of cURL working? Thanks again.

Last edited by mlissner; 11-06-2006 at 06:53 PM.
 
  


Reply

Tags
curl, forms


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how do I incorporate yahoo online data with oocalc ? anindyanuri Linux - Software 0 07-11-2006 10:08 PM
web based data entry ScottReed Programming 3 07-10-2006 12:04 AM
Tool to develop data entry software in linux which has internet support kunalcool1 Linux - Software 0 09-06-2005 03:53 PM
Manage data and maintenance of MySQL database online rhuser Linux - Software 2 02-17-2003 08:31 PM
Gui Interface For Data Entry JAMESBEATTY Linux - Newbie 2 10-23-2002 05:53 PM


All times are GMT -5. The time now is 01:27 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration