LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-30-2015, 10:08 AM   #1
l33y
Member
 
Registered: Jan 2014
Location: Gulf Coast, USA
Distribution: Xubuntu
Posts: 50
Blog Entries: 1

Rep: Reputation: Disabled
curl question regarding security


If I use curl to copy the html off of a website, would the owners of the website think I was trying to hack them?

As an example, say I decided to build a mysql database program that records daily rainfall in a specific area of the country. I would be using curl to copy the website daily, then using gawk to find the amount of rainfall in the html file, then importing this value into a mysql database.

Or would the owners of the website even know I was using curl?

Thanks in advance
 
Old 08-30-2015, 10:47 AM   #2
DavidMcCann
Senior Member
 
Registered: Jul 2006
Location: London
Distribution: CentOS, Salix
Posts: 4,437

Rep: Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360Reputation: 1360
As I understand it, curl downloads the source from a url. Since that's what a browser does, or wget, how does the server know (or its owners care) which tool is being used? Hacking is when you try to access information on the server that isn't intended to be downloaded, and that naturally gets noticed (although not always, unfortunately!)
 
1 members found this post helpful.
Old 08-30-2015, 04:14 PM   #3
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 1,860

Rep: Reputation: 238Reputation: 238Reputation: 238
Read the man page, report your self as a browser.
Code:
curl -A "Mozilla5/0 Firefox 28" http://www...com -o - > report.html
 
1 members found this post helpful.
Old 08-30-2015, 04:19 PM   #4
273
LQ Addict
 
Registered: Dec 2011
Location: UK
Distribution: Debian Sid AMD64, Raspbian Wheezy, various VMs
Posts: 6,602

Rep: Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882Reputation: 1882
Quote:
Originally Posted by teckk View Post
Read the man page, report your self as a browser.
Code:
curl -A "Mozilla5/0 Firefox 28" http://www...com -o - > report.html
I would be tempted to do something like this. Logs do show the user agent string* and while web scraping is common and accepted practice I think some people may see repeated uses of curl as some kind of attempted hack.

*I just have to drop in my story that, at an old place of work, I used to connect to our web-based Outlook solution with the user agent "Hey Steve :-)" as the guy who was running the pilot used to check the logs to see which browsers and OSs people were using.
 
2 members found this post helpful.
Old 08-31-2015, 08:54 AM   #5
Habitual
LQ Addict
 
Registered: Jan 2011
Posts: 8,249
Blog Entries: 11

Rep: Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289Reputation: 2289
Quote:
Originally Posted by l33y View Post
Or would the owners of the website even know I was using curl?
Yes, they would.
If they review access.log, that is.

Code:
My_host_Name - - [31/Aug/2015:09:53:38 -0400] "GET / HTTP/1.1" 200 53925 "-" "curl/7.35.0"
 
1 members found this post helpful.
Old 08-31-2015, 10:56 PM   #6
l33y
Member
 
Registered: Jan 2014
Location: Gulf Coast, USA
Distribution: Xubuntu
Posts: 50
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Many thanks for the information, folks. I will check out the man page for curl, and mark this thread as solved. I am having fun learning gawk and mysql.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Curl shipped with 14.1 has security issues and bugs! akschu Slackware 8 02-16-2014 04:03 PM
[SOLVED] curl Question desmond33 Linux - Software 9 09-13-2010 02:26 AM
LXer: Security update for cURL LXer Syndicated Linux News 0 03-05-2009 04:41 AM
Quick question on curl and tar hbinded Linux - General 2 04-24-2007 09:38 PM
cURL script question verbatim Programming 8 05-18-2005 04:50 PM


All times are GMT -5. The time now is 09:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration