LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-04-2009, 01:33 PM   #1
MikeGruz
LQ Newbie
 
Registered: Jun 2009
Posts: 2

Rep: Reputation: 0
Question Problem authenticating into Google with bash script


Hello all,

I've been attempting to write a bash script that will query Google Trends for specified keywords (instead of having to use the site). One doesn't have to be logged in to view data, but to download csv files you must use your Google account. As you will see from the code below, I haven't gotten much past the authentication phase of the process.

This page explains how to authenticate for Google Analytics, but it doesn't seem to work when applied to Trends.

When I send user/pass info to Google, it replies with an authentication key, which the script grabs; this key (presumably) must be used each time one makes a request for a csv file, but I can't seem to get it to work. It just replies that I must be logged in to download the files.

Script below (please forgive my naivety, I'm not really a programmer.. I'm a political scientist):

Code:
#!/bin/bash

# trends
# 
# A script to download Google Trends data
#


# Get login/password information
echo
echo -e Please Enter Google Login:
read login

echo
echo -e Please Enter Google Password:
stty -echo
read pass

stty echo

# Authenticate with Given Information

auth="$(curl https://www.google.com/accounts/ServiceLoginBoxAuth \
-d Email=$login -d Passwd=$pass \
-d accountType=GOOGLE \
-d source=mikegruz-cURL-0.1 \
-d service=trends \
| awk /Auth=.*/ | sed s/Auth=// )"

# Request search term from user

echo
echo -e Please enter desired search term:
echo
read search

# Transmit Trends Request to Google

feedUri="https://www.google.com/accounts/ServiceLoginBoxAuth"

curl $feedUri -s --header "Authorization: GoogleLogin auth=$auth" \ "http://www.google.com/trends/viz?q=$search&graph=all_csv&sa=N"
 
Old 06-05-2009, 06:25 AM   #2
secesh
Senior Member
 
Registered: Sep 2004
Location: Savannah, GA
Distribution: Ubuntu, Gentoo, Mythbuntu, ClarkConnect
Posts: 1,154

Rep: Reputation: 47
I think your problem is that google is actively working against you creating a script to harvest their trend data. Not knowing much about google trends, I can't be a whole lot of help here, but I did see others that had problems accessing trends.

From Google's general TOS:
Quote:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
...and since they don't seem to have an official trends API yet, I assume they don't want you to automate trend data harvesting.

There was an unofficial API here: http://zoastertech.com/projects/, but it looks to have been taken down... perhaps he's a nice guy who'd like to chat about it, though.

Python alternative? http://www.juiceanalytics.com/openju...le-trends-api/

There was a flurry of activity late '07 about an API coming soon... but I can't seem to find one.
 
Old 06-05-2009, 10:32 AM   #3
MikeGruz
LQ Newbie
 
Registered: Jun 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Unfortunately, I think you're correct. It is a bit annoying that the API hasn't been released, considering it was so talked up two years ago. Back to the drawing board, I guess.

Thanks for your response!

Mike
 
Old 06-06-2009, 05:24 AM   #4
Zoasterboy
LQ Newbie
 
Registered: Jun 2009
Posts: 3

Rep: Reputation: 1
You can get the CSV if you can make a GET/POST.

This URL:

Code:
http://www.google.com/trends/viz?q=apples&date=all&geo=all&graph=all_csv&sort=0&sa=N
will return a CSV for searches for the keyword "apples" in all dates and locations.

I was able to retrieve the file without any authentication.

It's still working as it was before (as far as I know). Personally I abandoned working on my own projects because of the "no automated scripts" clause in Google's policy, but I doubt they will try to hunt you down if it's a little project. I was going to extend their basic API, but sadly now I can not.

Start here for some more info on how their basic API works:
http://zoastertech.com/projects/goog...etting+Started
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] bash : getopts problem in bash script. angel115 Programming 2 03-02-2009 10:53 AM
LXer: Bash script to find all of your indexed web pages on Google LXer Syndicated Linux News 0 01-15-2009 06:30 AM
authenticating a linux user from a script? evank Linux - Security 2 02-02-2006 11:49 AM
Problem with bash script. cheater1034 Programming 9 11-21-2005 10:29 PM
Problem with this bash script cantabile Programming 3 06-06-2005 05:31 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 02:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration