LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-03-2013, 04:00 PM   #1
Weekend_warrior1999
LQ Newbie
 
Registered: Feb 2013
Posts: 11

Rep: Reputation: Disabled
Bash script help with cURL


I am trying to create a script that posts data to a site that uses both html auth and sessions cookies, the problem im having is that it doesnt always work.. if I uncomment the "rm" statements and un the script it completes with no error but doesn't post the data to the specified URL. But if I rerun it within a few minutes of the first run, it seems to complete exactly as designed. for the life of me I cant find whats wrong with the script.

Quote:
#!/bin/bash
#Login To Obtain Session Cookie
dbuser="dbadmin"
dbpass="dbpassword"
dbserver="1234124.hostedresource.com"
db="yourdb"
table="products"
csvfile="/home/content/72/10252872/html/catalog/temp/products_all_`date +%d%b%Y`.txt"
user="administrator" #OSCommerce User to use for update operation
pass="reallygoodpasswordlol" #OSCommerce password to use for update operation
admin_URL="http://yourURL.com/catalog/admin" #URL to admin directory, NOTE: No Traiing Slash in URL
admin_creds_URL="http://$user:$pass@$admin_URL"
store_URL="http://your-URL.com/catalog" #URL of store
local_dir="/home/content/72/10252872/html/catalog" #Local directory where files are stored, NOTE: No trailing slash
csv_filename="products_all_" #Prefix to attach to CSV filename
image_DL_URL="image-DL-url.com/image.tar.gz"
csv_DL_URL="CSV-DL-URL.com/CSV.txt"
supplier_name="supplier_name"
csv_ext="txt"
arch_ext="tar.gz"
catagory_image="catagory.jpg"


clear
echo "Login and store session information for phase2 of import process"
curl -v --cookie cookies.txt \
--cookie-jar cookies.txt \
-D Headers.txt \
--user-agent Mozilla/4.0 \
-u $user:$pass \
--data "username=$user&password=$pass" \
$admin_URL/login.php?action=process

echo "Clear Temp Data matching: $local_dir/temp/$csv_filename*"
rm $local_dir/temp/$csv_filename*
echo .
echo "H & D Supplier Update"
echo "Download Supplier Catalog Images to: $local_dir/temp/$csv_filename`date +%d%b%Y`.$arch_ext"
wget -O $local_dir/temp/$csv_filename`date +%d%b%Y`.$arch_ext "$image_DL_URL"
echo .
echo "Delete old Supplier Catalog Images in: $local_dir/images/images/*"
rm $local_dir/images/images/*
echo .
echo "Unarchive Supplier Catalog Images to: $local_dir/images/images"
tar -xzf $local_dir/temp/$csv_filename`date +%d%b%Y`.$arch_ext -C $local_dir/images/images
echo .
echo "Download Supplier Product Catalog: $local_dir/temp/$csv_filename`date +%d%b%Y`.$csv_ext"
wget -O $local_dir/temp/$csv_filename`date +%d%b%Y`.$csv_ext "$csv_DL_URL"
echo .
echo "Catalog file renamed for processing:$csv_filename`date +%d%b%Y`.$csv_ext to $supplier_name.$csv_ext in $local_dir/temp/"

mv $local_dir/temp/$csv_filename`date +%d%b%Y`.$csv_ext $local_dir/temp/$supplier_name.$csv_ext
echo "Operation Complete, ready for second phase of import procedure"

sed '1,4 d' cookies.txt > saved-cookies
while IFS=$'\t' read -r -a myArray
do
echo ${myArray[0]}
echo ${myArray[1]}
echo ${myArray[2]}
echo ${myArray[3]}
echo ${myArray[4]}
echo ${myArray[5]}
echo ${myArray[6]}
echo ${myArray[7]}

# Open Easy Populate and post variable to import newly downloaded CSV file
curl -v -b cookies.txt \
--user-agent Mozilla/4.0 \
-u $user:$pass \
-H Headers.txt \
--referer "$admin_URL/login.php?${myArray[5]}=${myArray[6]}" \
-F ${myArray[5]}=${myArray[6]} -F localfile=$supplier_name.$csv_ext -F imput_mode=normal -F buttoninsert=Insert%20into%20db \
$admin_URL/easypopulate.php
# Import Should be complete

echo "Import of $supplier_name products complete"

done < saved-cookies
echo "Update DB tables to disable products that have ZERO stock quantity"
mysql -u$dbuser -p$dbpass -h$dbserver -D$db <<< 'UPDATE `products` SET `products_status`="0" WHERE `products_quantity`="0";'
echo .
echo "Update DB tables to set a generic category image for newly added categories"
mysql -u$dbuser -p$dbpass -h$dbserver -D$db <<< 'UPDATE `categories` SET `categories_image`="$catagory_image" WHERE `categories_image`="";'

rm cookies.txt
rm Headers.txt
rm saved-cookies

Last edited by Weekend_warrior1999; 02-04-2013 at 05:11 PM.
 
Old 02-04-2013, 04:28 AM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,589

Rep: Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942Reputation: 1942
Please use [code][/code] tags instead of [quote][/quote] tags to retain the formatting.

My initial suggestion would be to turn on verbose output with -xv after the interpreter and redirect to a log file to see what is / is not happening.
 
Old 02-04-2013, 12:04 PM   #3
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949
I can't tell you where your problem is yet, but here are some suggestions from looking at the code.

0) As grail mentioned, please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.


1) Clean, consistent formatting makes code more readable and easily debuggable. Be liberal with whitespace, both horizontal and vertical. Indent all your sub-commands evenly and separate logical sections with empty lines. Never just line everything up on the left edge.

Many scripters also feel that it's better to place the "do/then" keywords on the same line as the "for/while/until/if" keywords, as they are not separate commands but the closing indicators for the test block, and so are paired directly with the opening keyword. Putting them together on one line thus more clearly separates the outside block from the inside block.

Scripting With Style


2) It's a really bad idea to hard-code user names and passwords into a script. A better option would be to store them in a separate "credentials" style file, readable only by the user running the script, and source that file into it.

Code:
#this is the password file

#OSCommerce User to use for update operation
user="administrator"

#OSCommerce password to use for update operation
pass="reallygoodpasswordlol"
Code:
#this imports the password file if found.
if [[ -r "$HOME/passwordfile ]]; then

    . "$HOME/passwordfile"

else

    echo "Password file not found or not readable. Exiting."
    exit 1

fi
Or you could set it up to ask the user for them directly.


3) You use the exact same date command over and over in the script. Why not just run it once and save the value into a variable to use after that?


4) $(..) is highly recommended over `..`.


5) When you have a long string of options for a command, you can use an array to set them up ahead of time.

Code:
curlopts+=( -v --cookie cookies.txt --cookie-jar cookies.txt         )
curlopts+=( -D Headers.txt --user-agent Mozilla/4.0 -u "$user:$pass" )
curlopts+=( --data "username=$user&password=$pass"                   )
curlopts+=( "$admin_URL/login.php?action=process"                    )

curl "${curlopts[@]}"
Also consider setting up functions for complex or oft-repeated commands. In addition to making the code more organized, they can also help to keep the script's running environment less cluttered, by setting to local variables not needed globally.


6) All strings with variables like this should be quoted. Just leave them off the globbing patterns and the like.
Code:
#wrong
# rm $local_dir/temp/$csv_filename*
#right
rm "$local_dir/temp/$csv_filename"*
It's vital in scripting to understand how the shell handles arguments and whitespace. Many script errors come down to improper quoting:
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes


7) Learn the power of printf.

Code:
while IFS=$'\t' read -r -a myArray ; do

    printf '%s\n' "${myArray[@]:0:7}"

done < saved-cookies
That's about it for now. Good luck!

Last edited by David the H.; 02-04-2013 at 12:20 PM. Reason: minor code correction
 
Old 02-04-2013, 08:45 PM   #4
Weekend_warrior1999
LQ Newbie
 
Registered: Feb 2013
Posts: 11

Original Poster
Rep: Reputation: Disabled
Sorry for posting earlier using the wrong tags, didn't see the Code tag, :P
But even once I reformatted the script as described it continues to do the exact same thing. As long as the `rm` statements at the bottom are commented out the script will complete correctly only if I run it twice in a row. The part that fails is the second curl statement "curlopts2 statement array". It doesn't give an error, nor does it (second curl statement) display anything as far as returned HTML/PHP coding on first run, it does show that the request in sent and uses the data obtained in the cookie file from the first curl statement from the beginning of the script but when I run the script a second time it completes correctly and returns the expected HTML/PHP coding from the server. Why isn't this script completing correctly the first time.

As for the passwords, the code that you gave, was that intended to be used with the HTPASSWD file?
This script is to be housed on a hosted webserver, and is saved a directory above HTTP access, the only access to this file is SSH, CRON, FTP and the member C-Panel File Manager. which BTW once Ive got things the way I want them, I am planning to disable SSH, and FTP access on the account. It is to be ran by CRON once a night without user input.


Code:
clear


echo "Login and store session information for phase2 of import process"
        curlopts+=( -v --cookie cookies.txt --cookie-jar cookies.txt         )
        curlopts+=( -D Headers.txt --user-agent Mozilla/4.0 -u "$user:$pass" )
        curlopts+=( --data "username=$user&password=$pass"                   )
        curlopts+=( "$admin_URL/login.php?action=process"                    )

                curl "${curlopts[@]}"




echo "Clear Temp Data matching: $local_dir/temp/$csv_filename*"
       rm "$local_dir/temp/$supplier_name"*
        echo .




echo "H & D Supplier Update"
        echo "Download H & D Catalog Images to: $local_dir/temp/$csv_filename$datetime.$arch_ext"
               wget -O "$local_dir/temp/$csv_filename$datetime.$arch_ext" "$image_DL_URL"
        echo .
        echo "Delete old H & D Catalog Images in: $local_dir/images/images/*"
                rm "$local_dir/images/images/"*
        echo .
        echo "Unarchive H & D Catalog Images to: $local_dir/images/images"
               tar -xzf "$local_dir/temp/$csv_filename$datetime.$arch_ext" -C "$local_dir/images/images"
        echo .
        echo "Download H & D Product Catalog: $local_dir/temp/$csv_filename$datetime.$csv_ext"
                wget -O "$local_dir/temp/$csv_filename$datetime.$csv_ext" "$csv_DL_URL"
        echo .
        echo "Catalog file renamed for processing:$csv_filename$datetime.$csv_ext to $supplier_name.$csv_ext in $local_dir/temp/"
                mv "$local_dir/temp/$csv_filename$datetime.$csv_ext" "$local_dir/temp/$supplier_name.$csv_ext"
        echo "Operation Complete, ready for second phase of import procedure"

#remove un-needed header info from cookie.txt & create array with cookie info
        sed '1,4 d' cookies.txt > saved-cookies
                while IFS=$'\t' read -r -a myArray ; do
                        printf '%s\n' "${myArray[@]:0:7}"

                # Open Easy Populate and post variable to import newly downloaded CSV file
                                curlopts2+=( -v -b cookies.txt                                          )
                                curlopts2+=( -u "$user:$pass"                                           )
                                curlopts2+=( --user-agent Mozilla/4.0                                   )
                                curlopts2+=( -H Headers.txt                                             )
                                curlopts2+=( -e "$admin_URL/login.php?${myArray[5]}=${myArray[6]}"      )
                                curlopts2+=( -F "${myArray[5]}=${myArray[6]}"                           )
                                curlopts2+=( -F "localfile=$supplier_name.$csv_ext"                     )
                                curlopts2+=( -F "imput_mode=normal"                                     )
                                curlopts2+=( -F "buttoninsert=Insert%20into%20db"                       )
                                curlopts2+=("$admin_URL/easypopulate.php"                               )

                                        curl "${curlopts2[@]}"



                                        echo "Import of $supplier_name products complete"
                done < saved-cookies





echo "Update DB tables to disable products that have ZERO stock quantity"
        mysql "-u$dbuser" "-p$dbpass" "-h$dbserver" "-D$db" <<< 'UPDATE `products` SET `products_status`="0" WHERE `products_quantity`="0";'
echo .




echo "Update DB tables to set a generic category image for newly added categories"
        mysql "-u$dbuser" "-p$dbpass" "-h$dbserver" "-D$db" <<< 'UPDATE `categories` SET `categories_image`="$catagory_image" WHERE `categories_image`="";'
echo .




#echo "Update Google AdWords"
#       curl $admin_URL/googlefeeder.php
#echo .


#echo "Update SiteIndex File"
#        curl $store_URL/usu5_sitemaps/index.php




#rm "cookies.txt"
#rm "Headers.txt"
#rm "saved-cookies"

Last edited by Weekend_warrior1999; 02-04-2013 at 10:40 PM.
 
Old 02-09-2013, 06:32 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949Reputation: 1949
That's much better, although now I'd say you've gone to the opposite end and added too much whitespace.

Here's how I would format it, with a couple of additional modifications (e.g. removing the need for the saved-cookies file):

Code:
clear

echo "Login and store session information for phase2 of import process"

curlopts=( -v
           --cookie cookies.txt
           --cookie-jar cookies.txt
           -D Headers.txt
           --user-agent Mozilla/4.0
           -u "$user:$pass"
           --data "username=$user&password=$pass"
           "$admin_URL/login.php?action=process"
         )

curl "${curlopts[@]}"
unset curlopts

echo "Clear Temp Data matching: $local_dir/temp/$csv_filename*"

rm "$local_dir/temp/$supplier_name"*

echo .
echo "H & D Supplier Update"
echo "Download H & D Catalog Images to: $local_dir/temp/$csv_filename$datetime.$arch_ext"

wget -O "$local_dir/temp/$csv_filename$datetime.$arch_ext" "$image_DL_URL"

echo .
echo "Delete old H & D Catalog Images in: $local_dir/images/images/*"

rm "$local_dir/images/images/"*

echo .
echo "Unarchive H & D Catalog Images to: $local_dir/images/images"

tar -xzf "$local_dir/temp/$csv_filename$datetime.$arch_ext" -C "$local_dir/images/images"

echo .
echo "Download H & D Product Catalog: $local_dir/temp/$csv_filename$datetime.$csv_ext"

wget -O "$local_dir/temp/$csv_filename$datetime.$csv_ext" "$csv_DL_URL"

echo .
echo "Catalog file renamed for processing:$csv_filename$datetime.$csv_ext to $supplier_name.$csv_ext in $local_dir/temp/"

mv "$local_dir/temp/$csv_filename$datetime.$csv_ext" "$local_dir/temp/$supplier_name.$csv_ext"

echo "Operation Complete, ready for second phase of import procedure"

#remove un-needed header info from cookie.txt & create array with cookie info

while IFS=$'\t' read -r -a myArray ; do

    printf '%s\n' "${myArray[@]:0:7}"

    # Open Easy Populate and post variable to import newly downloaded CSV file

    curlopts=( -v
               -b cookies.txt
               -u "$user:$pass"
               --user-agent Mozilla/4.0
               -H Headers.txt
               -e "$admin_URL/login.php?${myArray[5]}=${myArray[6]}"
               -F "${myArray[5]}=${myArray[6]}"
               -F "localfile=$supplier_name.$csv_ext"
               -F "imput_mode=normal"
               -F "buttoninsert=Insert%20into%20db"
               "$admin_URL/easypopulate.php"
             )

    curl "${curlopts2[@]}"

    echo "Import of $supplier_name products complete"

done < <( sed '1,4d' cookies.txt )

echo "Update DB tables to disable products that have ZERO stock quantity"

mysqlopts=( "-u$dbuser"
            "-p$dbpass"
            "-h$dbserver"
            "-D$db"
          )

mysqlcommand='UPDATE `products` SET `products_status`="0" WHERE `products_quantity`="0";'

mysql "${mysqlopts[@]}" <<<"$mysqlcommand"

echo .
echo "Update DB tables to set a generic category image for newly added categories"

mysqlcommand='UPDATE `categories` SET `categories_image`="$catagory_image" WHERE `categories_image`="";'

mysql "${mysqlopts[@]}" <<<"$mysqlcommand"

echo .

#echo "Update Google AdWords"
#       curl $admin_URL/googlefeeder.php
#echo .

#echo "Update SiteIndex File"
#       curl $store_URL/usu5_sitemaps/index.php

#rm "cookies.txt"
#rm "Headers.txt"
I can see no real errors in the syntax or logic, except perhaps for the fact that you're both reading values from cookies.txt in the while loop and using it directly in the curl commands inside it. Since I'm not privy to the ins and outs of your situation it's hard to say more than that.

It seems that the problem is most likely somewhere in your inputs or your variable values. Since there's no way I can test that myself, you'll just have to run through it carefully until you find the error. Include echos for all of your variables, and/or run the script with "#!/bin/bash -vx" to get full debugging output. Also check that all of your inputs are sane. Are there dos-style newlines in the input text files, for example? You can pipe text through cat -A to see if there are any embedded non-printing characters.


Finally, since this has something to do with server response, it is possible that the issue isn't on your end per-se, but with the way the server is set up to respond to the requests. Perhaps it needs to save additional cookies or something before it will allow you to complete the transaction? Can you do everything you want manually outside the script?


Re: passwords, I'm talking specifically about your $user and $pass variables and how they are being set inside the script. Even if you consider the script itself to be secure, it's still poor practice to hard-code values like those into one. Using an external means of storage works to keep the data separate from the code, something you should always strive for. Indeed, I would recommend importing all user-specific values from the same external file.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Using curl to post data in a bash script rather than use a web page. pobrika Programming 3 06-08-2012 03:32 AM
bash simple cleanup script then curl sir-lancealot Programming 1 02-02-2011 12:59 PM
CURL issues in bash iniuria Programming 1 03-08-2010 03:39 AM
cURL: Server has many IPs, how would I make a cURL script use those IPs to send data? guest Programming 0 04-11-2009 11:42 AM
script to parse variables to curl script and execute morphix Programming 17 11-30-2007 01:27 AM


All times are GMT -5. The time now is 06:42 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration