Latest LQ Deal: Latest LQ Deals
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 02-12-2010, 11:52 AM   #1
LQ Newbie
Registered: Aug 2003
Location: New Hampshire
Distribution: LFS 6.1 & Gentoo
Posts: 13

Rep: Reputation: 0
Bash Script: parse active process stderr, strip, dump into variables, use variables

I think some of my problem is trying to describe what I am trying to do. The script runs a process which dumps a lot info to stderr. While the process is still running I want to strip data from the stderr, dump it into variables, act upon the variables, then repeat the strip, dump, act.

I have a command that is the start of this, but I want to expand this.

The line I strip out has 3 parts. A http address, a file location, and a md5 hash. At present it strips, downloads the file to the location desired, and saves the md5 in filename.md5.
I want to prevent re-downloading the same files by testing to see if the filename.md5 exists and compare hash.

$ ./process 2>&1 |  sed -un 's/.*http/http/p' | grep --line-buffered -v "'" | awk '{ system("curl --create-dirs " $1 " -o " $2 " ; echo " $3 " > " $2".md5") }'
./process 2>&1 |
redirects stderr to stdout
sed -un 's/.*http/http/p' |
pulls the line we need and strips off any beginning garbage. -u is for unbuffered
grep --line-buffered -v "'" |
a catch for some bad http lines. Again unbuffered
awk '{ system("curl --create-dirs " $1 " -o " $2 " ; echo " $3 " > " $2".md5") }'
parse the line and pass the 1st segement to curl as download address, 2nd segment as file location, and put the 3rd agrument into same location as 2nd segment with a .md5 extension

Suggestions on how to test for making this into a proper script, and how to expand it to test to see if filename.md5 exists then compare the hashs if so?

Here is my attempt at doing it in the awk subsection. Is there a more elegant way?

./process 2>&1 |  sed -un 's/.*http/http/p' | grep --line-buffered -v "'" | awk '{ system("if [ -e " $2".md5 ] ; then if [ $(cat " $2".md5) != " $3 " ] ; then curl --create-dirs " $1 " -o " $2 " ; echo " $3 " > " $2".md5 ; fi ; else curl --create-dirs " $1 " -o " $2 " ; echo " $3 " > " $2".md5 ; fi") }'

Last edited by TimeFade; 02-12-2010 at 12:16 PM.
Old 02-13-2010, 06:09 AM   #2
Registered: Jan 2007
Location: Canton, MI
Distribution: CentOS, SuSE, Red Hat, Debian, etc.
Posts: 703

Rep: Reputation: 99
Let me ask, are you trying to pull down a set of files from a website
and keep an md5 sum file for each? If so, there is a much easier way
to do this with website mirroring tools. If not, can you briefly
explain what you trying to do?


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
bash: use file as input into array, parse out other variables from array using awk beeblequix Linux - General 2 11-20-2009 10:07 AM
script to parse variables to curl script and execute morphix Programming 17 11-30-2007 01:27 AM
Help w/ script to read file and parse variables cslink23 Linux - General 18 11-26-2006 02:22 AM
Bash - Getting a for loop to process variables with a space in them davee Linux - General 4 10-08-2005 11:21 AM
bash script variables twantrd Programming 7 11-17-2004 02:38 AM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 02:46 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration