[SOLVED] Scripting & Pipe Help

powerplyer · 10-26-2013, 06:42 PM

Hello long time lurker and new to Linux.

I am using CentOs 6.4 and I need some help setting up a script.

Here is what I am looking to automate and below is a sample of the script I setup.

1) looking to pipe the output of the data to a variable file each time the script is run. For example currently I have to run ./erase.sh >> SNxxxxxxx.txt. However optimally I would like to run just ./erase.sh and have the script file look at the output from the script and grab the SNxxxxxxx and time stamp the output file automatically.

2)Can a run a script inside a script? I think I can but just making sure.
3)Is there a way to output to screen and pipe the information at the same not. Not a big deal just wondering.

Here is the sample script<erase.sh> ddcli is a command for the card.
#!/bin/bash
# rsync using variables

SOURCEDIR=/home/user/Desktop/

./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -format -eraseflash -s
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=FORMAT SECUREERASE complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

essentially this is a command to pull data from storage card I have.

Thank you in advance for the help.

Firerat · 10-26-2013, 07:58 PM

Quote:

Originally Posted by powerplyer

Hello long time lurker and new to Linux.

I am using CentOs 6.4 and I need some help setting up a script.

Here is what I am looking to automate and below is a sample of the script I setup.

1) looking to pipe the output of the data to a variable file each time the script is run. For example currently I have to run ./erase.sh >> SNxxxxxxx.txt. However optimally I would like to run just ./erase.sh and have the script file look at the output from the script and grab the SNxxxxxxx and time stamp the output file automatically.

Quote:

Originally Posted by powerplyer

2)Can a run a script inside a script? I think I can but just making sure.

yes

Quote:

Originally Posted by powerplyer

3)Is there a way to output to screen and pipe the information at the same not. Not a big deal just wondering.

yes, there is a utility called tee, which does just this

Code:

echo "send to tee" | tee Log.log
cat Log.log
echo "send another to tee" | tee Log.log
cat Log.log
echo "send to tee" | tee -a Log.log
cat Log.log
echo "send another to tee" | tee -a Log.log
cat Log.log

notice what the -a did,
not much to read, but read anyway

Code:

man tee

Quote:

Originally Posted by powerplyer

Here is the sample script<erase.sh> ddcli is a command for the card.

Code:

#!/bin/bash
# rsync using variables

SOURCEDIR=/home/user/Desktop/


./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -format -eraseflash -s
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=FORMAT SECUREERASE complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

essentially this is a command to pull data from storage card I have.

Thank you in advance for the help.

please use code tags,, example
[code]
Your code here
[/code]

not much for me to say about that script
only that you might want to make it 'position independent'
that is, hardcode it to the location of ddcli
one way is to cd into the dir that contains ddcli
another is to use the full path, that assumes the ddcli script/program is hardcoded with relevant paths

also it will 'carry on regardless' of any failure, since I can't see ddcli, I don't know if you can make it 'smarter'

assuming ddcli uses exit codes ( with 0 meaning success and none-zero being failure )

Code:

em="=-=-=-=-=-=-=-=-=-=-=-=-=-="
./ddcli -c 4 -list \
    && echo "${em}Initial LIST complete${em}" \
    || ( echo "${em} some error msg ${em}";exit 1 )

that is a list construct
basically it is this

Code:

em="=-=-=-=-=-=-=-=-=-=-=-=-=-=" # makes life easier later
# feel free to make it longer
# EqualMinus="=-=-=-=-=-=-=-=-=-=-=-=-=-="
# if you use something like vim, then there is a nice feature
# example , note <ctrl+p> is the ctrl key + p :)
# echo "${E<ctrl+p>
# it will either 'complete' the keyword, or present a list ;)
# 
./ddcli -c 4 -list
if [[ "$?" == "0" ]]; then
    echo "${em}Initial LIST complete${em}"
else
    echo "${em} some error msg ${em}"
    exit 1
fi

some reading
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/
http://www.tldp.org/LDP/abs/html/
http://mywiki.wooledge.org/BashGuide
http://www.gnu.org/software/bash/manual/bashref.html

The tldp stuff is great, however there are some nasty bad habits in it,
The mywiki.wooledge does a very good job of 'fixing' those habits

powerplyer · 10-26-2013, 08:56 PM

Hi Firerat thank you for the quick reply. Sorry about not putting the code in a box...

I am still unclear about item 1. I think I get the jist of item 3. Again, sorry some of what you said is still above my head.

Code:

#!/bin/bash
# rsync using variables

SOURCEDIR=/home/user/Desktop/


./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -format -eraseflash -s
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=FORMAT SECUREERASE complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -list
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

Here is a sample output from the file. Sorry if I was unclear but this is what I am trying accomplish

-- run all 5 commands (via the script) automatically output to a file called temp_output.txt
-- read the output and automatically get the serial number from the temp_output.txt
-- automatically rename the file temp_output to SN_SP24447758_<time stamp>.txt

I think my biggest question is how to automatically rename temp_output.txt to SN_SPxxxxxxxx_<timestamp>.txt. The problem is I do not know what to put in a script to read the file, get the board tracer number, rename that file with board tracer number and time stamp to a text file.

Here is a sample of the output from ddcli

Code:

WarpDrive Selected is NWD-RLP4-1860
------------------------------------------------------------------------
WarpDrive Information
------------------------------------------------------------------------
  WarpDrive ID                           : 3
  PCI Address                            : 00:81:00:00
  PCI Slot Number                        : 0x05
  PCIe Link Width                        : X8 Lane(s)
  PCIe Link Rate                         : 0x01 (5.0 Gbps)
  PCI DeviceId                           : 0x7E
  PCI SubSystem DeviceId                 : 0x507
  PCI SubSystem VendorId                 : 0x1000
  SAS Address                            : 500605B 00568B0A0
  Package Version                        : 10.110.04.00
  Firmware Version                       : 110.00.01.00
  Legacy BIOS Version                    : 110.00.00.00
  UEFI BSD Version                       : 07.18.05.03
  Chip Name                              : Nytro WarpDrive
  Board Name                             : NWD-RLP4-1860
  Board Assembly Number                  : L3-25586-00A
  Board Tracer Number                    : SP24447758
  RAID Support                           : YES

If you have any more insight that would be great. Please consider me a 2 year old when it comes to Linux.

Thanks again for your help

Firerat · 10-26-2013, 09:16 PM

not clear when you get that output

Code:

#!/bin/bash
# rsync using variables

MyTemp=$(mktemp)
LogIt="|tee -a $MyTemp"
SOURCEDIR=/home/user/Desktop/


./ddcli -c 4 -list $Logit
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health $Logit
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -format -eraseflash -s $Logit
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=FORMAT SECUREERASE complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -list $Logit
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddcli -c 4 -health $Logit
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

FileName="SN_$( awk '/Board Tracer Number/{print $NF;exit}' "$MyTemp" )_$( date +%Y-%m-%d-%H%M%S ).txt"
mv "$MyTemp" "$FileName"

Warning:
I have made no effort to use correct paths
I don't know if "Board Tracer Number" will appear more than once
( actually, just added the ;exit to the awk, so after first result it exits )

powerplyer · 10-26-2013, 09:45 PM

you are very gracious for doing the script.

I get a SN_2013-10-260193116.txt (great) file but it is blank, also the filename did not pick up the board tracer number. Could it be because we are not piping the output to any file. I see the output on the screen. I grabbed all the output and attached the text file.

I made a minor modification to the script to comment out the eraseflash option. Just to test it out... The eraseflash option takes too long. I think we are almost there...

Firerat · 10-26-2013, 10:00 PM

well, if those ./ddcli commands do not print to stdout, then tee will not be able to append them

I'm blind as I can't see what ddcli does

if you see the relevant text 'fly by' when running the script it could that it is sending it to stderr

change LogIt="|tee -a $MyTemp" to

Code:

LogIt=" 2>&1 | tee -a $MyTemp"

that redirects stderr to stdout
tee should then be appending it to the tmpfile,

powerplyer · 10-26-2013, 10:19 PM

Quote:

well, if those ./ddcli commands do not print to stdout, then tee will not be able to append them

I'm blind as I can't see what ddcli does

basically ddcli is a utility to pull information from the card.
If I were to run ddcli -c 1 -list in terminal --> the utility selects the card 1 and displays the list (informational) output of the card (ie FW,and other device information)to the terminal window. If I want to save it to a file I would do a ddcli -c 1 -list >> list_output.txt, as an example. Like doing ls command it would output to termianl windows and if I did a ls >> dir.txt it would output the directory content to a file called dir.txt. BTW I under stand that doing a ">>" will continue to append to that one file.

Quote:

if you see the relevant text 'fly by' when running the script it could that it is sending it to stderr

I do see the revelent information fly by

Quote:

change LogIt="|tee -a $MyTemp" to

Code:

LogIt=" 2>&1 | tee -a $MyTemp"

that redirects stderr to stdout
tee should then be appending it to the tmpfile

Is there something information I can get to help further, please let me know?
Thank you

Firerat · 10-26-2013, 10:26 PM

ok, well I don't understand why the file was blank

lets keep it simple and eliminate tee ( it might not be installed? )

LogIt=">> $MyTemp"

powerplyer · 10-26-2013, 10:42 PM

I think we are almost there.

I modified the code slightly, I pointed to the $MyTemp. The output file is correct. The only problem is the output does not "fly by" on the screen. I can only see my echo commands.

Here is the modified code:

Code:

#!/bin/bash
# rsync using variables

MyTemp=$(mktemp)
LogIt=">> $MyTemp"
SOURCEDIR=/home/user/Desktop/LSI/6208


./ddoemcli -c 1 -list >> $MyTemp
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddoemcli -c 1 -health >> $MyTemp
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

#./ddoemcli -c 1 -format -eraseflash -s $Logit
#echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=FORMAT SECUREERASE complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddoemcli -c 1 -list >> $MyTemp
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

./ddoemcli -c 1 -health >> $MyTemp
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-="

FileName="SN_$( awk '/Board Tracer Number/{print$NF;exit}' "$MyTemp" )_$( date +%Y-%m-%d-%H%M%S ).txt"
mv "$MyTemp" "$FileName"

Here is the output on the terminal file:

Code:

./1C_Securerase.sh
=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-=
=-=-=-=-=-=-=-=-=-=-=-=-=-=Initial HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-=
=-=-=-=-=-=-=-=-=-=-=-=-=-=Final LIST complete=-=-=-=-=-=-=-=-=-=-=-=-=-=
=-=-=-=-=-=-=-=-=-=-=-=-=-=Final HEALTH complete=-=-=-=-=-=-=-=-=-=-=-=-=-=

and here is the output of the file name and it has data.

SN_SP32611130_2013-10-26-203612.txt

Firerat · 10-26-2013, 11:04 PM

yes, the reason you do not see the ddcli script output is because it is being sent to the temp file

at the moment I assume tee is not installed,

another way of doing what tee does is to start tail in the background

Code:

MyTemp=$(mktemp)
tail -f $MyTemp &
tailPID=$!
.. rest of code..
kill $tailPID
mv "$MyTemp" "$FileName"

& puts tail -f into the background, it is still running

the $! is the PID of the last background process
we get that so we can 'cleanup' before the script exits

powerplyer · 10-26-2013, 11:22 PM

Quote:

Originally Posted by Firerat

yes, the reason you do not see the ddcli script output is because it is being sent to the temp file

at the moment I assume tee is not installed,

another way of doing what tee does is to start tail in the background

Code:

MyTemp=$(mktemp)
tail -f $MyTemp &
tailPID=$!
.. rest of code..
kill $tailPID
mv "$MyTemp" "$FileName"

& puts tail -f into the background, it is still running

the $! is the PID of the last background process
we get that so we can 'cleanup' before the script exits

PERFECT!!! Thank you very much for you help, I tied to rep you but it looks like you have helped many. Many thanks!!! We can mark this as [SOLVED].