Question about Bash Shell script to compare file sizes

michaellopez12 · 07-18-2014, 10:18 AM

To whom it may concern:

I have a question about changing a bash shell script. What would I have to change in my code so that every week I only get the file sizes for the last two weeks.

So for example next week is the 25 and I only want my shell script to compare file sizes for the 25 and 18th.

What would that code look like?

TenTenths · 07-18-2014, 10:20 AM

Quote:

Originally Posted by michaellopez12

I have a question about changing a bash shell script. What would I have to change in my code so that every week I only get the file sizes for the last two weeks.

Try posting your existing code and what you've tried. We aren't mind-readers.

michaellopez12 · 07-18-2014, 10:33 AM

Quote:

Originally Posted by TenTenths

Try posting your existing code and what you've tried. We aren't mind-readers.

Okay I will.

michaellopez12 · 07-18-2014, 10:36 AM

Quote:

Originally Posted by TenTenths

Try posting your existing code and what you've tried. We aren't mind-readers.

I just have to get it hold on.

michaellopez12 · 07-18-2014, 11:17 AM

Quote:

Originally Posted by TenTenths

Try posting your existing code and what you've tried. We aren't mind-readers.

Here is my code.

Code:

#!/bin/sh

SHELL=/bin/sh

xx=`date "+%m-%d-%y"`


ROOT_DIR=/home/admin/production/sql-csv


ls -l $ROOT_DIR/compwat/*.zip > $ROOT_DIR/compwat1.log

ls -l $ROOT_DIR/creditsafe/*.zip > $ROOT_DIR/creditsafe.log

ls -l $ROOT_DIR/db-new/*.zip > $ROOT_DIR/db-new.log

ls -l $ROOT_DIR/ftse/*.zip > $ROOT_DIR/ftse.log

ls -l $ROOT_DIR/isi/*.zip > $ROOT_DIR/isi.log

ls -l $ROOT_DIR/isi-IND/*.zip > $ROOT_DIR/isi-IND.log

ls -l $ROOT_DIR/isi-new/*.zip > $ROOT_DIR/isi-new.log

ls -l $ROOT_DIR/isi-new2/*.zip > $ROOT_DIR/isi-new2.log

ls -l $ROOT_DIR/redmoney/*.zip > $ROOT_DIR/redmoney.log

ls -l $ROOT_DIR/skadvisory/*.zip > $ROOT_DIR/skadvisory.log

ls -l $ROOT_DIR/worldbox/*.zip > $ROOT_DIR/worldbox.log

ls -l $ROOT_DIR/xpi/*.zip > $ROOT_DIR/xpi.log

ls -l $ROOT_DIR/redmoney/*.zip > $ROOT_DIR/redmoney.log

cat $ROOT_DIR/compwat1.log $ROOT_DIR/creditsafe.log $ROOT_DIR/db-new.log $ROOT_DIR/ftse.log $ROOT_DIR/isi.log $ROOT_DIR/isi-IND.log $ROOT_DIR/isi-new.log $ROOT_DIR/isi-new2.log $ROOT_DIR/redmoney.log $ROOT_DIR/skadvisory.log $ROOT_DIR/worldbox.log $ROOT_DIR/xpi.log $ROOT_DIR/redmoney.log > $ROOT_DIR/sql-csv-$xx.log

#rm $ROOT_DIR/compwat1.log $ROOT_DIR/creditsafe.log $ROOT_DIR/db-new.log $ROOT_DIR/ftse.log $ROOT_DIR/isi.log $ROOT_DIR/isi-IND.log $ROOT_DIR/isi-new.log $ROOT_DIR/isi-new2.log $ROOT_DIR/redmoney.log $ROOT_DIR/skadvisory.log $ROOT_DIR/worldbox.log $ROOT_DIR/xpi.log $ROOT_DIR/redmoney.log

#rm $ROOT_DIR/compwat1.log $ROOT_DIR/creditsafe.log $ROOT_DIR/db-new.log $ROOT_DIR/ftse.log $ROOT_DIR/isi.log $ROOT_DIR/isi-IND.log $ROOT_DIR/isi-new.log $ROOT_DIR/isi-new2.log $ROOT_DIR/redmoney.log $ROOT_DIR/skadvisory.log $ROOT_DIR/worldbox.log $ROOT_DIR/xpi.log $ROOT_DIR/redmoney.log


cp $ROOT_DIR/sql-csv-$xx.log   $ROOT_DIR/test1.log


#rm $ROOT_DIR/compwat1.log $ROOT_DIR/creditsafe.log $ROOT_DIR/db-new.log $ROOT_DIR/ftse.log $ROOT_DIR/isi.log $ROOT_DIR/isi-IND.log $ROOT_DIR/isi-new.log $ROOT_DIR/isi-new2.log $ROOT_DIR/redmoney.log $ROOT_DIR/skadvisory.log $ROOT_DIR/worldbox.log $ROOT_DIR/xpi.log

unSpawn · 07-18-2014, 11:38 AM

Quote:

Originally Posted by michaellopez12

What would I have to change in my code so that every week I only get the file sizes for the last two weeks.

File sizes of what exactly? The zip files? The log files? The size of the files listed in the log files? (Think about how the information you're presenting can be used.) You could populate a variable with relevant subdirectory names, then use a "for" loop, and (don't use 'ls' but) 'stat -c %s'. Or without loop use 'find /path -printf "%something\n"' (where "something" is the byte value directive from reading 'man find'). The output could populate another variable or whatever you're comfortable with to count. If you haven't coded then here's some texts to get you going:

BASH intros:
http://www.gnu.org/software/bash/man...ode/index.html
http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://www.tldp.org/LDP/Bash-Beginne...tml/index.html

BASH scripting guides:
http://mywiki.wooledge.org/BashGuide
http://mywiki.wooledge.org/BashFAQ
http://mywiki.wooledge.org/BashPitfalls
# these are must reads so do read them!

Common questions / problems:
http://mywiki.wooledge.org/ParsingLs
http://mywiki.wooledge.orgDontReadLinesWithFor
http://mywiki.wooledge.org/UsingFind
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Also see:
http://www.linuxquestions.org/questi...l-links-35334/

The Advanced BASH scripting guide:
http://www.tldp.org/LDP/abs/html/

Bourne shell (compatibility):
http://www.grymoire.com/Unix/Sh.html

michaellopez12 · 07-18-2014, 11:43 AM

Quote:

Originally Posted by unSpawn

File sizes of what exactly? The zip files? The log files? The size of the files listed in the log files? (Think about how the information you're presenting can be used.) You could populate a variable with relevant subdirectory names, then use a "for" loop, and (don't use 'ls' but) 'stat -c %s'. Or without loop use 'find /path -printf "%something\n"' (where "something" is the byte value directive from reading 'man find'). The output could populate another variable or whatever you're comfortable with to count. If you haven't coded then here's some texts to get you going:

BASH intros:
http://www.gnu.org/software/bash/man...ode/index.html
http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://www.tldp.org/LDP/Bash-Beginne...tml/index.html

BASH scripting guides:
http://mywiki.wooledge.org/BashGuide
http://mywiki.wooledge.org/BashFAQ
http://mywiki.wooledge.org/BashPitfalls
# these are must reads so do read them!

Common questions / problems:
http://mywiki.wooledge.org/ParsingLs
http://mywiki.wooledge.orgDontReadLinesWithFor
http://mywiki.wooledge.org/UsingFind
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Also see:
http://www.linuxquestions.org/questi...l-links-35334/

The Advanced BASH scripting guide:
http://www.tldp.org/LDP/abs/html/

Bourne shell (compatibility):
http://www.grymoire.com/Unix/Sh.html

The file sizes of the zip files is what I am searching for.

Firerat · 07-18-2014, 12:07 PM

I don't really understand what you are comparing

but as a gift

Code:

#!/bin/sh
xx=$(date "+%m-%d-%y")
ROOT_DIR=/home/admin/production/sql-csv

for Zip in ${ROOT_DIR}/*/*.zip;do
    stat -c "%s ${Zip}" "${Zip}"
done > "${ROOT_DIR}/sql-csv-${xx}.log"

That will give you the size in bytes and full path to the file,

Make sure you read the links provided by unSpawn,
They will help you understand the script I wrote for you

michaellopez12 · 07-18-2014, 01:25 PM

Quote:

Originally Posted by Firerat

I don't really understand what you are comparing

but as a gift

Code:

#!/bin/sh
xx=$(date "+%m-%d-%y")
ROOT_DIR=/home/admin/production/sql-csv

for Zip in ${ROOT_DIR}/*/*.zip;do
    stat -c "%s ${Zip}" "${Zip}"
done > "${ROOT_DIR}/sql-csv-${xx}.log"

That will give you the size in bytes and full path to the file,

Make sure you read the links provided by unSpawn,
They will help you understand the script I wrote for you

I will read those pages. Thank you. But will this code work every single week with the cron job I attach to this script? For example will it compare the file sizes for Aug 1 and the 25 when it runs on Aug 1? Are you sure it will work?

Firerat · 07-18-2014, 02:20 PM

Sorry , I missed that part

The 'trick' to that is to use the find command.

Code:

man find

That is the manual for find
See if you can work out how to 'find' files no older than 14 days

You then need to feed that list into the loop

There are a number of ways to do that.

One is to put the output of find into an array (you will need bash instead of sh )

Another (not recommended)

Code:

for Zip in $(find $ROOT_DIR <your 14 day 'rules'> -name "*.zip");do
   stat ....
done

Falls over if spaces in file path/name

The best solution would be a while read loop, using heredocs

Code:

while read Zip;do
   stat ...
done < <(find ....) > $logfile

apologies for 'gaps', I hate using software keyboards on phones

All will be covered in unSpawn's links

The wooledge ones are especially good, as they point out bad habits found on the web (including some in tldp ! )

If you get stuck, show us what you have, we can then help you out

Edit;
While 'testing' leave off the > $log....
So output is on screen, no need to mess about with cat to see results

Edit2:
One week or two?

michaellopez12 · 07-23-2014, 04:43 PM

Quote:

Originally Posted by Firerat

Sorry , I missed that part

The 'trick' to that is to use the find command.

Code:

man find

That is the manual for find
See if you can work out how to 'find' files no older than 14 days

You then need to feed that list into the loop

There are a number of ways to do that.

One is to put the output of find into an array (you will need bash instead of sh )

Another (not recommended)

Code:

for Zip in $(find $ROOT_DIR <your 14 day 'rules'> -name "*.zip");do
   stat ....
done

Falls over if spaces in file path/name

The best solution would be a while read loop, using heredocs

Code:

while read Zip;do
   stat ...
done < <(find ....) > $logfile

apologies for 'gaps', I hate using software keyboards on phones

All will be covered in unSpawn's links

The wooledge ones are especially good, as they point out bad habits found on the web (including some in tldp ! )

If you get stuck, show us what you have, we can then help you out

Edit;
While 'testing' leave off the > $log....
So output is on screen, no need to mess about with cat to see results

Edit2:
One week or two?

To answer your question in Edit2. Two weeks. I need to see the example for the this week and last week. the 25th and the 18th. Just using an example.

I am stuck now. So here is what I have.

Code:

#!/bin/sh

xx=$(date "+%m-%d-%y")
ROOT_DIR=/home/wvbadmin/production/sql-csv

for Zip in `find ${ROOT_DIR}/*/*.zip   -mtime +7 -print`;do
    stat -c "%s ${Zip}" "${Zip}"
done > "${ROOT_DIR}/sql-csv-${xx}.log"

This is a snippet of what is in my log file for file sizes of the zip files. This is after I execute my bash shell script.

Code:

209487274 /home/admin/production/sql-csv/skadvisory/weekly-update-skadvisory-07-04-14.zip
210164470 /home/admin/production/sql-csv/skadvisory/weekly-update-skadvisory-07-11-14.zip
238882046 /home/admin/production/sql-csv/worldbox/worldbox-weekly-update-14-07-04.zip
239817760 /home/admin/production/sql-csv/worldbox/worldbox-weekly-update-14-07-11.zip

My question now is how would I get it so that I get only this weeks 25 and 18th date to show up in my script. Would it have something to do with the mtime function?

Any help will be greatly appreciated.