[SOLVED] Bash script needed to move files to another folder.

No_one_knows_me · 12-30-2009, 10:05 AM

Greetings One and All,

This is my first post in this forum. Let me tell you what I am needing. I need a bash script that will look at the 6th and 7th characters of a filename and if it is equal to a set number, then it will be moved to a specified folder. Here's an example of what I'm talking about:

If I have a listing of files

2009-12-18.fictitious.gz
2009-11-01.notreal.gz
2009-02-22.fake.gz

and I want to move 2009-12-18.fictitious.gz to the December folder, since the 6th and 7th digits of the filename = 12, then I would like it to create the December folder if it doesn't exist and move the file to that directory. Likewise, I want to move 2009-11-01.notreal.gz to the November folder and have it create the folder if it doesn't exist, etc. Anyone up to the challenge?

Also, can all the .gz files for the month go into one big tarball?

rweaver · 12-30-2009, 10:14 AM

Well you can get the sixth and seventh character using cut, awk, sed, etc...

Code:

#!/bin/bash
for i in `ls /directory`; do 
  SS=`echo $i | cut -c6,7`
  if [ $SS -eq 1 ]; then
    mkdir -p /sorted/january
    mv $i /sorted/january
  elsfi [ $SS -eq 2 ]; then
    mkdir -p /sorted/febuary
    mv $i /sorted/febuary
  fi
done

There's the basics of it. You'd need to add the remainder of the logic of course. Also keep in mind I didn't test this, but if it's off it should be simple to fix it. If you need to gzip each month just add the logic to the end of the script after the if, basically for each folder tar cfvz month.tgz /sorted/month

tredegar · 12-30-2009, 10:25 AM

As usual with linux, there are many possible solutions.

I think it might be easier to create the January ... December folders yourself instead of writing a script to do it.

Then you could just do

Code:

mv 2009-12* /path/to/December
mv 2009-11* /path/to/November

Etcetera.

You could then copy your command history to a file with history > foo
and then edit foo to turn it into a script you can run next year to save yourself creating all those monthly directories again.

Quote:

Also, can all the .gz files for the month go into one big tarball?

Yes, they can.

Welcome to LQ!

No_one_knows_me · 12-30-2009, 10:27 AM

rweaver,

Thank you for such a quick turn! It seems pretty straight forward and simple enough. I will have to make a modification since all the files in that directory don't begin with the date, but a simple find statement should be enough. I will try to finish writing it, and test it for my December files later on today to see if it works. Thanks once again!

No_one_knows_me · 12-30-2009, 10:32 AM

tredegar,

I had contemplated just creating the monthly folders myself, too, but I have like 10 servers that this script will need to run on, so I was hoping to put the script in a cron job and let it work its magic.

tredegar · 12-30-2009, 10:48 AM

Quote:

I was hoping to put the script in a cron job and let it work its magic.

You still can, but realise that at some stage you are going to have to type mkdir January February March ... and you might as well do this only once, then extract that (and anything else useful) from your history, and use it to help you write your script without too much re-typing.

Test your script.

When you are happy with it, copy the script to your servers.

No_one_knows_me · 12-30-2009, 11:29 AM

rweaver,

I have completed the proposed script you gave me, and I am getting a

Code:

./monthly_logs.sh: line 52: syntax error near unexpected token `done'
./monthly_logs.sh: line 52: `done'

I went to line 52, and all that is there is:

Code:

done

I am thinking that I will need more 'fi' statements since I had so many else if statements. Here is my script in its entirety, let me know of any suggestions:

Code:

#!/bin/bash
#Script to move daily log files to one big tar.gz at the end of the month
#and place them in their respective folders.

DIR=/var/log/
#TMP=$DIR/month/tmp

#mkdir $TMP

cd $DIR

for i in `ls $DIR`; do
         SS=`echo $i | cut -c6,7`
           if [ $SS -eq 01 ]; then
           mkdir -p /month/January
           mv $i /month/January
         else if [ $SS -eq 02]; then
           mkdir -p /month/February
           mv $i /month/February
          else if [ $SS -eq 03]; then
           mkdir -p /month/March
           mv $i /month/March
         else if [ $SS -eq 04]; then
           mkdir -p /month/April
           mv $i /month/April
         else if [ $SS -eq 05]; then
           mkdir -p /month/May
           mv $i /month/May
         else if [ $SS -eq 06]; then  
           mkdir -p /month/June
           mv $i /month/June
         else if [ $SS -eq 07]; then
           mkdir -p /month/July
           mv $i /month/July
          else if [$SS -eq 08]; then
           mkdir -p /month/August
           mv $i /month/August
         else if [$SS -eq 09]; then
           mkdir -p /month/September
           mv $i /month/September
         else if [$SS -eq 10]; then
           mkdir -p /month/October
           mv $i /month/October
         else if [$SS -eq 11]; then
           mkdir -p /month/November
           mv $i /month/November
         else if [$SS -eq 12]; then
           mkdir -p /month/December
           mv $i /month/December
     fi
done

tredegar · 12-30-2009, 11:49 AM

Quote:

I had so many else if statements....

How about:

Code:

case $SS in
    01)
           mkdir -p /month/January
           mv $i /month/January
;;
    02)           
           mkdir -p /month/February
           mv $i /month/February
;;
                  .
                  .
                  .
                  .
    12)           
           mkdir -p /month/December
           mv $i /month/December
;;
     *)
           Echo Bad month number: $SS; exit 1
;; 
esac

Not tested, but easier to read / understand /debug too

Edit: I think if you are going to use "-p" with mkdir, you should not have the leading slash before /month as otherwise the dir month will be made at /.
you'll have to experiment!

No_one_knows_me · 12-30-2009, 11:56 AM

tredegar,

I like your approach to this. It is clean and should eliminate the else if statements that seem to be causing me a little grief in scripting correctly. I will go back to the drawing board and see what happens. Thanks for your help!

David the H. · 12-30-2009, 11:57 AM

There's no need to use external tools like cut or awk to extract the substring. All you need is bash's built-in parameter substitution.

Also $() is recommended over ``.

Code:

for i in $(ls $DIR); do
         if [ ${i:5:2} -eq 01 ]; then
           mkdir -p /month/January
           mv $i /month/January

           ...etc...

The error messages you're getting are probably because you need to have spaces on both sides of the test brackets

Code:

if [ value to test ]; then
    ^             ^

And actually, I think a case structure would be more appropriate here than if.

Code:

for i in $(ls $DIR); do

     case "${i:5:2}" in

          01) mkdir -p /month/January
           mv $i /month/January
          ;;

          02) mkdir -p /month/February
           mv $i /month/February
          ;;

          ...etc...

          *) echo "$i is not a valid filename."   #error message and exit if
             exit 1                               #filename is invalid.
          ;;
     esac
done

Edit: Beaten by tredegar.

tredegar · 12-30-2009, 12:02 PM

Quote:

Beaten by tredegar.

- Not at all, you always have something to contribute, in this case parameter substitution.
bash-scripting is not my strong point, so I like to learn too

No_one_knows_me · 12-30-2009, 12:51 PM

David the H.,

Quote:

Also $() is recommended over ``.

Is this true for the find command string or not?

So where I would normally do a:

Code:

for i in $DIR ; do `find . 2* -name -type f -print`

as a recommendation it should be:

Code:

$(find . 2* -name -type f -print)

I'm just trying to get up to speed, as it appears my ways may be somewhat antiquated.

David the H. · 12-30-2009, 01:04 PM

That's true. I did have more to add. I was mostly just complaining about all the time I took writing up the case statement, only to find someone beat me to it.

But this stuff happens all the time, so no worries.

I see that I also failed to point out that there's no need for the embedded ls command (use of ls in loops is considered by many to be very ungraceful). Bash can read the directory listing directly; just use a * wildcard in the directory path. You can use variable substitution to provide the path itself.

Code:

for i in $DIR/*; do     # or specify a matching pattern
                        # with something like "$DIR/*.gz"
...commands...

done

David the H. · 12-30-2009, 01:16 PM

Embedded commands work similarly to variable substitution. Any commands placed inside will be executed first, and their output substituted before the line containing it is executed.

http://www.tldp.org/LDP/abs/html/commandsub.html

Backticks (``) are a very old way to use embedded commands in bourne-type shells.

$() is a newer form of the same thing. I say newer, but it's been around long enough now that, unless you work on very old legacy systems, you shouldn't have any problems using it.

This page enumerates the reasons the newer form is preferred.
http://mywiki.wooledge.org/BashFAQ/082

No_one_knows_me · 12-30-2009, 01:24 PM

David the H.,

Thank you for clarifying that for me, and I will bear that in mind in future scripting efforts. As it stands, everything is working as it should, but I neglected to create a find statement to grab just the '2*.gz' files in the directory and then utilize the parameter substitution you pointed out earlier. So I've just created that statement, and here's what I have:

Code:

for i in $DIR ; $(find . '2*.gz' -name -type f -print); do case "${i:5:2}" in
01) mkdir -p month/January
    mv $i month/January
;;

....etc.....

I'm getting

Code:

syntax error near unexpected token `$(find . '2*.gz' -name -type f -print)'

Now, I'm not testing this in a script, I'm just running it from the command-line. That shouldn't matter, though, right?