LinuxQuestions.org - Create directories from a file list.

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Create directories from a file list. (https://www.linuxquestions.org/questions/linux-newbie-8/create-directories-from-a-file-list-889227/)

bv2925

06-30-2011 12:03 PM

Create directories from a file list.

Hi All,
I am trying to write a script to pick the directory name from a list of file. Here is a detailed picture.

Have a file name LIST which contains the follwing for example
/apps/oracle/product/test1
/apps/oracle/product/test2
/apps/oracle/product/test3

I need a script that reads these line from LIST and creates folders in /apps/oracle/product/test1
/backup/date/test1 after reading the first line
/backup/date/test2 after readin the second line
/backup/date/test3 and so on.

I am very new to scripting and seek all you r help.
Thanks.

schneidz

06-30-2011 12:09 PM

this smells like homework so what have you tried so far ?

i would look into programs like basename, cut, mkdir ...

edit: i googled 'bash read line' and the first link is:
http://www.linuxquestions.org/questi...84/#post714625

markush

06-30-2011 12:17 PM

Hello bv2925, welcome to LQ,

you can use the manpages, execute (for example) the command

Code:

man mkdir

in a terminal. More about scripting you'll find here: http://tldp.org/LDP/Bash-Beginners-Guide/html/ as a first step and here: http://tldp.org/LDP/abs/html/ very elaborate.

Markus

tronayne

06-30-2011 01:05 PM

OK, I'm feeling generous today -- here's a bare-bones shell program that will do what you want:

Code:

#!/bin/bash

#      we need the date

stamp=$(date +%F)

#      open the file "list" for reading as file descriptor 3

exec 3< list

#      while there is something to read...

while read -u 3 fname

do

        #      change "app" to "backup," "oracle" to the date, and remove "/product"

        #      note, this progam must be able to write to /backup!

        string=`echo ${fname} | sed 's/app/backup/;s/oracle/'${stamp}'/;s/\product//'`

        mkdir -p ${string}

done

#      close the input file

exec 3<&-

You need to read the manual pages for date, sed and mkdir if you want to understand what's going on (or to make any changes you might find useful).

Hope this helps some.

bv2925

06-30-2011 01:11 PM

Hi schneidz
i have no clue about sctipting but somehow managed to write a small script to cat the file and mkdir a new file.
I do not know where to start to just picking the file name in the end and create a new directory.
Would be helpful if you can provide an example.

tronayne

06-30-2011 02:46 PM

Here's another shot at it (with lots of displays of what it's doing).

Let's assume that the last element of your input list is a file name (rather than a directory name):

Code:

#!/bin/bash

#      we need the date

stamp=$(date +%F)

echo ${stamp}

#      open the file "list" for reading as file descriptor 3

exec 3< list

#      while there is something to read...

while read -u 3 fname

do

        #      let's assume that we need the name of the file; i.e., the last

        #      element of the string we read is a file, not a directory

        file=$(echo ${fname} | sed 's#.*/\(test[^/]*\).*#\1#')

        echo "The file name is " ${file}

        #      and lets' assume that we need the name of the directory tree; i.e.,

        #      "/apps/oracle/product"

        dirnam=${fname%/*}

        echo "The directory tree is " ${dirnam}

        #      now, let's change "app" to "backup," "oracle" to the date

        #      and remove "/product"

        #      note, this progam must be able to write to /backup!

        string=`echo ${dirnam} | sed 's/app/backup/;s/oracle/'${stamp}'/;s/\product//'`

        #      now, make the directory tree

        echo "mkdir -p ${string}"

        #      and then copy the content of "file name" (the last element) into that

        #      directory -- if "test1" is a file it'll get copied or, if test1 is a

        #      directory, that will get copied (that's what "-pr" does).

        #

        #      to actually do this, remove "echo" and the double quotes from the above

        #      "mkdir" line and the below "cp" line. This is just showning you the

        #      commands that would be executed.

        echo "cp -pr ${string} ${fname}"

done

#      close the input file

exec 3<&-

The above "echoes" a lot of stuff to show you what it's doing; you can comment those out (put a pound sign at the beginning of the line) or leave 'em in. As it is it will not execute the "mkdir" or "cp" lines -- it just displays them -- you would need to make those look like this:

Code:

        mkdir -p ${string}

        cp -pr ${string} ${fname}

Hope this helps some.

bv2925

07-01-2011 11:53 AM

Hi Tronayne;

I have made some modifications to your script and this is the error i get. Can you please guide me as to where i am going wrong.

+ read -u 3 fname
++ echo /apps/hyperion/archivefiles/Sa_Xchgrate
++ sed 's#.*/$test[^/]*$.*#\1#'
+ file=/apps/hyperion/archivefiles/Sa_Xchgrate
+ echo 'The file name is ' /apps/hyperion/archivefiles/Sa_Xchgrate
The file name is /apps/hyperion/archivefiles/Sa_Xchgrate
+ dirnam=/apps/hyperion/archivefiles
+ echo 'The directory tree is ' /apps/hyperion/archivefiles
The directory tree is /apps/hyperion/archivefiles
++ echo /apps/hyperion/archivefiles
++ sed 's/apps/apps/;s/hyperion/stage;s/archivefiles/misc/;s/2011-07-01/;'
sed: -e expression #1, char 33: unknown option to `s'
+ string=
+ mkdir -p
mkdir: too few arguments
Try `mkdir --help' for more information.
+ cp -pr /apps/hyperion/archivefiles/Sa_Xchgrate
cp: missing destination file
Try `cp --help' for more information.

The modified script is as follows:
#!/bin/bash
set -x
stamp=$(date +%F)
list=/apps/hyperion/archivefiles/list
echo ${stamp}
exec 3< ${list}
while read -u 3 fname
do
file=$(echo ${fname} | sed 's#.*/$test[^/]*$.*#\1#')
echo "The file name is " ${file}
dirnam=${fname%/*}
echo "The directory tree is " ${dirnam}
string=`echo ${dirnam} | sed 's/apps/apps/;s/hyperion/stage;s/archivefiles/misc/;s/'${stamp}'/;'`
mkdir -p ${string}
cp -pr ${string} ${fname}
done
# close the input file
exec 3<&-

schneidz

07-01-2011 09:39 PM

Quote:

Originally Posted by bv2925 (Post 4400531)

Hi schneidz
i have no clue about sctipting [sic] but somehow managed to write a small script to cat the file and mkdir a new file.
I do not know where to start to just picking the file name in the end and create a new directory.
Would be helpful if you can provide an example.

can you post your attempt in [ code ] tags; here's a sample:

Code:

mkdir -p /backup/`date`/`basename /apps/oracle/product/test1`

i will allow you to read in your file and put it in a loop.

edit: ask your teacher about `backticks`.

tronayne

07-02-2011 09:42 AM

OK, let's back up here and take another look at what it is we're trying to accomplish.

You have a list of files or a list of directories (I'm not clear on which) that you want to back up to a different directory.

The first thing you need to do is isolate the last element (that is either a file name or a directory name, but that doesn't matter). There are a number of ways to do that, one of which as suggested by others, is to use the basename utility; e.g.,

Code:

basename /apps/oracle/product/test1

test1

Code:

basename /apps/hyperion/archivefiles/Sa_Xchgrate

Sa_Xchgrate

That can also be done with the cut utility but only if the number of fields is constant; i.e., there are always (and forever) going to be four elements:

Code:

echo /apps/hyperion/archivefiles/Sa_Xchgrate | cut -d'/' -f5

Sa_Xchgrate

(there are five slashes before the last element). cut is probably not the best idea because your development tree may change and, when it does, the whole thing falls apart.

So, instead of using the fancy-schmancy substring extraction and sed code

Code:

sed 's#.*/$test[^/]*$.*#\1#')

(which requires that the pattern "test" is the beginning of the file name), it probably faster, better, cleaner, less confusing to simply use basename to isolate the last element.

Now, you want to have a directory tree, using your original post, something like this

Code:

/backup/2011-07-02/Sa_Xchgrate

That about right? Maybe not exact but similar enough that we can get on with it.

So, we need a data stamp and we get that with

Code:

stamp=$(date +%F)

and then we need to make the first parts of the back up directory with

Code:

dirnam=/backup/${stamp}

mkdir -p ${dirnam}

The nice thing about the -p option to mkdir is that it won't fail if the directory tree already exists so that's why we do that.

Then, because we've isolate the last element of the list -- which is either a file name or a directory name, doesn't matter -- we can use

Code:

cp -pr /apps/hyperion/archivefiles/Sa_Xchgrate ${dirnam}

and that will copy either a file named Sa_Xchgrate or a directory tree named Sa_Xchgrate to /backup/2011-07-01/Sa_Xchgrate.

That's pretty simple -- if, that is, it's what you really want, eh?

So

Code:

#!/bin/bash

#

#      It's a good idea to put some comments at the beginning of a shell program (or

#      any program for that matter) describing what you're doing and why you're doing it

#      ('cause you won't remember next week what you did and why).

#

#      we need a date stamp to name a back up directory

#

stamp=$(date +%F)

#

#      we need to read zero or more file- or direcotry names from a file

#      we're going to open that file as unit 3 (units 0, 1, and 2 are stadard

#      input, standard output and standard error)

#

exec 3< your_list_file_name

while read -u 3 list

do

        #

        #      we need the last element of what we just read, either a file name or a

        #      directory name, doesn't matter; we'll use the basename utillity to get it

        #

        filenam=basename ${list}

        #

        #      we've got what we need, so let's create a dirctory

        #      this is just "backup/${stamp}" change the hard-coded

        #      path name to whay you want it to be

        #

        dirnam=/backup/${stamp}

        mkdir -p ${dirnam}

        #

        #      and then copy the stuff

        #

        #      how about a message telling us what we're doing?

        echo "On ${stamp} creating back-up of ${list} to ${dirnam}"

        cp -pr ${list} ${dirnam}

done

#

#      and close the input file

#

exec 3<&-

#

#      and exit cleanly

#

exit 0

You need to edit "your_list_file_name" to the actual list file name and that ought to do it.

Hope this helps some.

bv2925

07-05-2011 09:37 AM

Hi Tronayne,
Thanks for the script. I have moved a little ahead with this where it creates the directory structure till the date.

I would want to describe the scenario so that i could provide an in depth picture of the script.

We have a file (List) which has a bunch of folder location names mentioned below.
/apps/hyperion/archivefiles/
/apps/hyperion/archivefiles/Demo
/apps/hyperion/archivefiles/Sample
/apps/hyperion/archivefiles/Sa_Xchgrate
/apps/hyperion/archivefiles/list

The above files (each one) have a different list of filename's under each one which have to be backed up in to the new directories that are being create under the date directory in the backup location.
So an example of the contents under Demo is
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct

So the script has to read filename (List), Create a new directory (Demo), read filelist from demo(Demo.app,Basic.otl etc) and backup them up to the new backup location ex: /backup/Date/Demo/{Demo.app,Basic.otl etc}

Hope I am clear.
Thanks.

tronayne

07-06-2011 11:01 AM

The only important thing you need to know is the last element in you list; that will be the name of the directory you want to back up and it is extracted with the basename utility as in the shell program above. That is, you do not need to know the names of files in the directory, you only need the directory name because cp -pr does two things, -p preserves mode, ownership, timestamps and -r copy directories recursively. And, if you want to watch it do it, replace -pr with -prv.

The other only thing you have to do is edit the shell program and make

Code:

        #

        #      we've got what we need, so let's create a dirctory

        #      this is just "backup/${stamp}" change the hard-coded

        #      path name to whay you want it to be

        #

        dirnam=/backup/${stamp}

        mkdir -p ${dirnam}

        #

to look like you want it to (although, right now, it's what you've described).

If you have one line in your file "List," say, /apps/hyperion/archivefiles/Demo and execute the shell program, you'll get a backup of the entire Demo tree (you do not need to enter anything else in List.

If the content of List were to be

Code:

/apps/hyperion/archivefiles/Demo

/apps/hyperion/archivefiles/Sample

/apps/hyperion/archivefiles/Sa_Xchgrate

You will get the entire content of each of those directories in

Code:

/backup/${stamp}/Demo

/backup/${stamp}/Sample

/backup/${stamp}/Sa_Xchgrate

where ${stamp} is the date.

That ought to do it -- try it.

Hope this helps some.

schneidz

07-06-2011 12:21 PM

Quote:

Originally Posted by bv2925 (Post 4405726)

^ still unsure what you are trying to do. can you post a before-and-after picture ?
here is my attempt to understand what you need:

Code:

[schneidz@hyper bv]$ cat demo.lst

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm

/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct

[schneidz@hyper bv]$ echo mkdir -p /backup/`date +%Y-%j`; cat demo.lst | while read line; do echo cp $line /backup/`date +%Y-%j`/`basename $line`; done

mkdir -p /backup/2011-187

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app /backup/2011-187/Demo.app

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl /backup/2011-187/Basic.otl

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db /backup/2011-187/Basic.db

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind /backup/2011-187/Basic.ind

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm /backup/2011-187/Basic.esm

cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct /backup/2011-187/Basic.tct

good luck,
s. henry

bv2925

07-27-2011 09:18 AM

Hi schneidz
Thanks for the input but i am unable to achive what i am supposed to do.
Here is the scenario in detail.
I have a folder under which three files exist as listed below. The number of files will change depending on the number of applications.
$ cd /apps/hyperion/archivefiles/
$ ls
Demo Sample Sa_Xchgrate
Each of these have a list of files to be backed up with there full path as shown below.
$ cat Demo
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct
I have to write a program to read Demo,
Create a new directory Demo under mkdir -p ${Backuplocation}/${DATE}
and backup the files listed in demo Ex:Demo.app, Basic.otl etc into the new location ${Backuplocation}/${DATE}/Demo/
The number of files may change during the course of time.
Next read Sample and repeat the procedure untill all the files in /apps/hyperion/archivefiles/ have been read and backed up.
in the end tar ${Backuplocation}/${DATE}.
Thanks in advance.

bv2925

07-27-2011 09:29 AM

Hi Schneidz,
I have set the Variables as mentioned below.
set -x
DATE=`date +"%d-%b-%Y"`
Log='/apps/stage/misc/bkuptest/backup.log' #Location of the log file
Backuplocation='/apps/stage/misc/bkuptest/backup'
BACKUP_DIRECTORY=${Backuplocation}/${DATE}
Archivelist='/apps/hyperion/archivefiles/'

Thanks.

schneidz

07-29-2011 08:05 AM

Quote:

Originally Posted by bv2925 (Post 4426707)

looks good so far. fyi, i think one of these two date formats will sort better:
date +"%Y-%m-%d"
date +"%Y-%j"

most of your criteria is fairly elemental, at this point you just need to combine those conditions in an automated fashion using the variables you set above.
this should get you started:

Code:

[schneidz@hyper bv]$ mkdir -p $BACKUP_DIRECTORY/demo; cat /apps/hyperion/archivefiles/demo | while read line; do cp "$line" $BACKUP_DIRECTORY/demo; done

you just need to modify the script to include all the archive files you care about (e.g.- sample, sa_xchgrate)

All times are GMT -5. The time now is 03:57 PM.