LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-30-2011, 01:03 PM   #1
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Rep: Reputation: Disabled
Question Create directories from a file list.


Hi All,
I am trying to write a script to pick the directory name from a list of file. Here is a detailed picture.

Have a file name LIST which contains the follwing for example
/apps/oracle/product/test1
/apps/oracle/product/test2
/apps/oracle/product/test3

I need a script that reads these line from LIST and creates folders in /apps/oracle/product/test1
/backup/date/test1 after reading the first line
/backup/date/test2 after readin the second line
/backup/date/test3 and so on.

I am very new to scripting and seek all you r help.
Thanks.
 
Old 06-30-2011, 01:09 PM   #2
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,027

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
this smells like homework so what have you tried so far ?

i would look into programs like basename, cut, mkdir ...

edit: i googled 'bash read line' and the first link is:
http://www.linuxquestions.org/questi...84/#post714625

Last edited by schneidz; 06-30-2011 at 01:13 PM.
 
Old 06-30-2011, 01:17 PM   #3
markush
Senior Member
 
Registered: Apr 2007
Location: Germany
Distribution: Slackware
Posts: 3,979

Rep: Reputation: 850Reputation: 850Reputation: 850Reputation: 850Reputation: 850Reputation: 850Reputation: 850
Hello bv2925, welcome to LQ,

you can use the manpages, execute (for example) the command
Code:
man mkdir
in a terminal. More about scripting you'll find here: http://tldp.org/LDP/Bash-Beginners-Guide/html/ as a first step and here: http://tldp.org/LDP/abs/html/ very elaborate.

Markus
 
Old 06-30-2011, 02:05 PM   #4
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,477

Rep: Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020
OK, I'm feeling generous today -- here's a bare-bones shell program that will do what you want:
Code:
#!/bin/bash
#       we need the date
stamp=$(date +%F)
#       open the file "list" for reading as file descriptor 3
exec 3< list
#       while there is something to read...
while read -u 3 fname
do
        #       change "app" to "backup," "oracle" to the date, and remove "/product"
        #       note, this progam must be able to write to /backup!
        string=`echo ${fname} | sed 's/app/backup/;s/oracle/'${stamp}'/;s/\product//'`
        mkdir -p ${string}
done
#       close the input file
exec 3<&-
You need to read the manual pages for date, sed and mkdir if you want to understand what's going on (or to make any changes you might find useful).

Hope this helps some.
 
1 members found this post helpful.
Old 06-30-2011, 02:11 PM   #5
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi schneidz
i have no clue about sctipting but somehow managed to write a small script to cat the file and mkdir a new file.
I do not know where to start to just picking the file name in the end and create a new directory.
Would be helpful if you can provide an example.
 
Old 06-30-2011, 03:46 PM   #6
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,477

Rep: Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020
Here's another shot at it (with lots of displays of what it's doing).

Let's assume that the last element of your input list is a file name (rather than a directory name):
Code:
#!/bin/bash
#       we need the date
stamp=$(date +%F)
echo ${stamp}
#       open the file "list" for reading as file descriptor 3
exec 3< list
#       while there is something to read...
while read -u 3 fname
do
        #       let's assume that we need the name of the file; i.e., the last
        #       element of the string we read is a file, not a directory
        file=$(echo ${fname} | sed 's#.*/\(test[^/]*\).*#\1#')
        echo "The file name is " ${file}
        #       and lets' assume that we need the name of the directory tree; i.e.,
        #       "/apps/oracle/product"
        dirnam=${fname%/*}
        echo "The directory tree is " ${dirnam}
        #       now, let's change "app" to "backup," "oracle" to the date
        #       and remove "/product"
        #       note, this progam must be able to write to /backup!
        string=`echo ${dirnam} | sed 's/app/backup/;s/oracle/'${stamp}'/;s/\product//'`
        #       now, make the directory tree
        echo "mkdir -p ${string}"
        #       and then copy the content of "file name" (the last element) into that
        #       directory -- if "test1" is a file it'll get copied or, if test1 is a
        #       directory, that will get copied (that's what "-pr" does).
        #
        #       to actually do this, remove "echo" and the double quotes from the above
        #       "mkdir" line and the below "cp" line. This is just showning you the
        #       commands that would be executed.
        echo "cp -pr ${string} ${fname}"
done
#       close the input file
exec 3<&-
The above "echoes" a lot of stuff to show you what it's doing; you can comment those out (put a pound sign at the beginning of the line) or leave 'em in. As it is it will not execute the "mkdir" or "cp" lines -- it just displays them -- you would need to make those look like this:
Code:
        mkdir -p ${string}
        cp -pr ${string} ${fname}
Hope this helps some.
 
1 members found this post helpful.
Old 07-01-2011, 12:53 PM   #7
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi Tronayne;

I have made some modifications to your script and this is the error i get. Can you please guide me as to where i am going wrong.

+ read -u 3 fname
++ echo /apps/hyperion/archivefiles/Sa_Xchgrate
++ sed 's#.*/\(test[^/]*\).*#\1#'
+ file=/apps/hyperion/archivefiles/Sa_Xchgrate
+ echo 'The file name is ' /apps/hyperion/archivefiles/Sa_Xchgrate
The file name is /apps/hyperion/archivefiles/Sa_Xchgrate
+ dirnam=/apps/hyperion/archivefiles
+ echo 'The directory tree is ' /apps/hyperion/archivefiles
The directory tree is /apps/hyperion/archivefiles
++ echo /apps/hyperion/archivefiles
++ sed 's/apps/apps/;s/hyperion/stage;s/archivefiles/misc/;s/2011-07-01/;'
sed: -e expression #1, char 33: unknown option to `s'
+ string=
+ mkdir -p
mkdir: too few arguments
Try `mkdir --help' for more information.
+ cp -pr /apps/hyperion/archivefiles/Sa_Xchgrate
cp: missing destination file
Try `cp --help' for more information.

The modified script is as follows:
#!/bin/bash
set -x
stamp=$(date +%F)
list=/apps/hyperion/archivefiles/list
echo ${stamp}
exec 3< ${list}
while read -u 3 fname
do
file=$(echo ${fname} | sed 's#.*/\(test[^/]*\).*#\1#')
echo "The file name is " ${file}
dirnam=${fname%/*}
echo "The directory tree is " ${dirnam}
string=`echo ${dirnam} | sed 's/apps/apps/;s/hyperion/stage;s/archivefiles/misc/;s/'${stamp}'/;'`
mkdir -p ${string}
cp -pr ${string} ${fname}
done
# close the input file
exec 3<&-
 
Old 07-01-2011, 10:39 PM   #8
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,027

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
Quote:
Originally Posted by bv2925 View Post
Hi schneidz
i have no clue about sctipting [sic] but somehow managed to write a small script to cat the file and mkdir a new file.
I do not know where to start to just picking the file name in the end and create a new directory.
Would be helpful if you can provide an example.
can you post your attempt in [ code ] tags; here's a sample:
Code:
mkdir -p /backup/`date`/`basename /apps/oracle/product/test1`
i will allow you to read in your file and put it in a loop.

edit: ask your teacher about `backticks`.

Last edited by schneidz; 07-04-2011 at 11:38 PM. Reason: forgot to put backticks around date command
 
1 members found this post helpful.
Old 07-02-2011, 10:42 AM   #9
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,477

Rep: Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020
OK, let's back up here and take another look at what it is we're trying to accomplish.

You have a list of files or a list of directories (I'm not clear on which) that you want to back up to a different directory.

The first thing you need to do is isolate the last element (that is either a file name or a directory name, but that doesn't matter). There are a number of ways to do that, one of which as suggested by others, is to use the basename utility; e.g.,
Code:
basename /apps/oracle/product/test1
test1
or
Code:
basename /apps/hyperion/archivefiles/Sa_Xchgrate
Sa_Xchgrate
That can also be done with the cut utility but only if the number of fields is constant; i.e., there are always (and forever) going to be four elements:
Code:
echo /apps/hyperion/archivefiles/Sa_Xchgrate | cut -d'/' -f5
Sa_Xchgrate
(there are five slashes before the last element). cut is probably not the best idea because your development tree may change and, when it does, the whole thing falls apart.

So, instead of using the fancy-schmancy substring extraction and sed code
Code:
sed 's#.*/\(test[^/]*\).*#\1#')
(which requires that the pattern "test" is the beginning of the file name), it probably faster, better, cleaner, less confusing to simply use basename to isolate the last element.

Now, you want to have a directory tree, using your original post, something like this
Code:
/backup/2011-07-02/Sa_Xchgrate
That about right? Maybe not exact but similar enough that we can get on with it.

So, we need a data stamp and we get that with
Code:
stamp=$(date +%F)
and then we need to make the first parts of the back up directory with
Code:
dirnam=/backup/${stamp}
mkdir -p ${dirnam}
The nice thing about the -p option to mkdir is that it won't fail if the directory tree already exists so that's why we do that.

Then, because we've isolate the last element of the list -- which is either a file name or a directory name, doesn't matter -- we can use
Code:
cp -pr /apps/hyperion/archivefiles/Sa_Xchgrate ${dirnam}
and that will copy either a file named Sa_Xchgrate or a directory tree named Sa_Xchgrate to /backup/2011-07-01/Sa_Xchgrate.

That's pretty simple -- if, that is, it's what you really want, eh?

So
Code:
#!/bin/bash
#
#       It's a good idea to put some comments at the beginning of a shell program (or
#       any program for that matter) describing what you're doing and why you're doing it
#       ('cause you won't remember next week what you did and why).
#
#       we need a date stamp to name a back up directory
#
stamp=$(date +%F)
#
#       we need to read zero or more file- or direcotry names from a file
#       we're going to open that file as unit 3 (units 0, 1, and 2 are stadard
#       input, standard output and standard error)
#
exec 3< your_list_file_name
while read -u 3 list
do
        #
        #       we need the last element of what we just read, either a file name or a
        #       directory name, doesn't matter; we'll use the basename utillity to get it
        #
        filenam=basename ${list}
        #
        #       we've got what we need, so let's create a dirctory
        #       this is just "backup/${stamp}" change the hard-coded
        #       path name to whay you want it to be
        #
        dirnam=/backup/${stamp}
        mkdir -p ${dirnam}
        #
        #       and then copy the stuff
        #
        #       how about a message telling us what we're doing?
        echo "On ${stamp} creating back-up of ${list} to ${dirnam}"
        cp -pr ${list} ${dirnam}
done
#
#       and close the input file
#
exec 3<&-
#
#       and exit cleanly
#
exit 0
You need to edit "your_list_file_name" to the actual list file name and that ought to do it.

Hope this helps some.
 
1 members found this post helpful.
Old 07-05-2011, 10:37 AM   #10
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi Tronayne,
Thanks for the script. I have moved a little ahead with this where it creates the directory structure till the date.

I would want to describe the scenario so that i could provide an in depth picture of the script.

We have a file (List) which has a bunch of folder location names mentioned below.
/apps/hyperion/archivefiles/
/apps/hyperion/archivefiles/Demo
/apps/hyperion/archivefiles/Sample
/apps/hyperion/archivefiles/Sa_Xchgrate
/apps/hyperion/archivefiles/list

The above files (each one) have a different list of filename's under each one which have to be backed up in to the new directories that are being create under the date directory in the backup location.
So an example of the contents under Demo is
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct

So the script has to read filename (List), Create a new directory (Demo), read filelist from demo(Demo.app,Basic.otl etc) and backup them up to the new backup location ex: /backup/Date/Demo/{Demo.app,Basic.otl etc}

Hope I am clear.
Thanks.
 
Old 07-06-2011, 12:01 PM   #11
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,477

Rep: Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020Reputation: 1020
The only important thing you need to know is the last element in you list; that will be the name of the directory you want to back up and it is extracted with the basename utility as in the shell program above. That is, you do not need to know the names of files in the directory, you only need the directory name because cp -pr does two things, -p preserves mode, ownership, timestamps and -r copy directories recursively. And, if you want to watch it do it, replace -pr with -prv.

The other only thing you have to do is edit the shell program and make
Code:
        #
        #       we've got what we need, so let's create a dirctory
        #       this is just "backup/${stamp}" change the hard-coded
        #       path name to whay you want it to be
        #
        dirnam=/backup/${stamp}
        mkdir -p ${dirnam}
        #
to look like you want it to (although, right now, it's what you've described).

If you have one line in your file "List," say, /apps/hyperion/archivefiles/Demo and execute the shell program, you'll get a backup of the entire Demo tree (you do not need to enter anything else in List.

If the content of List were to be
Code:
/apps/hyperion/archivefiles/Demo
/apps/hyperion/archivefiles/Sample
/apps/hyperion/archivefiles/Sa_Xchgrate
You will get the entire content of each of those directories in
Code:
/backup/${stamp}/Demo
/backup/${stamp}/Sample
/backup/${stamp}/Sa_Xchgrate
where ${stamp} is the date.

That ought to do it -- try it.

Hope this helps some.
 
Old 07-06-2011, 01:21 PM   #12
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,027

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
Quote:
Originally Posted by bv2925 View Post
Hi Tronayne,
Thanks for the script. I have moved a little ahead with this where it creates the directory structure till the date.

I would want to describe the scenario so that i could provide an in depth picture of the script.

We have a file (List) which has a bunch of folder location names mentioned below.
/apps/hyperion/archivefiles/
/apps/hyperion/archivefiles/Demo
/apps/hyperion/archivefiles/Sample
/apps/hyperion/archivefiles/Sa_Xchgrate
/apps/hyperion/archivefiles/list

The above files (each one) have a different list of filename's under each one which have to be backed up in to the new directories that are being create under the date directory in the backup location.
So an example of the contents under Demo is
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct

So the script has to read filename (List), Create a new directory (Demo), read filelist from demo(Demo.app,Basic.otl etc) and backup them up to the new backup location ex: /backup/Date/Demo/{Demo.app,Basic.otl etc}

Hope I am clear.
Thanks.
^ still unsure what you are trying to do. can you post a before-and-after picture ?
here is my attempt to understand what you need:
Code:
[schneidz@hyper bv]$ cat demo.lst
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct
[schneidz@hyper bv]$ echo mkdir -p /backup/`date +%Y-%j`; cat demo.lst | while read line; do echo cp $line /backup/`date +%Y-%j`/`basename $line`; done
mkdir -p /backup/2011-187
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app /backup/2011-187/Demo.app
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl /backup/2011-187/Basic.otl
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db /backup/2011-187/Basic.db
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind /backup/2011-187/Basic.ind
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm /backup/2011-187/Basic.esm
cp /apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct /backup/2011-187/Basic.tct
good luck,
s. henry

Last edited by schneidz; 07-06-2011 at 01:32 PM.
 
1 members found this post helpful.
Old 07-27-2011, 10:18 AM   #13
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi schneidz
Thanks for the input but i am unable to achive what i am supposed to do.
Here is the scenario in detail.
I have a folder under which three files exist as listed below. The number of files will change depending on the number of applications.
$ cd /apps/hyperion/archivefiles/
$ ls
Demo Sample Sa_Xchgrate
Each of these have a list of files to be backed up with there full path as shown below.
$ cat Demo
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Demo.app
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.otl
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.db
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.ind
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.esm
/apps/hyperion/products/Essbase/EssbaseServer/app/Demo/Basic/Basic.tct
I have to write a program to read Demo,
Create a new directory Demo under mkdir -p ${Backuplocation}/${DATE}
and backup the files listed in demo Exemo.app, Basic.otl etc into the new location ${Backuplocation}/${DATE}/Demo/
The number of files may change during the course of time.
Next read Sample and repeat the procedure untill all the files in /apps/hyperion/archivefiles/ have been read and backed up.
in the end tar ${Backuplocation}/${DATE}.
Thanks in advance.
 
Old 07-27-2011, 10:29 AM   #14
bv2925
LQ Newbie
 
Registered: Jun 2011
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi Schneidz,
I have set the Variables as mentioned below.
set -x
DATE=`date +"%d-%b-%Y"`
Log='/apps/stage/misc/bkuptest/backup.log' #Location of the log file
Backuplocation='/apps/stage/misc/bkuptest/backup'
BACKUP_DIRECTORY=${Backuplocation}/${DATE}
Archivelist='/apps/hyperion/archivefiles/'

Thanks.
 
Old 07-29-2011, 09:05 AM   #15
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,027

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
Quote:
Originally Posted by bv2925 View Post
Hi Schneidz,
I have set the Variables as mentioned below.
set -x
DATE=`date +"%d-%b-%Y"`
Log='/apps/stage/misc/bkuptest/backup.log' #Location of the log file
Backuplocation='/apps/stage/misc/bkuptest/backup'
BACKUP_DIRECTORY=${Backuplocation}/${DATE}
Archivelist='/apps/hyperion/archivefiles/'

Thanks.
looks good so far. fyi, i think one of these two date formats will sort better:
date +"%Y-%m-%d"
date +"%Y-%j"


most of your criteria is fairly elemental, at this point you just need to combine those conditions in an automated fashion using the variables you set above.
this should get you started:
Code:
[schneidz@hyper bv]$ mkdir -p $BACKUP_DIRECTORY/demo; cat /apps/hyperion/archivefiles/demo | while read line; do cp "$line" $BACKUP_DIRECTORY/demo; done
you just need to modify the script to include all the archive files you care about (e.g.- sample, sa_xchgrate)

Last edited by schneidz; 07-29-2011 at 09:08 AM.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] List of directories in a file TheBigH Linux - General 6 04-04-2011 10:43 AM
Trying to create directories based on file names JackieBrown Linux - General 3 05-15-2009 10:13 PM
script that create a blank file in all sub-directories. samengr Programming 8 02-18-2009 06:00 PM
How to list the directories of a file system by their sizes? joaotelles Solaris / OpenSolaris 3 12-04-2008 03:11 PM
Create an archive file which includes files in sub-directories? amaze Linux - General 1 09-26-2003 09:43 AM


All times are GMT -5. The time now is 12:41 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration