Assistance in Shell script on placing files across mount points.

manickaraja · 02-19-2010, 02:23 AM

Hi Experts,
Need assistance from you. Given below is my current requirement. Currently help me on the same.

I have a database with x number of files (192 at the moment, but will vary from time to time). I am going to copy these files to another location on the same server thorugh shell script. Problem with total size of 192 files is approx 900 GB (again this will vary from time to time).
My shell script should calculate the free space available at present in the server on each of the mount point (can be filled till it reaches 95%). Always 5% free space should be available free for future growth.

After calculating, it should prepare another flat file with following details

FileName1 - /ora01 - 30 GB
FileName2 - /ora01 - 70 GB
(lets assume it can put only 100GB worth data in /ora01 and so it should switch to next available mount)
FileName3 - /ora05 - 30 GB
and so on....

Could you please help me with the same? Please let me know if anyone needs further clarification on the above requirement.

Thanks,
Manick.

manickaraja · 02-19-2010, 02:25 AM

Forgot to mention that input to the program would be another flat file which consists of Filenames and current sizes which needs to be copied.

ex:
FileName1 - 30GB
FileName2 - 70GB
FileName3 - 30GB
FileName4 - 50GB
etc..

GrapefruiTgirl · 02-19-2010, 02:27 AM

Not sure about anyone else, but I don't understand the question. Please clarify it some more

Also, you may find the `df -h` command helpful in this scenario:

Code:

sasha@reactor: df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root              15G   13G  1.6G  89% /
tmpfs                 2.0G     0  2.0G   0% /dev/shm
sasha@reactor:

Disillusionist · 02-19-2010, 02:31 AM

Are we to assume that you are moving the datafiles that comprise an Oracle database?

Please advise what you are trying to achieve and whether you have archive log mode turned on.

manickaraja · 02-19-2010, 02:56 AM

Hi,
Yes. These are oracle databfiles and database is in archive log. So should be fine.
Now let me tell how we do this process at present manually so that it will easy for everyone to understand the requirement.

a) First we pull the data file names and current sizes from source database in a spread sheet
b) We execute df command on server and calculate free space available in each of the mount points
c) We try to split the files and place it across 'n' number of mount points depending upon file size and free space available.
d) once we decided we create a flat file with above information (file-name & mount point to which it needs to be copied).
e) This flat file will be input for the process which does the copy job.

Now we are trying to automate the process of flat file creation (from step b to d through shell script and step a will be input for this process.

Please let me know if still the requirement is unclear.

Thanks,
Manick.

manickaraja · 02-19-2010, 04:09 AM

Quote:

Originally Posted by GrapefruiTgirl

Not sure about anyone else, but I don't understand the question. Please clarify it some more

Also, you may find the `df -h` command helpful in this scenario:

Code:

sasha@reactor: df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root              15G   13G  1.6G  89% /
tmpfs                 2.0G     0  2.0G   0% /dev/shm
sasha@reactor:

Hi,
I have updated with more clarification. Could you please check and advice? Thanks a lot for your valuable time.

Disillusionist · 02-20-2010, 03:57 AM

I must admit I am still somewhat unclear as to why you are doing what you say you are doing.

Questions:

Are you trying to create a backup copy of the database?
If not are you trying to resolve space issues with the current DB?

If you are trying to create a backup copy of the database, use the RMAN duplicate database command.

If you are trying to resolve space issues, why move the existing files?

You can add new datafiles to the current tablespaces.

If I have missed the point here please let me know.

konsolebox · 02-21-2010, 10:43 AM

dd might do the trick

for example, first file:

Code:

dd if=file of=ofile1 bs=<size1>

Code:

dd if=file of=ofile2 skip=<size1> bs=<size2>

Please test these commands on a small file then check if the outputs' size are really what are expected. 'man dd' for more details.

Disillusionist · 02-22-2010, 02:01 AM

Hi Konsolebox,

That would work if you wanted to paste multiple file parts together, I don't believe that is what the OP is after.

It looks like the OP wants to make a copy of the original DB (either as a cold backup, or as a duplicate of the DB)

Hi, manickaraja,

As the DB is in Archive Log mode, it would make more sense (to me) to take a hot backup (why else would you use Archive Log mode?) this removes the need to shutdown the DB in order to back it up.

Once you have a hot backup, you can duplicate the DB. You can use the parameter file of the Duplicate DB to transform your datafile filenames from the production locations to the new duplicate locations.

Disillusionist · 02-22-2010, 02:12 AM

My process for automatically fitting the files to the new area's would be:

Get the list of all the datafiles
Sort in size order (largest file first)
Get the current size, used, available information of the new locations
Sort the mountpoint in size order (least space first)
Copy the datafile into the smallest mountpoint that it will fit (with the required space remaining)
Repeat

I don't have time at the moment to write this in actual code, but that's the process I would use

manickaraja · 02-22-2010, 08:50 AM

Hi,
We are doing a cloning and it is from a active data guard. We dont want to use RMAN Duplicate database for various reasons and hence we are using RMAN Image copy for copying datafiles after which we will be restoring the database by applying archive logs from data guard standby database.
Now if any of you have any ready made code or can easily give me one to fit the datafiles into various mount points depending upon space available on each of them, it will be of great help.

Thanks,
Manick.

AnanthaP · 02-22-2010, 05:32 PM

In theory, you can run out of free space in all mount points.

The things the script must do are:
(1) Create a list (or array) of mounted file systems.
(2) For each file,
(2a) For each mounted file system, compare the size with the available area or try the next file system.

What have you got so far?

End

manickaraja · 02-23-2010, 01:59 AM

Hi,
The actual problem I am facing is I cannot copy the file , check free space after copy and proceed with another file copt to same file system depending upon space available.
I have to write a script first for performing file copies and then execute it. So for writing the script I need to decide which all files will fit in which mount point. So somehow I have calculate within the program before availing the space. How can I do it?

Thanks

konsolebox · 02-23-2010, 04:36 AM

some hint:

Code:

while
    <check if fs still has space, store the size to a variable for later use>
do
    <with the value of the variable, look for the file that best fits the remaining space then copy it>
done

Disillusionist · 02-25-2010, 02:24 AM

Here is a start.

Code:

#!/usr/bin/perl
###
### Read datafiles into hash containing the datafile and it's size in KB
###

open(DATAFILES, "datafiles") or die "Could not open datafiles\n";
while(<DATAFILES>) {
   chomp;    # remove charage return
   ($v_file, $v_size)=split(/ /, $_);
   $datafiles{"$v_file"}="$v_size";
}
close(DATAFILES);

###
### Read mountpoints into hash containing mountpoint and available
###  space - 5%
###

open(MOUNTPOINTS, "mountpoints") or die "Could not open mountpoints\n";
while(<MOUNTPOINTS>) {
   chomp;
   ($v_mount, $v_size, $v_avail)=split(/ /, $_);
   $max_avail=int($v_avail - ($v_size / 20));
   $mountpoints{"$v_mount"}="$max_avail";
}
close(MOUNTPOINTS);

At this stage, you have read the information into the script and stored the bits you need to work with.

Left to do
==========

List the datafiles in size order (largest first)
List the mountpoints in size order (smallest first)
Find which mountpoints the datafiles fit in
`-- Store the information about which datafile to allocate to which mountpoint
`-- reduce the max_avail of the relevant mountpoint
`-- repeat until either all datafiles are allocated or you run out of space
`-- create a log of errors & warnings

That's all I have the time for at the moment, if you progress the script before I do, please post the code.