Shell script question

dipeshvshah · 08-16-2010, 05:11 PM

Hi Experts!!

I have backup_server and application_server.

backup_server has directory AAA. I need to check from application server
that is there any new files created today in the AAA dirctory. if yes, all files were created today or partial files?.

I am not expert in scripting. Please suggest me the way.

Thanks
Dip

MBybee · 08-16-2010, 05:39 PM

I would recommend using the find command. Find has atime and mtime switches that will let you know which files were created or modified today.

Find docs: http://linux.die.net/man/1/find

jschiwal · 08-16-2010, 05:58 PM

You can use two tests in you find statment to find files created in last 24 hours and files more than 5 minutes old which might filter out files still being written to. (Depending on the file size)

For currently open files, look at using the lsof command. To track files being created, look at the famd daemon.

You will need to provide more information on what you mean by partial files, such as what kinds of files you are talking about (can integrity be tested ). How they were created, what log files if any will indicate a failure ( e.g. ftp logs )

dipeshvshah · 08-16-2010, 06:22 PM

Thanks for the reply.

Quote:

#Defining variables and assigning values
USR='admin'
PASSWD='test'
HT='test.linux.ca'
FILE='S*.pdf'
DIRNAME='/docs'

echo $HT
ftp -n -i -v $HT <<end_ftp
user $USR $PASSWD
cd $DIRNAME
CT=ls -RF | sed -e '/^$/d' -e '/.*[/@:=|>]$/d' |wc -l
echo $CT
bye
end_ftp

I was doing testing.

It says invalid command. I believe because its accessing remote server.

Can someone modify the code to make it work. I also tried 'find' same way but it says

usage: ls remote-directory local-file

Thanks,
dip

konsolebox · 08-16-2010, 06:30 PM

Can I ask how files are considered as partial?

To find all the files in AAA directory:

Code:

find "<path to AAA directory>" -type f -daystart

To find all the files in AAA directory that were created today:

Code:

find "<path to AAA directory>" -type f -daystart -not -ctime +0

To find all the files in AAA directory that were not created today:

Code:

find "<path to AAA directory>" -type f -daystart -ctime +0

To count the results from a directory search, just add '| wc -l':

Code:

find ... | wc -l

Putting everything inside a script:

Code:

#!/bin/bash

AAA="<path to AAA directory>"

declare -i TODAYCOUNT NOTTODAYCOUNT

TODAYCOUNT=$(find "$AAA" -type f -daystart -not -ctime +0 | wc -l)

if [[ TODAYCOUNT -eq 0 ]]; then
	echo "No file was created today."
else
	NOTTODAYCOUNT=$(find "$AAA" -type f -daystart -ctime +0 | wc -l)

	if [[ NOTTODAYCOUNT -eq 0 ]]; then
		echo "There are new files in $AAA and all of them were created today ($TODAYCOUNT)."
	else
		echo "Some of the files in $AAA were created today ($TODAYCOUNT)."
	fi
fi

or

Code:

#!/bin/bash

AAA="<path to AAA directory>"

declare -a TODAY
declare -i TODAYCOUNT=0 NOTTODAYCOUNT I

while read; do
	TODAY[TODAYCOUNT++]=$REPLY
done < <(find "$AAA" -type f -daystart -not -ctime +0 | wc -l)

if [[ TODAYCOUNT -eq 0 ]]; then
	echo "No file was created today."
else
	NOTTODAYCOUNT=$(find "$AAA" -type f -daystart -ctime +0 | wc -l)

	if [[ NOTTODAYCOUNT -eq 0 ]]; then
		echo "There are new files in $AAA and all of them were created today ($TODAYCOUNT)."
	else
		echo "Some of the files in $AAA were created today ($TODAYCOUNT)."
	fi

	echo --------------------

	for (( I = 0; I < TODAYCOUNT; ++I )); do  # for bash 2.05+
		echo "${TODAY[I]}"
	done

	echo --------------------
fi

konsolebox · 08-16-2010, 06:33 PM

I was already done with post I just made and was about to make the post when I read your new post in preview so please don't mind the scripts. I think they're not easy to apply if you're using ftp.

TB0ne · 08-16-2010, 09:43 PM

Quote:

Originally Posted by dipeshvshah

Thanks for the reply.

I was doing testing.

It says invalid command. I believe because its accessing remote server.

Can someone modify the code to make it work. I also tried 'find' same way but it says

usage: ls remote-directory local-file

Thanks,
dip

Yes...YOU can modify the code to make it work. This is your job...why should we do it for you, when you just want to take the lazy way out and give up? We don't know your environment, or what you really need this to do. There are thousands of scripting tutorials available through Google...you can easily find one.

Also, if you're going to use FTP, try either using an Expect script, or using SCP/SFTP with a key swap, so it won't prompt you for passwords.

r3sistance · 08-17-2010, 01:05 AM

Isn't this all re-inventing the wheel?

dipeshvshah, Are you after actually knowing if the files were updated or are you more specifically just after updating all new and all updated files? if so I would recommend the rsync utility as this will do most of this on your behalf tho I do not know if it can be used to create a report as I myself never use rsync but I would presume it would be possible (as I back-up whole 30~50GB VM images as opposed to single files).

jschiwal · 08-17-2010, 03:49 AM

You could use the find command locally to produce a list of filenames that meet your criteria. You don't use find inside an ftp script. In your ftp script you generated, you are including shell script commands. An ftp script can only include ftp commands.

You could run find on the remote server via ssh to return a list of files on the remote server matching certain criteria.

One technique to consider is to use find's -printf command to generate the parts of the ftp script where you put the files.
find /<dir>/ -type f -cmin +5 -ctime -1 -printf "put %p\n" >>sendfiles.ftp
echo "bye" >>sendfiles.ftp

Assemble the ftp script and then run it like
ftp <sendfiles.ftp >logfile

Be sure to read the info file for find. You can return the names of files and ignore directories, for example.
----

Using rsync or tar incremental dumps may be more suitable for what you want to do. You need ssh access to do this.

dipeshvshah · 08-17-2010, 09:26 AM

Thanks jschiwal & konsolebox.

I used following approach to get listing of all files of remote server.
Now I have remote server file information on same server.

Quote:

#Defining variables and assigning values
USR='admin'
PASSWD='abc'
HT='xyz.aaa.com'
FILE='S*.pdf
DIRNAME='/doc'

LsFiles()
{
ftp -n -i $HT <<end_ftp
user $USR $PASSWD
cd $DIRNAME
ls "-lrt *.pdf"
bye
end_ftp
}

LsFiles > list_of_files.txt

I am getting listing in the output.txt

Quote:

-rwxr--r-- 1 504 504 176201 Aug 17 19:28 1652001_5-31-2010.pdf
-rwxr--r-- 1 504 504 178860 Aug 17 19:28 1648403_5-31-2010.pdf
-rwxr--r-- 1 504 504 175665 Aug 17 19:28 1655876_5-31-2010.pdf
-rwxr--r-- 1 504 504 177563 Aug 17 19:29 1661000_5-31-2010.pdf
-rwxr--r-- 1 504 504 302814 Aug 17 19:29 1650920_5-31-2010.pdf
-rwxr--r-- 1 504 504 174484 Aug 15 19:29 1665618_5-31-2010.pdf
-rwxr--r-- 1 504 504 177675 Jun 15 19:29 1655604_5-31-2010.pdf

Now I want to count today's created files and old files. COuld you pl. give
me a clue.

Thanks

jschiwal · 08-17-2010, 06:40 PM

Code:

ls "-lrt *.pdf"

Are these options listed in the remote servers `ls' command options? If not you are still injecting shell commands inside an ftp script.

If not, I think it would be better (if you have access) to ssh into the remote server so you can run the find command. As already mentioned using sftp as well would be more secure.
---
You can match the date "Aug 17" in this case, and count the number of occurrences using awk.
or use sed or grep as before & wc to count the number of matches.
TODAY=$( date '+%d %b' )
newfiles=$( sed -n "/$TODAY/p" list_of_files.txt | wc -l )

or
newfiles=$( grep -c "$TODAY" list_of_files.txt)