LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-16-2010, 06:11 PM   #1
dipeshvshah
LQ Newbie
 
Registered: Aug 2010
Posts: 3

Rep: Reputation: 0
Shell script question


Hi Experts!!

I have backup_server and application_server.

backup_server has directory AAA. I need to check from application server
that is there any new files created today in the AAA dirctory. if yes, all files were created today or partial files?.

I am not expert in scripting. Please suggest me the way.

Thanks
Dip
 
Old 08-16-2010, 06:39 PM   #2
MBybee
Member
 
Registered: Jan 2009
Location: wherever I can make a living
Distribution: PC-BSD / FreeBSD / Debian / Ubuntu / Win7 / OpenVMS
Posts: 438

Rep: Reputation: 57
I would recommend using the find command. Find has atime and mtime switches that will let you know which files were created or modified today.

Find docs: http://linux.die.net/man/1/find
 
Old 08-16-2010, 06:58 PM   #3
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670
You can use two tests in you find statment to find files created in last 24 hours and files more than 5 minutes old which might filter out files still being written to. (Depending on the file size)

For currently open files, look at using the lsof command. To track files being created, look at the famd daemon.

You will need to provide more information on what you mean by partial files, such as what kinds of files you are talking about (can integrity be tested ). How they were created, what log files if any will indicate a failure ( e.g. ftp logs )
 
Old 08-16-2010, 07:22 PM   #4
dipeshvshah
LQ Newbie
 
Registered: Aug 2010
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks for the reply.
Quote:
#Defining variables and assigning values
USR='admin'
PASSWD='test'
HT='test.linux.ca'
FILE='S*.pdf'
DIRNAME='/docs'

echo $HT
ftp -n -i -v $HT <<end_ftp
user $USR $PASSWD
cd $DIRNAME
CT=ls -RF | sed -e '/^$/d' -e '/.*[/@:=|>]$/d' |wc -l
echo $CT
bye
end_ftp
I was doing testing.

It says invalid command. I believe because its accessing remote server.

Can someone modify the code to make it work. I also tried 'find' same way but it says

usage: ls remote-directory local-file

Thanks,
dip
 
Old 08-16-2010, 07:30 PM   #5
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
Blog Entries: 8

Rep: Reputation: 235Reputation: 235Reputation: 235
Can I ask how files are considered as partial?

To find all the files in AAA directory:
Code:
find "<path to AAA directory>" -type f -daystart
To find all the files in AAA directory that were created today:
Code:
find "<path to AAA directory>" -type f -daystart -not -ctime +0
To find all the files in AAA directory that were not created today:
Code:
find "<path to AAA directory>" -type f -daystart -ctime +0
To count the results from a directory search, just add '| wc -l':
Code:
find ... | wc -l
Putting everything inside a script:
Code:
#!/bin/bash

AAA="<path to AAA directory>"

declare -i TODAYCOUNT NOTTODAYCOUNT

TODAYCOUNT=$(find "$AAA" -type f -daystart -not -ctime +0 | wc -l)

if [[ TODAYCOUNT -eq 0 ]]; then
	echo "No file was created today."
else
	NOTTODAYCOUNT=$(find "$AAA" -type f -daystart -ctime +0 | wc -l)

	if [[ NOTTODAYCOUNT -eq 0 ]]; then
		echo "There are new files in $AAA and all of them were created today ($TODAYCOUNT)."
	else
		echo "Some of the files in $AAA were created today ($TODAYCOUNT)."
	fi
fi
or
Code:
#!/bin/bash

AAA="<path to AAA directory>"

declare -a TODAY
declare -i TODAYCOUNT=0 NOTTODAYCOUNT I

while read; do
	TODAY[TODAYCOUNT++]=$REPLY
done < <(find "$AAA" -type f -daystart -not -ctime +0 | wc -l)

if [[ TODAYCOUNT -eq 0 ]]; then
	echo "No file was created today."
else
	NOTTODAYCOUNT=$(find "$AAA" -type f -daystart -ctime +0 | wc -l)

	if [[ NOTTODAYCOUNT -eq 0 ]]; then
		echo "There are new files in $AAA and all of them were created today ($TODAYCOUNT)."
	else
		echo "Some of the files in $AAA were created today ($TODAYCOUNT)."
	fi

	echo --------------------

	for (( I = 0; I < TODAYCOUNT; ++I )); do  # for bash 2.05+
		echo "${TODAY[I]}"
	done

	echo --------------------
fi
 
1 members found this post helpful.
Old 08-16-2010, 07:33 PM   #6
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
Blog Entries: 8

Rep: Reputation: 235Reputation: 235Reputation: 235
I was already done with post I just made and was about to make the post when I read your new post in preview so please don't mind the scripts. I think they're not easy to apply if you're using ftp.
 
Old 08-16-2010, 10:43 PM   #7
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 17,956

Rep: Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693Reputation: 3693
Quote:
Originally Posted by dipeshvshah View Post
Thanks for the reply.


I was doing testing.

It says invalid command. I believe because its accessing remote server.

Can someone modify the code to make it work. I also tried 'find' same way but it says

usage: ls remote-directory local-file

Thanks,
dip
Yes...YOU can modify the code to make it work. This is your job...why should we do it for you, when you just want to take the lazy way out and give up? We don't know your environment, or what you really need this to do. There are thousands of scripting tutorials available through Google...you can easily find one.

Also, if you're going to use FTP, try either using an Expect script, or using SCP/SFTP with a key swap, so it won't prompt you for passwords.
 
1 members found this post helpful.
Old 08-17-2010, 02:05 AM   #8
r3sistance
Senior Member
 
Registered: Mar 2004
Location: UK
Distribution: CentOS 5.4, Mac OS 10.4 (tiger)
Posts: 1,005

Rep: Reputation: 79
Isn't this all re-inventing the wheel?

dipeshvshah, Are you after actually knowing if the files were updated or are you more specifically just after updating all new and all updated files? if so I would recommend the rsync utility as this will do most of this on your behalf tho I do not know if it can be used to create a report as I myself never use rsync but I would presume it would be possible (as I back-up whole 30~50GB VM images as opposed to single files).
 
Old 08-17-2010, 04:49 AM   #9
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670
You could use the find command locally to produce a list of filenames that meet your criteria. You don't use find inside an ftp script. In your ftp script you generated, you are including shell script commands. An ftp script can only include ftp commands.

You could run find on the remote server via ssh to return a list of files on the remote server matching certain criteria.

One technique to consider is to use find's -printf command to generate the parts of the ftp script where you put the files.
find /<dir>/ -type f -cmin +5 -ctime -1 -printf "put %p\n" >>sendfiles.ftp
echo "bye" >>sendfiles.ftp

Assemble the ftp script and then run it like
ftp <sendfiles.ftp >logfile

Be sure to read the info file for find. You can return the names of files and ignore directories, for example.
----

Using rsync or tar incremental dumps may be more suitable for what you want to do. You need ssh access to do this.

Last edited by jschiwal; 08-17-2010 at 04:55 AM.
 
Old 08-17-2010, 10:26 AM   #10
dipeshvshah
LQ Newbie
 
Registered: Aug 2010
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks jschiwal & konsolebox.

I used following approach to get listing of all files of remote server.
Now I have remote server file information on same server.

Quote:
#Defining variables and assigning values
USR='admin'
PASSWD='abc'
HT='xyz.aaa.com'
FILE='S*.pdf
DIRNAME='/doc'

LsFiles()
{
ftp -n -i $HT <<end_ftp
user $USR $PASSWD
cd $DIRNAME
ls "-lrt *.pdf"
bye
end_ftp
}

LsFiles > list_of_files.txt
I am getting listing in the output.txt

Quote:
-rwxr--r-- 1 504 504 176201 Aug 17 19:28 1652001_5-31-2010.pdf
-rwxr--r-- 1 504 504 178860 Aug 17 19:28 1648403_5-31-2010.pdf
-rwxr--r-- 1 504 504 175665 Aug 17 19:28 1655876_5-31-2010.pdf
-rwxr--r-- 1 504 504 177563 Aug 17 19:29 1661000_5-31-2010.pdf
-rwxr--r-- 1 504 504 302814 Aug 17 19:29 1650920_5-31-2010.pdf
-rwxr--r-- 1 504 504 174484 Aug 15 19:29 1665618_5-31-2010.pdf
-rwxr--r-- 1 504 504 177675 Jun 15 19:29 1655604_5-31-2010.pdf
Now I want to count today's created files and old files. COuld you pl. give
me a clue.

Thanks

Last edited by dipeshvshah; 08-17-2010 at 10:37 AM.
 
Old 08-17-2010, 07:40 PM   #11
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670
Code:
ls "-lrt *.pdf"
Are these options listed in the remote servers `ls' command options? If not you are still injecting shell commands inside an ftp script.

If not, I think it would be better (if you have access) to ssh into the remote server so you can run the find command. As already mentioned using sftp as well would be more secure.
---
You can match the date "Aug 17" in this case, and count the number of occurrences using awk.
or use sed or grep as before & wc to count the number of matches.
TODAY=$( date '+%d %b' )
newfiles=$( sed -n "/$TODAY/p" list_of_files.txt | wc -l )

or
newfiles=$( grep -c "$TODAY" list_of_files.txt)
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Shell script question poorboyiii Linux - General 3 02-13-2010 06:42 PM
Shell Script Question ytnexus Linux - Newbie 3 06-28-2009 10:22 PM
shell script question icecubeflower Linux - Newbie 4 03-31-2009 07:24 AM
question about shell script tmbigrigg Linux - Newbie 4 11-05-2007 03:35 PM
Shell script question. siaful Linux - Newbie 1 04-29-2004 06:01 PM


All times are GMT -5. The time now is 11:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration