Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 09-04-2012, 02:04 PM   #1
LQ Newbie
Registered: Jun 2012
Posts: 15

Rep: Reputation: Disabled

I was wondering if anyone can help me figure out what it going on with a CRON job. The scenario:

I have a script that transfers file between two machines using sftp. The script sftps to to remote machine, does an ls on the directory, and pipes that to a file, "remote_files.txt". It then transfers this file to the local machine. There is a file named "old_files.txt" on the local machine that has a list of all the files that were transferred. When the "remote_files.txt" file is transferred to the local machine, the script runs a grep against them, and writes the deltas to "new_files.txt". It then sftps the files that are listed in the "new_files.txt" file, and appends them to the "old_files.txt" list.

The CRON job was set to run every 10 minutes, and has had to be increased to 30 minutes. When I run the script manually, it takes about 48 seconds, but when it runs through CRON, its taking over 30 minutes. We dont want to have to keep increasing it, and are trying to figure out why its taking so long. When I run top whiles its running, its the grep command that is taking about 28 minutes to run. This is the command that we have:

grep -vf $HOME/old_files.txt $HOME/remote_files.txt > $HOME/new_files.txt
I dont see anything in the cron logs that would tell me anything.. Any ideas?
Old 09-04-2012, 07:14 PM   #2
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,873

Rep: Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600
Are you capturing all the output eg
*/30 * * * * /path/ >/home/me/my.log  2>&1
Also, try adding the 'set -xv' debug code near the top
set -xv
Old 09-04-2012, 07:57 PM   #3
Registered: Nov 2007
Location: Northeastern USA
Distribution: kubuntu
Posts: 204

Rep: Reputation: 106Reputation: 106
What access do you have to both machines?

If what you're doing is essentially backing up one machine to the other, then using rsync/rsyncd would probably give you a lot more options, flexibility, and maybe even speed (since it's very good with transferring just the parts of things that have changed.)

For this to work, though, you'd have to be able to configure both machines to open the proper port and for the greatest flexibility, you might want to run the whole thing using rsyncd on the remote machine and you'd have to make sure that rsync/rsyncd are properly installed and accessible to you on both ends. I use it all the time on my local machine, but haven't worked with remote ones yet.

For more info, there's tons on the web or see the list at "rsync" <> where the experts hang out.


Old 09-04-2012, 10:00 PM   #4
Registered: Jul 2006
Location: Maryland, USA
Distribution: Fedora and others
Posts: 756

Rep: Reputation: 144Reputation: 144
It sounds like you're not sure which line in your cron script is actually taking a long time. You can find out with the time command. Just put in front of each command and it will tell you how much time each is taking.

Of course, it writes to stderr, so you'll have to be sure to capture it, like chrism01 did.

As for scp, I have code like this in a couple of my cron scripts:
test -r ~/.keychain/$(hostname)-sh  &&  . ~/.keychain/$(hostname)-sh
if [ -z "$SSH_AGENT_PID" ] || [ ! -d "/proc/$SSH_AGENT_PID" ]; then
    echo "ERROR: No ssh agent, SSH_AGENT_PID=\"$SSH_AGENT_PID\""
    exit 1
Old 09-04-2012, 10:28 PM   #5
LQ Newbie
Registered: Jun 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thank you all for the replies..

@chrism01 - Yes, we have it set up to capture to a log like that. I will try adding that to the code to see if I can see anything else..

@josephj - Are you asking what kind of access I personally have? I have root access to both machines. We have it set up so there is a user with the same username on both machines, and that is what we use to initiate the transfer. The way our network is set up: We have one web machine that is exposed to the internet, and one application machine that is not exposed to the internet. The application and web machine can only talk to each other but only when the application machine initiates an SFTP connection. Our clients upload a file through our UI that is on the web machine, and it gets transfered to the application machine to be processed. We have explored using rsync, but it requires both machines directories to match, and when the file on the application machine is processed, its moved to an archive directory, so that wont work..

@KenJackson - I will look into adding the time command to the script. Would I add it on the line above, or in line with the current commands?

echo This is code

time echo this is code
Old 09-04-2012, 10:34 PM   #6
Registered: Jul 2006
Location: Maryland, USA
Distribution: Fedora and others
Posts: 756

Rep: Reputation: 144Reputation: 144
Add it to the front of the command
time echo this is code
Old 09-05-2012, 04:19 AM   #7
Registered: Sep 2009
Location: Perth, W.A.
Distribution: Slackware, Debian, Gentoo, FreeBSD, OpenBSD
Posts: 208

Rep: Reputation: 41
Originally Posted by blsimpson View Post
We have explored using rsync, but it requires both machines directories to match, and when the file
Are you able to set up links during the archiving process on the remote machine, such as 'ln -s DIR/archive/FILE DIR/FILE' so that rsync can find them? Would make this easy if you can.

Re the grep, is the cron job disabled when you tested this? I'm wondering if perhaps cron is tripping over itself by touching one of the txt files while the previous cron job was still running.
Old 09-05-2012, 08:33 AM   #8
Registered: Jan 2012
Posts: 42

Rep: Reputation: 2
Runs as?

For me personally 9 out of 10 problems with cron are dealing with the fact that my script runs as a different user. And I am betting it's your problem this time too.

You use $HOME but never mention under which user the cron task runs as.

Test your script by
su -c '/usr/local/bin/script par1 par2' useritshouldrunas

And if you are using $HOME that might not even be sufficient. Echo it in your script so you know its value!
Old 09-05-2012, 02:39 PM   #9
LQ Newbie
Registered: Sep 2006
Distribution: Slackware
Posts: 3

Rep: Reputation: 1
Just a thought

Separately from the CRON vs manually-run timing issue, you might also think about this design issue.

If I understand your process correctly, it looks like your old_files.txt file keeps getting appended to. (If I'm mistaken, you can ignore all this.) I'm not sure how many lines yours currently contains, but as it keeps getting larger, your grep may take longer and longer to process, I suppose depending on how beefy your machine is.

I ran a little test on an older (1.6 GHz) PC I have with Slackware 13.37. I created a file of file names with ls to simulate your remote_files.txt, and then another set of files simmulating your old_files.txt. I timed several runs of grep matching yours, using old_files.txt with increasing numbers of lines. The run-time of the grep increased as the number of lines in old_files.txt increased:

number of lines in old_files.txt -- grep run time
15 -- near-instantaneous
2000 -- 9 seconds
8000 -- 88 seconds
32000 -- 13 minutes 48 seconds

Again, if I understand your process correctly, you may find it keeps taking longer and longer as your old_files.txt grows. Just a thought.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Strange Cron directories: /var/spool/cron/cron.GfqqGO Woodsman Slackware 4 05-11-2011 02:37 PM
cron.hourly cron.weekly cron.monthly and 0anacron. Are they necessary? glore2002 Debian 2 09-30-2009 08:57 PM
adding a perl script to cron.daily / cron.d to setup a cron job CrontabNewBIE Linux - Software 6 01-14-2008 08:16 AM
cron not working from crontab nor form /etc/cron/cron.d. What did SuSE change? JZL240I-U SUSE / openSUSE 11 01-04-2007 01:57 AM
Can any one plz explain why/what for cron.d, cron.daily, cron.weekly etc are there. mavinashbabu Linux - Newbie 4 09-21-2006 01:50 PM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:21 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration