LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-14-2017, 12:44 AM   #1
timl
Member
 
Registered: Jan 2009
Location: Sydney, Australia
Distribution: Fedora,CentOS
Posts: 711

Rep: Reputation: 152Reputation: 152
rsync runs out of memory in cron job


Hi, I recently bought 2*5TB USB hard drives which I have attached to my Raspberry Pi 1 with 512M of memory. The Pi serves as a bit torrent server. I needed a bit more room for storage and saw these 2 items going cheap. A bit bigger than I need at the moment but should keep me going for a while - one is the active disk and the other is the backup.

Having populated both drives I brought them online yesterday. I have a cron job set up to invoke a script which backs up the active drive onto the backup.
Quote:
rsync -r -a -v -t /mnt/from /mnt/to/
There are 9 individual folders which are backed up and the script simply submits the rsync jobs one after the other. I ran into a problem with the cron job where I got these messages:
Quote:
Jul 13 06:25:09 resurgam rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2055" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 13 06:25:10 resurgam rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2055" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 14 06:24:37 resurgam kernel: [89763.004792] rsync invoked oom-killer: gfp_mask=0x2146d0, order=1, oom_score_adj=0
That looks like I am out of memory! However the script runs fine from the command line. This worked fine on my old 2TB drives so I am guessing it is related to the drive size but I have no ideas what. Any suggestions?

Cheers
 
Old 07-14-2017, 05:24 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,614

Rep: Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607Reputation: 5607
It is a bit unclear for me: where is this rsync command executed? on rpi? Did you check the memory you have somehow? Did you try to add some swap space? Was there any other app running that time?
 
Old 07-14-2017, 09:34 AM   #3
kilgoretrout
Senior Member
 
Registered: Oct 2003
Posts: 2,854

Rep: Reputation: 340Reputation: 340Reputation: 340Reputation: 340
You might want to post your script in it's entirety.
Quote:
However the script runs fine from the command line.
When a script runs fine from the command line but fails when run as a cron job, it's usually a PATH problem. The cron environment is different than your usual command line environment. Relative paths that run fine from the command line often fail in cron. Try revising your script to give full paths.
 
Old 07-14-2017, 10:30 AM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,605

Rep: Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165
In addition to what pan64 posted, typically "rsyslogd was HUPed" is indicative of the logs being rotated. Looks like your running your job from cron.daily or a cron job that happens at the same time. Typically rsync reads the entire directory structure and stores it in memory so as suggested this might cause a memory issue if other stuff is running at the same time. Since it works running from command line try setting up a cron job after the cron.daily jobs are complete or earlier and so that it completes before the daily jobs start. Adding swap space should help too.

Last edited by michaelk; 07-14-2017 at 10:52 AM.
 
Old 07-14-2017, 11:37 AM   #5
HappyTux
Senior Member
 
Registered: Mar 2003
Location: Nova Scotia, Canada
Distribution: Debian AMD64
Posts: 3,918

Rep: Reputation: 176Reputation: 176
I would add sync each folder one at a time in the script that should cut down on the memory used as each new process will allow the releasing of the old completed ones memory.
 
Old 07-14-2017, 12:08 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,555

Rep: Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088
Quote:
Originally Posted by michaelk View Post
In addition to what pan64 posted, typically "rsyslogd was HUPed" is indicative of the logs being rotated. Looks like your running your job from cron.daily or a cron job that happens at the same time.
Note that those "HUP" messages were from the day before. The "run-parts" script (in the Red Hat family, at least) runs each of the scheduled jobs in sequence, waiting for each to complete before starting the next, so there should be no competition if all the jobs are being run from cron.daily. Of course if if you have your own cron job that happens to run at the same time as cron.daily, that's another matter.
 
Old 07-14-2017, 04:02 PM   #7
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,605

Rep: Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165Reputation: 4165
Your correct. Trying to fill in some missing holes in the data...
 
Old 07-22-2017, 12:29 AM   #8
timl
Member
 
Registered: Jan 2009
Location: Sydney, Australia
Distribution: Fedora,CentOS
Posts: 711

Original Poster
Rep: Reputation: 152Reputation: 152
Sorry for not responding earlier. The problem is not solved but I made it go away. I simply changed the time of the cron job from 5am to 6pm and it has run fine ever since. I note that cron.daily runs at 6:25am so should not have affected a 5am start but this is what prompted me to make the change.

For the purposes of documentation and to fill in a few blanks...The rpi runs off its SD card and has 2 USB HDDs attached (as mentioned above). The rsync is run from the Pi. I have a script on the Pi called backup-files and it looks like this (names changed to protect the innocent)
Quote:
rsync -r -a -v -t /mnt/from1 /mnt/two1/
rsync -r -a -v -t /mnt/from2 /mnt/two2/
rsync -r -a -v -t /mnt/from3 /mnt/two3/
rsync -r -a -v -t /mnt/from4 /mnt/two4/
rsync -r -a -v -t /mnt/from5 /mnt/two5/
rsync -r -a -v -t /mnt/from6 /mnt/two6/
rsync -r -a -v -t /mnt/from7 /mnt/two7/
rsync -r -a -v -t /mnt/from8 /mnt/two8/
rsync -r -a -v -t /mnt/from9 /mnt/two9/
crontab -l shows
Quote:
* 5 * * * /home/pi/backup-files
0 23 * * 0-1,3-4 /home/pi/scripts/music
The music backup should take a matter of minutes so I am not expecting any contention but ya never know.

Not marking it as solved then as I don't really think I solved anything but, at this point, no longer an issue.

Cheers
 
Old 07-26-2017, 08:21 PM   #9
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,873

Rep: Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600Reputation: 2600
Code:
* 5 * * * /home/pi/backup-files
Isn't that every minute from 05:00-05:59 ?
Did you mean '0 5 * * *' ?
 
1 members found this post helpful.
Old 07-26-2017, 08:28 PM   #10
timl
Member
 
Registered: Jan 2009
Location: Sydney, Australia
Distribution: Fedora,CentOS
Posts: 711

Original Poster
Rep: Reputation: 152Reputation: 152
You are right. Why didn't I spot that! I'll correct ASAP

[EDIT] Okay, I get it now. I guess I was being lucky before with that line. The job was being submitted every minute. Not sure of the implications of multiple very similar rsyncs when there is a bit of volume to copy.

When I made the change mentioned above I changed the time to include a minute value so the problem was masked. I'll return to 5am and see if this is okay. Then I'll mark this as solved.

Last edited by timl; 07-26-2017 at 11:25 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync not running in cron job; please help Ravishankarappa Linux - Newbie 8 08-17-2011 12:11 PM
Who runs a cron job? fredrated Linux - General 4 12-23-2010 11:06 PM
Cannot execute rsync as a cron job jdaniels73 Linux - Software 2 09-03-2006 05:03 AM
cron job only runs halfway aunquarra Linux - General 15 01-31-2005 04:53 PM
Why Does One Cron Job Runs Twice? cpope67 Programming 4 11-16-2004 06:09 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:57 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration