Hi, I recently bought 2*5TB USB hard drives which I have attached to my Raspberry Pi 1 with 512M of memory. The Pi serves as a bit torrent server. I needed a bit more room for storage and saw these 2 items going cheap. A bit bigger than I need at the moment but should keep me going for a while - one is the active disk and the other is the backup.

Having populated both drives I brought them online yesterday. I have a cron job set up to invoke a script which backs up the active drive onto the backup.
There are 9 individual folders which are backed up and the script simply submits the rsync jobs one after the other. I ran into a problem with the cron job where I got these messages:
That looks like I am out of memory! However the script runs fine from the command line. This worked fine on my old 2TB drives so I am guessing it is related to the drive size but I have no ideas what. Any suggestions?

Cheers

It is a bit unclear for me: where is this rsync command executed? on rpi? Did you check the memory you have somehow? Did you try to add some swap space? Was there any other app running that time?

You might want to post your script in it's entirety.
When a script runs fine from the command line but fails when run as a cron job, it's usually a PATH problem. The cron environment is different than your usual command line environment. Relative paths that run fine from the command line often fail in cron. Try revising your script to give full paths.

In addition to what pan64 posted, typically "rsyslogd was HUPed" is indicative of the logs being rotated. Looks like your running your job from cron.daily or a cron job that happens at the same time. Typically rsync reads the entire directory structure and stores it in memory so as suggested this might cause a memory issue if other stuff is running at the same time. Since it works running from command line try setting up a cron job after the cron.daily jobs are complete or earlier and so that it completes before the daily jobs start. Adding swap space should help too.

I would add sync each folder one at a time in the script that should cut down on the memory used as each new process will allow the releasing of the old completed ones memory.

Note that those "HUP" messages were from the day before. The "run-parts" script (in the Red Hat family, at least) runs each of the scheduled jobs in sequence, waiting for each to complete before starting the next, so there should be no competition if all the jobs are being run from cron.daily. Of course if if you have your own cron job that happens to run at the same time as cron.daily, that's another matter.

Your correct. Trying to fill in some missing holes in the data...

Sorry for not responding earlier. The problem is not solved but I made it go away. I simply changed the time of the cron job from 5am to 6pm and it has run fine ever since. I note that cron.daily runs at 6:25am so should not have affected a 5am start but this is what prompted me to make the change.

For the purposes of documentation and to fill in a few blanks...The rpi runs off its SD card and has 2 USB HDDs attached (as mentioned above). The rsync is run from the Pi. I have a script on the Pi called backup-files and it looks like this (names changed to protect the innocent)
crontab -l shows
The music backup should take a matter of minutes so I am not expecting any contention but ya never know.

Not marking it as solved then as I don't really think I solved anything but, at this point, no longer an issue.

Cheers

Isn't that every minute from 05:00-05:59 ?
Did you mean '0 5 * * *' ?

1 members found this post helpful.

You are right. Why didn't I spot that! I'll correct ASAP

[EDIT] Okay, I get it now. I guess I was being lucky before with that line. The job was being submitted every minute. Not sure of the implications of multiple very similar rsyncs when there is a bit of volume to copy.

When I made the change mentioned above I changed the time to include a minute value so the problem was masked. I'll return to 5am and see if this is okay. Then I'll mark this as solved.