I'm having a problem doing rsync to another machine. Here's the setup...

I've got my first machine with all of my data. I've got a second machine with a 160gig drive mounted as a slave. I've got the 160gig mounted over NFS to the first machine. So what I'm doing is... every night via cron, I'm running rsync on the first machine and it's copying data to the 160gig drive that is mounted on the first via NFS.

When I run it normally during the week, there is not a lot of data being moved. Usually everything works fine. However, on the weekends I'm doing a bigger backup and its throwing kernel errors on the machine that holds the 160gig drive. When the error happens, it completely freezes the backup machine and I have to reboot to get everything back to normal. Thankfully the cron is just paused while the backup machine is frozen and when it comes back online the backup continues to run. So the backups are doing good, but I still hate rebooting and I'm sure the kernel errors need to be fixed.

When I boot up, I have been running fsck on the 160gig and the check seems to pass without any problems.

Here's the error...

This is all that shows up in /var/log/messages. This message is also echo'd at the terminal and it ends with...

Note that I'm paraphrasing the above error because it didn't actually show in the logs, but it shows on the terminal.

My kernel is 2.4.22-1.2197 nptl and my OS is FC1.

Any ideas where to begin? Should I join the kernel mailing list and post it there? I hesitate to do that because I have done similar stuff in the past and unless you research your problem very well before posting, usually the list members will chew you a new arse, so I think I need to get my ducks in a row before I go posting on some list.

Thanks in advance.

From the paging request error message it looks like the kernel might be having trouble finding enough memory to complete the copy operation. This is particularly interesting since create_empty_buffers is the top entry on your call trace. How much RAM do you have on the machine and how much swap? Maybe you could try running a big rsync load and watch free and vmstat to see if your taxing your memory beyond what it can take.

Hey! Excellent ideas! I will certainly try that.

Here's what I've got...

I have a single stick of 512MB PC-133 (32x8) 64X64. The item lists as "Generic Low Density". When I bootup, I get this in dmesg:

479MB LOWMEM available.
Memory: 481904k/491456k available (1482k kernel code, 9164k reserved, 1110k data, 136k init, 0k highmem)

Here's my "free" output...

As you can see, I've got a little under a gig of swap allocated and nearly none of it being used. I'm basically not doing much at all on the system. rsync is running on the sending machine and this machine is just doing the receiving, so it isn't working too hard. ;)

Here's my dmesg. Doesn't seem to be anything that looks bad (out of the ordinary) but maybe you'll see something I don't.

Thanks for the ideas, and I will be happy to post anything you think may be of help.

Sheesh that was confusing. I thought this was a listing from my own box for a minute and got toatally confufuse... almost same VIA chipset (I have 686a) and my drives are:

hda: IC35L040AVVN07-0, ATA DISK drive
hdb: WDC WD800JB-00FMA0, ATA DISK drive

almost the same as well!!! Well, at a quick glance

Very confusiong for me Sorry, nothing on the problem tho Have you ever had any trouble with the IBM drive? I've never fully trusted mine and it makes some funny noises and seems to reset itself when it's working a lot.... Probably unrelated to this problem tho...

Well, I don't know. You see, originally this drive came as a USB device, but I got sick of running like that and took the drive out of the case (rather easily too) and mounted it as a slave in one of my machines for a long time. During that time it was used as a backup drive and its worked pretty good for that. However, now I have removed all the backups, formatted to ext3 and loaded an OS and now it's my master drive in another machine. So I really haven't had that much experience with this drive while running an OS on it, but I can tell you that mine has never made any noises, and in fact runs pretty quiet.

The 160gig I picked up at Sams for a decent price (I think, anyway) and it's doing fine also, but again, using it as a backup drive, so don't know how well it would perform with an OS on it.

My motherboard is a stanky old ASRock mobo that carried an Athlon chip. I say stanky because it doesn't have an AGP slot and has only 2 measly PCI slots, but thankfully there is a lot of stuff on the board itself like sound and video. For a backup server it's more than adequate and didn't cost me a lot. But anyway, I digress.

Amazing coincidence.

I have OS spread over both and use heavy (tho the WD is new) and can't really complain. Just nerves maybe as I got the IBM for free (sealed in a bag tho). It runs quiet too, except on rare occassions it goes CHWEEEAPWEEEAP-CLICK CHWEEEAPWEEEAP-CLICK ... then quiet again.

I hold my breath a moment, then continue on working

Anyway, better get back to the real issue here...

Just wanted to give an update on my problem here.

I rearranged my backups from "push" type to "pull" type. So now everything seems to be OK, as far as I can tell.

Now I am doing it so that the source computers (containing the data that needs to be backed up) are being exported through /etc/exports and the backup machine is pulling the data from the remote machines to the local machine.

During the runs, the destination machine runs about 80% CPU and never uses more than the RAM I have installed. It uses all the RAM I have available but never taps into the swap space... it always remains zero. On the source machines, there are about 6 or 8 NFS daemons that are using zero memory but each of them uses about less than 1% CPU.

No more kernel errors, but I'm going to give it another week and see what happens.