Reload kernel module without a reboot

Phaethar · 10-08-2013, 08:39 AM

Hey all,

I'm having some trouble with a file server along with some replication software, and I'm looking for a way to get things working again without rebooting the server, as it's in production.

First, a quick description of the setup. The file server is running RHEL 5. The OS is local, and is on an ext3 file system. The storage array is external and connected via SAS, and is an ext4 file system on /dev/sdb. Looks like this:

Code:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdb on /Share type ext4 (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

We run some real time bi-directional replication software on this system, as well as a backup file server at another location. It's been working great for the past couple of years, up until this weekend when we had to reboot the file server. When the file server was restarted, it did not mount /dev/sdb automatically. Instead, I manually issued a mount command as soon as the system was up. This is to prevent conflicts, as there is also a backup file server also connected to the storage array via SAS, but only 1 can be connected at a time. End result is we can't have the system set to auto-mount the array.

The issue now is that the real-time replication portion is not working. I can manually sync folders, so the software is working and able to see both servers. The system that was not rebooted at the remote location continues to work in real time and push changes back here. But I can't get the system that was restarted to track changes in real time. I've worked with support for the software company, and they tell me it's the kernel module that's loaded at boot that's causing the issue. They're telling me that the file system needs to be automatically mounted (via fstab) when the system starts up in order for changes to be tracked. So, while that's easy enough to account for in the future, it doesn't help much now.

What I need to know is this: Can I someone load or reload the required kernel module on a live file server in order for the system to start tracking file system changes?

Also, in case it helps (or anyone else uses the software), we're using a program called Netvault Replicator. I'm trying to get the module they use, but I'm not sure if the issue here is their module, or if it's a system module that their software just taps into. I do see this listing in the lsmod output:

Code:

nvr_dr                306116  2 [permanent]

Most of the Netvault Replicator stuff is preceded by nvr, so I thought this might be theirs as well. I'm waiting to hear back from them on whether or not it is, and if I can somehow reload it. In the meantime, any other input would really be appreciated.

Thanks!

TB0ne · 10-08-2013, 09:21 AM

Quote:

Originally Posted by Phaethar

Hey all,
I'm having some trouble with a file server along with some replication software, and I'm looking for a way to get things working again without rebooting the server, as it's in production.

First, a quick description of the setup. The file server is running RHEL 5. The OS is local, and is on an ext3 file system. The storage array is external and connected via SAS, and is an ext4 file system on /dev/sdb. Looks like this:

Code:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdb on /Share type ext4 (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

We run some real time bi-directional replication software on this system, as well as a backup file server at another location. It's been working great for the past couple of years, up until this weekend when we had to reboot the file server. When the file server was restarted, it did not mount /dev/sdb automatically. Instead, I manually issued a mount command as soon as the system was up. This is to prevent conflicts, as there is also a backup file server also connected to the storage array via SAS, but only 1 can be connected at a time. End result is we can't have the system set to auto-mount the array.

The issue now is that the real-time replication portion is not working. I can manually sync folders, so the software is working and able to see both servers. The system that was not rebooted at the remote location continues to work in real time and push changes back here. But I can't get the system that was restarted to track changes in real time. I've worked with support for the software company, and they tell me it's the kernel module that's loaded at boot that's causing the issue. They're telling me that the file system needs to be automatically mounted (via fstab) when the system starts up in order for changes to be tracked. So, while that's easy enough to account for in the future, it doesn't help much now.

What I need to know is this: Can I someone load or reload the required kernel module on a live file server in order for the system to start tracking file system changes?

Also, in case it helps (or anyone else uses the software), we're using a program called Netvault Replicator. I'm trying to get the module they use, but I'm not sure if the issue here is their module, or if it's a system module that their software just taps into. I do see this listing in the lsmod output:

Code:

nvr_dr                306116  2 [permanent]

Most of the Netvault Replicator stuff is preceded by nvr, so I thought this might be theirs as well. I'm waiting to hear back from them on whether or not it is, and if I can somehow reload it. In the meantime, any other input would really be appreciated.

Take a look at the modprobe command. Doing something like "modprobe -r <module name>" will remove (unload) the module, and "modprobe <module name>" will reload it. Which MAY (or may not) be a good thing, especially when it comes to disk access. Proceed with caution, but I'd reboot the server instead.

And Red Hat support can help you with such things, since you're using RHEL and paying for a subscription. Since you're using Netvault and RHEL, and have logged a support case, that would be the quickest and safest thing to do.

voleg · 10-08-2013, 09:48 AM

You can't reload module until it is in use. According to your lsmod
this module is in use (number 2 say that). I assume that it is a
service ov Netvault, then you have to stop service. If you'll get
down time for service, you can reboot all server for sure.

Phaethar · 10-08-2013, 10:01 AM

Quote:

Originally Posted by voleg

You can't reload module until it is in use. According to your lsmod
this module is in use (number 2 say that). I assume that it is a
service ov Netvault, then you have to stop service. If you'll get
down time for service, you can reboot all server for sure.

Voleg, that's correct, the module is in use, as the service (nvr_replicator) is currently running. It's running so I can manually sync files from this server to the remote backup server. However, I can stop the service in order to reload the module if need be, as it's not doing anything in real time anyway. Stopping the service changes the 2 to a 0 (just tried it).

TB0ne, I'm working with the developer right now on determining whether or not the module can be reloaded. They didn't know for sure, so I figured I'd start doing some checking on my own. If I can reload it, I was planning on using modprobe to do it.

Something odd about the module. It shows up in lsmod as nvr_dr. However, I'm not able to get any information on the module:

Code:

# modinfo nvr_dr
modinfo: could not find module nvr_dr

TB0ne · 10-08-2013, 10:31 AM

Quote:

Originally Posted by Phaethar

Voleg, that's correct, the module is in use, as the service (nvr_replicator) is currently running. It's running so I can manually sync files from this server to the remote backup server. However, I can stop the service in order to reload the module if need be, as it's not doing anything in real time anyway. Stopping the service changes the 2 to a 0 (just tried it).

TB0ne, I'm working with the developer right now on determining whether or not the module can be reloaded. They didn't know for sure, so I figured I'd start doing some checking on my own. If I can reload it, I was planning on using modprobe to do it.

Something odd about the module. It shows up in lsmod as nvr_dr. However, I'm not able to get any information on the module:

Code:

# modinfo nvr_dr
modinfo: could not find module nvr_dr

The odd part lies in the modinfo command. modinfo fetchs the list of known modules by reading the /lib/modules/$(uname -r)/modules.* files, which are usually updated with depmod. If depmod -a has not been run after installing the module in question, then modinfo knows nothing about it. This does not prevent anybody from loading the module with insmod and lsmod will show it just fine if loaded.

This isn't necessarily a PROBLEM...just odd. Honestly, though....when it comes to disk access and other goodies (ESPECIALLY since you're doing replication), I'd not risk it at all. Schedule downtime and reboot, to be safe.