Swapping hard drives to another chassis

Siper2 · 09-09-2013, 02:20 PM

Hey all.

I recently posted this question over on FedoraForum, but as yet I've not had any feedback. Thought I'd try here, as well. For simplicity's sake, I'll just cut and paste:

Quote:

Hello all. I have a question that's driving me a little crazy, so this is the place to be!

I am working with some IBM x3500 Blade servers running FC6. Long story short, I ran in to a problem at a remote site last week. Ended up getting it fixed, but one of the issues I ran across was that I do not seem able to swap hard drives from the previous/production chassis to another one (if there's a hardware problem), and have it come online.

INFO: All of them run the exact same OS, FC6. We have one proprietary application to connect to our terminals. All of the servers are identical, hardware-wise. We run a RAID1 and are using IBM/Adaptec ServeRAID 8K cards for controllers/management. They all have 192.168.x.x IPs on a 255.255.255.0 subnet.

So... I went to swap the two drives from the old chassis to the new. I kept them in the same order (SCSI 1, 0 left to right). When I booted up the new chassis using the old drives, Fedora started *however* during the network initialization, the IP process failed (right after the successful loopback test). I had made no changes to the IP addressing prior to shutting down the previous chassis and pulling the drives.

It boots up, I log in to command line and discover that the /etc/sysconfig ifcfg-eth0 IP is gone (we only use one port, so eth0). No IPADDR entry nor SUBNET. They can be manually entered via "vi" at which point, if I recall correctly, once the network service was stopped/started I was able to ping across the LAN, however I was not able to get the system fully working. We connect via PCI cards to serial cables that connect to end user equipment. The configuration information was not coming across, which is actually communicated to the server via an off-site server on our WAN.

A LITTLE MORE INFO: We have some older servers running Debian, also IBM Blade servers but an older version that use ServeRAID 7K controllers. Drive swaps in those have always worked just fine. During boot, when the RAID is initializing, after a drive swap the controller recognizes the drives and prompts to "accept the configuration" from the new drives. Do so, and it boots up and POOF everything works.

Well, with the server with the problem, as I said we use ServeRAID 8K. At boot, when drives are swapped there is NO prompt for configuration acceptance, it just boots up.

I was initially thinking that it might be a problem with the controller, but I'm not sure about that. To me it seems more like a problem with the IP configuration binding to the MAC address of the eth0 port, and then when the drives are swapped, it does not work. However what is different about Debian that would allow this to work, when Fedora does not? That may be a pivotal question, if I'm going about this the right way.

Essentially, we SHOULD be able to swap chassis, maybe with an extra step or two... I just don't know what those are. Yet.

Thanks! I just know this is going to be a facepalm moment... something stupid that I'm not aware of yet...

Two unavoidable things:

A moderator on FedoraForum pointed out was that Fedora should not be used on a production server. I agree... sort of... but this was not my call. This is several separate LANs of a dozen or more servers apiece, all over the service area I drive to. It's unavoidable.

Also, yes, they run FC6. I realize that we went beyond support for FC6 in about 2007. This server type was installed in 2007-2008 initially, and for our purposes we had to remain constant. We will not be upgrading. It is what it is. These servers run one proprietary application, that's it.

So that's out of the way...

Any advice? Since I posted the above, my efforts in troubleshooting have rather fallen flat. The most helpful documents I found are here:

http://zenit.senecac.on.ca/wiki/inde...sses_on_Fedora

This unfortunately does not work. We don't use Gnome/GUI (console only) for our purposes, but even under the system-network-config command line utility, there is no option for unbinding a MAC address associated to an ethernet port.

https://bugzilla.redhat.com/show_bug...iple&id=212869

That bug seems to comment on the file /etc/udev/rules.d/70-persistent-net.rules not being present in older Fedora releases. Looks like maybe it started with FC8, not sure. Maybe relevant, maybe not...

I've tried changing the HWADDR settings under /etc/sysconfig/network-scripts/ifg-eth0 (we only use the one port), but no dice. Yet.

This is a tricky one. Sorry if I got confusing, but any help would be grand.

-S2-

Siper2 · 09-09-2013, 02:26 PM

One thing: I did mention that our older servers, running Debian work fine. Swap the drives into the other chassis, accept the drive configuration in ServeRAID, and the OS loads and that's it. One thing I'd really love to know is *why* it works there, but not Fedora.... could be helpful in troubleshooting this.

http://forums.debian.net/viewtopic.php?f=7&t=54894

The third response in that thread is long and has a few key points, but nothing that goes so far as to give a real answer, to me at least.

jefro · 09-09-2013, 02:43 PM

Don't post to your own post. Makes it look like someone answered it.

Yep, I have seen Fedora do that. I forgot now what the issue was and the fix. I just ended up reloading it. From what I recall it was still trying to load up the other nic's. It was actually calling the new nics (supposed to be the same hardware) by new names like eth5 and eth6. There is a simple sort of fix out there but not sure what it was.

Siper2 · 09-09-2013, 02:53 PM

A simple fix is what I'm hoping for.....I've figured for the past few days that it would be something simple-ish, at least. I did notice that after the drives were swapped, the eth0 and eth1 addresses changed, or went like you said to eth1 and eth2 or something like that. So I'm almost wondering if it IS a RAID controller configuration that has to be set? Only trouble is that these ServeRAID 8K modules don't have any kind of acceptance prompt when a new drive pair is inserted. It just starts to boot normally.

IBM servers have a CTRL+A option at boot during the RAID initialization phase, but that's just to modify or delete the existing RAID. I don't think it has a mini-configuration option anymore.

As for posting to my post, sorry I'm not quite sure what you mean - the quoted section perhaps?

TenTenths · 09-10-2013, 02:23 AM

Have you considered that in your original networking config you have directives to bind eth0/eth1/ethX to specific MAC addresses?

If the MAC address isn't seen in the new backplane then maybe your OS is assuming that NIC has gone and disables the config for it.

We get something similar when we "clone" VMs in XenCenter which is what gave me the idea to suggest it.

Siper2 · 09-10-2013, 04:54 AM

That's sort of what I think is going on, but I do not know how to fix it.

Siper2 · 09-12-2013, 02:59 PM

One thing I've noticed is that when the drives are initially swapped to the other chassis, the MAC addresses swap. eth0 becomes eth1 and vice-versa.

Very, very peculiar. I simply cannot figure this out!

jefro · 09-12-2013, 07:36 PM

I've only seen this issue on Fedora. I did find the answer after looking for it but I forgot what it was. There is a solution out there but it isn't exactly worded easy to find as I recall. It is one of those things that I said that I will remember and then forgot.

Siper2 · 09-12-2013, 07:49 PM

Well it happens on RHeL too, I've seen it firsthand. Curiously, with that it kind of resolved itself after a number of reboots.

I think it's got to be something related to just changing the MAC. Post-swap perhaps since the OS seems to pick up on the hardware once it's installed (as designed).