I don't know the rules about linking to other forum threads, but I posted this on the Scientific Linux forums and haven't had much in the way of responses, so I thought I'd try a more Linux-generic forum with more users. Rather than re-typing or copy/pasting everything, I figured I'd just link to it:

http://scientificlinuxforum.org/inde...=ST&f=9&t=2597

I've provided most of my computer information there, but if additional information is needed let me know. The computer I'm working with is not on the network and there's no USB interfaces on it, so pasting command output results, if requested, will be difficult.

Fast forward, the issue I'm currently having (using the old version of Fedora) is this:

Computer 1 Arp Request to computer 2.
Computer 2 receives arp request.
Computer 2 sends arp reply.
Computer 1 never receives ARP reply.

When I had Scientific linux on the computer, it was communicating fine, but appeared to have a NIC limitation (would reliably communicate with approx 20 out of 24 NICs). I updated to SL previously because Fedora was no longer communicating with those NICs, and I thought it was an issue with communicating with newer link partners (protocols?).

Anyway, any help you can provide would be appreciated.

Too many interfaces. It could exceed NIC driver limit. You should check dmesg output whether any error in there.
By the way, does computer 1 or 2 have a lot of interface?

Computer 2 has 6 PCI cards with 4 fiber optic NICs each for a total of 24 interfaces.

How can I find out the driver limit?

My concern though is how with Fedora 2, it could handle that number of interfaces, but with Scitentific Linux 6, it seems that it can't. So, I reimaged Computer 2 back to Fedora because at least all of the interfaces are receiving data, where with Scientific Linux, even though it would receive AND send, but seems to run into a limit of 20 interfaces.

The only thing that changed with the Fedora setup is that the "Computer 1s" are newer devices than what the setup was working with before and I don't know if there is some sort of protocol mismatch now.

Just a little while ago, I noticed that TCPDUMP on Computer 2 was seeing a message: along the lines of "ethertype Unknow (0x88cc). Googling it tells me that it's something to do with Link Layer Discovery Protocol. If this is what's causing my issue, I don't know where to begin trying to fix it.

How can I find out the driver limit?
1)Check dmesg output. Normally, NIC driver should report something when exceeding its limit.
2)Unplug 1 or more PCI card, whether issue is gone. You should find out how many interface Fedora 2 support?

Can you explain why you have to use Fedora 2? Fedora 2 is very old, now is Fedora 20. New Fedora could fix the issue.

The dmesg output shows that "pcnet32: 24 cards found" with no errors indicated both on Scientific Linux and Fedora 2

To give a back story on Fedora this: This system was set up years ago, approximately 2004 when Fedora was at 2. The previous person responsible for this system had done just what you indicated, moving/removing cards to see how many interfaces Fedora could support and was able to get the system working with 24. For the system we've set up we actually needed 48 interfaces, but since only 24 could be supported we ended up with 2 identical computers, 24 ports each even though we had enough PCI slots for all 12 PCI cards (12x4 each).

This system is used to test optical networking switches in various environmental situations, which is why we're using so many individual interfaces.

The system worked with the optical network switches we were connecting to in the past. The only thing that changed is that we're connecting to a newer model switch. When we did that, we started seeing our communication issue.

Fast forward, I tried newer versions of Linux via live CDs and found that they would communicate, but don't appear to be able to handle the same number of interfaces.

It appears that something in the protocol has changed which makes our Fedora network setup incompatible, and with our Scientific Linux version it's compatible, but can't handle the number of interfaces. So, I have one system that can handle the number, but not the communication, and the other, vice versa.

What I'd like to do is either figure out how to get Scientific Linux to handle the number of interfaces, or figure out how to get Fedora to communicate with newer fiber optic link partners.

By the way, the network interface hardware is a Ramix (GE Fanuc?) 4 port fiber optic networking PMC card in a PCI carrier.

I may just go ahead and get a hold of a copy of the latest version of Fedora, or whichever is compatible with my hardware and give it a shot to see what happens. Meanwhile, if there is anything else I can try, please let me know.

Another way is a little hard. Newer Fedora can communicate with newer model switch. That means that new driver can talk to newer switch. If you can get Fedora 2 kernel and new driver source code, new driver can be built against Fedora 2 and then is uploaded to your system. It should solve everything.
If above way can't be done, using newer Fedora is right direction. Yes, you need resolve new issue.

1 members found this post helpful.

The newest version of Fedora that appears to be compatible with the hardware I'm running (RAM, etc.) is Fedora 14. I'm going to give that a shot on Monday and go from there. If that doesn't work, I'll try recompiling the kernel with a newer version of the driver. I haven't recompiled a kernel in a loooooooong time, so we'll see how much I remember. I'll be sure to post results.

Thanks for the advice thus far!

I've installed Fedora 14 on one of the two computers and after some configuration, it appears to be working as needed/expected. I'm installing on the second computer as I'm typing this.

I'm still wondering why Scientific Linux appears to have a limitation with the number of NICs.

Something I didn't try with SL6 was to install the full DVD (I installed it from a LiveCD), whereas with Fedora 14, I downloaded the entire DVD. I wouldn't think that would make much of a difference, but I could be wrong.

Fedora 14 up on both computers and is working as needed. Thanks all for pointing me in the right direction. I don't know if I would consider this to be [SOLVED] as I wasn't able to fix the original issues with the original distributions, but at least my problem has been resolved (and my bosses will be happy).

Thanks again!

Each kernel and driver have limitation. Normally, they don't mention because 99% case just use maximum 4 or 5 NICs.
Newer always is better. That's why every provider keep updating their software.

Are you not just simply running out of irqs?

http://docs.oracle.com/cd/E24290_01/...e_perftune.htm

See 6.1.6

I had thought that was an issue earlier on and looked at the IRQ/PIRQ/APIC stuff and did some tweaking, but noted no change in behavior.

That is peculiar. Why would a system digress after so many years. Have you confitrmed irq's being avalable. Have you changed hardware. Big difference between consumer and workstation class.

I wondered the same thing. No hardware on the computers had changed. The only hardware change was what the NICs were connected to. I'm thinking some newer protocol or something was preventing communication. The NICs are fiber-optic and circa 2004 or so, the fiber connections on the other end were manufactured in 2012.

As I said, I checked the IRQs and everything and did some tweaking. Nothing looked unusual and no tweaks changed the behavior.