my setup is to PXE boot from eth0 and the OS (ubuntu 10.04) is loaded via NFS3. Each machine has four NICS. Is it possible to have static IP's on eth1-3 on the PXE booted machine if several machines are using the same image?

Someone in IRC suggested using udev rules to specify MAC<>ifname mappings, and using interface names in /etc/network/interfaces to configure them per-node.

What I need is to be able to have these machines boot off the same image and have different IP's on eth1-3

You have several issues...

You could use the same generic image to clone for each client, but I don't know if will work for the clients after setup... Each machine needs to be separately identifiable on the network... And each will have unique MACs, hardware issues, etc.

The boot device will be configured in the BIOS, and all others will have a unique MAC address (independent of one another), meaning your udev rules will have quite a lot of parameters to sort that out... unique for each machine. The only thing I can think of that would work would be to write the udev rules to set the udev namings based on the numerical order of the MAC addresses, for consistency. That doesn't even address IP assignment. Normally that is handled by a DHCP server. Whatever they are interfacing to will have unique MAC addresses too. So, there's no difference in accessing specific devices over the network using more standard topology, they can be called out by their MAC addresses.

It really depends on what you are trying to accomplish with this setup... Is it a branching/distributed network? Is it a network of machines that interfaces with some kind of connection to Ethernet interfaced devices? Are you setting up these machines as some kind of smart switches? It really depends on what you are trying to do, in terms of how you approach it.

I get the feeling that what you are doing is overkill. I can't understand why you would have client machines with 4 NICs each, when a few $15 switches, and a centralized DHCP server would likely accomplish the same thing. Most Ethernet interface devices don't care how or from where they are controlled over the network.

Might take a script into the image maybe include machine info to make selections or mac addresses to assign.

I get the feeling there is a way to also do that at boot time or via pxe.

it's a NFS diskless cluster. The head node has the image and the compute nodes PXE boot then pull the image via NFS. All compute nodes use the same shared NFS image so if I make a change on one compute node, it affects all compute nodes.

I do know that each node is assigned a unique host identifier after it's booted. Can something like that be done using mac addresses?

someone in IRC suggested using udev rules to specify MAC<>ifname mappings, and using interface names in /etc/network/interfaces to configure them per-node but I'm not sure if this would have to be done post startup/boot.

It occurs to me that dhcp has the ability to specify a fixed IP-address based on the MAC address of the inquiring machine. (Maybe an off-the-shelf router can do it!) In this case, you could simply set up each Linux box to ask for its address from dhcp, knowing in advance which server it will ask and what its answer will always be. Q.E.D.