[SOLVED] How to fix error: "bind: Cannot assign requested address"

Azazwa · 06-18-2010, 06:22 AM

Hi!

I have a small cluster (OSCAR, Fedora 8) and I was able to run some application software on it. Then lightning struck very close to the building. Fortunately I had unplugged all the power cables (because the cluster has not yet been moved to where the power lines are protected), but it seems that the institution didn't have any protection on their LAN cables, and so the whole building's public network cards are damaged. A costly lesson.

Anyway, when I tried to run the application software in parallel across the cluster (using the private network which is unscathed) I get the error message given in the subject line. I contacted the application software's help department as I thought I had perhaps forgotten to set something, but according to them it is a normal network problem. They gave some suggestions as to what the problem may be, but I have checked it and it doesn't cure the problem.

I have included it here so that you don't waste time by suggesting the same things.

Quote:

Check the /etc/hosts file and make sure that the nodes all have a
single definition and you don't have lines like

127.0.0.1 localhost normnode3

and that normnode3 has the same address both on the master and on the
node.

You can try

ping normnode3

from the master and see what address comes back

64 bytes from 164.190.57.105: icmp_seq=1 ttl=64 time=0.306 ms

or is it 127.0.0.1. Then do the reverse.

Also double check that you can ssh between nodes without password
but I would expect a different error then.

The command "hostname" returns gnlserv01, which is the public NIC.

After the lightning I had trouble getting the nodes to communicate "automatically" with each other, but it can be cured by starting the xinetd service and disabling the firewall on the master node (it's not too dangerous since I don't have a public interface at present and since I'm sitting behind the institution's firewall as well.) Just by the way, I would think that ther should be a file somewhere in which I could specify those two commands to take place when the master node is switched on. Could you perhaps enlighten me as to where and how I could specify it?

I was wondering whether I would need to explicitly start a bind-type service or something like that? (Since I had to explicitly start xinetd)

I'm rather clueless really. I googled around and found that there is a named service, so I tried to start it, but I don't think it's installed on the computer. Therefore, since I have managed to run the application software in parallel previously, the named service is probably not the problem. Hm?

Here is a copy of how my /etc/hosts file looks like:

Code:

# Do not remove the following line, or various programs
# that require network functionality will fail.

# These entries are managed by SIS, please don't modify them.
127.0.0.1       localhost.localdomain   localhost
192.168.1.254   snode0.oscardomain.za   snode0  oscar_server    nfs_oscar       pbs_oscar
abc.xyz.104.218 gnlserv01.ab.cx.yz      gnlserv01
192.168.1.1     normnode1.ab.cx.yz      normnode1
192.168.1.2     normnode2.ab.cx.yz      normnode2
192.168.1.3     normnode3.ab.cx.yz      normnode3

Here is the output of ifconfig -a:

Code:

[compchem@gnlserv01 /root]$ ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:1C:C0:AF:10:18  
          inet addr:192.168.1.254  Bcast:192.168.255.255  Mask:255.255.0.0
          inet6 addr: fe80::21c:c0ff:feaf:1018/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2587 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3109 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:332943 (325.1 KiB)  TX bytes:409521 (399.9 KiB)
          Base address:0x20c0 Memory:e0300000-e0320000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:5009 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5009 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3813184 (3.6 MiB)  TX bytes:3813184 (3.6 MiB)

I would really appreciate suggestions and comments!

;-)

MensaWater · 06-18-2010, 08:18 AM

I don't think iptables or xinetd would be dependent on which node of the cluster is up.

In RedHat style distros like Fedora you can set what items come up automatically at boot with the chkconfig command.

If you run "chkconfig --list xinetd" you should see something like:

Code:

xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off

This shows that xinetd is configure to start automatically in run levels 3, 4 and 5 and stop (or NOT start) in run levels 0, 1, and 6.

You can change which run levels to start it automatically in with the chkconfig --levels. For example to tell it to also start in run level 6 you would type: "chkconfig --levels 6 xinetd on".

For iptables you simply substitute iptables for xinetd in the above commands.

Note that the message you got may be more related to the port it is trying to use than the interface. Ports for services are defined in /etc/services (sometimes - some people don't update this file when they start using new ports and random ports aren't in it).

xinetd is used to determine what daemons should be started when connected to by listening on the ports for the daemons. Without it you'd have to have the daemons running themselves all the time and listening for themselves which could tie up system resources unnecessarily. (e.g. if you had vsftpd and someone only ftp's to you once a day why have that daemon up 24 hours?) By listening again we're talking about ports. If you look in /etc/xinetd.d you should things for the various daemons and each of those will show you which port that particular daemon uses. I'm guessing your issue is that you're using a cluster port (or maybe some other) that isn't being listened for by xinetd until you start it.
Note: Not all the files in xinetd.d are enabled - make sure you check to see which ones are as those are the relevant ones for you.

iptables is used as a firewall (and for other purposes like NATting). Since it is often used to block everything EXCEPT what is allowed by it then the fact you have to stop it suggests that it does not contain a line allowing the port that is being listened for by whichever daemon xinetd has configured. (Or possibly not allowing it on a specific NIC.) You can modify the iptables settings to allow the port once you identify it.

Azazwa · 07-22-2010, 07:42 AM

Hi!

My apologies for my long silence.

Thanks for your advice so far MensaWater. Perhaps giving some of the logs before the original error will help.

Code:

ntsnet: starting master process on gnlserv01.zz.zz.zz
/soft/gaussianBinaries/g03/linda7.1/opteron-linux-I8/bin/linda_sh /soft/gaussianBinaries/g03/linda-exe/l302.exel 0 water.chk 1 /scratch/compchem/Gau-4534.int 0 /scratch/compchem/Gau-4534.rwf 0 /scratch/compchem/Gau-4534.d2e 0 /scratch/compchem/Gau-4534.scr 0 /scratch/compchem/Gau-4533.inp 0 junk.out 0 +LARGS 2 0 -kainterval 1 -master 48667 -tsnetport 21028 -maxworkers 1 -minworkers 1 -minwait 600 -maxwait 600 -nodename gnlserv01.zz.zz.zz -kaon
ntsnet: starting 1 worker on normnode3.zz.zz.zz
/soft/gaussianBinaries/g03/linda7.1/opteron-linux-I8/bin/linda_rsh normnode3.zz.zz.zz -r ssh /soft/gaussianBinaries/g03/linda-exe/l302.exel 0 water.chk 1 /scratch/compchem/Gau-4534.int 0 /scratch/compchem/Gau-4534.rwf 0 /scratch/compchem/Gau-4534.d2e 0 /scratch/compchem/Gau-4534.scr 0 /scratch/compchem/Gau-4533.inp 0 junk.out 0 +LARGS 2 1 -maxworkers 1 -chdir /home/compchem/gaussianTests -worker gnlserv01.zz.zz.zz:48667 -workerwait 900 -tsnetref 1 -nodename normnode3.zz.zz.zz
ntsnet: exec'ing /soft/gaussianBinaries/g03/linda7.1/opteron-linux-I8/bin/LindaLauncher
bind: Cannot assign requested address

Okay, so from the above it seems that ports 48667 and 21028 are used. I looked in the /etc/services file as you suggested. Um, so do I just add these two ports with um, their related services... perhaps tsnet and something else with normal vi commands?

In xinetd.d:

Code:

[compchem@gnlserv01 xinetd.d]$ ls -ll
total 64
-rw-r--r-- 1 root root 1157 2007-09-18 13:15 chargen-dgram
-rw-r--r-- 1 root root 1159 2007-09-18 13:15 chargen-stream
-rw-r--r-- 1 root root  523 2007-09-18 11:10 cvs
-rw-r--r-- 1 root root 1157 2007-09-18 13:15 daytime-dgram
-rw-r--r-- 1 root root 1159 2007-09-18 13:15 daytime-stream
-rw-r--r-- 1 root root 1157 2007-09-18 13:15 discard-dgram
-rw-r--r-- 1 root root 1159 2007-09-18 13:15 discard-stream
-rw-r--r-- 1 root root 1148 2007-09-18 13:15 echo-dgram
-rw-r--r-- 1 root root 1150 2007-09-18 13:15 echo-stream
-rw-r--r-- 1 root root  332 2007-10-14 01:05 rsync.presis~
-rw-r--r-- 1 root root 1212 2007-09-18 13:15 tcpmux-server
-rw-r--r-- 1 root root  491 2009-03-31 14:26 tftp
-rw-r--r-- 1 root root 1149 2007-09-18 13:15 time-dgram
-rw-r--r-- 1 root root 1150 2007-09-18 13:15 time-stream

It didn't speak to me. Hopefully it will make sense someday. Though, when I looked at it using the GUI alternative it helped a bit.

Your advice will be much appreciated!

;-)

Azazwa · 07-22-2010, 10:35 AM

Hi!

Well, I think it has been solved. ;-)

I put in a new network card, and then checked what the new IP address was (because it's given by a DHCP server from outside) and then changed it in /etc/hosts. I requested a fixed IP address from the public network's Sys admin, so that it won't change the whole time.

Thanks anyway!

;-D