-   *BSD (
-   -   OpenBSD 3.4 dropping network connection (

iainr 04-12-2004 07:35 AM

OpenBSD 3.4 dropping network connection
I'm a real newbie with OpenBSD. I installed 3.4 for the first time yesterday and I've got a very odd network problem.

At first, the network is fine. I can ping and ssh in and out and routing through to the Internet works.

However, I notice two problems that crop up over time (and I'm talking an hour at most).
1. If I log on from another system with SSH and leave the session, it hangs after a while. The session isn't dropped cleanly (and TMOUT=0), but hangs at the command prompt.
2. I try to ping the server from another system and neither ping nor ssh works.

Here's where it gets really wierd. The server doesn't seem to accept any incoming connections, even ping, but outgoing traffic is still working fine : web browsing, ssh, ping. As soon as I make an outgoing connection from the OpenBSD console, the incoming stuff starts working again and I can ping and ssh to the server.

My only though at the moment is that somehow the network (which is a DSL/Cable router) is losing the MAC address of the BSD server and the outward traffic refreshes it. However, I can't see why this would be happening and have no idea how to fix it. I've had a variety of different Linux distros connected up (including Red Hat 9 installed on the same server that OpenBSD is now on) and haven't seen this problem before.

chort 04-12-2004 12:18 PM

That is really, really weird. I read just about every OpenBSD mailing lists and I've never seen a problem like that. How did you install, from the retail CD, or from the -release packages, or from a snapshot (I'm guessing one of the first two since the latest snapshot should be 3.5-current).

Have you tried running tcpdump on that segment when the box exhibits the odd behavior? tcpdump on the box itself may be interesting, but if it's on a hub probably the best thing would be to run simultaneous tcpdump (or Ethereal, or whatever floats your boat) and compare. That will probably get you to the bottom of the issue very quickly by comparing what another box on the same network sees and what the OBSD box sees.

Oh yeah, what network card are you using and which driver does OpenBSD use for it? You can find the OpenBSD driver by doing ifconfig -a, the first entry should be your ethernet card (fxp0, vr0, bge0, etc).

iainr 04-12-2004 01:02 PM

Thanks chort,

I did indeed install from th retail CD. The interface comes up as fxp0.

I haven't tried doing any low level network checks, I'll give that a try. The problem has persisted over several reboots and a change in IP address, though I haven't tried a reinstall yet.

iainr 04-12-2004 02:24 PM

The ethereal trace shows nothing very interesting, unfortunately :

1. Linux tries to ping BSD. Arp requests go out, asking who has BSD's IP address but no one replies.
2. BSD pings Linux. Ask's for Linux's MAC address and received immediate answer. Ping proceeds.
3. Linux pings BSD. This time Linux gets an immediate arp response and ping proceeds.

No log messages on the BSD side, unless they are hidden away somewhere other than /var/log. This is running ethereal on Linux. I'll try running ethereal on BSD and see what shows up.

chort 04-12-2004 02:29 PM

The fxp cards and the driver itself have been very stable, in my experience (and they appear to be a favorite of my OBSD users). I have an fxp card in every machine I've built, including 3 in my firewall. I definitely haven't seen a problem like that. I also started my firewall from the 3.4 retail CDs, and didn't see anything like the problem you're having.

You didn't compile a custom kernel by any chance, did you? Do you have PF enabled? How are your network subnets setup? Is traffic being forwarded through another device, like a router?

iainr 04-12-2004 03:48 PM

No custom kernel. Just did the installation and the afterboot steps.

It might be useful to post some command outputs : it all looks normal to me, but there might be something BSDish that I'm missing.

I've got the basic home network with several computers directly attached to a Cable/DSL router, all on the same subnet (, routing to the Internet.

BSD IP address is, Linux IP is, router is


# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
lo1: flags=8008<LOOPBACK,MULTICAST> mtu 33224
        address: 00:50:8b:0b:2a:a1
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet netmask 0xffffff00 broadcast
        inet6 fe80::250:8bff:fe0b:2aa1%fxp0 prefixlen 64 scopeid 0x1

# netstat -rn
Routing tables

Destination        Gateway            Flags    Refs    Use    Mtu  Interface
default          UGS        0      36      -  fxp0
127/8              UGRS        0        0  33224  lo0          UH          2      424  33224  lo0
192.168.1/24      link#1            UC          2        0      -  fxp0        0:6:25:a5:1d:f8    UHLc        1      87      -  fxp0      0:90:7d:f3:f:67    UHLc        1      117      -  fxp0          UGHS        0        2  33224  lo0

chort 04-13-2004 04:44 AM

Try running Ethereal (or tcpdump) on both hosts at once and correlate the logs, that way you can see both sides of the story. For instance, maybe the BSD box is trying to reply to the Linux box with the ARP reply, but it some how gets blocked?

It probably won't be useful, but could you paste the output of

netstat -i fxp0


netstat -ss


If you can manage it (i.e the boxen can be disconnected from the Internet for a while), try connecting both boxen directly with a crossover cable (using the same IPs and network settings will be fine) and see if you get the same behavior. There's an outside possibility that it's a bad switch port on your router.

iainr 04-13-2004 04:28 PM

The behaviour seems quite interesting. It appears that the connection is consistently dropped after 20-30 minutes with no outgoing traffic. Incoming traffic can continue (e.g. if I bring it up, then ping the interface from the Linux box, it suddenly stops working after 20-30 minutes).

I've tried changing switch ports (but not using a crossover cable) and that made no difference.

I tried running ethereal on the BSD box but it looks like the act of running ethereal keeps the interface up . Run ethereal, keeps working. Stop ethereal, interface comes down 25 mins later. :mad:

Here's the output of netstat -ss and netstat -i fxp0 (taken when the interface was not accepting incoming traffic; though I can't see any significant difference with the same commands when the interface is accepting traffic).


Name    Mtu  Network    Address              Ipkts Ierrs    Opkts Oerrs Colls
lo0    33224 <Link>                            1427    0    1427    0    0
lo0    33224 loopback    localhost            1427    0    1427    0    0
lo0    33224 localhost.m ::1                  1427    0    1427    0    0
lo0    33224 fe80::%lo0/ fe80::1%lo0          1427    0    1427    0    0
lo1*    33224 <Link>                              0    0        0    0    0
fxp0    1500  <Link>      00:50:8b:0b:2a:a1    70164    0    10401    0    0
fxp0    1500  192.168.1/2 charlie              70164    0    10401    0    0
fxp0    1500  fe80::%fxp0 fe80::250:8bff:fe    70164    0    10401    0    0
pflog0* 33224 <Link>                              0    0        0    0    0
pfsync0 1896  <Link>                              0    0        0    0    0
sl0*    296  <Link>                              0    0        0    0    0
sl1*    296  <Link>                              0    0        0    0    0
ppp0*  1500  <Link>                              0    0        0    0    0
ppp1*  1500  <Link>                              0    0        0    0    0
tun0*  3000  <Link>                              0    0        0    0    0
tun1*  3000  <Link>                              0    0        0    0    0
enc0*  1536  <Link>                              0    0        0    0    0
bridge0 1500  <Link>                              0    0        0    0    0
bridge1 1500  <Link>                              0    0        0    0    0
vlan0*  1500  <Link>      00:00:00:00:00:00        0    0        0    0    0
vlan1*  1500  <Link>      00:00:00:00:00:00        0    0        0    0    0
gre0*  1450  <Link>                              0    0        0    0    0
gif0*  1280  <Link>                              0    0        0    0    0
gif1*  1280  <Link>                              0    0        0    0    0
gif2*  1280  <Link>                              0    0        0    0    0
gif3*  1280  <Link>                              0    0        0    0    0

        13520 total packets received
        12847 packets for this host
        673 packets for unknown/unsupported protocol
        11936 packets sent from this host
        4 output packets discarded due to no route
        674 calls to icmp_error
        Output packet histogram:
                echo reply: 7842
                destination unreachable: 674
        Input packet histogram:
                echo reply: 35
                destination unreachable: 674
                echo: 7842
        7842 message responses generated
        2504 packets sent
                2315 data packets (287129 bytes)
                6 data packets (1056 bytes) retransmitted
                1 fast retransmitted packet
                102 ack-only packets (914 delayed)
                4 window update packets
                78 control packets
        3000 packets received
                2038 acks (for 286067 bytes)
                22 duplicate acks
                933 packets (92175 bytes) received in-sequence
                4 completely duplicate packets (96 bytes)
                1 packet with some duplicate data (48 bytes duplicated)
                3 out-of-order packets (0 bytes)
                1 window update packet
        61 connection requests
        10 connection accepts
        17 connections established (including accepts)
        146 connections closed (including 1 drop)
        54 embryonic connections dropped
        2045 segments updated rtt (of 1499 attempts)
        6 retransmit timeouts
        10 keepalive timeouts
                9 keepalive probes sent
                1 connection dropped by keepalive
        1016 correct ACK header predictions
        838 correct data packet header predictions
        37 PCB cache misses
                        cwr by fastrecovery: 1
                        cwr by timeout: 6
        2001 datagrams received
        674 dropped due to no socket
        1284 broadcast/multicast datagrams dropped due to no socket
        43 delivered
        875 datagrams output
        717 missed PCB cache
        32 total packets received
        32 packets for this host
        39 packets sent from this host
        Input packet histogram:
                TCP: 32
        Mbuf statistics:
                1 one mbuf
                30 one ext mbufs
                1 two or more ext mbuf
        source addresses on an outgoing I/F
                1 node-local
        source addresses of same scope
                1 node-local
        Output packet histogram:
                multicast listener report: 6
                neighbor solicitation: 1
        Histogram of error messages to be generated:

chort 04-13-2004 06:15 PM

Hmmm, well this is all very odd. It seems like at initial start-up the machine will answer ARPs, and if the interface is in promiscuous mode it will answer, or if outbound data is initiated to the machine that is asking for the ARP reply, then it will reply to that machine...

The only thing I could find that might be relevant is this note in the changelog from 3.4 to 3.5:
"Add some delay when reading the address off fxp(4) eeproms, otherwise the result may be garbage."

Now I don't know if that's at all related, but it's possible.

You could try downloading a snapshot, found on any of the FTP mirros in the path pub/OpenBSD/snapshots/i386, specifically download the bsd.rd and place it in your / directory, reboot, and at the boot> prompt type in bsd.rd (before it automatically proceeds with booting the normal kernel). At the prompt, choose the (u)pgrade option and proceed. You can download all the updated sets via ftp or http through the upgrade process (i.e. it will ask you if you which disk has your root partition, then it asks which partition it's on, then it asks you if you want to mount the other partitions found in /etc/fstab, then it will ask you if it should start networking (yes!), then a few optional steps that you shouldn't need to do, then ask you where the update sets are, choose (f)tp or (h)ttp, then view a list of mirrors to choose from, pick your mirrior, etc...

One thing to be aware of is that when you update via snapshots, they don't make any changes to /etc so you'll need to do those by hand. You can see a list of what changed between 3.4 and 3.5 right here

The changes that would apply are the new users & groups (for privilege separation), and the changes to /etc/security and /etc/netstart. If you're familiar with CVS you should be able to just check out /usr/src/etc/netstart and /usr/src/etc/security and copy them over, otherwise you could download them from the CVSweb. You should not need to recompile any of the tools as described, since that's for updating from source. The binary package should include those changes.

iainr 04-14-2004 05:13 PM

I have upgraded to 3.5 (very interesting - thanks for the instructions). Unfortunately it hasn't made any difference. This suggests to me that it's most likely a hardware problem with the PC or ethernet adapter which I either didn't spot before or has only started recently (the PC has been sitting unused for a couple of months).

To test this, I'll install OpenBSD on a different PC and see if that has the same problem; or I might try a different NIC in the same PC. Either way, I'll post back when I have more to report.

I'm not sure, but it looks like when the incoming connection is down, doing anything at all on the PC is enough to bring it up (e.g. just doing command recall in the shell seems to be sufficient), so it could be some sort of hardware problem where the box is hibernating in some way. That might explain the consistency of the 20-30 minute period after which the network goes down : maybe there's some internal hiberation counter.

chort 04-15-2004 04:23 AM

Silly question, but did you try simply reseating the PCI card, or trying it in a different slot? It really does sound like a hardware problem.

By the way, if you appreciate the simplicity of the upgrade process (that was a full version bump BTW, 3.5 just went gold several days ago, so basically you have -current from a few days after the latest -release), then you will really dig updating to multiple machines. I do this almost every weekend: follow the instructions from the release(8) man page to recompile the kernel and userland (I rarely do X, since it rarely gets updated), then you do the process to build the sets (make release). When you're done, you get all the same sets and files that are on the FTP sites in the snapshot directory. You can place these on your own HTTP or FTP server (or NFS I think? I can't remember) and simply copy the bsd.rd to any other OBSD machine, reboot with bsd.rd, and viola you can do the (u)pgrade process using your locally compiled snapshot! That is, of course, if you have multiple OBSD boxen on the same arch (i.e. i386, or all PPC, or all SPARC, etc). You can do something similar with the ports system, too. If you build a port on one box, you can copy the package to another box (of the same arch) and just do pkg_add <package name>.

iainr 04-17-2004 05:36 AM

Thanks Chort - I thought the network upgrade process was excellent and I'll certainly be having a play with the more advanced stuff.

I've tried instaling OpenBSD on another local PC and it works fine this time, so definitely something hardware related. I'll have to play around at fixing the first machine when I have some time on my hands.

I'm actually very pleased. The PC I've just installed it on is one I bought for 20 a while ago and never found a use for. With only 32MB of RAM and a P166 processor, it was too light for the new Linux distros. I tried installing older versions (e.g. RH6) but that doesn't have a driver for the network adapter. Being lazy/busy (take your pick) I left it for a few months so I was very pleased that OpenBSD installed fine, seems to run at a reasonable speed, and configured the network card perfectly during the installation.

chort 04-17-2004 08:15 PM

On that note of installing on old machines, you can fit OpenBSD in far less than 200MB. The only sets you *must* install are base3?.tgz, etc3?.tgz, and bsd (the kernel). In fact, I've been toying with the idea of trying to install OpenBSD on a 256MB USB flash drive, I just haven't figured out how I would get it to boot... yet :)

newpenguin 04-17-2004 10:12 PM

well its only becoz of system standby settings of ur bios.
i have a pc i used it for linux and freebsd and never got any problem but when i installed openbsd it started to go in standbymode after sometime becoz of bios power was strange for me.

chort 04-18-2004 01:21 AM

That's a good point. Try going into your BIOS settings and disabling all power management.

All times are GMT -5. The time now is 12:28 AM.