RH AS 3.2 External Packets Disappear but iptables show accepted
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
RH AS 3.2 External Packets Disappear but iptables show accepted
I'm going nuts. I've got an academic license for RH; so, I cannot get any support from them. I hope that you can help!
The problem: I cannot ssh (or ftp) to my machine from outside my lan. I receive syn packets from the outside world. However, my machine fails to reply with a syn,ack (as verified by ethereal). I have iptables show that the packet is accepted by putting a log before and after the accept line for the ssh port.
LOG all -- anywhere anywhere LOG level warning tcp-options ip-options prefix `DOSSH'
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ftp
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
LOG all -- anywhere anywhere LOG level warning tcp-options ip-options prefix `IPTABLESFAIL'
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
It shows that the packet comes in by showing the before accept line. It does not show the after accept log line.
So, I feel confident that my iptables are ok. In fact, I have the same trouble when iptables are cleared. So, that leads me to think that tcpwrappers are at fault. However, that too does not seem to be the case, I have hosts.allow look like:
That shows ok when I try to ssh locally. However, nothing appears when I try to ssh in from outside my network (other than the log message showing the incomming syn attempts).
PAM should not even be involved yet. So, what is going on?
I have even tried putting a simple TCP echo program on port 22 (no wrappers, no pam). No luck there either. The packets make it to iptables but seem to get dropped before getting to the accept in the echo program.
Things were working fine, but I guess I changed something. However, I have no idea what it was and how to put things back.
Do you have any ideas of where I could look? The logs are empty .... no one wants to claim they dropped the packet....
What does your OUTPUT chain look like (use iptables -vnL and including the default chain policy)? netstat -pant? Also, try stopping iptables with: service iptables stop
My thought process here: first packet is making it though the firewall, so that looks good. But you don't see any outbound packets, so either sshd isn't listening for the proper address (though you'd expect a RST if that was the case) or iptables is blocking the packets going back out (simply flushing the filter table will still leave the default chain policies in place, so if OUTPUT is DROP, that could explain what you're seeing.
I don't think this is part of the problem (yet), but with these two rules you are only allowing the NEW state:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ftp
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
Do you have any other rules allowing the ESTABLISHED state? If not, then the first incoming packet will make it through but any other packets after that will not.
Here is a bad session from a "foreign network". I have iptables turned off. I have sshd run with -d -d -d D:
Server listening on 68.208.65.46 port 22.
Generating 768 bit RSA key.
RSA key generation complete.
------------------------
ssh -vvv
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: connecting to 68.208.65.46 [68.208.65.46] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: connect to address 68.208.65.46 port 22: Connection timed out
-------------------
Ethereal
No. Time Source Destination Protocol Info
1038 13.025159 68.208.65.50 68.208.65.46 TCP 2462 > ssh [SYN] Seq=0 Ack=0 Win=16384 Len=0 MSS=1380
....
---
/var/log/messages
EMPTY!
Here is a good session which is on the same network as the target machine.
[root@linux1 root]# /sbin/service iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
[root@linux1 root]# /sbin/service sshd stop
Stopping sshd: [ OK ]
[root@linux1 root]# sshd -d -d -d -D
debug2: read_server_config: filename /etc/ssh/sshd_config
debug1: sshd version OpenSSH_3.6.1p2
debug1: private host key: #0 type 0 RSA1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: Bind to port 22 on 68.208.65.46.
Server listening on 68.208.65.46 port 22.
Generating 768 bit RSA key.
RSA key generation complete.
debug1: Server will not fork when running in debugging mode.
Connection from 68.208.65.34 port 55442
debug1: Client protocol version 2.0; client software version PuTTY-Release-0.55
debug1: no match: PuTTY-Release-0.55
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-1.99-OpenSSH_3.6.1p2
debug3: privsep user:group 74:74
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug2: Network child is on pid 27627
debug3: preauth child monitor started
debug3: mm_request_receive entering
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes256-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,rijndael192-cbc,aes128-cbc,rijndael128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: aes256-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,rijndael192-cbc,aes128-cbc,rijndael128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,none
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,none
debug2: kex_parse_kexinit: none,zlib,none
debug2: kex_parse_kexinit: none,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-sha1
debug1: kex: client->server aes256-cbc hmac-sha1 none
debug2: mac_init: found hmac-sha1
debug1: kex: server->client aes256-cbc hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST_OLD received
debug3: mm_request_send entering: type 0
debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI
debug3: mm_request_receive_expect entering: type 1
debug3: mm_request_receive entering
debug3: monitor_read: checking request 0
debug3: mm_answer_moduli: got parameters: 1024 2048 8192
debug3: mm_request_send entering: type 1
debug2: monitor_read: 0 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_choose_dh: remaining 0
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug2: dh_gen_key: priv key bits set: 284/512
debug2: bits set: 1569/3191
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug2: bits set: 1615/3191
debug3: mm_key_sign entering
debug3: mm_request_send entering: type 4
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN
debug3: mm_request_receive_expect entering: type 5
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_sign
debug3: mm_answer_sign: signature 0x967b410(143)
debug3: mm_request_send entering: type 5
debug2: monitor_read: 4 used once, disabling now
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug3: mm_request_receive entering
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user jbriner service ssh-connection method none
debug1: attempt 0 failures 0
debug3: mm_getpwnamallow entering
debug3: mm_request_send entering: type 6
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM
debug3: mm_request_receive_expect entering: type 7
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_pwnamallow
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: mm_request_receive entering
debug2: input_userauth_request: setting up authctxt for jbriner
debug3: mm_start_pam entering
debug3: mm_request_send entering: type 41
debug3: monitor_read: checking request 41
debug3: mm_inform_authserv entering
debug1: Starting up PAM with username "jbriner"
debug3: mm_request_send entering: type 3
debug2: input_userauth_request: try method none
debug3: mm_auth2_read_banner entering
debug3: mm_request_send entering: type 8
debug3: mm_request_receive_expect entering: type 9
debug3: mm_request_receive entering
debug3: Trying to reverse map address 68.208.65.34.
debug1: PAM setting rhost to "68.208.65.34"
debug2: monitor_read: 41 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 3
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 3 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug3: mm_request_receive entering
debug1: userauth_banner: sent
Failed none for jbriner from 68.208.65.34 port 55442 ssh2
debug1: userauth-request for user jbriner service ssh-connection method keyboard-interactive
debug1: attempt 1 failures 1
debug2: input_userauth_request: try method keyboard-interactive
debug1: keyboard-interactive devs
debug1: auth2_challenge: user=jbriner devs=
debug1: kbdint_alloc: devices ''
debug2: auth2_challenge_start: devices
Failed keyboard-interactive for jbriner from 68.208.65.34 port 55442 ssh2
debug1: userauth-request for user jbriner service ssh-connection method passworddebug1: attempt 2 failures 2
debug2: input_userauth_request: try method password
debug3: mm_auth_password entering
debug3: mm_request_send entering: type 10
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD
debug3: mm_request_receive_expect entering: type 11
debug3: mm_request_receive entering
debug3: monitor_read: checking request 10
debug1: PAM password authentication accepted for jbriner
debug3: mm_answer_authpassword: sending result 1
debug3: mm_request_send entering: type 11
debug3: mm_auth_password: user authenticated
Accepted password for jbriner from 68.208.65.34 port 55442 ssh2
debug3: mm_send_keystate: Sending new keys: 0x9674008 0x9673be8
debug3: mm_newkeys_to_blob: converting 0x9674008
debug3: mm_newkeys_to_blob: converting 0x9673be8
debug3: mm_send_keystate: New keys have been sent
debug2: pam_acct_mgmt() = 0
debug3: mm_send_keystate: Sending compression state
debug3: mm_request_send entering: type 24
Accepted password for jbriner from 68.208.65.34 port 55442 ssh2
debug1: monitor_child_preauth: jbriner has been authenticated by privileged process
debug3: mm_send_keystate: Finished sending state
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 24
debug3: mm_request_receive entering
debug3: mm_newkeys_from_blob: 0x967ff68(139)
debug2: mac_init: found hmac-sha1
debug3: mm_get_keystate: Waiting for second key
debug3: mm_newkeys_from_blob: 0x967ff68(139)
debug2: mac_init: found hmac-sha1
debug3: mm_get_keystate: Getting compression state
debug3: mm_get_keystate: Getting Network I/O buffers
debug3: mm_share_sync: Share sync
debug3: mm_share_sync: Share sync end
debug1: PAM establishing creds
debug1: permanently_set_uid: 500/500
debug2: set_newkeys: mode 0
debug2: set_newkeys: mode 1
debug1: Entering interactive session for SSH2.
debug1: fd 7 setting O_NONBLOCK
debug1: fd 8 setting O_NONBLOCK
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 256 win 16384 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: init
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug2: User child is on pid 27628
debug3: mm_request_receive entering
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: lastlog_openseek: Couldn't open /var/log/lastlog: Permission denied
debug1: Allocating pty.
debug3: mm_request_send entering: type 25
debug3: mm_pty_allocate: waiting for MONITOR_ANS_PTY
debug3: mm_request_receive_expect entering: type 26
debug3: mm_request_receive entering
debug3: monitor_read: checking request 25
debug3: mm_answer_pty entering
debug1: session_new: init
debug1: session_new: session 0
debug3: mm_request_send entering: type 26
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug3: tty_parse_modes: SSH2 n_bytes 11
debug3: tty_parse_modes: ispeed 38400
debug3: tty_parse_modes: ospeed 38400
debug3: mm_answer_pty: tty /dev/pts/1 ptyfd 3
debug3: mm_request_receive entering
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: PAM setting tty to "/dev/pts/1"
debug1: PAM establishing creds
debug1: Setting controlling tty using TIOCSCTTY.
debug1: channel 0: rfd 10 isatty
debug1: fd 10 setting O_NONBLOCK
debug2: fd 9 is O_NONBLOCK
debug2: channel 0: rcvd adjust 669
debug2: channel 0: rcvd adjust 21
debug2: channel 0: rcvd adjust 26
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 2
debug2: channel 0: rcvd adjust 45
debug2: channel 0: rcvd adjust 45
debug2: channel 0: rcvd adjust 90
debug2: channel 0: rcvd adjust 45
debug2: channel 0: rcvd adjust 45
debug2: channel 0: rcvd adjust 51
debug2: channel 0: rcvd adjust 21
debug2: channel 0: rcvd adjust 26
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 1
debug2: channel 0: rcvd adjust 2
debug2: channel 0: rcvd adjust 8
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 27629
debug1: session_exit_message: session 0 channel 0 pid 27629
debug1: channel 0: request exit-status
debug1: session_exit_message: release channel 0
debug1: channel 0: write failed
debug1: channel 0: close_write
debug1: channel 0: output open -> closed
debug1: session_close: session 0 pid 27629
debug3: mm_request_send entering: type 27
debug1: channel 0: read<=0 rfd 10 len -1
debug3: monitor_read: checking request 27
debug1: channel 0: read failed
debug3: mm_answer_pty_cleanup entering
debug1: channel 0: close_read
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: channel 0: input open -> drain
debug3: mm_session_close: session 0 pid 27628
debug1: channel 0: ibuf empty
debug3: mm_session_close: tty /dev/pts/1 ptyfd 3
debug1: channel 0: send eof
debug1: session_pty_cleanup: session 0 release /dev/pts/1
debug1: channel 0: input drain -> closed
debug1: channel 0: send close
debug2: notify_done: reading
debug3: channel 0: will not send data after close
debug3: mm_request_receive entering
debug2: channel 0: rcvd adjust 7
debug3: channel 0: will not send data after close
debug1: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug1: channel 0: is dead
debug1: channel 0: garbage collecting
debug1: channel_free: channel 0: server-session, nchannels 1
debug3: channel_free: status: The following connections are open:\015
#0 server-session (t4 r256 i3/0 o3/0 fd -1/-1)\015
debug3: channel_close_fds: channel 0: r -1 w -1 e -1
Connection closed by 68.208.65.34
debug1: krb5_cleanup_proc called
Closing connection to 68.208.65.34
debug3: mm_request_send entering: type 42
debug3: monitor_read: checking request 42
debug3: mm_answer_term: tearing down sessions
-------- /var/log/messages
Oct 11 14:28:50 linux1 iptables: succeeded
Oct 11 14:28:50 linux1 last message repeated 2 times
Oct 11 14:29:16 linux1 sshd[4939]: Received signal 15; terminating.
Oct 11 14:29:16 linux1 sshd: sshd -TERM succeeded
Oct 11 14:31:53 linux1 PAM-warn[27618]: service: sshd [on terminal: NODEVssh]
Oct 11 14:31:53 linux1 PAM-warn[27618]: user: (uid=0) -> jbriner [remote: ?nobody@68.208.65.34]
Oct 11 14:31:53 linux1 sshd(pam_unix)[27628]: session opened for user jbriner by root(uid=500)
Oct 11 14:32:06 linux1 sshd(pam_unix)[27628]: session closed for user jbriner
Also could you explain what you mean by "foreign" network, as they all appear to be on the same subnet in terms of IP addresses. Are they seperated by a router or second firewall/gateway?
Also check general connectivity, can the 2 machines even ping each other with both of their firewalls off. What about nmap with the firewalls on and off? Also verify that all hosts are using the same SSH protocol (though normally a protocol miss-match results in a connection refused or an error msg)
The clients are on different subnets (255.255.255.240).
I don't see how turning on IPTABLES will help. However, I will try it.
As per firewalls, I do have a firewall/router that I cannot control. It
is supposed to allow 21/22/80 packets through. As you see, the packet
does get through. My machine however never sends the syn,ack (as viewed
on either side of the firewall).
Are there any kernel switches that I can turn on to see more about how the tcp
buffers and algorithms are working? Something that will force more output to the
log files?
My machine however never sends the syn,ack (as viewed on either side of the firewall).
How did you determine this? I took the ethereal dump you posted above to be run from the ssh client (the foreign host). Have you done an ethereal dump from the server side as well? It pretty clear that the first packet is making it to the sshd server (the iptables log shows that. But we don't ever see one back out. So either it's not being generated or it's leaving the sshd server and disappearing somewhere in between. I don't think it has anything to do with the firewall on the sshd server or with sshd itself (you can connect from other hosts on the LAN).
If you put a log message in the OUTPUT chain of the sshd server, you'll log the outgoing packets. But as far as looking at the tcp stack itself, I don't think you'll have much luck. Though if you turn iptables on, you'll be able to see the connection tracking table in /proc/net/ip_conntrack which should show you the current connection states.
The Ethereal was done on the server side. I have done it on both sides (although the client side is not shown here).
I will add the addition to the output chain. I will also see if I can view the /proc/net/ip_conntrack.
Now we are getting somewhere! The reply message is a reset. So, someone is canceling the message. It is interesting to note that Ethereal does not capture the reset packet.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
