LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 10-05-2009, 08:31 AM   #1
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Rep: Reputation: 1
FTP authentication almost always slow


I have a perplexing problem for which I would be very grateful for some assistance.
We are a small predominantly Windows site with a few Linux machines (Red Hat ES 3) and about 70 Windows Servers (mixture of physical and virtual).
When connecting TO a Linux machine via FTP, authentication is taking about 40 seconds (very slow). The username and password prompts come up instantly. I put in the password, press enter and then it hangs for approx 40 seconds before it eventually connects fine. Transferring files via FTP, once connected, is fast. It's just the authemtication that is slow.
To help the diagnosis, I set up a simple shell script executed 24x7 every 5 minutes by cron, which connects to another linux box via FTP. It records the elapsed time to authenticate.
I have established the following facts:
(1) Authenticating via FTP from Windows to a Windows machine is always instantaneous.
(2) Authenticating via FTP from Windows to a Linux machine is almost always very slow.
(3) Authenticating via FTP from Linux to a Linux machine is almost always very slow
(4) We have occasional periods lasting between 10 and 15 minutes each time, where authenticating via FTP from Linux to a Linux machine is instanteneous. There is no apparent pattern to when these periods occur, but they always last 10 to 15 minutes. For example one day we had 7 of the "fast" periods, another day just 2.
(5) We have 3 Linux machines. When we have one of these "fast" periods, it is fast for all the Linux machines. (2 of the Linux machines are physical, one is virtual).
So there we have it. The evidence suggests to me that the problem is caused by something external to the Linux machines because all the Linux machines seem to be affected (fast or slow) at the same time. We have a proxy server, DNS server and firewalls here, but I'm not a network administrator so I'm not well up on those things.
Any ideas gratefully received - this is one wierd problem where Google has failed to help me!
Many Thanks!
 
Old 10-05-2009, 09:13 AM   #2
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Three possibilities;
- tcpdump to inspect traffic involved in your exchange
- More verbose debugging when executing the command.
- Inspect transactions using strace

You might get more information by using -d debug in your ftp command. At least it would show where you are hanging, or if any exceptional steps are involved.

wfh@wisp:~$ ftp -d ftp.smartftp.com
Connected to smartftp.com.
220 SmartFTP Server ready...
...etc.

Another *MUCH* more noisey solution is to run ftp using 'strace'...and be patient, <ctrl>+c when it "pauses", then cut-n-paste screen output (as opposed to logging), then examine what happens after you type your password and hit <enter>.

There will probably be something in there which is non-standard, or you will get an idea of what's happening during the long delay.

Last edited by wfh; 10-05-2009 at 09:17 AM.
 
Old 10-05-2009, 10:26 AM   #3
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Original Poster
Rep: Reputation: 1
Results from ftp-d and strace

Thanks to wfh for the suggestions. Tcpdump is going to take a bit of getting to grips with (which I will do next), but in the meantime I have tried the "ftp -d" and "strace" suggestions. These both confirm that the thing hangs after the password is sent for authentication, and there don't seem to be any extra clues on there, at least so it seems to me.
Here are the interesting bits (I have globally editted hostnames and passwords for security reasons) - I will post any other clues from tcpdump later, when I can work out how to use it....
$ ftp -d fred
Connected to fred.int.bongo.co.uk.
220 fred.int.bongo.co.uk FTP server (Version 5.60) ready.
---> AUTH GSSAPI
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
Trying to authenticate to <ftp@fred.int.bongo.co.uk>
calling gss_init_sec_context
Trying to authenticate to <host@fred.int.bongo.co.uk>
calling gss_init_sec_context
GSSAPI error major: Miscellaneous failure
GSSAPI error minor: No credentials cache found
GSSAPI error: initializing context
GSSAPI authentication failed
---> AUTH KERBEROS_V4
334 Using authentication type KERBEROS_V4; ADAT must follow
KERBEROS_V4 accepted as authentication type
Kerberos V4 krb_mk_req failed: You have no tickets cached
Name (fred:dsdteam): root
---> USER root
331 Password required for root.
Password:
---> PASS XXXX
<< It hangs here for approx 40 seconds >>
230 User root logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

########
This is the strace output either side of the hang....

ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(4, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon -echo ...}) = 0
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon -echo ...}) = 0
write(2, "Password:", 9) = 9
read(4, "!", 1) = 1
read(4, "l", 1) = 1
read(4, "1", 1) = 1
read(4, "n", 1) = 1
read(4, "u", 1) = 1
read(4, "x", 1) = 1
read(4, "b", 1) = 1
read(4, "0", 1) = 1
read(4, "x", 1) = 1
read(4, "!", 1) = 1
read(4, "\n", 1) = 1
write(2, "\n", 1) = 1
ioctl(4, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
rt_sigaction(SIGINT, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb74f90, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
close(4) = 0
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
write(3, "PASS bongopassword\r\n", 17) = 17
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0

<< It hangs here for approx 40 seconds >>

read(3, "230 User root logged in.\r\n", 1024) = 26
write(1, "230 User root logged in.\n", 25) = 25
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
rt_sigaction(SIGINT, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
write(3, "SYST\r\n", 6) = 6
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
read(3, "215 UNIX Type: L8\r\n", 1024) = 19
rt_sigaction(SIGINT, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
rt_sigaction(SIGINT, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb6f5e0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
write(1, "Remote system type is UNIX.\n", 28) = 28
write(1, "Using binary mode to transfer fi"..., 37) = 37
rt_sigaction(SIGINT, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb760a0, [INT], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
rt_sigaction(SIGPIPE, {0xb760c8, [PIPE], SA_RESTORER|SA_RESTART, 0x2508e8}, {0xb760c8, [PIPE], SA_RESTORER|SA_RESTART, 0x2508e8}, 8) = 0
write(1, "ftp> ", 5) = 5
read(0,
 
Old 10-06-2009, 02:59 AM   #4
nilocj.d
LQ Newbie
 
Registered: Sep 2009
Location: Sheffield, UK
Distribution: Debian
Posts: 5

Rep: Reputation: 0
Things like this are very often caused by a DNS timeout when the FTP server is trying to resolve the connecting IP address back to a hostname. Worth checking that out.
 
Old 10-07-2009, 01:33 AM   #5
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Quote:
Originally Posted by nilocj.d View Post
Things like this are very often caused by a DNS timeout when the FTP server is trying to resolve the connecting IP address back to a hostname. Worth checking that out.
I agree, niloc. Perhaps a PTR record would speed things up. Although, why would this work *SOME* of the time if the root cause was DNS?
 
Old 10-07-2009, 04:51 AM   #6
nilocj.d
LQ Newbie
 
Registered: Sep 2009
Location: Sheffield, UK
Distribution: Debian
Posts: 5

Rep: Reputation: 0
Quote:
Originally Posted by wfh View Post
I agree, niloc. Perhaps a PTR record would speed things up. Although, why would this work *SOME* of the time if the root cause was DNS?
Not sure, but the fact that the Windows machines can always connect instantly to each other suggests that the problem is with just the Linux machines doing reverse DNS lookups.

I'd suggest logging into one of the Linux boxes that runs an FTP server and doing:

host 123.123.123.123

where 123.123.123.123 is the IP address of one of the connecting IP addresses. I suspect the command will timeout or take a while. Try it a few times.

Obviously not a fix, but to prove a point it might be worth putting an entry for a connecting IP address and hostname in to /etc/hosts on a Linux FTP server and then trying again. The corresponding client should then connect instantly.
 
Old 10-07-2009, 09:12 AM   #7
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Let's assume that it is a reverse lookup issue. If computerbongo edits his the Windows 'hosts' file (possibly at %SystemRoot%\system32\drivers\etc\), adding the name and IP of the Linux machines, the problem might go away. Then it's a matter of fixing DNS...adding a PTR for each Linux box.

Worth a try, IMO.
 
Old 10-07-2009, 09:23 AM   #8
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Original Poster
Rep: Reputation: 1
Hello! Based on the suggestions above (from nilocj.d), and also some other ideas, I have tried the following...
(1) host 123.123.123.123 (the IP address of one of the connecting IP addresses) - the command completes instantly and comes back with the correct ip address.
(2)I chose a pair of Linux machines, and made sure that both /etc/host files had entries for both hostnames. - FTP authentication is still slow.
(3) ftp localhost - FTP authentication is still slow.
(4) ftp fred (where I am signed on to machine fred) - FTP authentication is still slow.
(5) ftp 127.0.0.1 - FTP authentication is still slow.
(6) I have put an "nslookup" into the 24x7 FTP shell script, immediately before and after the ftp connection. This has shown that it is always using the same DNS server to do the lookup, during the "fast" and "slow" FTP authentication periods.
 
Old 10-07-2009, 11:47 AM   #9
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Regarding 'tcpdump', this is what I captured when connecting:

ftp ftp.smartftp.com

...authenticating with "anonymous/me@mymain.com".

To start tcpdump, note the syntax used:

tcpdump src or dst port ftp

------------- snip ------------
root@wisp:/home/wfh# tcpdump src or dst port ftp
tcpdump: WARNING: wmaster0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wmaster0, link-type IEEE802_11 (802.11), capture size 96 bytes
09:39:50.829407 IP wisp.local.60662 > www.smartftp.com.ftp: S 1144995630:1144995630(0) win 5840 <mss 1460,sackOK,timestamp 143157066 0,nop,wscale 7>
09:39:50.951585 IP wisp.local.60662 > www.smartftp.com.ftp: . ack 4112303731 win 5840 <nop,nop,timestamp 143157096 49469439>
09:39:51.072951 IP wisp.local.60662 > www.smartftp.com.ftp: . ack 31 win 5840 <nop,nop,timestamp 143157127 49469451>
09:39:55.049536 IP wisp.local.60662 > www.smartftp.com.ftp: P 0:16(16) ack 31 win 5840 <nop,nop,timestamp 143158121 49469451>
09:39:55.172144 IP wisp.local.60662 > www.smartftp.com.ftp: . ack 69 win 5840 <nop,nop,timestamp 143158151 49469861>
09:39:59.373696 IP wisp.local.60662 > www.smartftp.com.ftp: P 16:30(14) ack 69 win 5840 <nop,nop,timestamp 143159202 49469861>
09:39:59.497652 IP wisp.local.60662 > www.smartftp.com.ftp: . ack 100 win 5840 <nop,nop,timestamp 143159233 49470292>
09:39:59.497715 IP wisp.local.60662 > www.smartftp.com.ftp: P 30:36(6) ack 100 win 5840 <nop,nop,timestamp 143159233 49470292>
09:39:59.661396 IP wisp.local.60662 > www.smartftp.com.ftp: . ack 119 win 5840 <nop,nop,timestamp 143159274 49470305>
------------- /snip ------------

This might help, at least give you a base to start from.
 
Old 10-09-2009, 08:44 AM   #10
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Original Poster
Rep: Reputation: 1
Results from tcpdump

Thanks wfh. I have used tcpdump and wireshark to try to work out what is happening. As far as I can tell. I have established the following facts:
(A) The delay is occurring on the server end, not the client end. Just in case I've got my terminology wrong, by this I mean if I am invoking the FTP application on host FRED and establishing a connection to remote host DELORES, then the delay is happening on DELORES.
(B) On the remote server (DELORES in the example) I get about 30 DNS protocol network packets being sent to the primary DNS server. The "info" column in the Wireshark display shows about 3 different types of similar description
[1] Standard query A kerberos.example.com
[2] Standard query AAAA kerberos.example.com
[3] Standard query AAAA kerberos.example.com.bongo.co.uk (where bongo.co.uk is our domain name, changed here for security reasons).
Now I don't know anything about kerberos, except that on the Linux machines it hasn't been intentionally set up. But the "kerberos.example.com" suggests an unconfigured kerberos setup (kerberos.example.com appears in the /etc/krb5.conf file which apparently is one of the places you configure kerberos).
I have stopped at this point as I don't want to go changing stuff to do with kerberos as I don't know what I'm doing, and I don't want to risk locking myself out of the machine by meddling with authentication from a position of ignorance.
Any ideas? Does kerberos need to be disabled in some way?
 
Old 10-10-2009, 01:02 AM   #11
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Quote:
Originally Posted by computerbongo View Post
I have stopped at this point as I don't want to go changing stuff to do with kerberos as I don't know what I'm doing, and I don't want to risk locking myself out of the machine by meddling with authentication from a position of ignorance.
Any ideas? Does kerberos need to be disabled in some way?
Good job, computerbongo!

Kerboros is a more exotic form of authentication requiring, in the case of "SecureID", a little fob that is synced with your kerboros server. You have to answer it's query as to "what comes up on your fob" in order to authenticate. What is a "Kerberos client", "Kerberos server", and "application server" - Navy kerberos FAQ

You could choose to disable kerboros, but as you suggest, you are wise to consider the "why is kerboros installed" issue *BEFORE* taking action.

Can your server use local authentication or maybe anonymous login?

Last edited by wfh; 10-10-2009 at 01:03 AM.
 
Old 10-12-2009, 09:27 AM   #12
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Original Poster
Rep: Reputation: 1
Wow - I think it's finally solved!
I had set up all the machines to run gssftp via xinetd. I chose gssftp for no particular reason at the time. Apparently gssftp is "a kerberized xinetd-based FTP daemon which does not pass authentication information over the network". The big clue being the word "kerberized". I have now disabled gssftp on the 2 test servers, and have set up vsftpd to run as a service via xinetd instead of gssftp. This has solved the problem. Connecting by FTP is now fast all the time. So I just need to roll it out to the live machines now.
There's a noticeable difference in the messages that appear when I initiate the ftp connection from the client. Here's what I get when gssftp is running as the server:
# ftp fred
Connected to fred.bongo.co.uk.
220 fred.bongo.co.uk FTP server (Version 5.60) ready.
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
GSSAPI error major: Miscellaneous failure
GSSAPI error minor: No credentials cache found
GSSAPI error: initializing context
GSSAPI authentication failed
334 Using authentication type KERBEROS_V4; ADAT must follow
KERBEROS_V4 accepted as authentication type
Kerberos V4 krb_mk_req failed: You have no tickets cached
Name (fred:root):
And here's what I get when vsftpd is running as the server:
# ftp dolores
Connected to dolores.bongo.co.uk.
220 (vsFTPd 2.0.1)
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (dolores:root):
So hopefully that's it. The step that led me to discover kerberos being involved was using tcpdump and wireshark to notice the packets being sent to a non-existent kerberos server throughout the delay period. Many thanks to wfh who provided the tcpdump suggestion that lead to the breakthrough!
 
Old 10-12-2009, 09:45 AM   #13
computerbongo
LQ Newbie
 
Registered: Oct 2009
Posts: 6

Original Poster
Rep: Reputation: 1
Ooops, I didn't mean to say packets where being sent to a non-existent kerberos server. I should have said packets were being sent to the primary DNS server and the contents of the packets quoted a non-existent DNS server.
 
Old 10-12-2009, 11:34 PM   #14
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Quote:
Originally Posted by computerbongo View Post
Connecting by FTP is now fast all the time. So I just need to roll it out to the live machines now.
...
So hopefully that's it. The step that led me to discover kerberos being involved was using tcpdump and wireshark to notice the packets being sent to a non-existent kerberos server throughout the delay period. Many thanks to wfh who provided the tcpdump suggestion that lead to the breakthrough!
Still, it was *YOU* who ran with the suggestion! Good detective work. Good analysis of the results!
 
Old 11-23-2010, 06:05 AM   #15
ersameed
LQ Newbie
 
Registered: Nov 2010
Location: India
Distribution: CentOS
Posts: 3

Rep: Reputation: 0
Just simply comment all entries in /etc/resolve.conf
and try again, Hope it will work.
i got same problem. After removing reslove.conf entries, it works fine.
 
  


Reply

Tags
authentication, ftp


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
slow ssh authentication stefaandk Linux - Networking 5 10-12-2008 04:28 PM
ssh authentication slow now linuxhippy Slackware 1 03-08-2007 08:29 PM
Slow authentication twantrd Linux - Enterprise 4 09-23-2004 02:29 AM
Authentication slow with Sendmail jgaraas Linux - Software 0 05-01-2003 11:09 AM
Help Authentication Dead Slow kirankv Linux - Newbie 3 03-21-2003 03:07 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 03:33 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration