bash: scp: command not found \ lost connection

maccas17 · 06-03-2012, 06:07 PM

Running RHEL 6.2 and I am having an issue using scp:

Code:

scp /etc/hosts host1:/tmp
bash: scp: command not found
lost connection

openssh-clients.x86_64 - 5.3p1-70.el6_2.2 - package is installed and if I execute just scp I see:

Code:

scp
usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
           [-l limit] [-o ssh_option] [-P port] [-S program]
           [[user@]host1:]file1 ... [[user@]host2:]file2

which scp
/usr/bin/scp

The scp command works fine if one is copying files locally on the same host.

Code:

scp /etc/hosts host
# ll host
-rw-r--r--. 1 root root 201 Jun  1 18:19 host

SSH and SFTP work fine between the two systems. I also enabled Protocol version 1 support for sshd and ssh (client) as I seem to remember that is what scp uses. All to no avail.

Both SELinux and iptables are disabled on the source and target systems.

Has anybody seen similar issues with using scp on RHEL 6.2?

rosehosting.com · 06-03-2012, 06:43 PM

It seems like scp is not in the $PATH on the remote server. Execute

Code:

which scp

on the remote host to see the location of scp (most likely /usr/bin/) and append it to the $PATH variable in .bashrc for example.

Code:

export PATH=$PATH:/usr/bin/

unSpawn · 06-03-2012, 07:10 PM

Quote:

Originally Posted by rosehosting.com

It seems like scp is not in the $PATH on the remote server.

If you execute 'scp' on a host then the binary is loaded on the host and not on the remote host?..
* BTW I think you're way overdoing it with your five-link-signature. LQ is provided to you for free and you repay that in this way with ad overkill?

Quote:

Originally Posted by maccas17

Code:

scp /etc/hosts host1:/tmp
bash: scp: command not found
lost connection

What's curious about this is that if the binary would be outside your path you should see a "-bash: scp: No such file or directory" message and not "command not found". And really not "lost connection" which implies something tries to build a connection. Are you using some alias or function? What shows if you try the long and verbose command line: '/usr/bin/scp -vv /etc/hosts user@host1:/tmp/'? Can you SFTP to host1?

Quote:

Originally Posted by maccas17

I also enabled Protocol version 1 support for sshd and ssh (client) as I seem to remember that is what scp uses.

AFAIK SSHv1 should only be used for accessing SSHv2 incompatible hosts and not for other reasons.

Quote:

Originally Posted by maccas17

Both SELinux and iptables are disabled on the source and target systems.

There's no need to disable those features and weaken the security posture: if there's routing or filtering issues they can be diagnosed using "-j LOG" rules and fixed and if there's SELinux AVC messages they can be fixed building a local policy.

rosehosting.com · 06-03-2012, 07:27 PM

Quote:

Originally Posted by unSpawn

If you execute 'scp' on a host then the binary is loaded on the host and not on the remote host?..
* BTW I think you're way overdoing it with your five-link-signature. LQ is provided to you for free and you repay that in this way with ad overkill?

I'm really sorry that was not intended for those purposes at all, but thanks for letting us know. We are just enjoying the forums and helping the open source community. Actually, an email was sent to the moderators about this and we have not heard back yet if our sign. is ok and allowed. Anyhow, we will immediately remove it from there.

On the topic: It looks like the $PATH is not sourced in a non-interactive ssh session. So that's why was suggested to export scp's binary to the path on the remote host and see if that helps.

maccas17 · 06-04-2012, 03:47 AM

Thanks for the responses.

/usr/bin is set in $PATH on both the systems (which scp works fine) and as highlighted, the command works locally.

It is the "lost connection" error that perplexes me.

As I mentioned, ssh and sftp (file download and upload) to the host is working fine.

I have tried some ssh debugging already, but didn't want to overload my original post! Here is the client output:

Code:

scp -P 2222 /etc/hosts root@hvw100:/tmp
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: Copy environment: SELINUX_ROLE_REQUESTED=
debug3: Copy environment: SELINUX_LEVEL_REQUESTED=
debug3: Copy environment: SELINUX_USE_CURRENT_RANGE=
Environment:
  LANG=en_GB
  USER=root
  LOGNAME=root
  HOME=/root
  PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  MAIL=/var/mail/root
  SHELL=/bin/bash
  SSH_CLIENT=10.10.0.248 34521 2222
  SSH_CONNECTION=10.10.0.248 34521 10.10.0.31 2222
  SELINUX_ROLE_REQUESTED=
  SELINUX_LEVEL_REQUESTED=
  SELINUX_USE_CURRENT_RANGE=
debug3: channel 0: close_fds r -1 w -1 e -1 c -1
bash: scp: command not found
lost connection

Output on the host server I'm copying to:

Code:

/usr/sbin/sshd -ddd -p2222
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 565
debug2: parse_server_config: config /etc/ssh/sshd_config len 565
debug3: /etc/ssh/sshd_config:21 setting Protocol 2,1
debug3: /etc/ssh/sshd_config:36 setting SyslogFacility AUTHPRIV
debug3: /etc/ssh/sshd_config:66 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:70 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:81 setting GSSAPIAuthentication yes
debug3: /etc/ssh/sshd_config:83 setting GSSAPICleanupCredentials yes
debug3: /etc/ssh/sshd_config:97 setting UsePAM yes
debug3: /etc/ssh/sshd_config:100 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:101 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:102 setting AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
debug3: /etc/ssh/sshd_config:103 setting AcceptEnv XMODIFIERS
debug3: /etc/ssh/sshd_config:109 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:132 setting Subsystem sftp /usr/libexec/openssh/sftp-server
debug1: sshd version OpenSSH_5.3p1
debug1: private host key: #0 type 0 RSA1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p2222'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
Generating 1024 bit RSA key.
RSA key generation complete.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 565
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.10.0.248 port 34526
debug1: Client protocol version 1.5; client software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Local version string SSH-1.99-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 2035
debug3: preauth child monitor started
debug3: mm_request_receive entering
debug3: privsep user:group 74:74
debug1: permanently_set_uid: 74/74
debug3: Wrote 436 bytes for a total of 457
debug1: Sent 1024 bit server key and 2048 bit host key.
debug3: mm_request_send entering: type 78
debug3: mm_request_receive_expect entering: type 79
debug3: mm_request_receive entering
debug3: monitor_read: checking request 78
debug3: mm_request_send entering: type 79
debug3: mm_request_receive entering
debug1: Encryption type: 3des
debug3: mm_request_send entering: type 29
debug3: mm_request_receive_expect entering: type 30
debug3: mm_request_receive entering
debug3: monitor_read: checking request 29
debug3: mm_request_send entering: type 30
debug2: monitor_read: 29 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_ssh1_session_id entering
debug3: mm_request_send entering: type 31
debug2: cipher_init: set keylen (16 -> 32)
debug2: cipher_init: set keylen (16 -> 32)
debug1: Received session key; encryption turned on.
debug3: Wrote 12 bytes for a total of 469
debug3: monitor_read: checking request 31
debug3: mm_answer_sessid entering
debug2: monitor_read: 31 used once, disabling now
debug3: mm_request_receive entering
debug1: Installing crc compensation attack detector.
debug3: mm_getpwnamallow entering
debug3: mm_request_send entering: type 7
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM
debug3: mm_request_receive_expect entering: type 8
debug3: mm_request_receive entering
debug3: monitor_read: checking request 7
debug3: mm_answer_pwnamallow
debug3: Trying to reverse map address 10.10.0.248.
debug2: parse_server_config: config reprocess config len 565
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 8
debug2: monitor_read: 7 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_start_pam entering
debug3: mm_request_send entering: type 50
debug1: Attempting authentication for root.
debug3: Wrote 12 bytes for a total of 481
debug3: monitor_read: checking request 50
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "hostdp01.local.net"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 50 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_auth_password entering
debug3: mm_request_send entering: type 11
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD
debug3: mm_request_receive_expect entering: type 12
debug3: mm_request_receive entering
debug3: monitor_read: checking request 11
debug3: PAM: sshpam_passwd_conv called with 1 messages
debug1: PAM: password authentication accepted for root
debug3: mm_answer_authpassword: sending result 1
debug3: mm_request_send entering: type 12
debug3: mm_auth_password: user authenticated
debug3: mm_do_pam_account entering
debug3: mm_request_send entering: type 51
debug3: mm_request_receive_expect entering: type 52
debug3: mm_request_receive entering
debug3: mm_request_receive_expect entering: type 51
debug3: mm_request_receive entering
debug1: do_pam_account: called
debug3: PAM: do_pam_account pam_acct_mgmt = 0 (Success)
debug3: mm_request_send entering: type 52
debug3: mm_do_pam_account returning 1
debug3: Wrote 12 bytes for a total of 493
debug3: mm_send_keystate: Sending ssh1 KEY+IV
debug3: ssh1_3des_iv: Copying 3DES IV
debug3: ssh1_3des_iv: Copying 3DES IV
debug3: mm_send_keystate: Sending compression state
debug3: mm_request_send entering: type 25
debug3: mm_send_keystate: Finished sending state
Accepted password for root from 10.10.0.248 port 34526
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 25
debug3: mm_request_receive entering
debug3: mm_get_keystate: Getting compression state
debug3: mm_get_keystate: Getting Network I/O buffers
debug3: mm_share_sync: Share sync
debug3: mm_share_sync: Share sync end
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: SELinux support enabled
debug3: ssh_selinux_setup_variables: setting execution context
debug1: PAM: establishing credentials
debug3: PAM: opening session
debug2: cipher_init: set keylen (16 -> 32)
debug2: cipher_init: set keylen (16 -> 32)
debug3: ssh1_3des_iv: Installed 3DES IV
debug3: ssh1_3des_iv: Installed 3DES IV
debug2: session_new: allocate (allocated 0 max 10)
debug3: session_unused: session id 0 unused
debug1: session_new: session 0
debug1: Installing crc compensation attack detector.
debug1: Exec command 'scp -t /tmp'
debug2: fd 3 setting TCP_NODELAY
debug1: Entering interactive session.
debug2: fd 6 setting O_NONBLOCK
debug2: fd 7 setting O_NONBLOCK
debug2: fd 9 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug2: fd 8 setting O_NONBLOCK
debug1: server_init_dispatch_13
debug1: server_init_dispatch_15
debug3: Wrote 60 bytes for a total of 81
debug3: Wrote 52 bytes for a total of 133
debug3: Wrote 68 bytes for a total of 201
debug3: Wrote 68 bytes for a total of 269
debug3: Wrote 68 bytes for a total of 337
debug3: Wrote 28 bytes for a total of 365
debug3: Wrote 28 bytes for a total of 393
debug3: Wrote 36 bytes for a total of 429
debug3: Wrote 28 bytes for a total of 457
debug3: Wrote 84 bytes for a total of 541
debug3: Wrote 36 bytes for a total of 577
debug3: Wrote 36 bytes for a total of 613
debug3: Wrote 60 bytes for a total of 673
debug3: Wrote 76 bytes for a total of 749
debug3: Wrote 44 bytes for a total of 793
debug3: Wrote 44 bytes for a total of 837
debug3: Wrote 44 bytes for a total of 881
debug2: notify_done: reading
debug3: Wrote 44 bytes for a total of 925
debug1: End of interactive session; stdin 0, stdout (read 0, sent 0), stderr 603 bytes.
debug1: Command exited with status 127.
debug3: Wrote 20 bytes for a total of 945
debug1: Received exit confirmation.
debug1: session_close: session 0 pid 2038
debug3: session_unused: session id 0 unused
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug3: PAM: sshpam_thread_cleanup entering
Transferred: sent 1622, received 40 bytes
Closing connection to 10.10.0.248 port 34526

It looks to me like after connecting fine using keys it then starts writing and fails with a 127 exit code. Have no idea why.
I think there must be some issue on the host I'm copying to - going to look at this further.

As to SELinux and firewalls - this is a temporary measure and those layers will be added back after I can get scp working; less things in the mix generally makes troubleshooting easier!