Need help with wput (proxy configuration)

blnl · 06-20-2014, 03:23 PM

The proxy is configured as follows:

Code:

user123@acc4012 $ export ftp_proxy=this.is.confidential:8080
user123@acc4012 $ echo $ftp_proxy
this.is.confidential:8080
user123@acc4012 $

I can successfully retrieve files using wget:

Code:

user123@acc4012 $ wget -e use_proxy=yes ftp://user.name:secret@home.telfort.nl/TestFile.ods
--2014-06-20 21:50:25--  ftp://user.name:*password*@home.telfort.nl/TestFile.ods
Resolving this.is.confidential... 134.27.232.42
Connecting to this.is.confidential|134.27.232.42|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 26296 (26K)
Saving to: `TestFile.ods'

100%[=====================================================================================================================================================>] 26,296      --.-K/s   in 0.1s    

Last-modified header invalid -- time-stamp ignored.
2014-06-20 21:50:26 (244 KB/s) - `TestFile.ods' saved [26296/26296]

user123@acc4012 $

Still, wput refuses to upload a file:

Code:

user123@acc4012 $ wput -V    
wput version: 0.6.1
user123@acc4012 $ wput --proxy=http TestFile.ods ftp://user.name:secret@home.telfort.nl
--21:37:18-- `TestFile.ods'
    => ftp://user.name:xxxxx@home.telfort.nl:21/TestFile.ods
Connecting to home.telfort.nl:21... connected! 
Logging in as user.name ... Logged in!
Error: Connection could not be established.
Proxy states 'HTTP/1.0 403 Forbidden
Mime-Version: 1.0
Date: Fri, 20 Jun 2014 19:37:19 GMT
Content-Type: text/html
Connection: close
Proxy-Connection: close
Content-Length: 2303

<!DOCTYPE html PUBLIC "-//W3C//Dtd XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/Dtd/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

<title>Notification: Policy: General</title>
<style type="text/css">
body {
  font-family: Ari'Warning: Using port-mode. Unable to use the http-proxy for this connection
Receive-Error: Invalid FTP-answer (24 bytes): PORT 92,120,70,35,178,56
Reconnecting to be sure, nothing went wrong
Waiting 10 seconds... Receive-Error: read() failed. Read '' so far. (errno: Operation now in progress (115))
Receive-Error: Connection broke down.
Connecting to :21... Error: Connection could not be established.
Proxy states 'HTTP/1.0 503 Service Unavailable
Mime-Version: 1.0
Date: Fri, 20 Jun 2014 19:37:30 GMT
Content-Type: text/html
Connection: close
Proxy-Connection: close
Content-Length: 2246

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Notification: DNS Failure</title>
<style type="text/css">
body {
  font-family: Arial, Helvetica, sans-serif;
  font-size: 14px;
  color:'failed!
Waiting 10 seconds... *** glibc detected *** wput: corrupted double-linked list: 0x00000000019153c0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3bc9e75916]
/lib64/libc.so.6[0x3bc9e786a4]
wput[0x406cf4]
wput[0x405dff]
wput[0x40d06b]
wput[0x4023c2]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3bc9e1ecdd]
wput[0x401ef9]
======= Memory map: ========
00400000-00416000 r-xp 00000000 00:1e 69928006                           /home/user123/APPS/wput
00615000-00616000 rw-p 00015000 00:1e 69928006                           /home/user123/APPS/wput
01913000-01934000 rw-p 00000000 00:00 0                                  [heap]
3bc9600000-3bc9620000 r-xp 00000000 fd:00 34554                          /lib64/ld-2.12.so
3bc981f000-3bc9820000 r--p 0001f000 fd:00 34554                          /lib64/ld-2.12.so
3bc9820000-3bc9821000 rw-p 00020000 fd:00 34554                          /lib64/ld-2.12.so
3bc9821000-3bc9822000 rw-p 00000000 00:00 0 
3bc9e00000-3bc9f89000 r-xp 00000000 fd:00 34555                          /lib64/libc-2.12.so
3bc9f89000-3bca189000 ---p 00189000 fd:00 34555                          /lib64/libc-2.12.so
3bca189000-3bca18d000 r--p 00189000 fd:00 34555                          /lib64/libc-2.12.so
3bca18d000-3bca18e000 rw-p 0018d000 fd:00 34555                          /lib64/libc-2.12.so
3bca18e000-3bca193000 rw-p 00000000 00:00 0 
3bcba00000-3bcba16000 r-xp 00000000 fd:00 3770                           /lib64/libresolv-2.12.so
3bcba16000-3bcbc16000 ---p 00016000 fd:00 3770                           /lib64/libresolv-2.12.so
3bcbc16000-3bcbc17000 r--p 00016000 fd:00 3770                           /lib64/libresolv-2.12.so
3bcbc17000-3bcbc18000 rw-p 00017000 fd:00 3770                           /lib64/libresolv-2.12.so
3bcbc18000-3bcbc1a000 rw-p 00000000 00:00 0 
3bd0200000-3bd0216000 r-xp 00000000 fd:00 49774                          /lib64/libgcc_s-4.4.6-20120305.so.1
3bd0216000-3bd0415000 ---p 00016000 fd:00 49774                          /lib64/libgcc_s-4.4.6-20120305.so.1
3bd0415000-3bd0416000 rw-p 00015000 fd:00 49774                          /lib64/libgcc_s-4.4.6-20120305.so.1
7fc5d8add000-7fc5d8ae2000 r-xp 00000000 fd:00 3270                       /lib64/libnss_dns-2.12.so
7fc5d8ae2000-7fc5d8ce1000 ---p 00005000 fd:00 3270                       /lib64/libnss_dns-2.12.so
7fc5d8ce1000-7fc5d8ce2000 r--p 00004000 fd:00 3270                       /lib64/libnss_dns-2.12.so
7fc5d8ce2000-7fc5d8ce3000 rw-p 00005000 fd:00 3270                       /lib64/libnss_dns-2.12.so
7fc5d8ce3000-7fc5d8cef000 r-xp 00000000 fd:00 3272                       /lib64/libnss_files-2.12.so
7fc5d8cef000-7fc5d8eef000 ---p 0000c000 fd:00 3272                       /lib64/libnss_files-2.12.so
7fc5d8eef000-7fc5d8ef0000 r--p 0000c000 fd:00 3272                       /lib64/libnss_files-2.12.so
7fc5d8ef0000-7fc5d8ef1000 rw-p 0000d000 fd:00 3272                       /lib64/libnss_files-2.12.so
7fc5d8ef1000-7fc5d8ef4000 rw-p 00000000 00:00 0 
7fc5d8f1a000-7fc5d8f1d000 rw-p 00000000 00:00 0 
7fff75eac000-7fff75ec1000 rw-p 00000000 00:00 0                          [stack]
7fff75fab000-7fff75fac000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
user123@acc4012 $

What am I doing wrong?

dijetlo · 06-21-2014, 02:35 AM

Quote:

Proxy states 'HTTP/1.0 403 Forbidden

If I had to guess?
You're trying to write into a forbidden location.
Why don't you tell us where you're trying to write the web page to because it looks like your trying to write to the html address you pulled the web page from. What are the permissions on the directory you're trying to FTP into ?

blnl · 06-21-2014, 03:59 PM

The location that I'm writing to is http://home.telfort.nl/chaos_and_order.com/
To prove that the location is not forbidden I can perform wput/wget operation to/from the specified location using my fedora machine which is not behind a proxy:

Code:

[boris@e6410dl ~]$ wput TestFile.ods ftp://user.name:secret@home.telfort.nl
--22:46:39-- `TestFile.ods'
    => ftp://user.name:xxxxx@213.205.34.110:21/TestFile.ods
Connecting to 213.205.34.110:21... connected! 
Logging in as user.name ... Logged in!
Length: 26,296
97% [=========================================================================>  ] 25,600            
22:46:42 (TestFile.ods) - `42.82K/s' [26296]

FINISHED --22:46:42--
Transfered 26,296 bytes in 1 file at  8.35K/s
[boris@e6410dl ~]$ mv TestFile.ods TestFile.bak 
[boris@e6410dl ~]$ wget ftp://user.name:secret@home.telfort.nl/TestFile.ods
--2014-06-21 22:47:19--  ftp://user.name:*password*@home.telfort.nl/TestFile.ods
           => \u2018TestFile.ods\u2019
Resolving home.telfort.nl (home.telfort.nl)... 213.205.34.110
Connecting to home.telfort.nl (home.telfort.nl)|213.205.34.110|:21... connected.
Logging in as user.name ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD not needed.
==> SIZE TestFile.ods ... 26296
==> PASV ... done.    ==> RETR TestFile.ods ... done.
Length: 26296 (26K) (unauthoritative)

100%[==============================================================================>] 26,296      59.1KB/s   in 0.4s   

2014-06-21 22:47:22 (59.1 KB/s) - \u2018TestFile.ods\u2019 saved [26296]

[boris@e6410dl ~]$ diff TestFile.*
Files TestFile.bak and TestFile.ods are identical
[boris@e6410dl ~]$

blnl · 06-21-2014, 04:59 PM

By the way, when I leave out "--proxy=http" option, there is no "Proxy states 'HTTP/1.0 403 Forbidden"

Code:

user123@acc4012 $ wput TestFile.ods ftp://user.name:secret@home.telfort.nl
--23:15:05-- `TestFile.ods'
    => ftp://user.name:xxxxx@home.telfort.nl:21/TestFile.ods
Connecting to home.telfort.nl:21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!
Waiting 10 seconds... Connecting to :21... Error: recv() timed out. No data received
failed!

dijetlo · 06-21-2014, 07:00 PM

Which proxy server are you using?
These instructions pertain to squid.

Quote:

[root@server ~#]vi /etc/squid/squid.conf
acl SSL_ports port 443 21 #(Edit add the ftp port 21 to SSL port acl)
acl ftp proto FTP #(Create a new acl for ftp protocol)
http_access allow ftp #(Allow browsing above created ftp acl)
#save the squid.conf file

However I'd think most proxies are going to need something similar in their config setup. The "acl" calls on the SSL ports and ftp protocols would indicate that at least with Squid, without the configuration modification, you'd probably get a "forbidden" response.

blnl · 06-22-2014, 09:40 AM

I'm not able to change/configure the proxy. Proxy is being administrated by the company IT staff.

However if ncftp (see below) can connect to that location I suppose wput should also be able to do the same.

ncftp session:

Code:

user123@acc3066 ~ $ ncftp  ftp://user.name:secret@home.telfort.nl
NcFTP 3.2.5 (Jan 10, 2011) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to home.telfort.nl via this.is.confidential...                                                              
Ready.
Logging in...                                                                                                       
User user.name logged in.
Quotas on: using 30123.10 of 51200.00 kb
Logged in to home.telfort.nl.                                                                                       
Current remote directory is /.
ncftp / > ll
drwxrwxrwx   2 user.name@telfort.nl storegrp         4096 Jan 10  2008 cgi-art
drwxr-xr-x  12 user.name@telfort.nl storegrp         4096 Jun 15 21:24 Fedora
drwxr-xr-x   2 user.name@telfort.nl storegrp         4096 Oct 31  2008 octave
drwxr-xr-x   9 user.name@telfort.nl storegrp         4096 Aug 28  2010 openSuSE
drwxr-xr-x   2 user.name@telfort.nl storegrp         4096 Apr 26 16:11 Ubuntu
drwxr-xr-x   2 user.name@telfort.nl storegrp         4096 Dec 27  2012 Windows
ncftp / > exit
user123@acc3066 ~ $

This is my proxy configuration for ncftp:

Code:

# NcFTP firewall preferences
# ==========================
#
# If you need to use a proxy for FTP, you can configure it below.
# If you do not need one, leave the ``firewall-type'' variable set
# to 0.  Any line that does not begin with the ``#'' character is
# considered a configuration command line.
#
# NOTE:  NcFTP does NOT support HTTP proxies that do FTP, such as "squid"
#        or Netscape Proxy Server.  Why?  Because you have to communicate with
#        them using HTTP, and this is a FTP only program.
#
# Types of firewalls:
# ------------------
#
#    type 1:  Connect to firewall host, but send "USER user@real.host.name"
#
#    type 2:  Connect to firewall, login with "USER fwuser" and
#             "PASS fwpassword", and then "USER user@real.host.name"
#
#    type 3:  Connect to and login to firewall, and then use
#             "SITE real.host.name", followed by the regular USER and PASS.
#
#    type 4:  Connect to and login to firewall, and then use
#             "OPEN real.host.name", followed by the regular USER and PASS.
#
#    type 5:  Connect to firewall host, but send
#             "USER user@fwuser@real.host.name" and
#             "PASS pass@fwpass" to login.
#
#    type 6:  Connect to firewall host, but send
#             "USER fwuser@real.host.name" and
#             "PASS fwpass" followed by a regular
#             "USER user" and
#             "PASS pass" to complete the login.
#
#    type 7:  Connect to firewall host, but send
#             "USER user@real.host.name fwuser" and
#             "PASS pass" followed by
#             "ACCT fwpass" to complete the login.
#
#    type 8:  Connect to firewall host, but send "USER user@real.host.name:port"
#
#    type 9:  Connect to firewall host, but send "USER user@real.host.name port"
#
#    type 0:  Do NOT use a firewall (most users will choose this).
#
firewall-type=1
#
#
#
# The ``firewall-host'' variable should be the IP address or hostname of
# your firewall server machine.
#
firewall-host=this.is.confidential
#
#
#
# The ``firewall-user'' variable tells NcFTP what to use as the user ID
# when it logs in to the firewall before connecting to the outside world.
#
#firewall-user=user123
#
#
#
# The ``firewall-password'' variable is the password associated with
# the firewall-user ID.  If you set this here, be sure to change the
# permissions on this file so that no one (except the superuser) can
# see your password.  You may also leave this commented out, and then
# NcFTP will prompt you each time for the password.
#
#firewall-password=fwpass
#
#
#
# Your firewall may require you to connect to a non-standard port for
# outside FTP services, instead of the internet standard port number (21).
#
firewall-port=21
#
#
#
# You probably do not want to FTP to the firewall for hosts on your own
# domain.  You can set ``firewall-exception-list'' to a list of domains
# or hosts where the firewall should not be used.  For example, if your
# domain was ``probe.net'' you could set this to ``.probe.net''.
#
# If you leave this commented out, the default behavior is to attempt to
# lookup the current domain, and exclude hosts for it.  Otherwise, set it
# to a list of comma-delimited domains or hostnames.  The special token
# ``localdomain'' is used for unqualified hostnames, so if you want hosts
# without explicit domain names to avoid the firewall, be sure to include
# that in your list.
#
firewall-exception-list=localhost,localdomain
#
#
#
# You may also specify passive mode here.  Normally this is set in the
# regular $HOME/.ncftp/prefs file.  This must be set to one of
# "on", "off", or "optional", which mean always use PASV,
# always use PORT, and try PASV then PORT, respectively.
#
#passive=on
#
#
#
# NOTE:  This file was created for you on Thu Jun 19 21:50:59 2014
#        by NcFTP 3.2.5.  Removing this file will cause the next run of NcFTP
#        to generate a new one, possibly with more configurable options.
#
# ALSO:  A /usr/local/etc/ncftp.firewall file, if present, is processed before this file,
#        and a /usr/local/etc/ncftp.firewall.fixed file, if present, is processed after.

dijetlo · 06-22-2014, 01:28 PM

Quote:

Proxy is being administrated by the company IT staff.

Have you spoken with them?

blnl · 06-22-2014, 06:02 PM

Yes, I opened a support ticket.
First line of support does not have the knowledge to give a useful advice, while this is not high enough priority to get to the second line of support.

dijetlo · 06-22-2014, 08:04 PM

blnl, I'm going to refer you back to post #2, you are trying to write to a forbidden location. I'm not going to tell you how to bypass your corporate proxy server, other than to suggest that if you have a cell phone that you can hook to your laptop and get internet that way... as long as it's OK with them that would fix your problem AND has the added advantage of not getting you fired for being a pernicious threat to domain security.

blnl · 06-24-2014, 03:32 PM

Quote:

Originally Posted by dijetlo

blnl, I'm going to refer you back to post #2, you are trying to write to a forbidden location. I'm not going to tell you how to bypass your corporate proxy server, other than to suggest that if you have a cell phone that you can hook to your laptop and get internet that way... as long as it's OK with them that would fix your problem AND has the added advantage of not getting you fired for being a pernicious threat to domain security.

There is no reason to be concerned about my job security

Firstly, I'm not trying to bypass the proxy server. I need a solution that does not violate the security.
Secondly, it is in fact the IT department that advised me to try out wget and wput as means to exchange large files with the supplier ftp account. For some reason ,unknown to me, after upgrading our server farm to RHEL 6, there is neither ftp nor gftp available on new servers. We only have a few RHEL 5 machines left that still have the ftp capability (but these machines are going to be decommissioned any time soon).
Finally, I already have found a working solution (ncftp) to exchange files with supplier (as it would be inappropriate to use supplier account for testing, I'm using my private ftp account instead). ncftp works fine with corporate proxy as I demonstrated earlier in this thread. Getting the wput working is out of pure curiosity (not the necessity)

It seams that wput is not complementary to wget, as wget can connect trough proxy and wput cannot. I'll try posting this question at wput support...