LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Red Hat
User Name
Password
Red Hat This forum is for the discussion of Red Hat Linux.

Notices


Reply
  Search this Thread
Old 04-17-2014, 06:23 PM   #1
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
RHEL5.9 curl to https openssl/heartbleed issue


We’ve run into a problem wherein a 3rd party vendor’s https site recently updated for the openssl/heartbleed issue is now intermittently failing when we try to use curl to access it.

The cogent portion of the error from curl is:
SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 *
Empty reply from server

We’re seeing this when using RHEL5 (which uses openssl 0.9.8 and is therefore not *directly* impacted by the heartbleed issue) running the curl 7.15.5-15.el5 (which has curl upstream 7.15.5 as its upstream base).
On speaking with the vendor and testing on one of our CentOS5 (which has same versioning since it is compiled from RHEL5 source) we found that the latest repository supplied versions of curl 7.15.5 and openssl 0.9.8e do not resolve the problem. However on installing the latest upstream version (7.3.6) directly from Curl’s site the issue doesn’t seem to exist any longer.

This makes it seem that something in the new openssl 1.0.1g wbe servers is breaking older curl versions even when run from supposedly unaffected openssl 0.9.8e servers like RHEL 5.9.

I’m wondering if anyone else has run into this with any of their vendors.

Note that I HAVE actually searched for the above error and found it has occurred for various people (often to Amazon apparently) but none of the threads I've found relate it to the openssl update (mainly because they predate it). None of the discussion seems germane to what we're seeing.

P.S. Updating to curl 7.3.6 was a pain. One must:
1) Download curl and libcurl as well as various libraries mentioned at the top of the page:
http://mirror.city-fan.org/ftp/contr...ils/Mirroring/
2) Remove some software that relies on a library that goes away when old version of curl/libcurl is updated to new version of curl/libcurl.

We've opened a ticket with RedHat to see if they can backport whatever it is in 7.3.6 that makes it work to their extended 7.15.5 version so we can avoid updating to the upstream on our Production server. I just wanted to see if anyone else has gone through this already.
 
Old 04-17-2014, 11:11 PM   #2
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,622

Rep: Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651
for RHEL5.9 ( assuming you did auto upgrade to 5.10 or are locking it at 5.9)
redhat patched it
all you need to do is update the install and the openssl bug is patched

is the centos updated to 5.10
if it is not ,then 5.9 is unsupported and will never receive the openssl fix
 
Old 04-18-2014, 09:09 AM   #3
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831

Original Poster
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
I'm not asking how to patch for heartbleed. In RHEL5.x you do not need to update openssl because it is running 0.9.8e based version which is unaffected by heartbleed. It is only RHEL6.5 and greater that had the affected version of openssl (1.0.1e).

As noted my issue is with the curl on RHEL5 making an outbound connection to an https site and it is intermittent. It started when the external site updated their side for the heartbleed issue.

For the record, in testing we DID first update to latest curl and openssl on the CentOS 5 server from its repositories (which has same updates as the RHEL5) even though the openssl update shouldn't have been necessary. Despite that we still had the issue. The 3rd party vendor had also been testing on CentOS5 and saw the same thing on their end.

Overnight I also updated a RHEL5.5 and another RHEL5.9 server to the latest curl and openssl versions from the RHEL5 repositories (which as noted above is the same as the one CentOS previously tested) but still have the issue on those servers also.

So far the only fix I've found is the one the 3rd party vendor suggested which was download and update using curl 7.3.6 from curl site directly. We don't want to do that on our RHEL5 Production server and have engaged RedHat support to explain why the repository version of curl is having this issue. That is to say in our view it is a "bug" in the extended RHEL 5 version of curl.

My post here was to see if anyone else has run into it.
 
Old 04-21-2014, 08:28 AM   #4
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831

Original Poster
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
Update and probable resolution:

First a little more detail on the output of the "curl -v" we were running sanitized to remove the URL, IP and data we were passing in curl -v:

* About to connect() to <somesite.somedomain.com> port 443
* Trying 209.49.147.26... connected
* Connected to creditserver.microbilt.com <(IP of somesite)> port 443
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using RC4-MD5
* Server certificate:
* subject: /C=US/ST=Tennessee/L=Nashville/O=<Somedomain>
Corporation/CN=<somesite.somedomain.com>
* start date: 2013-02-22 00:00:00 GMT
* expire date: 2015-04-28 12:00:00 GMT
* subjectAltName: <somesite.somedomain.com> matched
* issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High
Assurance CA-3
* SSL certificate verify ok.
> GET
> /SDK/PostData.asp?<data passed in the curl -v> HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: <somesite.somedomain.com>
> Accept: */*
>
* SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
* Empty reply from server

* Connection #0 to host <somesite.somedomain.com> left intact
curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
* Closing connection #0"

We had found that this WAS working OK in RHEL6 which has a curl version based on 7.19.

After other testing suggested by RedHat we found that adding "-1" or "-3" to the curl -v we did not get the failure on multiple attempts (where as without them we got it frequently - usually by 3rd to 5th attempt but often on the 1st attempt). We also found (not suggested by RedHat) that it ALWAYS failed if we added "-2". These 3 options determine the type of connection one is making to an https site.

After that we added the "-3" option to ~.curlrc in the home directory of the affected users (Oracle administrative accounts in our case) so that they would always use "-3" any time curl was invoked.

That was Friday night at 10 PM US ET. That night we saw the next 3 automated jobs that ran in the next hour all completed successfully. On Saturday we saw the automated process run 41 times and only return negative results twice. My assumption is the 2 negative results were "normal" results.

Today we should see an even higher activity of the automated processes.

Just wanted to update in case anyone else is struggling with this. It appears that the newer versions of curl after 7.15 are not defaulting to SSLv2 so if you're seeing the above errors try running command line tests with the "-3" and/or the "-1".
 
Old 04-23-2014, 03:50 PM   #5
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831

Original Poster
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
Resolution:

On last check we didn't seem to be having this issue any longer and had seen a lot more traffic. Since then I've heard no complaints so it appears adding the "-3" option for curl resolved our issue.

We found out later that the vendor did not update openssl version - They were using an F5 load balancer on the front end and the 10.2.4 version of F5 OS (an embedded Linux) was not directly affected by heartbleed because it did not have the affeccted openssl version. Despite that they put in an F5 Irule change and it appears to be this that caused the issue to start occurring.

Other notes on the F5 in case anyone reading this has same issue:

Versions 11.5 & 11.5.1 DO have the affected openssl version.

The Irule they put on the F5 was likely not necessary.

On F5 you can put the SSL certificate on the load balancer itself so that internet traffic uses it rather than the back end servers. The F5 in turn makes its connections to the the back end servers using http rather than https (since the https was already done on the front end facing the internet).

Alternatively you can put the certificate(s) on the back end servers and have the F5 use https to them. The F5 documentation indicates one CAN put in an Irule to mitigate the issue if the back end web servers are affected and not patched in such a case.
It seems instead you'd want to patch your web servers but I guess F5 was just offering additional assistance for those who don't.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] OpenSSL Heartbleed and my old Slackware 12 czezz Slackware 5 05-06-2014 09:42 AM
LXer: Test Sites for Heartbleed OpenSSL Vulnerability LXer Syndicated Linux News 0 04-09-2014 01:00 PM
LXer: Heartbleed: Serious OpenSSL zero day vulnerability revealed LXer Syndicated Linux News 1 04-08-2014 07:38 AM
[SOLVED] Need suggestion:->>Failed HTTPS transfer to https://supportfiles.sun.com/curl manalisharmabe Solaris / OpenSolaris 11 01-10-2014 12:58 AM
Curl HTTPS OpenSSL Certificate issue Manjunath1847 Linux - General 1 08-09-2010 10:13 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Red Hat

All times are GMT -5. The time now is 09:01 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration