CentOS-7: curl & wget only working infrequently, but yum works fine

tyler2016 · 05-10-2019, 06:11 AM

Have you tried Python's urllib? Since yum is Python, I wonder if it is doing something different:

Code:

$ python2
Python 2.7.13 (default, Sep 26 2018, 18:42:22) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> u = urllib2.urlopen('https://www.lua.org/ftp/lua-5.3.5.tar.gz')
>>> f = open('/tmp/lua.tgz', 'wb')
>>> f.write(u.read())
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> exit()
$ ls -l /tmp/lua.tgz 
-rw-r--r-- 1 tyler tyler 303543 May 10 07:10 /tmp/lua.tgz

Just make sure that the URL you try doesn't have any redirects.

cheddarcheese · 05-10-2019, 07:59 AM

Nice thinking. Okay, I used the same Python code as you offered, amending the URL to one which I've previously tried unsuccessfully with curl and wget, and it worked just fine -- fast and complete. In fact I tried it a couple of times to make sure it wasn't a fluke, and it worked just fine each time.

Hmmm. So, I guess that determines something at least, which is useful, as it kinda rules out a bunch of other stuff, but I'm not sure how to use this latest knowledge to effect a solution with non-Python tools such as curl/wget/etc.

I guess there must be some common library or something which the latter is using which isn't playing nicely for some reason, but I've no clue how to figure that out or how to fix it. My first inclination would be simply to ensure that CentOS is up to date, but I regularly do yum updates, and the machine doesn't think there's currently anything to update.

Thank you very much indeed for pointing me in a new direction; am hoping that maybe someone can now help me across the finish line!

tyler2016 · 05-10-2019, 08:17 AM

What does your /usr/local look like? Do you have anything you built from source in there? Did you build curl or wget from source? Your version of curl is newer than the one in the standard repositories. Assuming a yes, verify the compile time options and try setting LD_LIBRARY_PATH to /usr/local/lib or wherever you put it.

cheddarcheese · 05-10-2019, 08:44 AM

Actually, I don't recall exactly where I got the version of curl from, but it wasn't built from source; the reason it is a very recent version is that - in attempting to find a solution to the download problem - I'd been endeavoring to ensure everything that could possibly be updated was updated (i.e. the problem existed before this most recent version of curl was installed), but obviously it didn't make any difference in the end.

Nothing in /usr/local/ of any consequence (I don't think): /usr/local/src/ is empty, as is /usr/local/lib/.