CentOS-7: curl & wget only working infrequently, but yum works fine

cheddarcheese · 03-08-2019, 01:58 PM

I hope I'm posting this in the right place, as I tend to think that it's more likely a network sort of thing than anything else, but I could be wrong.

My problem is that curl and wget usually fail to retrieve more than just the first part of any given file, and often nothing at all. Both have worked previously on the same server, and nothing else on my network has changed - and I'm not really sure what's changed on my server to now make them not work, although very intermittently it will work just fine.

I'm not exactly quite sure when this started happening, but it's been like it for a while (a few months), and I just put off doing anything about it - and also because I have no idea how to resolve, other than what I've already tried.

So, my server is running CentOS Linux release 7.6.1810

My version of curl is very recent:

Code:

curl 7.64.0 (x86_64-redhat-linux-gnu) libcurl/7.64.0 NSS/3.36
zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.8.0 nghttp2/1.31.1

The issue was first apparent to me when I was trying to do WordPress updates (and it's been an issue for a while), when updates will fail with, for example:

Code:

Download failed. cURL error 28: Operation timed out after 300000 milliseconds with 1355444 out of 5007255 bytes received

Occasionally the updates will work, but I think that tends to be with smaller files. However, even so, every now and then it will work just fine with bigger files.

So, with regards to trying to resolve the issue with WordPress, I tried:

- increasing the timeout in Wordpress files: class-wp-http-curl.php and class-http.php
- increasing max_execution_time in /etc/php.ini to 120

... but no change. Also tried from the command line using both curl and wget, but same issue.

Code:

e.g. $ curl -O -v https://downloads.wordpress.org/plugin/nimble-builder.1.5.0.zip  
	....  
	curl: (56) I/O operation timed out

Also:

a) resolv.conf
- previously had added: options single-request-reopen, which resolved a conflict with IPv4/IPv6 many months ago
- also used several different DNS server setups
b) tried fetching stuff from both http and https sites
c) tried fetching using IP addresses instead of domain names, and tried downloading different stuff from different sites.
d) firewall - temporarily disabled iptables and tried more curl requests, but made no difference. Re-enabled iptables.
e) rebooted
f) all OS updates are installed

I can use yum with no problems.

Sample strace output is given below:

Code:

    Near beginning, a lot like this:
    open("/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
    read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220H\0\0\0\0\0\0"..., 832) = 832
    fstat(3, {st_mode=S_IFREG|0755, st_size=210824, ...}) = 0
    mmap(NULL, 2302664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff0cbf8c000
    mprotect(0x7ff0cbfbd000, 2093056, PROT_NONE) = 0
    mmap(0x7ff0cc1bc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x30000) = 0x7ff0cc1bc000
    close(3)                                = 0
    
    Part way through, stuff like this:
    access("/home/xyz/.pki/nssdb/.57765795_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765796_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765797_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765798_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765799_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765800_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    ... 
    and quite a few of these:
    recvfrom(3, 0x184e0d5, 12115, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    poll([{fd=3, events=POLLIN}], 1, 1000)  = 1 ([{fd=3, revents=POLLIN}])
    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])
    recvfrom(3, "\322U\200l9\332+\31w\235*\322dt\233\273\37]\244~\264{\24g\2\301\320\276\271\207\337["..., 12115, 0, NULL, NULL) = 1448
    recvfrom(3, 0x184e67d, 10667, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    
    Close to end, a lot like this:
    write(2, "k", 1)                        = 1
    poll([{fd=3, events=POLLIN}], 1, 1000)  = 0 (Timeout)
    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
    write(2, "\r", 1)

Should be no issues with memory:

Memory:

Code:

                  total        used        free      shared  buff/cache   available
    Mem:        8175404     1734268      310528       20832     6130608     5936760
    Swap:       8257532       85472     8172060

Really not sure what else to try. It's very frustrating. Any assistance appreciated.

ondoho · 03-08-2019, 02:09 PM

please play with curl's/wget's options to see if you can narrow it down or at least get more information.

it's a wild guess, but i've twice had the issue that half the internet seemed to have stopped working, and both times it was an ipv4/ipv6 issue.

cheddarcheese · 03-08-2019, 02:24 PM

Well, I've tried pretty much everything I can think of, and verbose mode in curl didn't really reveal too much. Even the same file download will fail a hundred times but then work perfectly fine once or twice. I don't doubt the ipv4/ipv6 possibilities, having experienced that before (hence the addition to resolv.conf), but I'm really not very clued up on networky type of stuff, and kinda need some pointers in the right direction ... wherever that is.

Also just tried disabling IPv6:

Code:

$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
$ sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

.. but it seems to make no difference.

ondoho · 03-10-2019, 06:27 AM

^ i hope you undid that again.

why are you mentioning wordpress?
how are you executing the curl commands?
i don't see how wordpress settings can have an impact on a command issued from the command line.

please show us the commands & their complete output.

if this is a wordpress issue, a search for "cURL error 28" gives relevant-looking results.

cheddarcheese · 03-10-2019, 06:52 AM

- why are you mentioning wordpress?
- As clearly explained, because that's how I discovered the problem

- how are you executing the curl commands?
- As one would expect ... from the command line

- i don't see how wordpress settings can have an impact on a command issued from the command line.
- I'm sure they wouldn't.

- if this is a wordpress issue, a search for "cURL error 28" gives relevant-looking results.
- As already [obviously] discovered, the problem is not WP, it's most likely network related ... hence the question is posted in "Networking".

- please show us the commands & their complete output.
- The original post already more or less does this, unless there's something very specific you're looking for.