LinuxQuestions.org - CentOS-7: curl & wget only working infrequently, but yum works fine

- Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)

- - CentOS-7: curl & wget only working infrequently, but yum works fine (https://www.linuxquestions.org/questions/linux-software-2/centos-7-curl-and-wget-only-working-infrequently-but-yum-works-fine-4175650688/)

CentOS-7: curl & wget only working infrequently, but yum works fine

I originally posted this in the networking section a couple of weeks back, but I've not had any suggestions for solutions, so I've reposted here.

My problem is that curl and wget usually fail to retrieve more than just the first part of any given file, and often nothing at all. Both have worked previously on the same server, and nothing else on my network has changed - and I'm not really sure what's changed on my server to now make them not work, although very intermittently it will work just fine.

I'm not exactly quite sure when this started happening, but it's been like it for a while (a few months), and I just put off doing anything about it - and also because I have no idea how to resolve, other than what I've already tried.

So, my server is running CentOS Linux release 7.6.1810

My version of curl is very recent:

Code:

curl 7.64.0 (x86_64-redhat-linux-gnu) libcurl/7.64.0 NSS/3.36

zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.8.0 nghttp2/1.31.1

The issue was first apparent to me when I was trying to do WordPress updates (and it's been an issue for a while), when updates will fail with, for example:

Code:

Download failed. cURL error 28: Operation timed out after 300000 milliseconds with 1355444 out of 5007255 bytes received

However, I'm certain this is not a WP issue, as I've also tried from the command line using both curl and wget, and have the same issue.

Occasionally the downloads will work, but I think that tends to be with smaller files. However, even so, every now and then it will work just fine with bigger files.

Code:

e.g. $ curl -O -v https://downloads.wordpress.org/plugin/nimble-builder.1.5.0.zip  

        ....  

        curl: (56) I/O operation timed out

I've tried several things, including:

a) resolv.conf
- previously had added: options single-request-reopen, which resolved a conflict with IPv4/IPv6 many months ago
- also used several different DNS server setups
b) tried fetching stuff from both http and https sites
c) tried fetching using IP addresses instead of domain names, and tried downloading different stuff from different sites.
d) firewall - temporarily disabled iptables and tried more curl requests, but made no difference. Re-enabled iptables.
e) rebooted
f) all OS updates are installed

I can use yum with no problems.

Sample strace output is given below:

Code:

    Near beginning, a lot like this:

    open("/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3

    read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220H\0\0\0\0\0\0"..., 832) = 832

    fstat(3, {st_mode=S_IFREG|0755, st_size=210824, ...}) = 0

    mmap(NULL, 2302664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff0cbf8c000

    mprotect(0x7ff0cbfbd000, 2093056, PROT_NONE) = 0

    mmap(0x7ff0cc1bc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x30000) = 0x7ff0cc1bc000

    close(3)                                = 0

    

    Part way through, stuff like this:

    access("/home/xyz/.pki/nssdb/.57765795_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    access("/home/xyz/.pki/nssdb/.57765796_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    access("/home/xyz/.pki/nssdb/.57765797_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    access("/home/xyz/.pki/nssdb/.57765798_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    access("/home/xyz/.pki/nssdb/.57765799_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    access("/home/xyz/.pki/nssdb/.57765800_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)

    ... 

    and quite a few of these:

    recvfrom(3, 0x184e0d5, 12115, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

    poll([{fd=3, events=POLLIN}], 1, 1000)  = 1 ([{fd=3, revents=POLLIN}])

    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])

    recvfrom(3, "\322U\200l9\332+\31w\235*\322dt\233\273\37]\244~\264{\24g\2\301\320\276\271\207\337["..., 12115, 0, NULL, NULL) = 1448

    recvfrom(3, 0x184e67d, 10667, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

    

    Close to end, a lot like this:

    write(2, "k", 1)                        = 1

    poll([{fd=3, events=POLLIN}], 1, 1000)  = 0 (Timeout)

    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)

    write(2, "\r", 1)

Should be no issues with memory:

Memory:

Code:

                  total        used        free      shared  buff/cache  available

    Mem:        8175404    1734268      310528      20832    6130608    5936760

    Swap:      8257532      85472    8172060

Also tried disabling IPv6:

Code:

$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

$ sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Really not sure what else to try. It's very frustrating. Any assistance appreciated.

Quote:

Originally Posted by cheddarcheese (Post 5976587)

Have you tried to download anything else besides WP files?

How about:
- This will be several GB: wget http://isoredirect.centos.org/altarc...thing-1810.iso
- Several MB: wget http://isoredirect.centos.org/altarc...stall-1810.iso

Also, is the time correct in BIOS? I have had similar odd issues then realized BIOS time was WAY off (this was after motherboard replacement).

Hi,

Thanks for the comments. Yeah, I already tried lots of different stuff besides downloading WP files. (among them the test files from https://www.thinkbroadband.com/download).

Your other idea with the BIOS clock was interesting: I hadn't checked that, but subsequently I've now learned about hwclock and set BIOS to system time. I've now tried a couple of downloads from different places (WP and non-WP), but it's seemed to have made no difference.

Any other ideas welcome. Thx!

As noted in the first line of the post, "I originally posted this in the networking section a couple of weeks back, but I've not had any suggestions for solutions, so I've reposted here." ... where at least I've had one useful response so far, even if it didn't pan out.

Quote:

Originally Posted by cheddarcheese (Post 5976652)

Normally, if the above is the case, then you can report the opening post in your original thread and ask a mod/admin to move the thread to a different subforum, thus avoiding thread duplication.

Ah, okay; I didn't know that. I did deliberately mention that I'd previously posted in the other section, to avoid any annoyance on anyone's part, but I'll know for future reference; thanks.

Anyway, if anyone is able to offer some sparks of genius to help resolve the problem then I'll be hugely grateful; it seems I've tried just about everything, but to no avail.

Quote:

Originally Posted by cheddarcheese (Post 5976603)

I'm curious, have you tried with "--no-check-certificate"

Code:

wget --no-check-certificate <url>

While you get the error with wget and curl, is there anything else going into messages / syslog?

I think I had already checked messages in /var/log/messages before, but just cleared it out and started trying downloads again, with both curl and wget, but nothing new appeared in the log when I did try.

I hadn't previously tried wget with

Code:

--no-check-certificate

, but just did so; same problem. Also tried curl with -k:

Code:

$ curl -O -k http://ipv4.download.thinkbroadband.com/512MB.zip

- which I think is curl's equivalent to wget's --no-check-certificate, but it made no difference. Tried several different downloads, and each of them begins to download, but after maybe three or four seconds it just stalls as almost always.

Have you ruled out layer one issues?

Well, being no expert in such things I can't really rule anything out unless I can explicitly test for something. Still, with regards to physical layer stuff then I don't really think it would be (but welcome to be proven wrong), as all other network functionality seems normal from the same machine (yum works, and a web-server works on there, etc, etc), and the same firewall/router is also used by another machine (Windows) where I have no problems, and I'm not sure what else I would really check.

If you have any spare cables, try swapping cables, same with NICs, switch ports.

I can do a switcheroo with cables and ports, but not possible with NICs. I'll play around with that tomorrow when I'm back at the machine.

However, with due respect, I don't see how that could make any difference when it's already established that no other network services on the same machine have any problems with the same setup.

Quote:

Originally Posted by cheddarcheese (Post 5977373)

To be honest, I'm pretty baffled myself. I suggested hardware since hardware failures can do some weird things and are easy to test. I remember the old story about some engineers that couldn't figure out why a complicated machine wouldn't work. Some less knowledgeable person asked an obvious question about a cable being plugged in. It turns out they were right, somebody had tripped over it.

Well, a month later, but still no progress. I cannot think of anything else to try. Would be eternally grateful if anyone ever gets any ideas on what the problem could be.