LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-22-2019, 01:32 PM   #1
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Rep: Reputation: 7
Unhappy CentOS-7: curl & wget only working infrequently, but yum works fine


I originally posted this in the networking section a couple of weeks back, but I've not had any suggestions for solutions, so I've reposted here.

My problem is that curl and wget usually fail to retrieve more than just the first part of any given file, and often nothing at all. Both have worked previously on the same server, and nothing else on my network has changed - and I'm not really sure what's changed on my server to now make them not work, although very intermittently it will work just fine.

I'm not exactly quite sure when this started happening, but it's been like it for a while (a few months), and I just put off doing anything about it - and also because I have no idea how to resolve, other than what I've already tried.

So, my server is running CentOS Linux release 7.6.1810

My version of curl is very recent:

Code:
curl 7.64.0 (x86_64-redhat-linux-gnu) libcurl/7.64.0 NSS/3.36
zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.8.0 nghttp2/1.31.1
The issue was first apparent to me when I was trying to do WordPress updates (and it's been an issue for a while), when updates will fail with, for example:

Code:
Download failed. cURL error 28: Operation timed out after 300000 milliseconds with 1355444 out of 5007255 bytes received
However, I'm certain this is not a WP issue, as I've also tried from the command line using both curl and wget, and have the same issue.

Occasionally the downloads will work, but I think that tends to be with smaller files. However, even so, every now and then it will work just fine with bigger files.

Code:
e.g. $ curl -O -v https://downloads.wordpress.org/plugin/nimble-builder.1.5.0.zip  
	....  
	curl: (56) I/O operation timed out
I've tried several things, including:

a) resolv.conf
- previously had added: options single-request-reopen, which resolved a conflict with IPv4/IPv6 many months ago
- also used several different DNS server setups
b) tried fetching stuff from both http and https sites
c) tried fetching using IP addresses instead of domain names, and tried downloading different stuff from different sites.
d) firewall - temporarily disabled iptables and tried more curl requests, but made no difference. Re-enabled iptables.
e) rebooted
f) all OS updates are installed

I can use yum with no problems.

Sample strace output is given below:

Code:
    Near beginning, a lot like this:
    open("/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
    read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220H\0\0\0\0\0\0"..., 832) = 832
    fstat(3, {st_mode=S_IFREG|0755, st_size=210824, ...}) = 0
    mmap(NULL, 2302664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff0cbf8c000
    mprotect(0x7ff0cbfbd000, 2093056, PROT_NONE) = 0
    mmap(0x7ff0cc1bc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x30000) = 0x7ff0cc1bc000
    close(3)                                = 0
    
    Part way through, stuff like this:
    access("/home/xyz/.pki/nssdb/.57765795_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765796_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765797_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765798_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765799_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    access("/home/xyz/.pki/nssdb/.57765800_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory)
    ... 
    and quite a few of these:
    recvfrom(3, 0x184e0d5, 12115, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    poll([{fd=3, events=POLLIN}], 1, 1000)  = 1 ([{fd=3, revents=POLLIN}])
    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])
    recvfrom(3, "\322U\200l9\332+\31w\235*\322dt\233\273\37]\244~\264{\24g\2\301\320\276\271\207\337["..., 12115, 0, NULL, NULL) = 1448
    recvfrom(3, 0x184e67d, 10667, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    
    Close to end, a lot like this:
    write(2, "k", 1)                        = 1
    poll([{fd=3, events=POLLIN}], 1, 1000)  = 0 (Timeout)
    poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
    write(2, "\r", 1)
Should be no issues with memory:

Memory:
Code:
                  total        used        free      shared  buff/cache   available
    Mem:        8175404     1734268      310528       20832     6130608     5936760
    Swap:       8257532       85472     8172060
Also tried disabling IPv6:
Code:
$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
$ sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Really not sure what else to try. It's very frustrating. Any assistance appreciated.

Last edited by cheddarcheese; 03-22-2019 at 02:30 PM. Reason: Additional info about disabling IPv6.
 
Old 03-22-2019, 02:03 PM   #2
dc.901
Senior Member
 
Registered: Aug 2018
Location: Atlanta, GA - USA
Distribution: CentOS/RHEL, openSuSE/SLES, Ubuntu
Posts: 1,005

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Quote:
Originally Posted by cheddarcheese View Post
However, I'm certain this is not a WP issue, as I've also tried from the command line using both curl and wget, and have the same issue.

Occasionally the downloads will work, but I think that tends to be with smaller files. However, even so, every now and then it will work just fine with bigger files.
Have you tried to download anything else besides WP files?

How about:
- This will be several GB: wget http://isoredirect.centos.org/altarc...thing-1810.iso
- Several MB: wget http://isoredirect.centos.org/altarc...stall-1810.iso

Also, is the time correct in BIOS? I have had similar odd issues then realized BIOS time was WAY off (this was after motherboard replacement).
 
Old 03-22-2019, 02:21 PM   #3
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
Hi,

Thanks for the comments. Yeah, I already tried lots of different stuff besides downloading WP files. (among them the test files from https://www.thinkbroadband.com/download).

Your other idea with the BIOS clock was interesting: I hadn't checked that, but subsequently I've now learned about hwclock and set BIOS to system time. I've now tried a couple of downloads from different places (WP and non-WP), but it's seemed to have made no difference.

Any other ideas welcome. Thx!
 
Old 03-22-2019, 04:09 PM   #4
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 19,872
Blog Entries: 12

Rep: Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053
duplicate of https://www.linuxquestions.org/quest...ne-4175649810/
 
Old 03-22-2019, 05:03 PM   #5
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
As noted in the first line of the post, "I originally posted this in the networking section a couple of weeks back, but I've not had any suggestions for solutions, so I've reposted here." ... where at least I've had one useful response so far, even if it didn't pan out.
 
Old 03-22-2019, 05:12 PM   #6
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 21 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925Reputation: 2925
Quote:
Originally Posted by cheddarcheese View Post
As noted in the first line of the post, "I originally posted this in the networking section a couple of weeks back, but I've not had any suggestions for solutions, so I've reposted here." ... where at least I've had one useful response so far, even if it didn't pan out.
Normally, if the above is the case, then you can report the opening post in your original thread and ask a mod/admin to move the thread to a different subforum, thus avoiding thread duplication.
 
Old 03-22-2019, 05:19 PM   #7
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
Ah, okay; I didn't know that. I did deliberately mention that I'd previously posted in the other section, to avoid any annoyance on anyone's part, but I'll know for future reference; thanks.

Anyway, if anyone is able to offer some sparks of genius to help resolve the problem then I'll be hugely grateful; it seems I've tried just about everything, but to no avail.
 
Old 03-22-2019, 05:45 PM   #8
dc.901
Senior Member
 
Registered: Aug 2018
Location: Atlanta, GA - USA
Distribution: CentOS/RHEL, openSuSE/SLES, Ubuntu
Posts: 1,005

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Quote:
Originally Posted by cheddarcheese View Post
Hi,

Thanks for the comments. Yeah, I already tried lots of different stuff besides downloading WP files. (among them the test files from https://www.thinkbroadband.com/download).

Your other idea with the BIOS clock was interesting: I hadn't checked that, but subsequently I've now learned about hwclock and set BIOS to system time. I've now tried a couple of downloads from different places (WP and non-WP), but it's seemed to have made no difference.

Any other ideas welcome. Thx!
I'm curious, have you tried with "--no-check-certificate"

Code:
wget --no-check-certificate <url>
While you get the error with wget and curl, is there anything else going into messages / syslog?
 
Old 03-22-2019, 06:15 PM   #9
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
I think I had already checked messages in /var/log/messages before, but just cleared it out and started trying downloads again, with both curl and wget, but nothing new appeared in the log when I did try.

I hadn't previously tried wget with
Code:
--no-check-certificate
, but just did so; same problem. Also tried curl with -k:
Code:
$ curl -O -k  http://ipv4.download.thinkbroadband.com/512MB.zip
- which I think is curl's equivalent to wget's --no-check-certificate, but it made no difference. Tried several different downloads, and each of them begins to download, but after maybe three or four seconds it just stalls as almost always.

Last edited by cheddarcheese; 03-22-2019 at 06:20 PM.
 
Old 03-23-2019, 07:06 PM   #10
tyler2016
Member
 
Registered: Sep 2018
Distribution: Debian, CentOS, FreeBSD
Posts: 243

Rep: Reputation: Disabled
Have you ruled out layer one issues?
 
Old 03-23-2019, 08:55 PM   #11
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
Well, being no expert in such things I can't really rule anything out unless I can explicitly test for something. Still, with regards to physical layer stuff then I don't really think it would be (but welcome to be proven wrong), as all other network functionality seems normal from the same machine (yum works, and a web-server works on there, etc, etc), and the same firewall/router is also used by another machine (Windows) where I have no problems, and I'm not sure what else I would really check.
 
Old 03-24-2019, 05:42 AM   #12
tyler2016
Member
 
Registered: Sep 2018
Distribution: Debian, CentOS, FreeBSD
Posts: 243

Rep: Reputation: Disabled
If you have any spare cables, try swapping cables, same with NICs, switch ports.
 
Old 03-24-2019, 04:58 PM   #13
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
I can do a switcheroo with cables and ports, but not possible with NICs. I'll play around with that tomorrow when I'm back at the machine.

However, with due respect, I don't see how that could make any difference when it's already established that no other network services on the same machine have any problems with the same setup.
 
Old 03-24-2019, 07:13 PM   #14
tyler2016
Member
 
Registered: Sep 2018
Distribution: Debian, CentOS, FreeBSD
Posts: 243

Rep: Reputation: Disabled
Quote:
Originally Posted by cheddarcheese View Post
I can do a switcheroo with cables and ports, but not possible with NICs. I'll play around with that tomorrow when I'm back at the machine.

However, with due respect, I don't see how that could make any difference when it's already established that no other network services on the same machine have any problems with the same setup.
To be honest, I'm pretty baffled myself. I suggested hardware since hardware failures can do some weird things and are easy to test. I remember the old story about some engineers that couldn't figure out why a complicated machine wouldn't work. Some less knowledgeable person asked an obvious question about a cable being plugged in. It turns out they were right, somebody had tripped over it.
 
Old 05-09-2019, 03:32 PM   #15
cheddarcheese
Member
 
Registered: Aug 2011
Location: Massachusetts, USA
Distribution: Fedora; Centos; Puppy
Posts: 116

Original Poster
Rep: Reputation: 7
Well, a month later, but still no progress. I cannot think of anything else to try. Would be eternally grateful if anyone ever gets any ideas on what the problem could be.
 
  


Reply

Tags
centos, curl, wget


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
CentOS-7: curl & wget only working infrequently, but yum works fine cheddarcheese Linux - Networking 4 03-10-2019 06:52 AM
[SOLVED] yum works but wget/curl does not. PoleStar Linux - Newbie 4 12-03-2018 02:13 PM
2nd monitor is detected infrequently amukher Linux - Desktop 1 11-29-2015 07:06 AM
yum working fine on server and ftp is working f9 but not able to config yum on client joj123 Red Hat 12 01-13-2015 06:03 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration