LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-02-2016, 02:03 AM   #1
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Rep: Reputation: Disabled
Question cUrl not behaving the way it used to


I have followed every piece of advice, consulted the man pages numerous times, and no syntax seems to work when downloading pictures with the files I give them. They consistently arrive on my hard drive with the original filename (e.g.: 2.jpg) instead of the more descriptive name I assign in the "curl [options] [URL]" string that downloads the file.

wget? Use wget? I would except there are some sites where wget pulls down a 'dummy" image instead of the one I see on the webpage. I suspect that might have something to do with assigning Konqueror as user-agent in my wgetrc instead of a more widely-used browser like Firefox. (There again I'm the maverick)

Why use a CLI downloader at all? Until I can bash script a browser, I will prefer the flexibility of curl, wget, Axel (which I love except for the fact that it doesn't preserve server timestamps) and aria2.

Not too long ago, I had bash scripts written for cUrl. Of course, that was a few cUrl versions back; if these new behaviours came as part of the development arc, I'll write scripts that invoke wget instead.

Carver
 
Old 12-02-2016, 03:04 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,655
Blog Entries: 3

Rep: Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901
Which options are you actually giving to curl? Please show a sample that is giving you trouble.
 
Old 12-02-2016, 04:00 AM   #3
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
An example

Quote:
Originally Posted by Turbocapitalist View Post
Which options are you actually giving to curl? Please show a sample that is giving you trouble.
Here's a (somewhat edited) example from my bash history:
Code:
curl http://galleries.blahblah.com/77/7765/071/pic/5.jpg -o "blah77-7765-071-005.jpg"
You'll notice the URL is in front of the 'local' filename; however that particular file downloaded as 5.jpg.

As I tried to indicate in my OP, I've done it both ways (local name before and after the URL) with no difference in the result. Might I need to tweak the .curlrc file, if such a thing exists?

Carver

Last edited by L_Carver; 12-02-2016 at 04:05 AM. Reason: A little more info
 
Old 12-03-2016, 03:58 AM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,655
Blog Entries: 3

Rep: Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901
I'm not able to duplicate the problem with curl 7.35.0. Which version do you have that is causing the problem?
 
Old 12-03-2016, 05:03 AM   #5
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,401
Blog Entries: 10

Rep: Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212Reputation: 5212
i cannot reproduce this:
Code:
curl http://dt.iki.fi/stuff/powerline-shell.png -o "stuff.jpg"
ls
stuff.jpg
how much did you edit your example? only the domain, or more?
is it part of a script? show us the script? does it work outside the script?
which curl version?
 
Old 12-03-2016, 09:24 AM   #6
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ondoho View Post
i cannot reproduce this:
how much did you edit your example? only the domain, or more?
Just the domain name.
Quote:
is it part of a script?
No, I won't write a script until I see curl save files by the name I give them. I've been typing in my cUrl commands
Quote:
which curl version?
7.4.70, with libcurl/7.47.0.

Carver
 
Old 12-03-2016, 05:12 PM   #7
c0wb0y
Member
 
Registered: Jan 2012
Location: Inside the oven
Distribution: Windows
Posts: 417

Rep: Reputation: 74
strace it?
 
Old 12-04-2016, 12:54 PM   #8
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by c0wb0y View Post
strace it?
i did an strace on a file I downloaded today (one that, again, arrived with its remote name, not the one I tried to give it). Having no idea how to interpret the many lines of codegunk strace generates, I have no idea which of what is good or bad.

A pointless side-track, imo. I can post the code from another strace, certainly, if you can draw out what might be going wrong here, but it would make the post in which I pasted it especially long.

I read on one page that redirecting the downloaded file (">") and naming it in that command should work. I tried it and got an empty file with the name I gave it.

Carver

Last edited by L_Carver; 12-04-2016 at 12:56 PM.
 
Old 12-04-2016, 01:07 PM   #9
c0wb0y
Member
 
Registered: Jan 2012
Location: Inside the oven
Distribution: Windows
Posts: 417

Rep: Reputation: 74
This is what I did:
Code:
c0wb0y@laptop:~> strace -e open -o strace-curl.txt curl -o /tmp/test.png http://dt.iki.fi/stuff/powerline-shell.png

c0wb0y@laptop:~> grep 'test.png\|curlrc' strace-curl.txt 
open("/home/c0wb0y/.curlrc", O_RDONLY)   = -1 ENOENT (No such file or directory)
open("/tmp/test.png", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
As you can see from the output, the png was downloaded and correctly named.
The output above can help you trace if you have any ~/.curlrc or even a global rc at /etc?
 
Old 12-07-2016, 08:45 PM   #10
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
Still using the server end filename

I tried your test string, minus the redirect to /tmp.

Quote:
Originally Posted by c0wb0y View Post
The output above can help you trace if you have any ~/.curlrc or even a global rc at /etc?
I have a ~/.curlrc, unedited from a downloaded sample (I forget the site on which I found it). It looks like this:
Code:
#this is a sample .curlrc file

# store the trace in curl_trace.txt file. beware that multiple executions of the curl command will overwrite this file
--trace curl_trace.txt

# store the header info in curl_headers.txt file. beware that multiple executions of the curl command will overwrite this file
--dump-header curl_headers.txt

#change the below referrer URL or comment it out entirely
-e "https://www.google.com"

#change the below useragent string. get your/other UA strings from http://www.useragentstring.com/
-A "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13"

#some headers
-H  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
-H  "Upgrade-Insecure-Requests: 1"
-H  "Accept-Encoding: gzip, deflate, sdch"
-H  "Accept-Language: en-US,en;q=0.8"
Aside: I did edit my .wgetrc, putting the user-agent string for, I believe, Konqueror 4.1 in it.

It occurred to me just now I've been enclosing my 'wanted' file names in double quotes (an old habit from using Cygwin). Could that be why I'm not getting the filename I ask for?


Carver
 
Old 12-07-2016, 09:25 PM   #11
c0wb0y
Member
 
Registered: Jan 2012
Location: Inside the oven
Distribution: Windows
Posts: 417

Rep: Reputation: 74
Did you try:
- renaming your curlc so it does not get used
- double quote should have no bearing to filename creation.
- run strace similar to what I did so you can see how filename is created.
 
Old 12-08-2016, 08:29 AM   #12
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
Seems now to do it right half the time

I'm getting just about every other download I do locally named; the others named as they are/were on the server. Not a script-able thing, as I see it, but as with everything else I'm finding different (and "bad") in this LinuxMint install, I feel like I'm edging closer to a solution. I'm starting to think some of the difficulty might be attributable to flaws, maybe even bugs, in my curl version that will, hopefully, be worked out in the next (or the next after that) dot-upgrade. And when you find something that's buggy, and you want more than to hope for resolution, the thing to do is communicate with the developers.

That may be the ultimate step.

Carver
 
Old 12-08-2016, 06:59 PM   #13
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 3,556

Rep: Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117Reputation: 1117
Code:
curl -A "Mozilla/5.0" http://www....com -o - > filename.xyz
 
Old 12-08-2016, 07:24 PM   #14
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 633

Rep: Reputation: 316Reputation: 316Reputation: 316Reputation: 316
Quote:
Originally Posted by L_Carver View Post
Here's a (somewhat edited) example from my bash history:
Code:
curl http://galleries.blahblah.com/77/7765/071/pic/5.jpg -o "blah77-7765-071-005.jpg"
You'll notice the URL is in front of the 'local' filename; however that particular file downloaded as 5.jpg.

As I tried to indicate in my OP, I've done it both ways (local name before and after the URL) with no difference in the result. Might I need to tweak the .curlrc file, if such a thing exists?

Carver
This is "Is it plugged in" level.. but make sure the url is quoted?
I have curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 and I don't have this behavior..
 
Old 12-23-2016, 05:31 AM   #15
L_Carver
Member
 
Registered: Sep 2016
Location: Webster MA USA
Posts: 241

Original Poster
Rep: Reputation: Disabled
Quoted URL does not work, either way

I tried
Code:
curl -o "imagegame12128-007.jpg" -R "http://img63.imagegame.com/i/12828/5c5lzwbrpnhn.jpg/img07.jpg"
which got me
Code:
ls: cannot access 'imagegame.jpg': No such file or directory
I tried
Code:
curl -R "http://img63.imagegame.com/i/12828/5c5lzwbrpnhn.jpg/img07.jpg" -o "imagegame12128-007.jpg"
and I get
Code:
ls: cannot access 'imagegame.jpg': No such file or directory
The same thing both ways. Double-quoting is a must for my terminal applications; every time when I use single quotes I get cannot stat errors. My terminal apps have been ruined by my bad Cygwin habits, and this isn't the first Linux install this has happened to.

Carver
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
CentOS 6.7 has really OLD curl. Best way to update curl? sneakyimp Linux - Server 4 04-26-2016 03:06 PM
[SOLVED] While loop not behaving ....... jv2112 Programming 4 08-26-2011 07:01 PM
ARP not behaving... sw1sst Linux - Networking 1 12-17-2009 04:51 AM
cURL: Server has many IPs, how would I make a cURL script use those IPs to send data? guest Programming 0 04-11-2009 11:42 AM
my computer is not behaving right way Paxmaster Linux - General 4 06-20-2005 12:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:19 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration