LinuxQuestions.org - cUrl not behaving the way it used to

Page 1 of 2

Show 50 post(s) from this thread on one page

- Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)

- - cUrl not behaving the way it used to (https://www.linuxquestions.org/questions/linux-software-2/curl-not-behaving-the-way-it-used-to-4175594633/)

L_Carver

12-02-2016 02:03 AM

cUrl not behaving the way it used to

I have followed every piece of advice, consulted the man pages numerous times, and no syntax seems to work when downloading pictures with the files I give them. They consistently arrive on my hard drive with the original filename (e.g.: 2.jpg) instead of the more descriptive name I assign in the "curl [options] [URL]" string that downloads the file.

wget? Use wget? I would except there are some sites where wget pulls down a 'dummy" image instead of the one I see on the webpage. I suspect that might have something to do with assigning Konqueror as user-agent in my wgetrc instead of a more widely-used browser like Firefox. (There again I'm the maverick)

Why use a CLI downloader at all? Until I can bash script a browser, I will prefer the flexibility of curl, wget, Axel (which I love except for the fact that it doesn't preserve server timestamps) and aria2.

Not too long ago, I had bash scripts written for cUrl. Of course, that was a few cUrl versions back; if these new behaviours came as part of the development arc, I'll write scripts that invoke wget instead.

Carver

Turbocapitalist

12-02-2016 03:04 AM

Which options are you actually giving to curl? Please show a sample that is giving you trouble.

L_Carver

12-02-2016 04:00 AM

An example

Quote:

Originally Posted by Turbocapitalist (Post 5636996)

Which options are you actually giving to curl? Please show a sample that is giving you trouble.

Here's a (somewhat edited) example from my bash history:

Code:

curl http://galleries.blahblah.com/77/7765/071/pic/5.jpg -o "blah77-7765-071-005.jpg"

You'll notice the URL is in front of the 'local' filename; however that particular file downloaded as 5.jpg.

As I tried to indicate in my OP, I've done it both ways (local name before and after the URL) with no difference in the result. Might I need to tweak the .curlrc file, if such a thing exists?

Carver

Turbocapitalist

12-03-2016 03:58 AM

I'm not able to duplicate the problem with curl 7.35.0. Which version do you have that is causing the problem?

ondoho

12-03-2016 05:03 AM

i cannot reproduce this:

Code:

curl http://dt.iki.fi/stuff/powerline-shell.png -o "stuff.jpg"

ls

stuff.jpg

how much did you edit your example? only the domain, or more?
is it part of a script? show us the script? does it work outside the script?
which curl version?

L_Carver

12-03-2016 09:24 AM

Quote:

Originally Posted by ondoho (Post 5637352)

i cannot reproduce this:
how much did you edit your example? only the domain, or more?

Just the domain name.

Quote:

is it part of a script?

No, I won't write a script until I see curl save files by the name I give them. I've been typing in my cUrl commands

Quote:

which curl version?

7.4.70, with libcurl/7.47.0.

Carver

c0wb0y

12-03-2016 05:12 PM

strace it?

L_Carver

12-04-2016 12:54 PM

Quote:

Originally Posted by c0wb0y (Post 5637612)

strace it?

i did an strace on a file I downloaded today (one that, again, arrived with its remote name, not the one I tried to give it). Having no idea how to interpret the many lines of codegunk strace generates, I have no idea which of what is good or bad.

A pointless side-track, imo. I can post the code from another strace, certainly, if you can draw out what might be going wrong here, but it would make the post in which I pasted it especially long.

I read on one page that redirecting the downloaded file (">") and naming it in that command should work. I tried it and got an empty file with the name I gave it.

Carver

c0wb0y

12-04-2016 01:07 PM

This is what I did:

Code:

c0wb0y@laptop:~> strace -e open -o strace-curl.txt curl -o /tmp/test.png http://dt.iki.fi/stuff/powerline-shell.png



c0wb0y@laptop:~> grep 'test.png\|curlrc' strace-curl.txt 

open("/home/c0wb0y/.curlrc", O_RDONLY)  = -1 ENOENT (No such file or directory)

open("/tmp/test.png", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4

As you can see from the output, the png was downloaded and correctly named.
The output above can help you trace if you have any ~/.curlrc or even a global rc at /etc?

L_Carver

12-07-2016 08:45 PM

Still using the server end filename

I tried your test string, minus the redirect to /tmp.

Quote:

Originally Posted by c0wb0y (Post 5637885)

The output above can help you trace if you have any ~/.curlrc or even a global rc at /etc?

I have a ~/.curlrc, unedited from a downloaded sample (I forget the site on which I found it). It looks like this:

Code:

#this is a sample .curlrc file



# store the trace in curl_trace.txt file. beware that multiple executions of the curl command will overwrite this file

--trace curl_trace.txt



# store the header info in curl_headers.txt file. beware that multiple executions of the curl command will overwrite this file

--dump-header curl_headers.txt



#change the below referrer URL or comment it out entirely

-e "https://www.google.com"



#change the below useragent string. get your/other UA strings from http://www.useragentstring.com/

-A "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13"



#some headers

-H  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"

-H  "Upgrade-Insecure-Requests: 1"

-H  "Accept-Encoding: gzip, deflate, sdch"

-H  "Accept-Language: en-US,en;q=0.8"

Aside: I did edit my .wgetrc, putting the user-agent string for, I believe, Konqueror 4.1 in it.

It occurred to me just now I've been enclosing my 'wanted' file names in double quotes (an old habit from using Cygwin). Could that be why I'm not getting the filename I ask for?

Carver

c0wb0y

12-07-2016 09:25 PM

Did you try:
- renaming your curlc so it does not get used
- double quote should have no bearing to filename creation.
- run strace similar to what I did so you can see how filename is created.

L_Carver

12-08-2016 08:29 AM

Seems now to do it right half the time

I'm getting just about every other download I do locally named; the others named as they are/were on the server. Not a script-able thing, as I see it, but as with everything else I'm finding different (and "bad") in this LinuxMint install, I feel like I'm edging closer to a solution. I'm starting to think some of the difficulty might be attributable to flaws, maybe even bugs, in my curl version that will, hopefully, be worked out in the next (or the next after that) dot-upgrade. And when you find something that's buggy, and you want more than to hope for resolution, the thing to do is communicate with the developers.

That may be the ultimate step.

Carver

teckk

12-08-2016 06:59 PM

Code:

curl -A "Mozilla/5.0" http://www....com -o - > filename.xyz

Sefyir

12-08-2016 07:24 PM

Quote:

Originally Posted by L_Carver (Post 5637002)

Here's a (somewhat edited) example from my bash history:

Code:

curl http://galleries.blahblah.com/77/7765/071/pic/5.jpg -o "blah77-7765-071-005.jpg"

This is "Is it plugged in" level.. but make sure the url is quoted?
I have curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 and I don't have this behavior..

L_Carver

12-23-2016 05:31 AM

Quoted URL does not work, either way

I tried

Code:

curl -o "imagegame12128-007.jpg" -R "http://img63.imagegame.com/i/12828/5c5lzwbrpnhn.jpg/img07.jpg"

which got me

Code:

ls: cannot access 'imagegame.jpg': No such file or directory

I tried

Code:

curl -R "http://img63.imagegame.com/i/12828/5c5lzwbrpnhn.jpg/img07.jpg" -o "imagegame12128-007.jpg"

and I get

Code:

ls: cannot access 'imagegame.jpg': No such file or directory

The same thing both ways. Double-quoting is a must for my terminal applications; every time when I use single quotes I get cannot stat errors. My terminal apps have been ruined by my bad Cygwin habits, and this isn't the first Linux install this has happened to.

Carver

All times are GMT -5. The time now is 08:21 AM.

Page 1 of 2

Show 50 post(s) from this thread on one page