LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-29-2017, 07:52 PM   #1
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Rep: Reputation: 16
Is curl the right tool for the job?


So I'm super lame at anything to do with http and related so I'm very keen to avoid writing a custom tool when (hopefully) something already exists.

Requirements:
Must be able to run unattended and play nice with cron
Must be able to send an xml request then retrieve an attachment from an xml response
Must be able to save that attachment to it's original filename, only on my local system (each attachment will be some document called document_id_number.format - usually .docx or .pdf)

Since the login details are encrypted with a bespoke method, the xml file sent as the request would be pre-built and need only to be put to the server to get the response from.

It looks to me like curl is the tool to do this ...
As I said, I want to avoid writing a tool so libcurl and c or python will almost certainly do the job ... but I'd prefer not to have to reinvent this wheel.
I have also looked at wget - it does appear to be somewhat less able to do what I need; I may be wrong.


If curl *can* do what I need, some snippets as to how to make that happen would be super-great
 
Old 03-30-2017, 02:02 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,684
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Either curl or wget will do the job.

wget is pre-installed in Ubuntu. If you look at the options --directory-prefix, --post-file or --post-data, --no-verbose or --quiet, and maybe --no-directories you should be able to get what you are looking for. As for receiving the file, wget will write to the disk whatever the server sends back, if you want to change the name then see also the --output-document option.
 
1 members found this post helpful.
Old 03-30-2017, 03:53 AM   #3
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
That's actually really good to know!
Hopefully I'm not stranded at home tomorrow (cyclone weather, you know!) and I can try it out!

Thanks so much - I'll report back with a hearty "solved" and share what I can
 
Old 03-30-2017, 09:45 AM   #4
KenJackson
Member
 
Registered: Jul 2006
Location: Maryland, USA
Distribution: Fedora and others
Posts: 756

Rep: Reputation: 144Reputation: 144
Quote:
Originally Posted by yakkmeister View Post
Must be able to send an xml request then retrieve an attachment from an xml response
Must be able to save that attachment to it's original filename, only on my local system
I routinely use both curl and wget to fetch files. They're both very flexible, so I'm sure they'll meet your needs.

But there's no "xml request" or "xml response"--you just fetch whatever file you want by specifying its URL. And only email has "attachments". HTTP just has files.

These two examples do the same thing. With curl use -s to silence the status noise and -O to save it as the same filename as on the server (or stdout by default). With wget use -q to silence the status noise (it saves to the same name by default).
Code:
curl -s -O https://www.kernel.org/feeds/all.atom.xml
wget -q https://www.kernel.org/feeds/all.atom.xml
 
Old 03-30-2017, 08:07 PM   #5
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
Quote:
And only email has "attachments". HTTP just has files.
Perhaps this is a definitions thing ... If it help, consider the response to be an xml portion with a binary portion following immediately.

Quote:
you just fetch whatever file you want by specifying its URL
Ok ... then it may not do what I need ...

There's no url for the file I want, it's a tad more complicated than that.
What happens is that an xml request is sent to a specific port on a server ... it does some sever magic ... then the server sends a 2-part response; xml and binary data.

This is an IBM IDM http interface (FileNet), if that helps ...
 
Old 03-30-2017, 11:36 PM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,684
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Both can actually use the POST to send information not just GET to fetch information. But if you are going to post a raw file, then it is more complicated in wget and easier in curl. So in curl, you'd post a file like this:

Code:
curl --form 'file=@yourxmlfile;filename=yourxmlfilesunofficialname' http://example.com/
You do have to have the web server set to receive and process HTML forms though. Can you say more about how your server is expecting the file and how it is set up to receive it?

Last edited by Turbocapitalist; 03-31-2017 at 01:40 AM. Reason: typo
 
1 members found this post helpful.
Old 03-30-2017, 11:45 PM   #7
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918
If you can't get curl to do what you want, and you fancy using some Python, have a look at the Requests library:

http://docs.python-requests.org/en/master/
 
1 members found this post helpful.
Old 03-31-2017, 01:21 AM   #8
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by Turbocapitalist View Post
Both can actually use the POST to send information not just GET to fetch information.
Yes, I did notice that. On that subject, if I POST a form successfully, does curl "know" to save the response? - it does - testing confirmed.

Quote:
Originally Posted by Turbocapitalist View Post
But if you are going to post a raw file, then it is more complicated in wget and easier in curl. So in curl, you'd post a file like this:

Code:
curl --form 'file=@yourxmlfile;filename=yourxmlfilesunofficialname' http://example.com/
When you say "raw file," do you mean a file which contains the xml itself?
If so, that's what I'm doing - I have a file called xml_test.xml. It has straight-up text with xml formatting. I need to, basically, dump that to http://theServer:1000/request.aspx which gives me back a document with a bit of xml - presumably file data but I don't know yet.

I don't know what filename= should be so I tried it with the filename - I'll read up on that in a bit ...
But, this definitely got me a response from the server! It dumped some html to stdout - that's perfectly workable.
Credentials are wrong (and they should be)so the server denied access ... but I got an actual response

Quote:
Originally Posted by Turbocapitalist View Post
You do have to have the web server set to receive and process HTML forms though. Can you say more about how your server is expecting the file and how it is set up to receive it?
I don't know what it's expecting to receive other than it's xml ... I am converting functionality from an old utility (2004) written in VB6 ... so ... it's hard going! Looks like it just dumped xml over a winsock-like port via some form of ocx jobby ... and I have no server access.

Last edited by yakkmeister; 03-31-2017 at 01:53 AM. Reason: typo
 
Old 03-31-2017, 01:44 AM   #9
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by hydrurga View Post
If you can't get curl to do what you want, and you fancy using some Python, have a look at the Requests library:

http://docs.python-requests.org/en/master/
This looks really neat!
While it's not the solution I was after, I'll probably have a look into it
 
Old 03-31-2017, 01:45 AM   #10
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,684
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by yakkmeister View Post
When you say "raw file," do you mean a file which contains the xml itself?
I mean one that's not formatted as MIME multipart/form-data. So I guess, yes, I mean your XML file.

https://www.w3.org/TR/html401/intera...html#h-17.13.4

You'd put xml_test.xml for file=@ That field applies to the content / data that you want uploaded.

For filename= you can put whatever you want. It can be the same name or a different name. That field refers to the label you want applied to the data.

PS. I fixed a typo in my previous post. url -> curl
 
Old 03-31-2017, 01:51 AM   #11
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,684
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by hydrurga View Post
If you can't get curl to do what you want, and you fancy using some Python, have a look at the Requests library:

http://docs.python-requests.org/en/master/
It's also quite easy to do with perl's ( CGI or HTML::Template / HTML::Template::Complied ) and LWP
 
Old 03-31-2017, 01:53 AM   #12
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by Turbocapitalist View Post
I mean one that's not formatted as MIME multipart/form-data. So I guess, yes, I mean your XML file.

https://www.w3.org/TR/html401/intera...html#h-17.13.4

You'd put xml_test.xml for file=@ That field applies to the content / data that you want uploaded.

For filename= you can put whatever you want. It can be the same name or a different name. That field refers to the label you want applied to the data.

PS. I fixed a typo in my previous post. url -> curl
Thanks for the clarification!
This was really very useful; thanks again for the help!
I'll be digging around in the curl man pages, etc, to see how much more it can do on its own - like saving the response as a file instead of xml/text with a binary blob
 
Old 03-31-2017, 03:51 AM   #13
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918
Quote:
Originally Posted by yakkmeister View Post
Thanks for the clarification!
This was really very useful; thanks again for the help!
I'll be digging around in the curl man pages, etc, to see how much more it can do on its own - like saving the response as a file instead of xml/text with a binary blob
Let us know how you get on!
 
Old 04-04-2017, 02:39 AM   #14
yakkmeister
Member
 
Registered: Jan 2005
Location: QLD Australia
Distribution: Debian 10 'Buster'
Posts: 74

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by hydrurga View Post
Let us know how you get on!
Sorry it's taken so long!

Yes, everything seems to be working fine - I've yet to get the server to authenticate the requests (whole other kettle of bacon) but they are, indeed, being received and acted upon by the server. Curl gives me a response and dumps it to StdOut, which I can script for. I hope :P

At this point, I've really just used Turbocapitalist's solution without a problem
 
1 members found this post helpful.
Old 04-04-2017, 03:42 AM   #15
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918Reputation: 2918
Quote:
Originally Posted by yakkmeister View Post
Sorry it's taken so long!

Yes, everything seems to be working fine - I've yet to get the server to authenticate the requests (whole other kettle of bacon) but they are, indeed, being received and acted upon by the server. Curl gives me a response and dumps it to StdOut, which I can script for. I hope :P

At this point, I've really just used Turbocapitalist's solution without a problem
Great to hear.
 
  


Reply

Tags
http


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync the right tool for the job? bioshock Linux - Newbie 5 02-20-2016 02:27 PM
LXer: Finding the right tool for the job LXer Syndicated Linux News 0 12-03-2015 01:11 PM
Streaming Video - Best Tool(s) For The Job? shadoxx428 Linux - Software 3 04-08-2009 01:48 AM
best tool for the job or how to tell what distro is best for me Desidarius Linux - Distributions 1 10-08-2004 09:55 AM
Cron Job with Curl andy18 Linux - General 0 05-16-2003 07:00 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration