What's a good language to use for polling WEB API's and parsing JSON results?

usao · 08-05-2016, 04:21 PM

I would like to write a program to poll remote WEB API's and parse the JSON results.
Im familiar with C/Fortran,KSH/Bash,Perl and SQL.

Names I have heard floating around include Python and Java (although this is a complex language). I wold prefer something simple and easy to work with.

Looking for a simple Linux bases scripting language I might interface with postgres. Needs to be able to manipulate Linux shell jobs.

dugan · 08-05-2016, 04:38 PM

BASH, cURL, and a command-line JSON parser.

usao · 08-05-2016, 04:49 PM

Im good with Bash, but don't know anything about curl. Ill have to look that one up and see what it does.
Is there a name for the "command line JSon" parser? I don't know where to find that. What package is it installed in? Does it work on CentOS? Does it have a yum install 'package' name?
Does it populate postgres or do you have to do that manually in the script?

dugan · 08-05-2016, 04:59 PM

For command-line Json parsers, I don't have a recommendation. However, "jq" and "Jshon" looked interesting last I checked. My best recommendation is to use a search engine to find information on either one, or on use "command-line JSON parser" as a search term.

Nothing will automatically populate postgress. You script that.

usao · 08-05-2016, 10:17 PM

I can do that, as long as I can get the JSON values into variables in the script. That's where the parser would help.

Doug G · 08-06-2016, 04:17 PM

I'd use php

sundialsvcs · 08-07-2016, 09:09 PM

It seems utterly ridiculous to me to use Bash for any purpose such as this.

When you write "a shell script," you can insert a #!commandprocessorname ("shebang") line at the top of the file, and the shell will invoke whatever command-processor you specify to execute the command. (The end-user will neither know nor care.) Therefore, "the world's your oyster" as to what language you prefer to use.

All of the languages you might pick ... uhh, other than 'bash scripting' ... come with rich, ready-made, fully debugged libraries to handle all of the core tasks that you will face: HTTP, JSON, even complete servers(!).

Thus, your task really becomes choosing a language. and then discovering and leveraging what has already been done(!) to accomplish your intended task. This "build vs. buy" decision is mostly "buy," and the wonderful news is that it won't cost you a dime. Not only "underlying plumbing," but quite possibly "more than 99% of the whole darned thing," might be "sitting on the shelf," waiting for you to find it, take it down and put it to work.

Quote:

Actum Ne Agas: Do Not Do A Thing Already Done.™

"Since you are 'a Good Programmer,'" the first thing that naturally happens when you're faced with a "new" (to you ...!) problem is to "start writing source-code in your head." Unconsciously, you assume that you are going to have to write (from scratch) your solution. These days, this is almost never the case.

Therefore, "Good Programmer" though you might be, your first task should be systems analysis, specification, and then research(!). First, to discover the cornucopia of existing software that might be brought to bear against your problem. Then, to discover (e.g. at GitHub) the many hundreds of existing applications, all of them available to you in source-code form, that already used that software to tackle a problem very similar to yours.

usao · 08-08-2016, 09:06 AM

Quote:

Originally Posted by Doug G

I'd use php

PHP appears more for creating web pages rather than reading them from a remote server.

usao · 08-08-2016, 09:10 AM

Quote:

Originally Posted by sundialsvcs

It seems utterly ridiculous to me to use Bash for any purpose such as this.

When you write "a shell script," you can insert a #!commandprocessorname ("shebang") line at the top of the file, and the shell will invoke whatever command-processor you specify to execute the command. (The end-user will neither know nor care.) Therefore, "the world's your oyster" as to what language you prefer to use.

All of the languages you might pick ... uhh, other than 'bash scripting' ... come with rich, ready-made, fully debugged libraries to handle all of the core tasks that you will face: HTTP, JSON, even complete servers(!).

Thus, your task really becomes choosing a language. and then discovering and leveraging what has already been done(!) to accomplish your intended task. This "build vs. buy" decision is mostly "buy," and the wonderful news is that it won't cost you a dime. Not only "underlying plumbing," but quite possibly "more than 99% of the whole darned thing," might be "sitting on the shelf," waiting for you to find it, take it down and put it to work.

"Since you are 'a Good Programmer,'" the first thing that naturally happens when you're faced with a "new" (to you ...!) problem is to "start writing source-code in your head." Unconsciously, you assume that you are going to have to write (from scratch) your solution. These days, this is almost never the case.

Therefore, "Good Programmer" though you might be, your first task should be systems analysis, specification, and then research(!). First, to discover the cornucopia of existing software that might be brought to bear against your problem. Then, to discover (e.g. at GitHub) the many hundreds of existing applications, all of them available to you in source-code form, that already used that software to tackle a problem very similar to yours.

This thread is part of what you call "Research".
My initial discovery phase indicated that "C" was not a good fit for what I was trying to do, in looking around, I found many possible languages which might be a better fit, and the purpose of this thread is to get feedback from those who have already done this (thus making use of what was already done, and not trying to re-invent the wheel).

Im already doing exactly what you suggest in your thread...

TenTenths · 08-08-2016, 09:18 AM

Quote:

Originally Posted by usao

PHP appears more for creating web pages rather than reading them from a remote server.

PHP is a rich language very suitable for scripting and certainly VERY suitable for working with APIs and not just to produce web pages. I've maintained PHP scripts that perform extremely complicate work on large (1M+ records) datasets that has had nothing to do with web pages.

schneidz · 08-08-2016, 09:48 AM

most kodi add-ons are written in python. they are essentially scraping urls and title/author/publication/... data from webpages such as you-toob.com/dis-nee.com/...

i usually bash it out using wget.

Habitual · 08-08-2016, 10:26 AM

I consult the API authoritative documentation.
and get ready to learn some python. Ugly? Hell yes.
Primitive? Runs faster than bash and friends.

keefaz · 08-08-2016, 11:58 AM

Quote:

Originally Posted by usao

PHP appears more for creating web pages rather than reading them from a remote server.

Code:

$ url="http://www.api.com/jsonpage"
$ php -r 'print_r(json_decode(file_get_contents($argv[1])));' $url

sundialsvcs · 08-08-2016, 05:55 PM

Quote:

Originally Posted by usao

This thread is part of what you call "Research".
My initial discovery phase indicated that "C" was not a good fit for what I was trying to do, in looking around, I found many possible languages which might be a better fit, and the purpose of this thread is to get feedback from those who have already done this (thus making use of what was already done, and not trying to re-invent the wheel).

Im already doing exactly what you suggest in your thread...

Reckon that, "across these many (koff koff ...) decades, now," I've probably used "most of 'em," and I don't reckon I ever did arrive at any "favorites."

"Pick a girl, ask her to marry you, and be genuinely(!) amazed when she says, "Yes!" Then, jump headlong into the rapids and hope for the best." Even though you might, from time to time, "not be entirely satisfied with your decision," the project will be achieved, and maybe she will still love you anyway.

astrogeek · 08-08-2016, 06:19 PM

Quote:

Originally Posted by usao

PHP appears more for creating web pages rather than reading them from a remote server.

Actually, PHP is a terrific scripting language as well. I'd definitely consider it for such an application.

I use it as a quick prototyping language for things eventually to be written in C/C++, and also have written several large applications in PHP that are part of my own critical infrastructure.