[SOLVED] sed not working as expect while parsing json

robertjinx · 04-13-2018, 01:21 PM

Hello, trying to parse json using sed, just to get a version number. The script is simple, but for whatever reason I get more text than I would expect.

Here is the code:

Code:

#!/bin/sh -e

api_url="https://api.bintray.com"

get_latest_version() {
        local version=0
	local tmpfile="/tmp/.version"

        rc=$(curl -sk "${api_url}/search/packages?name=jfrog-artifactory-pro&subject=jfrog&repo=artifactory-pro-rpms" -o $tmpfile -w '%{http_code}')
        if [ $rc -eq 200 ]; then
		# "latest_version":"5.10.2"
                version=$(sed -n 's/.*\"\(latest_version\)\":\"\(.*\)\"\,/\2/p' $tmpfile 2>/dev/null)
        fi

        rm -f $tmpfile
        echo $version
}

echo $(get_latest_version)

I know I'm doing something wrong, but can't figure out what. Any ideas are welcomed!

Turbocapitalist · 04-13-2018, 01:43 PM

The language sed would be the wrong tool for that. You'll benefit from a proper JSON parser.

You can have a standalone like jq or use one for the scripting language of your choice. For example there is CPAN's JSON for perl.

robertjinx · 04-13-2018, 01:56 PM

Quote:

Originally Posted by Turbocapitalist

The language sed would be the wrong tool for that. You'll benefit from a proper JSON parser.

You can have a standalone like jq or use one for the scripting language of your choice. For example there is CPAN's JSON for perl.

Thats not an option when running in a small environment without those tools, sed should be able to handle this, just I'm not seeing what I'm doing wrong.

Turbocapitalist · 04-13-2018, 02:09 PM

perl is in all environments, unless you've assembled something really custom. jq can be added to any. JSON, however, can be really messy from a whitespace perspective, and that includes linebreaks, which makes it hard to use something simple like sed.

But if you are dead set on (mis-)using sed in that, then please post a line from the sample data to be worked on and what the line should look like after the transformation. It's a bit hard to guess without seeing a before and after sample of the data.

robertjinx · 04-13-2018, 02:10 PM

Quote:

Originally Posted by Turbocapitalist

perl is in all environments, unless you've assembled something really custom. jq can be added to any. JSON, however, can be really messy from a whitespace perspective, and that includes linebreaks, which makes it hard to use something simple like sed.

But if you are dead set on (mis-)using sed in that, then please post a line from the sample data to be worked on and what the line should look like after the transformation. It's a bit hard to guess without seeing a before and after sample of the data.

This is the exact output:

Code:

robert@pandora:~> curl -sk -X GET "https://api.bintray.com/search/packages?name=jfrog-artifactory-pro&subject=jfrog&repo=artifactory-pro-rpms"
[{"name":"jfrog-artifactory-pro-rpm","repo":"artifactory-pro-rpms","owner":"jfrog","desc":"Artifactory is an advanced Binary Repository Manager for use by build tools (like Maven and Gradle), dependency management tools (like Ivy and NuGet) and build servers (like Jenkins, Hudson, TeamCity and Bamboo).\r\nRepository managers serve two purposes: they act as highly configurable proxies between your organization and external repositories and they also provide build servers with a deployment destination for your internally generated artifacts.","labels":["artifactory","powerpack"],"attribute_names":[],"licenses":[],"custom_licenses":["JFrog Artifactory EULA"],"followers_count":16,"created":"2015-05-21T13:52:20.704Z","website_url":"http://www.jfrog.com/","issue_tracker_url":"https://issues.jfrog.org/jira/browse/rtfact","linked_to_repos":[],"permissions":[],"versions":["5.10.2","5.10.1","5.10.0","5.9.3","5.9.1","5.9.0","5.8.4","5.8.3","5.8.2","5.8.1","5.8.0","5.7.2","5.7.1","5.7.0","5.6.3","5.6.2","5.6.1","5.6.0","5.5.2","5.5.1","5.5.0","5.4.6","5.4.5","5.4.4","5.4.3","5.4.2","5.4.1","5.4.0","5.3.2","5.3.1","5.3.0","5.2.1","5.2.0","5.1.4","5.1.3","5.1.0","5.0.1","5.0.0","4.16.1","4.16.0","4.15.0","4.14.3","4.14.2","4.14.1","4.14.0","4.13.2","4.13.1","4.13.0","4.12.2","4.12.1","4.12.0.1","4.11.2","4.11.1","4.11.0","4.10.0","4.9.1","4.9.0","4.8.2","4.8.1","4.8.0","4.7.7","4.7.6","4.7.5","4.7.4","4.7.3","4.7.2","4.7.1","4.7.0","4.6.1","4.6.0","4.5.2","4.5.1","4.5.0","4.4.3","4.4.2","4.4.1","4.4.0","4.3.3","4.3.2","4.3.1","4.3.0","4.2.2","4.2.1","4.2.0","4.1.3","4.1.2","4.1.0","4.0.2","4.0.1","4.0.0","3.9.5","3.9.4","3.9.3","3.9.2","3.9.1","3.9.0","3.8.0","3.7.0","3.6.0","3.5.3","3.5.2.1","3.5.2","3.5.1","3.5.0","2.4.0","3.0.0","2.3.3","3.0.3","2.5.0","3.1.0","3.2.0","2.6.0","3.3.0","2.3.3.1","2.3.4.1","2.6.3","3.5.2.1a","3.5.3b","3.4.0","3.6.0a","3.3.0.1","2.5.2","2.6.6","3.6.0-beta-1","3.2.2","3.5.2.1b","3.5.1d","3.0.1","2.5.1","2.6.5","3.1.1.1","3.4.2","3.2.1","3.3.1","3.5.3a","3.4.1","2.6.4","3.2.1.1","2.6.7","2.4.2","3.0.2","2.6.2","2.4.1","3.1.1","2.3.4","2.6.7.1","2.5.1.1","3.0.4","2.6.1"],"latest_version":"5.10.2","updated":"2018-04-12T18:43:35.725Z","rating_count":0,"system_ids":[],"vcs_url":"http://bit.ly/1U94K5i","maturity":""}]

Turbocapitalist · 04-13-2018, 02:24 PM

Thanks. It looks like you might be missing a [ in your pattern, that's a hard one so it might be best to just check for not a quote. And if you are just substituting out all the version numbers for the one single version number then you'll want a span of all characters except a quote. [^"]

What do you want that line to look like when you are done processing it? Just the one version by itself?

Code:

 sed -n -r 's/^.*(latest_version)"[^"]*"([^"]*)",.*$/\2/p'

robertjinx · 04-14-2018, 03:14 AM

Quote:

Originally Posted by Turbocapitalist

Thanks. It looks like you might be missing a [ in your pattern, that's a hard one so it might be best to just check for not a quote. And if you are just substituting out all the version numbers for the one single version number then you'll want a span of all characters except a quote. [^"]

What do you want that line to look like when you are done processing it? Just the one version by itself?

Code:

 sed -n -r 's/^.*(latest_version)"[^"]*"([^"]*)",.*$/\2/p'

Yes, just the one version. Thanks, that worked.

pan64 · 04-14-2018, 04:09 AM

just a comment:

Code:

# instead of
echo $(get_latest_version)
# you can write simply
get_latest_version

and also:

Code:

        if [ $rc -eq 200 ]; then
		# "latest_version":"5.10.2"
                sed -n 's/.*\"\(latest_version\)\":\"\(.*\)\"\,/\2/p' $tmpfile
        else
                echo 0
        fi

robertjinx · 04-14-2018, 07:23 AM

Quote:

Originally Posted by pan64

just a comment:

Code:

# instead of
echo $(get_latest_version)
# you can write simply
get_latest_version

and also:

Code:

        if [ $rc -eq 200 ]; then
		# "latest_version":"5.10.2"
                sed -n 's/.*\"\(latest_version\)\":\"\(.*\)\"\,/\2/p' $tmpfile
        else
                echo 0
        fi

Ahh no, this was a test code, just for guys to see it.

ondoho · 04-14-2018, 08:21 AM

if you really cannot use jq or jshon or perl for this, i think i'd rather write a shellscript than use sed...

i can think of setting the internal field separator first to '{', then read the file, then parse again with IFS=':', etc.

pan64 · 04-14-2018, 10:22 AM

you can use shell/sed/grep/awk/perl/python, yes, all of them are ok, but the regular way would be either jq or perl/python (or similar) with a json module. Otherwise the result will not be fully reliable.

For example:

Code:

REQUEST="${api_url}/search/packages?name=jfrog-artifactory-pro&subject=jfrog&repo=artifactory-pro-rpms"
curl -sk "${REQUEST}" | awk 'BEGIN { RS=","; FS=":" } $1 == "\"latest_version\"" { print $2; found++ } END { if (! found) { print 0 } } '

will "almost" work, you only need to remove " from the result.