PERL : anyone have a suggestion for a (pdf) book about JSON parsing?

////// · 02-18-2021, 01:21 PM

hello ladies and gents.

anyone who knows a good book of JSON parsing in perl, post here so i can do some

i have done quick google searches but i am not sure what modules i should use.

actually, i found one book just when i were writing this post : Practical perl tools - a little place for your stuff. <-- anyone heard about that one?

tia.

EDIT: i think ill be using "JSON::MaybeXS", is that "good one" ?

Turbocapitalist · 02-18-2021, 02:09 PM

I've used JSON. What is the larger task?

////// · 02-18-2021, 02:29 PM

Quote:

Originally Posted by Turbocapitalist

I've used JSON. What is the larger task?

i want to learn to parse suricata's filestore .json log files.
https://suricata.readthedocs.io/en/s...xtraction.html

here is example of .json log file :

Code:

{"timestamp":"2021-02-18T11:35:00.854323+0200","flow_id":975164619942783,"event_type":"fileinfo","src_ip":"205.185.216.10","src_port":80,"dest_ip":"192.168.10.99","dest_port":60316,"proto":"TCP","http":{"hostname":"download.windowsupdate.com","url":"/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab","http_user_agent":"Windows-Update-Agent/10.0.10011.16384 Client-Protocol/2.31","http_content_type":"application/vnd.ms-cab-compressed","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":7769},"app_proto":"http","fileinfo":{"filename":"/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab","sid":[],"gaps":false,"state":"CLOSED","md5":"b8d516587104a7347b3889357c4e2a78","sha1":"05325f9ce11ce1f49fffcf25307bcd43f92be912","sha256":"0ad50e6a8a7c654ea0d6df7c2363cb70c5ff5aa0b20c7582fe7f015914a93d71","stored":true,"file_id":480,"size":7769,"tx_id":10}}

shruggy · 02-18-2021, 02:46 PM

I ran your file through json_xs to make it more readable:

Code:

{
   "src_port" : 80,
   "fileinfo" : {
      "state" : "CLOSED",
      "md5" : "b8d516587104a7347b3889357c4e2a78",
      "stored" : true,
      "tx_id" : 10,
      "sid" : [],
      "sha1" : "05325f9ce11ce1f49fffcf25307bcd43f92be912",
      "size" : 7769,
      "file_id" : 480,
      "sha256" : "0ad50e6a8a7c654ea0d6df7c2363cb70c5ff5aa0b20c7582fe7f015914a93d71",
      "filename" : "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
      "gaps" : false
   },
   "app_proto" : "http",
   "proto" : "TCP",
   "src_ip" : "205.185.216.10",
   "http" : {
      "http_user_agent" : "Windows-Update-Agent/10.0.10011.16384 Client-Protocol/2.31",
      "url" : "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
      "protocol" : "HTTP/1.1",
      "http_method" : "GET",
      "length" : 7769,
      "http_content_type" : "application/vnd.ms-cab-compressed",
      "hostname" : "download.windowsupdate.com",
      "status" : 200
   },
   "flow_id" : 975164619942783,
   "event_type" : "fileinfo",
   "dest_ip" : "192.168.10.99",
   "dest_port" : 60316,
   "timestamp" : "2021-02-18T11:35:00.854323+0200"
}

Interestingly, json_pp and jq sort the keys differently.

json_pp

Code:

{
   "fileinfo" : {
      "gaps" : false,
      "tx_id" : 10,
      "sha1" : "05325f9ce11ce1f49fffcf25307bcd43f92be912",
      "size" : 7769,
      "sha256" : "0ad50e6a8a7c654ea0d6df7c2363cb70c5ff5aa0b20c7582fe7f015914a93d71",
      "file_id" : 480,
      "sid" : [],
      "filename" : "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
      "state" : "CLOSED",
      "stored" : true,
      "md5" : "b8d516587104a7347b3889357c4e2a78"
   },
   "flow_id" : 975164619942783,
   "src_ip" : "205.185.216.10",
   "proto" : "TCP",
   "app_proto" : "http",
   "dest_ip" : "192.168.10.99",
   "event_type" : "fileinfo",
   "src_port" : 80,
   "timestamp" : "2021-02-18T11:35:00.854323+0200",
   "dest_port" : 60316,
   "http" : {
      "http_method" : "GET",
      "http_user_agent" : "Windows-Update-Agent/10.0.10011.16384 Client-Protocol/2.31",
      "length" : 7769,
      "url" : "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
      "http_content_type" : "application/vnd.ms-cab-compressed",
      "protocol" : "HTTP/1.1",
      "hostname" : "download.windowsupdate.com",
      "status" : 200
   }
}

jq

Code:

{
  "timestamp": "2021-02-18T11:35:00.854323+0200",
  "flow_id": 975164619942783,
  "event_type": "fileinfo",
  "src_ip": "205.185.216.10",
  "src_port": 80,
  "dest_ip": "192.168.10.99",
  "dest_port": 60316,
  "proto": "TCP",
  "http": {
    "hostname": "download.windowsupdate.com",
    "url": "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
    "http_user_agent": "Windows-Update-Agent/10.0.10011.16384 Client-Protocol/2.31",
    "http_content_type": "application/vnd.ms-cab-compressed",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 7769
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/c/msdownload/update/others/2021/02/33653734_05325f9ce11ce1f49fffcf25307bcd43f92be912.cab",
    "sid": [],
    "gaps": false,
    "state": "CLOSED",
    "md5": "b8d516587104a7347b3889357c4e2a78",
    "sha1": "05325f9ce11ce1f49fffcf25307bcd43f92be912",
    "sha256": "0ad50e6a8a7c654ea0d6df7c2363cb70c5ff5aa0b20c7582fe7f015914a93d71",
    "stored": true,
    "file_id": 480,
    "size": 7769,
    "tx_id": 10
  }
}

////// · 02-18-2021, 02:55 PM

thanks shruggy, it is a lot easier to read like that.

i think that i might use JSON::MaybeXS
this guy says he is recommending it :
https://perlmaven.com/json

shruggy · 02-18-2021, 03:04 PM

Really cannot comment on this as I never used it. Actually, I never parsed JSON in Perl proper. CLI tools like jq, json_pp, json_xs, etc. are usually enough for my needs. E.g. when trying to tackle the problem from this thread, I wrote this:

Code:

#!/bin/bash
text='[[{"Key":"PORT",context:"5436"},{"Key":"HOST",context:"prod"},{"Key":"ADD",context:"location"}]]'

echo === sed+jq ===
sed 's/\<context\>/"&"/g' <<<"$text" | jq -c '.[][]|select(.Key=="HOST")'

# Remove the outer array with sed. Another possibility: jgrep -n
echo === sed+jgrep ===
sed 's/^.\(.*\).$/\1/;s/\<context\>/"&"/g' <<<"$text" | jgrep -c 'Key="HOST"'

# jt is similar to jshon
echo === sed+jt+grep ===
sed 's/\<context\>/"&"/g' <<<"$text" | jt [ Key % ] context % | grep -w ^HOST

# Remove the outer array with jt
echo === sed+jt+mlr ===
sed 's/\<context\>/"&"/g' <<<"$text" | jt % | mlr --json filter '$Key=="HOST"'

echo === pyjson5+jq ===
pyjson5 --as-json -c "$text" | jq -c '..|select(.Key?=="HOST")'

echo === json_pp+json_xs ===
json_pp -json_opt allow_barekey <<<"$text" |
json_xs -e '($_) = grep { $_->{Key} eq "HOST" } @{$_->[0]}'

The last part uses Perl tools, but only barely.

////// · 02-19-2021, 02:40 AM

yea, i could make it with bash also, but i really want to learn perl.