How to extract value from XML payload using Shell script

SasikumarBharanikumar · 11-13-2018, 10:56 AM

I have runtime XML file which has predefined format and Its being created in Jenkins workspace. I just want to parse the XML payload by using shell script.

Code:

<test-results>
     <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
       <params>
         <param index="0">
           <value>
           <![CDATA[ account_number:22988419 ]]>
           </value>
         </param>
         <param index="1">
           <value>
           <![CDATA[ txn_id:6wdadfsad2134330L ]]>
           </value>
         </param>
         <param index="2">
           <value>
           <![CDATA[ amount:1100 ]]>
           </value>
          </param>
          <param index="3">
            <value>
            <![CDATA[ currency:USD ]]>
            </value>
          </param>
          <param index="4">
            <value>
            <![CDATA[Id:11a09 ]]>
            </value>
          </param>
          <param index="5">
            <value>
            <![CDATA[Name:Consumer [testId=AS1-TC2, description=Txn amount; wallet -Bal, BA,CC,VISA Credit; Consumer - CC;No other preference set]]]>
            </value>
          </param>
        </params>
     </test-method>
     <test-method status="PASS" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
         <params>
           <param index="0">
              <value>
              <![CDATA[ account_number:22988419 ]]>
              </value>
           </param>
           <param index="1">
             <value>
             <![CDATA[ txn_id:6wdadfsad2134330L ]]>
             </value>
           </param>
        </params>
      </test-method>
      <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
        <params>
           <param index="0">
              <value>
              <![CDATA[ account_number:22988419 ]]>
              </value>
           </param>
           <param index="1">
              <value>
              <![CDATA[Name:Consumer [testId=AS1-TC3, description=Txn amount; wallet -Bal, BA,CC,VISA Credit; Consumer - CC;No other preference set]]]>
              </value>
           </param>
         </params>
      </test-method>
  </test-results>

The above payload I have to read the status of test-method if it's "FAIL" then I need to get the "testId" value from that particular test method. The above payload I have 3 test methods only two had status Failed I need to fetch both Test id and assign to variable as like below

Expected Output:

Code:

 fetchResult = AS1-TC2,AS1-TC3

I just need to fetch the "testId" of failed test-methods and assign it to variable with comma separated using shell script.

I tried the below lines but It doesn't return the entire test-method tags

Code:

failedTC=`grep "test-method.*FAIL" results.xml | sed -e 's/^.*test-instance-name="(.*)-----.*/\1/'

Output:

Code:

 <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">

I want to return entire <test-method> ... </test-method> for status="FAIL"

TB0ne · 11-13-2018, 11:09 AM

Quote:

Originally Posted by SasikumarBharanikumar

I have runtime XML file which has predefined format and Its being created in Jenkins workspace. I just want to parse the XML payload by using shell script.

Code:

<test-results>
     <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
       <params>
         <param index="0">
           <value>
           <![CDATA[ account_number:22988419 ]]>
           </value>
         </param>
         <param index="1">
           <value>
           <![CDATA[ txn_id:6wdadfsad2134330L ]]>
           </value>
         </param>
         <param index="2">
           <value>
           <![CDATA[ amount:1100 ]]>
           </value>
          </param>
          <param index="3">
            <value>
            <![CDATA[ currency:USD ]]>
            </value>
          </param>
          <param index="4">
            <value>
            <![CDATA[Id:11a09 ]]>
            </value>
          </param>
          <param index="5">
            <value>
            <![CDATA[Name:Consumer [testId=AS1-TC2, description=Txn amount; wallet -Bal, BA,CC,VISA Credit; Consumer - CC;No other preference set]]]>
            </value>
          </param>
        </params>
     </test-method>
     <test-method status="PASS" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
         <params>
           <param index="0">
              <value>
              <![CDATA[ account_number:22988419 ]]>
              </value>
           </param>
           <param index="1">
             <value>
             <![CDATA[ txn_id:6wdadfsad2134330L ]]>
             </value>
           </param>
        </params>
      </test-method>
      <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">
        <params>
           <param index="0">
              <value>
              <![CDATA[ account_number:22988419 ]]>
              </value>
           </param>
           <param index="1">
              <value>
              <![CDATA[Name:Consumer [testId=AS1-TC3, description=Txn amount; wallet -Bal, BA,CC,VISA Credit; Consumer - CC;No other preference set]]]>
              </value>
           </param>
         </params>
      </test-method>
  </test-results>

The above payload I have to read the status of test-method if it's "FAIL" then I need to get the "testId" value from that particular test method. The above payload I have 3 test methods only two had status Failed I need to fetch both Test id and assign to variable as like below
Expected Output:

Code:

 fetchResult = AS1-TC2,AS1-TC3

I just need to fetch the "testId" of failed test-methods and assign it to variable with comma separated using shell script. I tried the below lines but It doesn't return the entire test-method tags

Code:

failedTC=`grep "test-method.*FAIL" results.xml | sed -e 's/^.*test-instance-name="(.*)-----.*/\1/'

Output:

Code:

 <test-method status="FAIL" name="flowVISA" duration-ms="843329" data-provider="533w43werwer" finished-at="2018-11-01T23:30:48Z">

I want to return entire <test-method> ... </test-method> for status="FAIL"

While you probably *CAN* do this with a shell-script, it's going to take more than a single line. If you **KNOW** that you'll always get a Pass/fail before the ID, you can simply grep for those things, shovel them into variables, and do a basic match, outputting what you want, which is quick and dirty. And prone to failure. There are XML parsing libraries for perl and python that let you do just this, very easily:

Code:

#!/usr/bin/perl
use strict;
use warnings;
use XML::LibXML;

my $dom = XML::LibXML->load_xml(location => $InputXML);
 foreach my $title ($dom->findnodes('//test-results')) {
# This splits apart the tags and values into variables. Do things with them.
        $METHOD = $title->findvalue('./test-method');
        $ID = $title->findvalue('./param/value');
....
<grep $METHOD for Fail...if so, output $ID>

Above for example only. Totally untested, but will get you started.

SasikumarBharanikumar · 11-13-2018, 11:24 AM

Can you please give sample shell script to solve this problem.

TB0ne · 11-13-2018, 11:47 AM

Quote:

Originally Posted by SasikumarBharanikumar

Can you please give sample shell script to solve this problem.

No; we are not going to write your scripts for you, but we will be happy to HELP you. You were given suggestions for how to do this in bash, and provided an example of how to use perl to do it. It is now up to **YOU** to do your own work.

Read the "Question Guidelines" link in my posting signature. Post what you wrote/tried on your own, and tell us where you're stuck and we'll be glad to help.

l0f4r0 · 11-13-2018, 11:57 AM

Quote:

Originally Posted by SasikumarBharanikumar

Can you please give sample shell script to solve this problem.

I was about to give you one possible solution but I'm not so sure now...
So I will go like TB0ne, grep is not the good tool here because it's designed to work with lines so mainly with flat files and not with structured data like xml.
TB0ne suggested you to use Perl|Python but you can as well do it with xmllint or xmlstarlet. Considering the fact that xmlstarlet has a flatter learning curve and that your requirement is not too complicated, I would go with xmllint (I've just done your exercise myself so it works).

TB0ne · 11-13-2018, 12:04 PM

Quote:

Originally Posted by l0f4r0

I was about to give you one possible solution but I'm not so sure now...
So I will go like TB0ne, grep is not the good tool here because it's designed to work with lines so mainly with flat files and not with structured data like xml.
TB0ne suggested you to use Perl|Python but you can as well do it with xmllint or xmlstarlet. Considering the fact that xmlstarlet has a flatter learning curve and that your requirement is not too complicated, I would go with xmllint (I've just done your exercise myself so it works).

Indeed; both are good tools as well. I prefer perl, but that's just me. Aside from what I already gave the OP, it would probably take less than 10 more lines of code to get what they need.

tofino_surfer · 11-13-2018, 12:32 PM

Quote:

I have runtime XML file which has predefined format and Its being created in Jenkins workspace. I just want to parse the XML payload by using shell script.

If you want to parse XML you should use a more powerful language such as perl or python which have actual modules for parsing XML.

For perl:

https://metacpan.org/pod/XML::Parser

For python:

https://docs.python.org/2/library/xm...ementtree.html

Bash shell scripting will be comprised of primitive operations that work on a line by line basis and don't understand XML syntax.

Quote:

I tried the below lines but It doesn't return the entire test-method tags
I want to return entire <test-method> ... </test-method> for status="FAIL"

This would be simple with an XML parsing library. You would iterate over all test_method tags and then operate only on those with FAIL.

l0f4r0 · 11-14-2018, 06:47 AM

Quote:

Originally Posted by TB0ne

Aside from what I already gave the OP, it would probably take less than 10 more lines of code to get what they need.

Definitely less than 10 more lines to your existing 10

Actually I have a one-liner with no loop needed, like:

Code:

fetchResult=$(xmllint <...> results.xml | sed <...> | tr <...>)