BASH: Parse XML

worm5252 · 03-30-2010, 06:57 AM

Hey guys,
I am working on a script to list out attributes of an XML file and then add a new IP to the list of nodes on the load balancer. I have a script that queries the Load Balancer to get it's information. It returns me the following xml file.

Code:

<?xml version="1.0"?>
<gogrid>
  <response method="/grid/loadbalancer/list" status="success">
    <summary total="1" start="0" numpages="0" returned="1"/>
    <list>
      <object name="loadbalancer">
        <attribute name="id">6577</attribute>
        <attribute name="name">Web Server LB</attribute>
        <attribute name="description"/>
        <attribute name="virtualip">
          <object name="ipportpair">
            <attribute name="ip">
              <object name="ip">
                <attribute name="id">1654289</attribute>
                <attribute name="ip">173.204.16.196</attribute>
                <attribute name="subnet">173.204.16.192/255.255.255.240</attribute>
                <attribute name="public">true</attribute>
              </object>
            </attribute>
            <attribute name="port">80</attribute>
          </object>
        </attribute>
        <attribute name="type">
          <object name="option">
            <attribute name="id">1</attribute>
            <attribute name="name">Round Robin</attribute>
            <attribute name="description"/>
          </object>
        </attribute>
        <attribute name="persistence">
          <object name="option">
            <attribute name="id">1</attribute>
            <attribute name="name">None</attribute>
            <attribute name="description"/>
          </object>
        </attribute>
        <attribute name="realiplist">
          <list>
            <object name="ipportpair">
              <attribute name="ip">
                <object name="ip">
                  <attribute name="id">1654287</attribute>
                  <attribute name="ip">173.204.16.194</attribute>
                  <attribute name="subnet">173.204.16.192/255.255.255.240</attribute>
                  <attribute name="public">true</attribute>
                </object>
              </attribute>
              <attribute name="port">80</attribute>
            </object>
            <object name="ipportpair">
              <attribute name="ip">
                <object name="ip">
                  <attribute name="id">1654288</attribute>
                  <attribute name="ip">173.204.16.195</attribute>
                  <attribute name="subnet">173.204.16.192/255.255.255.240</attribute>
                  <attribute name="public">true</attribute>
                </object>
              </attribute>
              <attribute name="port">80</attribute>
            </object>
          </list>
        </attribute>
        <attribute name="os">
          <object name="option">
            <attribute name="id">1</attribute>
            <attribute name="name">F5</attribute>
            <attribute name="description">The F5 Load Balancer.</attribute>
          </object>
        </attribute>
        <attribute name="state">
          <object name="option">
            <attribute name="id">1</attribute>
            <attribute name="name">On</attribute>
            <attribute name="description">Loadbalancer is enabled and on.</attribute>
          </object>
        </attribute>
      </object>
    </list>
  </response>
</gogrid>

I need to figure out how to pull the Load Balancer ID# and the Load Balancer Name. Then I need to Pull the IP address and port number of ever single node on the load balancer. I do not need the IP and port for the load balancer itself. So far I am having trouble figuring out how to do it since everything uses an attribute tag.

Once I get this information I can go from there on adding the new IP and port to the list.

Any help is appriciated

PMP · 03-30-2010, 07:49 AM

Do you have an option of using perl ?

worm5252 · 03-30-2010, 09:02 AM

I prefer bash because it is the only scripting language I know. However this API I am using supports Java, PHP, Python, Perl, Ruby, C#, and bash.

The API is a REST system and I am doing a request for info of the load balancer. In the end I need to be able to iterate through all the IP addesses and ports of the nodes on the load balancer so I can add a new one. That is simple as generating the URL and using "curl -f" with the URL. However with a REST system I need to be able to list the Load Balancer ID, all the IPs and Port numbers as well as my authentication info within the URL. So that brings me back to why I am parsing this info to begin with. Crappy API in my opinion, but it is what I have to work with.

PMP · 03-30-2010, 09:03 AM

Well if this is bash
this thing worked for me, may be you have to modify a bit according to your need

Code:

echo `grep -A2 name=\"loadbalancer test.xml` | sed -e 's/.*name="id">\([0-9]\+\)\?.*name="name">\([-a-zA-Z0-9 ._]\+\)\?<.*/LB_ID=\1 LB_NAME=\2/'

With perl, you have lots of options, It has got good XML parsing modules

worm5252 · 03-30-2010, 09:11 AM

Thanks

PMP · 03-30-2010, 09:13 AM

If this worked for you please mark this thread as solved

worm5252 · 03-30-2010, 09:46 AM

I ran it and it seems to just hang. It is probably how I modified your command. I have the xml file located at /tmp/ggapi.list_load_balancer.xml so I modified the command to read this

Code:

echo `grep -A2 name=\"/tmp/ggapi.list_load_balancer.xml` | sed -e 's/.*name="id">\([0-9]\+\)\?.*name="name">\([-a-zA-Z0-9 ._]\+\)\?<.*/LB_ID=\1 LB_NAME=\2/'

I have a feeling that is not correct.

PMP · 03-30-2010, 12:54 PM

echo `grep -A2 name=\"loadbalancer test.xml` | sed -e 's/.*name="id">\([0-9]\+\)\?.*name="name">\([-a-zA-Z0-9 ._]\+\)\?<.*/LB_ID=\1 LB_NAME=\2/'

The test.xml is the name of the file, change this only not name=

The text in magenta is the text to be grepped and in the red is the file name you habe to change the filename to the file along with your path

tuxdev · 03-30-2010, 02:05 PM

Bash is completely and utterly inappropriate for parsing out XML files.

Quote:

You can't realistically parse tag-based markup languages like HTML and XML using Bash or utilities such as grep, sed or cut. If you just want to dump/render HTML, see (links|links2|lynx|w3m) -dump, html2text, vilistextum. For parsing out pieces of data, see tidy+(xmlstarlet|xmlgawk|xpath|xml2), or learn xslt. Ask #xml and #html for more help. See http://www.codinghorror.com/blog/archives/001311.html

- FreeNode #bash factoid

worm5252 · 03-30-2010, 02:15 PM

Well my challege is not that I cant parse it so to speak. The problem is the API provides a poorly structured response. You can see the xml above in my first post, instead of making it structured like <LoadBalancerID></LoadBalancerID> they use

Code:

<object name="loadbalancer">
        <attribute name="id">6577</attribute>

Everything ends up being an attribute of <object name="loadbalancer">. So There is not a real way for me to parse it to obtain information on the load balancer members, the load balncer ID, etc. So thats where bash comes into play. Aside from it is the only language I know

worm5252 · 03-30-2010, 02:17 PM

Quote:

Originally Posted by PMP

echo `grep -A2 name=\"loadbalancer test.xml` | sed -e 's/.*name="id">\([0-9]\+\)\?.*name="name">\([-a-zA-Z0-9 ._]\+\)\?<.*/LB_ID=\1 LB_NAME=\2/'

The test.xml is the name of the file, change this only not name=

The text in magenta is the text to be grepped and in the red is the file name you habe to change the filename to the file along with your path

I tried this on my xml. I get the following results

Code:

[root@WS1 scripts]# echo `grep -A2 name=\"loadbalancer /tmp/ggapi.list_load_balancer.xml` | sed -e 's/.*name="id">\([0-9]\+\)\?.*name="name">\([-a-zA-Z0-9 ._]\+\)\?<.*/LB_ID=\1 LB_NAME=\2/'
LB_ID=1 LB_NAME=On

so it works, just doesn't give me desired information

grail · 03-30-2010, 10:03 PM

Well I would like a little feedback on how to multidimensional arrays, but my workaround here seems to work with
the input you provided:

Code:

#!/usr/bin/awk -f

BEGIN{
		resp = 0
		load = 0
		ip = 0
		ippair = 0
}

/<response.*loadbalancer/{ resp = 1 }

resp && /<\/response>/{ resp = 0 }

resp && /me="loadbalancer/{ load = 1 }

load && /me="id/{ id = gensub(/.*id">|<\/at.*/, "", "g") }
load && /me="name/{ name = gensub(/.*me">|<\/at.*/, "", "g") }
load && /me="port/{ load = 0; ip = 1 }

ip && /me="ipportpair/{ ippair = 1 }

ippair && /me="id/{ 
		arr_index = gensub(/.*id">|<\/at.*/, "", "g") 
		getline arr[arr_index]
		gsub(/.*ip">|<\/at.*/, "", arr[arr_index])

}

ippair && /me="port/{ 
		arr[arr_index] = arr[arr_index]":"gensub(/.*rt">|<\/at.*/, "", "g") 
		ippair = 0
}

END{
		print "Loadbalancer ID# "id
		print "Loadbalancer name: "name

		for (x in arr){
				print "\nIP ID# "x" has the following address and port:"
				print "\tAddress: "gensub(/:.*/,"","g",arr[x])
				print "\tPort: "gensub(/.*:/,"","g",arr[x])
		}
}

PMP · 03-31-2010, 02:34 AM

I will suggest you to to this task with perl ,
have a look at
http://search.cpan.org/~grantm/XML-S.../XML/Simple.pm

Sergei Steshenko · 03-31-2010, 03:38 AM

Quote:

Originally Posted by PMP

I will suggest you to to this task with perl ,
have a look at
http://search.cpan.org/~grantm/XML-S.../XML/Simple.pm

Yep.

Or any other full-fledged XML (Perl is still my favorite language) parser.

The point is that XML is not line based while standard UNIX text munging tools are line-oriented, so ad-hoc parser written in them won't be robust.

bigearsbilly · 03-31-2010, 03:54 AM

crikey you are making life hard for yourself.

as said here:

Quote:

Originally Posted by tuxdev

Bash is completely and utterly inappropriate for parsing out XML files.

- FreeNode #bash factoid

would you service a car with a knife and a pair of pliers?
learn how to use the proper tools.

I would use perl and XML::Twig.