[SOLVED] sed match multiple lines

bradvan · 08-16-2016, 12:15 PM

I have an xml file that has multiple user sections. They look like:

Code:

<ADD_USER
   USERNAME = "Ralph Delp"
   USER_LOGIN = rpdelp
   PASSWORD = "%user_password%">
</ADD_USER>

There are multiple sections like this and I want to delete specific blocks. I am trying:

Code:

sed -i '/^<ADD_USER/{ N;/^   USERNAME = "Ralph Delp"/ {/^<ADD_USER/,/^<\/ADD_USER>/d}}' file

but I'm not getting a match and delete. I tried without the '^' in the username line:

Code:

sed -i '/^<ADD_USER/{ N;/USERNAME = "Ralph Delp"/ {/^<ADD_USER/,/^<\/ADD_USER>/d}}' file

It does seem to make the match, but only deletes the ADD_USER and USERNAME lines. Any suggestions?

syg00 · 08-16-2016, 06:40 PM

Quote:

Originally Posted by bradvan

Any suggestions?

Don't use sed.
You will get answers, most likely involving adding lines to the hold buffer until you hit the end line (</ADD_USER), then do the match and delete. If you just want the job done, use awk or perl or whatever - something that has better logic.

astrogeek · 08-16-2016, 07:03 PM

Agree, don't use sed!

The file is XML, so a simple XML tool like xmlstarlet would do the trick.

If the file is formatted as shown you could use awk with '<ADD_USER' or </ADD_USER> as the record separator.

Turbocapitalist · 08-17-2016, 02:54 AM

Quote:

Originally Posted by bradvan

Code:

<ADD_USER

Is the element really broken like that? It seems to be missing the closing less than ( > ) sign. If it's really not there, then you'll probably have to try "awk" as suggested. But if your file is well-formed XML then you can use some really easy XML parsers like XML::TreeBuilder or XML::TreeBuilder::XPath in Perl.

bradvan · 08-17-2016, 06:14 AM

Thanks! Taking a look at other tools.

Will post once I have a working answer.

syg00 · 08-17-2016, 06:17 AM

Do you have a blank (null) line after each entry ?. If yes, it will be real easy in awk.

ondoho · 08-17-2016, 02:31 PM

xml.
(and yes, the syntax is valid)
i have been succesfully using xmllint (part of libxml2) for things like this. the magic word to search for is xpath.

bradvan · 08-18-2016, 07:04 AM

Well, I have been looking at perl, but still not quite there. The xml files is the output of hponcfg which queries the local integrated lights out (iLO) interface on the HP server. I want to be able to add/remove users from iLO without having to go in to the graphical interface. A more complete listing of the xml is:

Code:

<!-- HPONCFG VERSION = "4.4.0" -->
<RIBLC VERSION="2.1">
 <LOGIN USER_LOGIN="Administrator" PASSWORD="password">
  <DIR_INFO MODE="write">
  <MOD_DIR_CONFIG>
    <DIR_AUTHENTICATION_ENABLED VALUE = "N"/>
    <SEVERAL MORE DIR LINES/>
  </MOD_DIR_CONFIG>
  </DIR_INFO>
  <RIB_INFO MODE="write">
  <MOD_NETWORK_SETTINGS>
    <SPEED_AUTOSELECT VALUE = "Y"/>
    <NIC_SPEED VALUE = "10"/>
    <SEVERAL MORE NETWORK LINES/>
  </MOD_NETWORK_SETTINGS>
  </RIB_INFO>
  <USER_INFO MODE="write">
  <ADD_USER
   USERNAME = "Ralph Delp"
   USER_LOGIN = rpdelp
   PASSWORD = "%user_password%">
   <ADMIN_PRIV value = "Y"/>
   <REMOTE_CONS_PRIV value = "Y"/>
   <RESET_SERVER_PRIV value = "Y"/>
   <VIRTUAL_MEDIA_PRIV value = "Y"/>
   <CONFIG_ILO_PRIV value = "Y"/>
  </ADD_USER>
  <ADD_USER
   USERNAME = "Root User"
   USER_LOGIN = "root"
   PASSWORD = "%user_password%">
   <ADMIN_PRIV value = "Y"/>
   <REMOTE_CONS_PRIV value = "Y"/>
   <RESET_SERVER_PRIV value = "Y"/>
   <VIRTUAL_MEDIA_PRIV value = "Y"/>
   <CONFIG_ILO_PRIV value = "Y"/>
  </ADD_USER>
 </USER_INFO>
 </LOGIN>
</RIBCL>

I am trying to delete elements based on the attribute values. In this case USERNAME. You edit out what you want to keep, and then you can re-submit what you want to delete to iLO with the edited xml file. So, if I want to preserve Ralph's account, I need to remove it from the xml file before sending it back and telling hponcfg to delete what is in the sent xml file. My perl is having a little problem. I can print all of the USERNAME values, but I can't seem to match against it. I have:

Code:

#!/usr/bin/perl
use strict;
use warnings;
use XML::LibXML;
die "Usage: $0 filename <name to delete>\n"
   unless ( @ARGV > 2 );
my $xml_file = shift;
my $bad_name = shift;
print "Searching for bad name: $bad_name\n";
my $xml = XML::LibXML->new;
my $dom = $xml->parse_file( $xml_file );
foreach my $user ( $dom->findnodes ( '/RIBCL/LOGIN/USER_INFO/ADD_USER' )) {
   my ($name) = $user->findnodes( './@USER_NAME');
   print $name->to_literal, "\n";
   if ( "$name->to_literal" eq "$bad_name" ) {
      my ($addu) = $name->parentNode;
      print "Removing element: ", $addu->to_literal, "\n";
   }
}

Which does print out all of the user names, but never matches. Instead of the 'eq' I tried:

Code:

if ( "$name->to_literal" =~ /"$bad_name"/ ) {

but that did not work either. I thought maybe there might be some leading or trailing spaces. So, I printed with a leading and trailing "X" but found not spaces. Any ideas on why I can't make a match?

bradvan · 08-18-2016, 07:05 AM

Oh, and no. No, blank or null lines.

bradvan · 08-18-2016, 08:49 AM

OK, now making the match.

Code:

#!/usr/bin/perl
use strict;
use warnings;
use XML::LibXML;
die "Usage: $0 filename <name to delete>\n"
   unless ( @ARGV > 2 );
my $xml_file = shift;
my $bad_name = shift;
chomp($bad_name);
print "Searching for bad name: $bad_name\n";
my $xml  = XML::LibXML->new;
my $dom  = $xml->parse_file( $xml_file );
my $root = $dom->getDocumentElement;
# Retrieve array of all users
my @users = $root->getElementsByTagName('ADD_USER');
foreach my $user ( @users ) {
   my $username = $user->getAttribute('USER_NAME');
   if ( "$username" eq "$bad_name" ) {
      print "Username = $username \n";
      print "user node type is: ", $user->nodeType, "\n";
      my ($addu) = $user->parentNode;
      print "addu node type is: ", $addu->nodeType, "\n";
      print "Removing element.\n";
      $addu->removeChild( $user );
      print $dom->toString, "\n";
   }
}

The user node type and addu node type are both 1 which is an element. My presumption is that $addu->removeChild( $user ) should remove the corresponding $user element from $root, correct? I then wanted to print out the new document with the removed element, but all I get is a lot of blank lines. So, two questions:

1. Do you think I have correctly removed the element?
2. How to print out the result?

I was using the wrong variable to print out and the wrong method. This is getting me the result, but with an extra blank line. Very close!

Thanks!

bradvan · 08-18-2016, 09:48 AM

Solved!

I can now delete multiple users at a time and no blank lines in the output. Thanks for all of the suggestions! Here is my final code:

Code:

#!/usr/bin/perl
use strict;
use warnings;
use XML::LibXML;
die "Usage: $0 filename <name to delete>\n"
   unless ( @ARGV > 1 );
my $xml_file  = shift;
my @bad_names = @ARGV;
my $output;
my $xml  = XML::LibXML->new;
my $dom  = $xml->parse_file( $xml_file );
my $root = $dom->getDocumentElement;
# Retrieve array of all users
my @users = $root->getElementsByTagName('ADD_USER');
foreach my $bad_name ( @bad_names ) {
   print "Searching for bad name: $bad_name\n";
   foreach my $user ( @users ) {
      my $username = $user->getAttribute('USER_NAME');
      if ( "$username" eq "$bad_name" ) {
         my ($addu) = $user->parentNode;
         print "Removing element for user name: ", $username, "\n";
         $addu->removeChild( $user );
         $output = $dom->toString(0);
         $outout =~ s/(?<=\n)\s*\n//g;
      }
   }
}
open( my $FH, '>', "newilo") or die "Could not open file newilo $!";
print $FH $output, "\n";
close $FH;

astrogeek · 08-18-2016, 05:14 PM

I am a little late getting back in here, but this might be worth mentioning...

Using xmlstarlet (find project at sourceforge), it reduces to this...

Code:

xml ed -d "//ADD_USER[@USERNAME='Ralph Delp']" test.xml

Your example file had a couple of errors corrected to this...

Code:

<!-- HPONCFG VERSION = "4.4.0" -->
<RIBLC VERSION="2.1">
 <LOGIN USER_LOGIN="Administrator" PASSWORD="password">
  <DIR_INFO MODE="write">
  <MOD_DIR_CONFIG>
    <DIR_AUTHENTICATION_ENABLED VALUE = "N"/>
  </MOD_DIR_CONFIG>
  </DIR_INFO>
  <RIB_INFO MODE="write">
  <MOD_NETWORK_SETTINGS>
    <SPEED_AUTOSELECT VALUE = "Y"/>
    <NIC_SPEED VALUE = "10"/>
    <SEVERAL MORE NETWORK LINES/> //Removed this line
  </MOD_NETWORK_SETTINGS>
  </RIB_INFO>
  <USER_INFO MODE="write">
  <ADD_USER
   USERNAME = "Ralph Delp"
   USER_LOGIN = rpdelp //Attr value must be quoted
   PASSWORD = "%user_password%">
   <ADMIN_PRIV value = "Y"/>
   <REMOTE_CONS_PRIV value = "Y"/>
   <RESET_SERVER_PRIV value = "Y"/>
   <VIRTUAL_MEDIA_PRIV value = "Y"/>
   <CONFIG_ILO_PRIV value = "Y"/>
  </ADD_USER>
  <ADD_USER
   USERNAME = "Root User"
   USER_LOGIN = "root"
   PASSWORD = "%user_password%">
   <ADMIN_PRIV value = "Y"/>
   <REMOTE_CONS_PRIV value = "Y"/>
   <RESET_SERVER_PRIV value = "Y"/>
   <VIRTUAL_MEDIA_PRIV value = "Y"/>
   <CONFIG_ILO_PRIV value = "Y"/>
  </ADD_USER>
 </USER_INFO>
 </LOGIN>
</RIBLC> /// L-C transposed

With those changes to make it valid XML and saved as test.xml the result is...

Code:

xml ed -d "//ADD_USER[@USERNAME='Ralph Delp']" test.xml

<?xml version="1.0"?>
<!-- HPONCFG VERSION = "4.4.0" -->
<RIBLC VERSION="2.1">
  <LOGIN USER_LOGIN="Administrator" PASSWORD="password">
    <DIR_INFO MODE="write">
      <MOD_DIR_CONFIG>
        <DIR_AUTHENTICATION_ENABLED VALUE="N"/>
      </MOD_DIR_CONFIG>
    </DIR_INFO>
    <RIB_INFO MODE="write">
      <MOD_NETWORK_SETTINGS>
        <SPEED_AUTOSELECT VALUE="Y"/>
        <NIC_SPEED VALUE="10"/>
      </MOD_NETWORK_SETTINGS>
    </RIB_INFO>
    <USER_INFO MODE="write">
      <ADD_USER USERNAME="Root User" USER_LOGIN="root" PASSWORD="%user_password%">
        <ADMIN_PRIV value="Y"/>
        <REMOTE_CONS_PRIV value="Y"/>
        <RESET_SERVER_PRIV value="Y"/>
        <VIRTUAL_MEDIA_PRIV value="Y"/>
        <CONFIG_ILO_PRIV value="Y"/>
      </ADD_USER>
    </USER_INFO>
  </LOGIN>
</RIBLC>

It is a great little tool for doing XML searches and edits - kind of like sed for xml!

bradvan · 08-19-2016, 06:24 AM

Thanks for that. One problem is that it is not installed on any of my servers. I did find it in the EPEL repository and was trying to work with it, but I was getting a little confused by what I was finding on the Internet versus what I found in the rpm. In my rpm there are several files under /usr/share/doc, a man page and one binary, /usr/bin/xmlstarlet. All of the examples I've seen and even the one you gave show using the command 'xml.' I'm a little confused. Shouldn't it be xmlstartlet ed -d ...?

astrogeek · 08-19-2016, 11:16 AM

I recall being a little confused when I first started using it, but I made myself notes on the basic operations I use so I have not had to look back at the docs for a while.

I would suggest that you review or grab a local copy of the online documentation from here - it is much more complete than the man page and has some really good examples of all operations.

From the top link on that page you will find the XmlStarlet Command Line XML Toolkit User's Guide which will get you directly to working examples from the indexed list of common operations.

If you are not familiar with XPath notation then you might want to do a little searching for that as well - XPath is the notation used to navigate your XML documents and specify internal elements. For example, the expression I used in my last post was...

Code:

xml ed -d "//ADD_USER[@USERNAME='Ralph Delp']" test.xml

... which means "//"= any path to an element named "ADD_USER" which has an attribute named "USERNAME" with a value of "Ralph Delp". Learning the basics of XPath is easy but essential.

And yes, the command is "xml", not xmlstarlet.

If you need a little more help please post back here, it is well worth learning your way around it and I and others will be glad to help!

bradvan · 08-19-2016, 12:30 PM

Thanks! Will continue to research.