Perl read file and parse blocks

konsolebox · 08-21-2010, 08:07 PM

grail · 08-22-2010, 01:40 AM

So it seems to work just fine for me (see output below based on the input provided by OP).
Also, the code does not care if there are more or less fields in each configuration.

Output:

Code:

{ 512-byte Blocks=10485767 Cylinders=5127 Attached VDEV TGT Device=N/A Device Symmetrix Name=1234 Device Serial ID=N/B Device Physical Name=Not Visibla Tracks=76807 Attached BCV Device=N/A MegaBytes=5127 KiloBytes=5242887 }
{ 512-byte Blocks=10485760 Cylinders=5120 Device Symmetrix Name=4567 Device Serial ID=N/A Device Physical Name=Not Visible Tracks=76800 MegaBytes=5120 KiloBytes=5242880 }

Obviously you can do with it as you will, I was just demonstrating that the output is correct.

NOTE: I altered the first lot of data to have different ending digits so I could see I was getting 2 lots of data.

konsolebox · 08-22-2010, 01:58 AM

@grail I was really bothered about post #9. It's sort of different.

grail · 08-22-2010, 02:08 AM

Well there is obviously one problem with the data from post #9:

Code:

Device Physical Name,Device Symmetrix Name,Symmetrix ID.....
Not Visible,1234,1234567
Not Visible,3456,1234567
Not Visible,8726,1234567
Not Visible,0000,1234567
Not Visible,1234,1234567

That would be that the data presented to us does not have the field - Symmetrix ID

But I presume that is just one of the many possibilities.

I do see the code I have presented easily producing said output

Goni · 08-22-2010, 03:04 AM

Quote:

Originally Posted by grail

So it seems to work just fine for me (see output below based on the input provided by OP).
Also, the code does not care if there are more or less fields in each configuration.

Output:

Code:

{ 512-byte Blocks=10485767 Cylinders=5127 Attached VDEV TGT Device=N/A Device Symmetrix Name=1234 Device Serial ID=N/B Device Physical Name=Not Visibla Tracks=76807 Attached BCV Device=N/A MegaBytes=5127 KiloBytes=5242887 }
{ 512-byte Blocks=10485760 Cylinders=5120 Device Symmetrix Name=4567 Device Serial ID=N/A Device Physical Name=Not Visible Tracks=76800 MegaBytes=5120 KiloBytes=5242880 }

Obviously you can do with it as you will, I was just demonstrating that the output is correct.

NOTE: I altered the first lot of data to have different ending digits so I could see I was getting 2 lots of data.

Thanks for the help grail. But I don't see the code producing the output correctly. If you see my first post, the data presented starts way before 512-byte Block. All those fields are missed from the output. Post #9 was just a sample output to illustrate how I should be expecting the output. That was not real data. Real data was in post 1, and 1 complete set of information is here (next post)

I think it is going to be a lot more complex than I think. Don't you think it would be better to define all the fields first, and then look for data for all those fields in each cycle of block from "Device Physical Name" to next "Device Physical Name"?

grail · 08-22-2010, 03:05 AM

So to get the output I cam up with:

Code:

#!/usr/bin/perl

use warnings;
#use strict;

open(HANDLE, "file1") || die "Unable to open file1";

@headings=("Device Physical Name","Device Symmetrix Name","Device Serial ID","Attached BCV Device");
push @headings, ("Attached VDEV TGT Device","Cylinders","Tracks","512-byte Blocks","MegaBytes","KiloBytes");
for (@headings){ push @headings_w_commas, "$_,"; }

$headings_w_commas[$#headings_w_commas] =~ s/,$//;

$counter = 0;

while($line = <HANDLE>){
    chomp($line);
    if ($line =~ /Device Physical Name/)
    {
        $counter++;
        $records = {};
    }

    if ($line ne "" && $line =~ /:/)
    {
        ($field,$value) = split(/:/, $line);
        $records->{trim($field)} = trim($value);
    }

    push @array, $records if ($line eq "") ;
}

push @array, $records if ($counter != scalar @array);

print "@headings_w_commas\n";

for $href ( @array ) {
    for $role ( @headings ) { 
        if ( exists $href->{$role} )
        {
            print "$href->{$role}, ";
        }
        else
        {
            print ", ";
        }
    }
    print "\n";
}

close(HANDLE);

sub trim
{
    my $string = shift;

    $string =~ s/^\s+//;
    $string =~ s/\s+$//;

    return $string;
}

Again there are probably much better alternatives

grail · 08-22-2010, 03:06 AM

Quote:

But I don't see the code producing the output correctly. If you see my first post, the data presented starts way before 512-byte Block

Yes the data is out of order as it is stored in a hash but if you look through it you will see that all the fields are there.

Goni · 08-22-2010, 03:15 AM

Here is a good set of sample data.

PHP Code:



    Device Physical Name     : Not Visible 
 
    Device Symmetrix Name    : 0521 
    Device Serial ID         : N/A 
    Symmetrix ID             : 000180100011 
 
    Attached BCV Device      : N/A 
 
    Attached VDEV TGT Device : N/A 
 
    Vendor ID                : EMC 
    Product ID               : SYMMETRIX        
    Product Revision         : 5773 
    Device WWN               : 60060480000290104621533030353231 
    Device Emulation Type    : FBA             
    Device Defined Label Type: N/A             
    Device Defined Label     : N/A             
    Device Sub System Id     : 0x0005 
    Cache Partition Name     : DEFAULT_PARTITION 
 
    Device Block Size        : 512 
 
    Device Capacity 
        { 
        Cylinders            :      18414 
        Tracks               :     276210 
        512-byte Blocks      :   35354880 
        MegaBytes            :      17263 
        KiloBytes            :   17677440 
        } 
 
    Effective Device Geometry: N/A 
        { 
        Sectors/Track        : N/A 
        Tracks/Cylinder      : N/A 
        Cylinders            : N/A 
        MegaBytes            : N/A 
        KiloBytes            : N/A 
        } 
 
    Device Configuration     : RAID-5          (Meta Member, 
                                                Non-Exclusive Access) 
 
    Device is WORM Enabled   : No 
    Device is WORM Protected : No 
 
    SCSI-3 Persistent Reserve: Disabled 
 
    Dynamic Spare Invoked    : No 
 
    Dynamic RDF Capability   : RDF2_Capable 
 
    STAR Mode                : No 
    STAR Recovery Capability : None 
    STAR Recovery State      : NA 
 
    Device Service State     : Normal          
 
    Device Status            : Ready            (RW) 
    Device SA Status         : Ready            (RW) 
 
    Front Director Paths (2): 
        { 
        ---------------------------------------------------------------------- 
                                 POWERPATH  DIRECTOR   PORT             LUN    
                                 --------- ----------  ---- -------- --------- 
        PdevName                 Type      Type Num    Sts  VBUS TID SYMM Host 
        ---------------------------------------------------------------------- 
        Not Visible              N/A       FA   08C:0  RW   000  00  0C8  N/A 
        Not Visible              N/A       FA   09C:0  RW   000  00  0C8  N/A 
        } 
 
    Meta Configuration       : Striped 
    Meta Stripe Size         : 960k    (1 Cylinders) 
    Meta Device Members (6)  : 
        { 
        ---------------------------------------------------------------------- 
                               BCV  DATA                    RDF  DATA          
                      ----------------------------  -------------------------- 
        Sym    Cap    Std Inv BCV Inv Pair          R1 Inv R2 Inv Pair         
        Dev    (MB)   Tracks  Tracks  State         Tracks Tracks State        
        ---------------------------------------------------------------------- 
        0517  17263        -       -  N/A                -      - N/A          
        0518  17263        -       -  N/A                -      - N/A          
        0523  17263        -       -  N/A                -      - N/A          
        0524  17263        -       -  N/A                -      - N/A          
    --> 0521  17263        -       -  N/A                -      - N/A          
        0526  17263        -       -  N/A                -      - N/A          
        ---------------------------------------------------------------------- 
             103579        -       -                     -      - 
        } 
 
    Mirror Set Type          : \[RAID-5,RAID-5,N/A,N/A\] 
 
    Mirror Set DA Status     : \[RW,RW,N/A,N/A\] 
 
    Mirror Set Inv. Tracks   : \[0,0,0,0\] 
 
    Back End Disk Director Information 
        { 
        Hyper Type                             : RAID-5 
        Hyper Status                           : Ready           (RW) 
        Disk \[Director, Interface, TID\]        : \[N/A,N/A,N/A\] 
        Disk Director Volume Number            : N/A 
        Hyper Number                           : N/A 
        Mirror Number                          : 1 
        Disk Group Number                      : 0 
        Disk Group Name                        : DISK_GROUP_000 
 
        Hyper Type                             : RAID-5 
        Hyper Status                           : Ready           (RW) 
        Disk \[Director, Interface, TID\]        : \[N/A,N/A,N/A\] 
        Disk Director Volume Number            : N/A 
        Hyper Number                           : N/A 
        Mirror Number                          : 2 
        Disk Group Number                      : 0 
        Disk Group Name                        : DISK_GROUP_000 
        } 
 
    RAID-5 Device Information 
        { 
        Number of Tracks in a Stripe                : 4       
        Overall Ready State of RAID-5 Device        : ReadyNoOtherMirror 
        Overall WriteProtect State of RAID-5 Device : EnabledNoOtherMirror 
        Member Number of the Failing Device         : None 
        Member Number that Invoked the Spare        : None 
        Disk Director (DA) that Owns the Spare      : None 
        Copy Direction                              : N/A 
        RAID-5 Hyper Devices (3+1): 
            { 
            Device : 0517 (M) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                02A:D8    560    16    5755    1 RW      N/A        0   286102 
                01C:C8    213    16    5755    2 RW      N/A        0   286102 
                15C:C8    213    16    5755    3 RW      N/A        0   286102 
                16A:D8    951    16    5755    4 RW      N/A        0   286102 
                } 
            Device : 0518 (m) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                15D:D0    364    17    5755    1 RW      N/A        0   286102 
                16B:C0     17    17    5755    2 RW      N/A        0   286102 
                02B:C0     17    17    5755    3 RW      N/A        0   286102 
                01D:D0    756    17    5755    4 RW      N/A        0   286102 
                } 
            Device : 0523 (m) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                15B:Dc    658    17    5755    1 RW      N/A        0   286102 
                16C:Dc   1054    17    5755    2 RW      N/A        0   286102 
                02C:Dc    663    17    5755    3 RW      N/A        0   286102 
                01B:Dc   1049    17    5755    4 RW      N/A        0   286102 
                } 
            Device : 0524 (m) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                15A:Cc    313    16    5755    1 RW      N/A        0   286102 
                16D:Cc    310    16    5755    2 RW      N/A        0   286102 
                02D:Cc    310    16    5755    3 RW      N/A        0   286102 
                01A:Cc    313    16    5755    4 RW      N/A        0   286102 
                } 
            Device : 0521 (m) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                02B:Dd    738    17    5755    1 RW      N/A        0   286102 
                01C:Dd   1056    17    5755    2 RW      N/A        0   286102 
                15C:Dd    737    17    5755    3 RW      N/A        0   286102 
                16B:Dd   1057    17    5755    4 RW      N/A        0   286102 
                } 
            Device : 0526 (m) 
                { 
                -------------------------------------------------------------- 
                 Disk     DA       Hyper       Member    Spare       Disk        
                DA :IT   Vol#   Num Cap(MB)  Num Status  Status  Grp#  Cap(MB) 
                -------------------------------------------------------------- 
                15A:D9    611    17    5755    1 RW      N/A        0   286102 
                16D:D9    927    17    5755    2 RW      N/A        0   286102 
                02D:D9    608    17    5755    3 RW      N/A        0   286102 
                01A:D9    930    17    5755    4 RW      N/A        0   286102 
                } 
            } 
        }

grail · 08-22-2010, 03:44 AM

Right ... so basically nothing at all like the original data. If you make the following change:

Code:

push @array, $records if ($line eq "") ;

#to

push @array, $records if ($line eq "" && $counter > scalar @array) ;

This will make it display the only record starting with "Device Physical Name" which was your original prerequisite.

Now you have added 23 other blank lines so my separator will no longer be of use and you have included data both with and without colons (:)
so now the field delimiter is also fairly much useless.

I think you would now need a Perl expert as this is way outside my expertise (using Perl that is)

Goni · 08-22-2010, 04:21 AM

Thanks grail and everyone else for your help. I guess I would have to look for a more options to fix this issue. Actually, I have this program working but it is compiled with perl2exe and the developer has left the company with no personal email contact. I want to modify the code since it only now works with some older hardware. Need to upgrade it. So, I thought to better rewrite instead of lurking around with assembler

My main aim to use perl was only to get an executable to be distributed to users, but I think even to use bash this can be achieved using SHC.

grail · 08-22-2010, 04:32 AM

Well I could probably knock something up in awk, but you would now need to redefine exactly what you want out of the above config (from post #23).

konsolebox · 08-22-2010, 05:02 AM

So that's the real data? That's way more complicated. Too bad I already created the harder method. Anyway I hope this will also help.

Code:

#!/usr/bin/perl

use strict;
use warnings;

use Text::Trim;

our %fields = ();
our @fieldsOrder = ();

our @entries = ();
our $entry = {};
our $field;
our $value;

sub addField($) {
	my $newField = shift;
	$fields{$newField} = 1;
	push @fieldsOrder, $newField;
}

addField('Device Physical Name');  ## reserve Device Physical Name as the first field

while (<>) {
	chomp;

	if ($_ eq '') {
		if (%$entry) {
			push @entries, $entry;
			$entry = {};
		}
	} elsif (/^[^:]+:[^:]+$/) {
		($field, $value) = split(/:/);

		$field = trim($field);
		$value = trim($value);

		unless (exists $fields{$field}) {
			addField($field);
		}

		$entry->{$field} = $value;
	}
}

if (%$entry) {
	push @entries, $entry;
}

print join(',', @fieldsOrder) . "\n";

foreach $entry (@entries) {
	my $first = 1;

	foreach $field (@fieldsOrder) {
		if (exists $entry->{$field}) {
			if ($first) {
				print $entry->{$field};
				$first = 0;
			} else {
				print ',' . $entry->{$field};
			}
		} else {
			unless ($first) {
				print ',';
			}
		}
	}

	print "\n";
}

edit: maybe more optimized version:

Code:

...

print join(',', @fieldsOrder) . "\n";

our $firstField = shift @fieldsOrder;

foreach $entry (@entries) {
	my $csv;

	if (exists $entry->{$firstField}) {
		$csv = $entry->{$firstField};
	} else {
		$csv = '';
	}

	foreach $field (@fieldsOrder) {
		$csv .= ',';

		if (exists $entry->{$field}) {
			$csv .= $entry->{$field};
		}
	}

	print $csv . "\n";
}

perl script.pl < data.txt

Anyway I should thank you grail for the first script you made. It really made things easier.

Trim function if Text::Trim does not exist.

Code:

sub trim($)
{
	my $string = shift;
	$string =~ s/^\s+//;
	$string =~ s/\s+$//;
	return $string;
}

P.S. Thank you Goni for starting this educational thread

.

konsolebox · 08-22-2010, 05:37 AM

I also noticed that there are some parts were a comma also exists. There are also too many values that have newlines. I think it's better to store these things in a different format instead.. how 'bout XML? If you use XML it will be easier since there are already many read and write parsers for it.

Goni · 08-22-2010, 05:48 AM

Quote:

Originally Posted by konsolebox

I also noticed that there are some parts were a comma also exists. There are also too many values that have newlines. I think it's better to store these things in a different format instead.. how 'bout XML?

Ok, I managed to get this output as xml. A sample is attached. But I must say, the original developer did one heck of a code to work with these text files

PHP Code:



<?xml version="1.0" standalone="yes" ?> 
<SymCLI_ML> 
  <Symmetrix> 
    <Symm_Info> 
      <symid>000290104621</symid> 
    </Symm_Info> 
    <Device> 
      <Dev_Info> 
        <pd_name>Not Visible</pd_name> 
        <dev_name>0000</dev_name> 
        <configuration>VAULT</configuration> 
        <attached_bcv>N/A</attached_bcv> 
        <emulation>VAULT_DEVICE</emulation> 
        <status>N/A</status> 
        <sa_status>N/A</sa_status> 
        <service_state>N/A</service_state> 
        <ssid>0x0000</ssid> 
        <cache_partition_name>N/A</cache_partition_name> 
      </Dev_Info> 
      <Attached> 
        <BCV>N/A</BCV> 
        <VDEV>N/A</VDEV> 
      </Attached> 
      <Product> 
        <vendor>EMC</vendor> 
        <name>SYMMETRIX</name> 
        <revision>5773</revision> 
        <serial_id>N/A</serial_id> 
        <symid>000290104621</symid> 
      </Product> 
      <Label> 
        <type>N/A</type> 
        <defined_label>N/A</defined_label> 
      </Label> 
      <Flags> 
        <ckd>False</ckd> 
        <worm_enabled>False</worm_enabled> 
        <worm_protected>False</worm_protected> 
        <dynamic_spare_invoked>False</dynamic_spare_invoked> 
        <dynamic_rdf_capability>None</dynamic_rdf_capability> 
        <star_mode>False</star_mode> 
        <star_recovery_capability>None</star_recovery_capability> 
        <star_recovery_state>N/A</star_recovery_state> 
        <radiant_managed>False</radiant_managed> 
        <restricted_access_dev>False</restricted_access_dev> 
        <rdb_checksum_enabled>False</rdb_checksum_enabled> 
        <non_exclusive_access>True</non_exclusive_access> 
        <scsi3_persist_res>Disabled</scsi3_persist_res> 
        <vcm>False</vcm> 
        <symmetrix_filesystem>False</symmetrix_filesystem> 
        <snap_save_device>False</snap_save_device> 
        <gatekeeper>False</gatekeeper> 
        <datadev>False</datadev> 
        <meta>None</meta> 
      </Flags> 
      <Capacity> 
        <block_size>512</block_size> 
        <cylinders>5120</cylinders> 
        <tracks>76800</tracks> 
        <blocks>10485760</blocks> 
        <megabytes>5120</megabytes> 
        <kilobytes>5242880</kilobytes> 
      </Capacity> 
      <device_geometry> 
        <geometry_type>N/A</geometry_type> 
        <sectors_per_tracks>N/A</sectors_per_tracks> 
        <tracks_per_cylinder>N/A</tracks_per_cylinder> 
        <cylinders>N/A</cylinders> 
        <megabytes>N/A</megabytes> 
        <kilobytes>N/A</kilobytes> 
      </device_geometry> 
      <Front_End> 
      </Front_End> 
      <Mirror_Set> 
        <Mirror> 
          <number>1</number> 
          <type>Data</type> 
          <status>N/A</status> 
          <invalid_tracks>0</invalid_tracks> 
        </Mirror> 
        <Mirror> 
          <number>2</number> 
          <type>N/A</type> 
          <status>N/A</status> 
          <invalid_tracks>0</invalid_tracks> 
        </Mirror> 
        <Mirror> 
          <number>3</number> 
          <type>N/A</type> 
          <status>N/A</status> 
          <invalid_tracks>0</invalid_tracks> 
        </Mirror> 
        <Mirror> 
          <number>4</number> 
          <type>N/A</type> 
          <status>N/A</status> 
          <invalid_tracks>0</invalid_tracks> 
        </Mirror> 
      </Mirror_Set> 
      <Back_End> 
        <Hyper> 
          <type>Data</type> 
          <status>N/A</status> 
          <number>1</number> 
          <mirror_number>1</mirror_number> 
          <Disk> 
            <director>01A</director> 
            <interface>C</interface> 
            <tid>0</tid> 
            <volume_number>1</volume_number> 
            <megabytes>286102</megabytes> 
            <actual_megabytes>286102</actual_megabytes> 
            <disk_group>0</disk_group> 
            <disk_group_name>DISK_GROUP_000</disk_group_name> 
          </Disk> 
        </Hyper> 
      </Back_End> 
    </Device> 
  </Symmetrix> 
</SymCLI_ML>

konsolebox · 08-22-2010, 05:59 AM

Btw what's special about this line? I mean the arrow:

Code:

    --> 0521  17263        -       -  N/A                -      - N/A

As I find that everything except that one contains has even leading spaces.