LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   grep/sed/awk multi line array (https://www.linuxquestions.org/questions/programming-9/grep-sed-awk-multi-line-array-4175430739/)

earthgecko 10-05-2012 02:34 PM
grep/sed/awk multi line array
 
This is not homework :)

Actually it is work at home. Need a ninja!

I want to match and covert ruby variables which are multiple line arrays to single line variable.

So considering the example:

/tmp/file

Code:
# comment blah
  file { '/etc/blah':
    something => Here['match to the first occurrence'],
  }

  $from_here = ['foo,
                'bar',
                "${to_somewhere_near_here}",
              ]

$extlookup_precedence = ["%{puppet_env}", "%{parent_domain}", "common", "api"]

filebucket { 'main': server => puppet }
File { backup => main }

# comment blah2
  file { '/etc/blah2':
    something => Here['match to the first occurrence'],
  }

  $and_this = ['foo,
                'bar',
                "${to_somewhere_near_here}",
              ]

 $not_relevant = 'not_this'
I am trying to grep/sed/awk that into:

Code:
$from_here="foo,bar,${to_somewhere_near_here}"
$extlookup_precedence="%{puppet_env},%{parent_domain},common,api"
$and_this="foo,bar,${to_somewhere_near_here}"
Having problems with matching each occurrence of the pattern and outputting only to the first match of the second pattern and then from the second match of the first pattern to the second match of the second pattern and so on.

cat /tmp/file | grep -v "^#" | sed -n '/\$.* = \[/,/.*\]$/p'

And various iterations of all sorts, \1, etc, etc and I am no closer. I have the remove quotes parts and double quote and comma separate, just failing to iterate the multi line arrays properly.

Any ninjas?

grail 10-06-2012 10:47 AM
One would ask that if you are comfortable using Ruby, why not use it for your task?

Also, is the goal to only print the global variables or are you intending to keep all the other data as well but convert the arrays to strings?

earthgecko 10-06-2012 12:11 PM
Hi grail

Hey if it works, I don't care ruby, shell, (perl, python) they can all be execd. But it is in shell at the moment so.

The intended goal is to get every variable which has the data in an array converted into a string shell variable. On that note my example was a little incorrect (copy paste job). There should be no leading $ in the result ;)

Code:
from_here="foo,bar,${to_somewhere_near_here}"
Anyway it is a cool pattern to have. Hell one day who knows maybe it could be groked.

But if you have a pattern for that in ruby :) I would sure appreciated it :)

danielbmartin 10-06-2012 02:50 PM
A brute-force solution. Not pretty but it works.
Code:
paste -s $InFile    \
|tr "\t" " "        \
|sed 's/\${/~{/g'  \
|tr -s ' '          \
|sed 's/\$/\n\$/g'  \
|sed 's/]/]\n/g'    \
|grep [$]          \
|cut -d "]" -f1    \
|grep [[]          \
|cut -d "$" -f2    \
|tr -d " '"        \
|tr '[' '"'        \
|tr "~" "$"        \
|sed 's/","/,/g'    \
|tr -s '"'          \
|sed 's/,*$//'      \
|sed 's/"${/${/'    \
> $OutFile
Daniel B. Martin

earthgecko 10-06-2012 04:00 PM
Yo danielbmartin thanks will try in the morning! Thanks!

amboxer21 10-06-2012 05:21 PM
Got it

sed 's/[ \t]*//g; s/\[//g; $d' test3.txt | awk '/\$/,/\]/ {ORS=""; gsub("]", "\n"); print; echo -e "\n"}' | cut -c2-;

amboxer21 10-06-2012 06:05 PM
......

grail 10-07-2012 08:48 AM
@amboxer21 - just ran your code which returned me:
Code:
$from_here='foo','bar',"${to_somewhere_near_here}",
$extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
$and_this='foo','bar',"${to_somewhere_near_here}",
Ignoring the '$' at the front, I do not see this working very well as a bash variable, ie you need to remove all the excess quoting.

@OP - Are we to assume that the ruby code you run this over will actually be incorrect data? I ask based on 2 examples:
Code:
$from_here = ['foo, # no quote after the word foo will cause ruby to error

"${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error
              ]
My current solution would work better if there are no errors in the original code. Please advise if we are to also account for this?

amboxer21 10-07-2012 11:46 AM
Quote:
Originally Posted by grail (Post 4799392)
@amboxer21 - just ran your code which returned me:
Code:
$from_here='foo','bar',"${to_somewhere_near_here}",
$extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
$and_this='foo','bar',"${to_somewhere_near_here}",
Ignoring the '$' at the front, I do not see this working very well as a bash variable, ie you need to remove all the excess quoting.

@OP - Are we to assume that the ruby code you run this over will actually be incorrect data? I ask based on 2 examples:
Code:
$from_here = ['foo, # no quote after the word foo will cause ruby to error

"${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error
              ]
My current solution would work better if there are no errors in the original code. Please advise if we are to also account for this?
My second solution takes care of the money symbols. The one with cut. I see what your saying about excess quoting though.

earthgecko 10-08-2012 02:09 AM
Thanks all - Daniel. Perfect, you are a ninja! [SOLVED]

amboxer21 thank you as well, however your second version erroneously includes the final variable. Not that I am complaining, thank you to all of you for the time.

Code:
gary@mc9:~$ paste -s /tmp/file    \
> |tr "\t" " "        \
> |sed 's/\${/~{/g'  \
> |tr -s ' '          \
> |sed 's/\$/\n\$/g'  \
> |sed 's/]/]\n/g'    \
> |grep [$]          \
> |cut -d "]" -f1    \
> |grep [[]          \
> |cut -d "$" -f2    \
> |tr -d " '"        \
> |tr '[' '"'        \
> |tr "~" "$"        \
> |sed 's/","/,/g'    \
> |tr -s '"'          \
> |sed 's/,*$//'      \
> |sed 's/"${/${/'
from_here="foo,bar,${to_somewhere_near_here}"
extlookup_precedence="%{puppet_env},%{parent_domain},common,api"
and_this="foo,bar,${to_somewhere_near_here}"
gary@mc9:~$
gary@mc9:~$
gary@mc9:~$ sed 's/[ \t]*//g; s/\[//g; $d' /tmp/file | awk '/\$/,/\]/ {ORS=""; gsub("]", "\n"); print; echo -e "\n"}' | cut -c2-;
from_here='foo,'bar',"${to_somewhere_near_here}",
extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
and_this='foo,'bar',"${to_somewhere_near_here}",
not_relevant='not_this'
gary@mc9:~$

grail 10-08-2012 06:06 AM
Any chance you might answer my questions? Just curious mainly. If I can assume that the original would be actual ruby code, ie without the issues mentioned:
Code:
$ cat earthgecko.infile
# comment blah
  file { '/etc/blah':
    something => Here['match to the first occurrence'],
  }

  $from_here = ['foo',
                'bar',
                "${to_somewhere_near_here}"
              ]

$extlookup_precedence = ["%{puppet_env}", "%{parent_domain}", "common", "api"]

filebucket { 'main': server => puppet }
File { backup => main }

# comment blah2
  file { '/etc/blah2':
    something => Here['match to the first occurrence'],
  }

  $and_this = ['foo',
                'bar',
                "${to_somewhere_near_here}"
              ]

 $not_relevant = 'not_this'
$ ruby -ne 'BEGIN{ squote="\047" }; # set value for single quote
>
> $_.scan(/^\s*\$([^ ]+).*\[(.*)/).each{ #find lines with whitespace before a $ that have a [ after it. Store name and rest of line
>    |x|
>   
>    print x[0] + "="; # print the stored name and append =
>    print squote + x[1].gsub(/["#{squote}\]\s]/,""); # print opening quote and rest of line minus " ' ] or whitespace
>   
>    if !(x[1] =~ /\]/) then # if rest of line does not have ]
>        print $_.gsub(/["#{squote}\]\s]/,"") until gets.chop! =~ /\]/ # get lines from file until finding a ] and remove all of the items listed above
>    end;
>       
>    puts squote # print closing quote
> }' earthgecko.infile

earthgecko 10-08-2012 06:38 AM
Hi grail

1. The 'foo variable was incorrect as you rightly point out it should be 'foo' - apologies that was put together too hastily.

2. comma after last value in array is present but would not necessarily need to be - e.g. "${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error. This is just a legacy formatting related to puppet resources.

Thanks for the input. Ironically when I applied Daniel's pattern to the proper large data set, it did not work :(

So I turned the problem inside out, from a point of view of "do not" change to a one of "rather extend". Changes often have unintended knock-on effects. So all this input really did help me to simplify the issue and just fix the incorrect outputs after the original operation, instead of trying to fix the big original operation. Not pretty, but effective.

amboxer21 10-08-2012 12:10 PM
I'm saying, anyone can group a bunch of excess calls together. I was going for efficiency. I feel it forces you to learn the language/tool your using. If I use the same approach as Daniel has, I could have provided a working solution as well.

earthgecko 10-08-2012 01:06 PM
Hi amboxer21 and grail

I titled the post grep/sed/awk not ruby. Things in the real world are sometimes complex. I know it is possible (probably) to use ruby for everything, but the world is heterogeneous.

Thank you for the ruby examples I will in no doubt reuse some of them in some place, in the future :)

Please do not take offence and if you do, just remember fences are expensive to replace.


All times are GMT -5. The time now is 06:40 PM.