LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-05-2012, 02:34 PM   #1
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Rep: Reputation: 0
grep/sed/awk multi line array


This is not homework

Actually it is work at home. Need a ninja!

I want to match and covert ruby variables which are multiple line arrays to single line variable.

So considering the example:

/tmp/file

Code:
# comment blah
  file { '/etc/blah':
    something => Here['match to the first occurrence'],
  }

  $from_here = ['foo,
                'bar',
                "${to_somewhere_near_here}",
               ]

$extlookup_precedence = ["%{puppet_env}", "%{parent_domain}", "common", "api"]

filebucket { 'main': server => puppet }
File { backup => main }

# comment blah2
  file { '/etc/blah2':
    something => Here['match to the first occurrence'],
  }

  $and_this = ['foo,
                'bar',
                "${to_somewhere_near_here}",
               ]

 $not_relevant = 'not_this'
I am trying to grep/sed/awk that into:

Code:
$from_here="foo,bar,${to_somewhere_near_here}"
$extlookup_precedence="%{puppet_env},%{parent_domain},common,api"
$and_this="foo,bar,${to_somewhere_near_here}"
Having problems with matching each occurrence of the pattern and outputting only to the first match of the second pattern and then from the second match of the first pattern to the second match of the second pattern and so on.

cat /tmp/file | grep -v "^#" | sed -n '/\$.* = \[/,/.*\]$/p'

And various iterations of all sorts, \1, etc, etc and I am no closer. I have the remove quotes parts and double quote and comma separate, just failing to iterate the multi line arrays properly.

Any ninjas?
 
Old 10-06-2012, 10:47 AM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,503

Rep: Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893
One would ask that if you are comfortable using Ruby, why not use it for your task?

Also, is the goal to only print the global variables or are you intending to keep all the other data as well but convert the arrays to strings?
 
Old 10-06-2012, 12:11 PM   #3
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Original Poster
Rep: Reputation: 0
Hi grail

Hey if it works, I don't care ruby, shell, (perl, python) they can all be execd. But it is in shell at the moment so.

The intended goal is to get every variable which has the data in an array converted into a string shell variable. On that note my example was a little incorrect (copy paste job). There should be no leading $ in the result

Code:
from_here="foo,bar,${to_somewhere_near_here}"
Anyway it is a cool pattern to have. Hell one day who knows maybe it could be groked.

But if you have a pattern for that in ruby I would sure appreciated it
 
Old 10-06-2012, 02:50 PM   #4
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,093

Rep: Reputation: 288Reputation: 288Reputation: 288
A brute-force solution. Not pretty but it works.
Code:
paste -s $InFile    \
|tr "\t" " "        \
|sed 's/\${/~{/g'   \
|tr -s ' '          \
|sed 's/\$/\n\$/g'  \
|sed 's/]/]\n/g'    \
|grep [$]           \
|cut -d "]" -f1     \
|grep [[]           \
|cut -d "$" -f2     \
|tr -d " '"         \
|tr '[' '"'         \
|tr "~" "$"         \
|sed 's/","/,/g'    \
|tr -s '"'          \
|sed 's/,*$//'      \
|sed 's/"${/${/'    \
> $OutFile
Daniel B. Martin
 
Old 10-06-2012, 04:00 PM   #5
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Original Poster
Rep: Reputation: 0
Yo danielbmartin thanks will try in the morning! Thanks!
 
Old 10-06-2012, 05:21 PM   #6
amboxer21
Member
 
Registered: Mar 2012
Location: New Jersey
Distribution: Gentoo
Posts: 277

Rep: Reputation: Disabled
Got it

sed 's/[ \t]*//g; s/\[//g; $d' test3.txt | awk '/\$/,/\]/ {ORS=""; gsub("]", "\n"); print; echo -e "\n"}' | cut -c2-;

Last edited by amboxer21; 10-07-2012 at 05:45 PM.
 
Old 10-06-2012, 06:05 PM   #7
amboxer21
Member
 
Registered: Mar 2012
Location: New Jersey
Distribution: Gentoo
Posts: 277

Rep: Reputation: Disabled
......

Last edited by amboxer21; 10-07-2012 at 01:01 PM.
 
Old 10-07-2012, 08:48 AM   #8
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,503

Rep: Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893
@amboxer21 - just ran your code which returned me:
Code:
$from_here='foo','bar',"${to_somewhere_near_here}",
$extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
$and_this='foo','bar',"${to_somewhere_near_here}",
Ignoring the '$' at the front, I do not see this working very well as a bash variable, ie you need to remove all the excess quoting.

@OP - Are we to assume that the ruby code you run this over will actually be incorrect data? I ask based on 2 examples:
Code:
$from_here = ['foo, # no quote after the word foo will cause ruby to error

"${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error
               ]
My current solution would work better if there are no errors in the original code. Please advise if we are to also account for this?
 
Old 10-07-2012, 11:46 AM   #9
amboxer21
Member
 
Registered: Mar 2012
Location: New Jersey
Distribution: Gentoo
Posts: 277

Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
@amboxer21 - just ran your code which returned me:
Code:
$from_here='foo','bar',"${to_somewhere_near_here}",
$extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
$and_this='foo','bar',"${to_somewhere_near_here}",
Ignoring the '$' at the front, I do not see this working very well as a bash variable, ie you need to remove all the excess quoting.

@OP - Are we to assume that the ruby code you run this over will actually be incorrect data? I ask based on 2 examples:
Code:
$from_here = ['foo, # no quote after the word foo will cause ruby to error

"${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error
               ]
My current solution would work better if there are no errors in the original code. Please advise if we are to also account for this?
My second solution takes care of the money symbols. The one with cut. I see what your saying about excess quoting though.

Last edited by amboxer21; 10-07-2012 at 11:50 AM.
 
Old 10-08-2012, 02:09 AM   #10
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Original Poster
Rep: Reputation: 0
Thanks all - Daniel. Perfect, you are a ninja! [SOLVED]

amboxer21 thank you as well, however your second version erroneously includes the final variable. Not that I am complaining, thank you to all of you for the time.

Code:
gary@mc9:~$ paste -s /tmp/file    \
> |tr "\t" " "        \
> |sed 's/\${/~{/g'   \
> |tr -s ' '          \
> |sed 's/\$/\n\$/g'  \
> |sed 's/]/]\n/g'    \
> |grep [$]           \
> |cut -d "]" -f1     \
> |grep [[]           \
> |cut -d "$" -f2     \
> |tr -d " '"         \
> |tr '[' '"'         \
> |tr "~" "$"         \
> |sed 's/","/,/g'    \
> |tr -s '"'          \
> |sed 's/,*$//'      \
> |sed 's/"${/${/'
from_here="foo,bar,${to_somewhere_near_here}"
extlookup_precedence="%{puppet_env},%{parent_domain},common,api"
and_this="foo,bar,${to_somewhere_near_here}"
gary@mc9:~$ 
gary@mc9:~$ 
gary@mc9:~$ sed 's/[ \t]*//g; s/\[//g; $d' /tmp/file | awk '/\$/,/\]/ {ORS=""; gsub("]", "\n"); print; echo -e "\n"}' | cut -c2-;
from_here='foo,'bar',"${to_somewhere_near_here}",
extlookup_precedence="%{puppet_env}","%{parent_domain}","common","api"
and_this='foo,'bar',"${to_somewhere_near_here}",
not_relevant='not_this'
gary@mc9:~$
 
Old 10-08-2012, 06:06 AM   #11
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,503

Rep: Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893Reputation: 1893
Any chance you might answer my questions? Just curious mainly. If I can assume that the original would be actual ruby code, ie without the issues mentioned:
Code:
$ cat earthgecko.infile
# comment blah
  file { '/etc/blah':
    something => Here['match to the first occurrence'],
  }

  $from_here = ['foo',
                'bar',
                "${to_somewhere_near_here}"
               ]

$extlookup_precedence = ["%{puppet_env}", "%{parent_domain}", "common", "api"]

filebucket { 'main': server => puppet }
File { backup => main }

# comment blah2
  file { '/etc/blah2':
    something => Here['match to the first occurrence'],
  }

  $and_this = ['foo',
                'bar',
                "${to_somewhere_near_here}"
               ]

 $not_relevant = 'not_this'
$ ruby -ne 'BEGIN{ squote="\047" }; # set value for single quote
> 
> $_.scan(/^\s*\$([^ ]+).*\[(.*)/).each{ #find lines with whitespace before a $ that have a [ after it. Store name and rest of line
>     |x| 
>     
>     print x[0] + "="; # print the stored name and append =
>     print squote + x[1].gsub(/["#{squote}\]\s]/,""); # print opening quote and rest of line minus " ' ] or whitespace
>     
>     if !(x[1] =~ /\]/) then # if rest of line does not have ]
>         print $_.gsub(/["#{squote}\]\s]/,"") until gets.chop! =~ /\]/ # get lines from file until finding a ] and remove all of the items listed above
>     end; 
>         
>     puts squote # print closing quote
> }' earthgecko.infile
 
Old 10-08-2012, 06:38 AM   #12
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Original Poster
Rep: Reputation: 0
Hi grail

1. The 'foo variable was incorrect as you rightly point out it should be 'foo' - apologies that was put together too hastily.

2. comma after last value in array is present but would not necessarily need to be - e.g. "${to_somewhere_near_here}", # here you have a comma to say there is another element but the next piece of data is the end of the array, this will also cause an error. This is just a legacy formatting related to puppet resources.

Thanks for the input. Ironically when I applied Daniel's pattern to the proper large data set, it did not work

So I turned the problem inside out, from a point of view of "do not" change to a one of "rather extend". Changes often have unintended knock-on effects. So all this input really did help me to simplify the issue and just fix the incorrect outputs after the original operation, instead of trying to fix the big original operation. Not pretty, but effective.
 
Old 10-08-2012, 12:10 PM   #13
amboxer21
Member
 
Registered: Mar 2012
Location: New Jersey
Distribution: Gentoo
Posts: 277

Rep: Reputation: Disabled
I'm saying, anyone can group a bunch of excess calls together. I was going for efficiency. I feel it forces you to learn the language/tool your using. If I use the same approach as Daniel has, I could have provided a working solution as well.
 
Old 10-08-2012, 01:06 PM   #14
earthgecko
LQ Newbie
 
Registered: Sep 2007
Distribution: ubuntu and centos
Posts: 21

Original Poster
Rep: Reputation: 0
Hi amboxer21 and grail

I titled the post grep/sed/awk not ruby. Things in the real world are sometimes complex. I know it is possible (probably) to use ruby for everything, but the world is heterogeneous.

Thank you for the ruby examples I will in no doubt reuse some of them in some place, in the future

Please do not take offence and if you do, just remember fences are expensive to replace.
 
  


Reply

Tags
regex, sed


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] BASH (grep / sed / awk): find string within a (line) range dragonetti Linux - Newbie 2 11-24-2011 10:16 PM
sed/awk/grep for multiple line data Coiby Programming 4 09-20-2011 06:16 AM
sed/awk/grep for multiple line data hotrodmacman Programming 8 10-18-2007 11:06 AM
Need to strip words from front of line. sed/awk/grep? joadoor Linux - Software 6 08-28-2006 04:39 AM
Multi-line return from grep into an array? rose_bud4201 Programming 2 06-14-2005 04:11 PM


All times are GMT -5. The time now is 07:15 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration