[SOLVED] parsing text string into array in bash script

rmori · 05-21-2014, 12:27 PM

Greetings all:
I haven't much experience with bash arrays, but this is my problem.

I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },

b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance

TB0ne · 05-21-2014, 01:16 PM

Quote:

Originally Posted by rmori

Greetings all:
I haven't much experience with bash arrays, but this is my problem. I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },
b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance

We'll be glad to help you. But based on what you've posted, there's little we can help with right now. You don't give us any samples of the data, what kind of array you're talking about, what you're going to do with it (output to file? another command?), version/distro of Linux, or show us what you've written already.

There is a tutorial for bash arrays:
http://www.thegeekstuff.com/2010/06/...rray-tutorial/

..which may be of help. Post what you've written/tried of your own, and show us the sample data, and we can try to help.

joe_2000 · 05-21-2014, 03:44 PM

Quote:

Originally Posted by rmori

Greetings all:
I haven't much experience with bash arrays, but this is my problem.

I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },

b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance

You could eg set the IFS variable to }, (which makes this character the line separator) and then iterate over the fields between. E.g.:

Code:

user@host:~$ STRING="one}two}three}"
user@host:~$ IFS=}
user@host:~$ for word in $STRING;do echo $word;done
one
two
three

If you really need an array there is e.g.

Code:

user@host:~$ echo "one}two}three}" > tmpfile
user@host:~$ IFS='}' read -d ' ' -r -a lines < tmpfile
user@host:~$ echo "${lines[@]}"
one two three

As TBOne said more info on what you actually want to achieve would help...

Firerat · 05-21-2014, 05:02 PM

Or

Untested

Code:

unset Array
OrigIFS="$IFS"
IFS="}"
while read cell;do
   Array+=($cell)
done < <$(curl http://some...url.....)
IFS="$OrigIFS"
echo ${#Array[@]}

Provided sample, I may test ..

grail · 05-22-2014, 05:45 AM

Without an example the question is rather moot

chrism01 · 05-22-2014, 06:23 AM

As above, please show us some data and how you want it split.
This is good for arrays http://wiki.bash-hackers.org/syntax/arrays

rmori · 05-22-2014, 09:10 AM

Sorry for the lack of information all. I'm running on CentOS 6.4 64-bit. The source text string looks like

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399510800000,"abcdefg" :0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399514400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwx yz":0},{"date":1399518000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399521600000,"a bcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399525200000,"abcdefg":0,"hijklmn":0,"opqrstu" :0,"vwxyz":0},{"date":1399528800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":139953240 0000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399536000000,"abcdefg":0,"hijklmn":0,"o pqrstu":0,"vwxyz":0},{"date":1399539600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":13 99543200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399546800000,"abcdefg":0,"hijklm n":0,"opqrstu":0,"vwxyz":0},{"date":1399550400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"d ate":1399554000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399557600000,"abcdefg":0, "hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399561200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz" :0},{"date":1399564800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399568400000,"abcd efg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399572000000,"abcdefg":0,"hijklmn":0,"opqrstu":0, "vwxyz":0},{"date":1399575600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":139957920000 0,"abcdefg":1220,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399582800000,"abcdefg":0,"hijklmn":0,"o pqrstu":0,"vwxyz":0},{"date":1399586400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":13 99590000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0}]

but it's all one continuous string containing no newline characters. I need to put it into an array with each of the array elements looking like:

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399510800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399514400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399518000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399521600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399525200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399528800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399532400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399536000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399539600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399543200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399546800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399550400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399554000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399557600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399561200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399564800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399568400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399572000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399575600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399579200000,"abcdefg":1220,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399582800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399586400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399590000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0}]

The leading and trailing braces ([]) can be removed.

The reason I wanted to place it into an array is because I have to perform some post-processing on various fields of each entry. Let me know if additional information is necessary.

Thank you

TB0ne · 05-22-2014, 09:37 AM

Quote:

Originally Posted by rmori

Sorry for the lack of information all. I'm running on CentOS 6.4 64-bit. The source text string looks like
but it's all one continuous string containing no newline characters. I need to put it into an array with each of the array elements looking like:

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399510800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},

The leading and trailing braces ([]) can be removed.

Makes sense now. You can do this either by using eval, like this:

Code:

ResultVariable=`eval cat test.txt | sed 's/\},/\n/g' | sed 's/\(\[\|\]\)//g'`

...which will shovel the result of that command-string (which splits the line as you want, and removes the brackets), into the ResultVariable variable. Read it from there. Or you can shovel the input text file through to a temp file, and read IT line-by-line, processing it as needed...should be easy to split the variables, since the data appears to be delimited by commas, and begins with a colon. Note that the code above is untested, and bash isn't my strong suit...your mileage may vary, and it may need some tweaking. The sed statements work, though.

rmori · 05-22-2014, 09:49 AM

The '\n' character is not valid in my sed substitution string. Does it actually work for you?

TB0ne · 05-22-2014, 10:02 AM

Quote:

Originally Posted by rmori

The '\n' character is not valid in my sed substitution string. Does it actually work for you?

Yes, it does. The "\n" is a newline. You might have to escape it with another backslash (as said, untested in bash script), but it does work from the command line.

rmori · 05-22-2014, 10:20 AM

Correct - it works from command line, but not from within a bash script. It actually inserts a space. When I escape it with another backslash, it inserts the '\n' string.

In a script, if I use

newvar=`cat test.txt |sed 's/\},/&\n/g'`

it will insert a newline as I wanted.

Thank you for your help.

TB0ne · 05-22-2014, 10:35 AM

Quote:

Originally Posted by rmori

Correct - it works from command line, but not from within a bash script. It actually inserts a space. When I escape it with another backslash, it inserts the '\n' string.

Try either:

Code:

\'$'\n

or

Code:

\o12

...as the replacement strings.

grail · 05-22-2014, 12:34 PM

Using just bash you could:

Code:

read line <file

IFS='}' read -a arr <<<"${line//[][{]/}"

echo ${#arr[@]}
echo ${arr[0]#,}
echo ${arr[-1]#,}