LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-21-2014, 01:27 PM   #1
rmori
LQ Newbie
 
Registered: Mar 2008
Location: Seguin, Texas
Distribution: Red Hat 9
Posts: 10

Rep: Reputation: 0
Question parsing text string into array in bash script


Greetings all:
I haven't much experience with bash arrays, but this is my problem.

I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },

b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance
 
Old 05-21-2014, 02:16 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,316

Rep: Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878
Quote:
Originally Posted by rmori View Post
Greetings all:
I haven't much experience with bash arrays, but this is my problem. I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },
b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance
We'll be glad to help you. But based on what you've posted, there's little we can help with right now. You don't give us any samples of the data, what kind of array you're talking about, what you're going to do with it (output to file? another command?), version/distro of Linux, or show us what you've written already.

There is a tutorial for bash arrays:
http://www.thegeekstuff.com/2010/06/...rray-tutorial/

..which may be of help. Post what you've written/tried of your own, and show us the sample data, and we can try to help.
 
1 members found this post helpful.
Old 05-21-2014, 04:44 PM   #3
joe_2000
Member
 
Registered: Jul 2012
Location: Aachen, Germany
Distribution: Void, Debian
Posts: 812

Rep: Reputation: 216Reputation: 216Reputation: 216
Quote:
Originally Posted by rmori View Post
Greetings all:
I haven't much experience with bash arrays, but this is my problem.

I am retrieving a long text string from a curl request. I need to:

a. break this data up into lines (I need to insert a newline character following this sequence },

b. once I have the string broken up into lines, I need to place the lines into an array (1 line
per cell)

I would greatly appreciate any assistance the community can offer. Thanks in advance
You could eg set the IFS variable to }, (which makes this character the line separator) and then iterate over the fields between. E.g.:
Code:
user@host:~$ STRING="one}two}three}"
user@host:~$ IFS=}
user@host:~$ for word in $STRING;do echo $word;done
one
two
three
If you really need an array there is e.g.
Code:
user@host:~$ echo "one}two}three}" > tmpfile
user@host:~$ IFS='}' read -d ' ' -r -a lines < tmpfile
user@host:~$ echo "${lines[@]}"
one two three
As TBOne said more info on what you actually want to achieve would help...
 
1 members found this post helpful.
Old 05-21-2014, 06:02 PM   #4
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444
Or

Untested

Code:
unset Array
OrigIFS="$IFS"
IFS="}"
while read cell;do
   Array+=($cell)
done < <$(curl http://some...url.....)
IFS="$OrigIFS"
echo ${#Array[@]}
Provided sample, I may test ..
 
1 members found this post helpful.
Old 05-22-2014, 06:45 AM   #5
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,424

Rep: Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823
Without an example the question is rather moot
 
Old 05-22-2014, 07:23 AM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,258

Rep: Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328Reputation: 2328
As above, please show us some data and how you want it split.
This is good for arrays http://wiki.bash-hackers.org/syntax/arrays
 
Old 05-22-2014, 10:10 AM   #7
rmori
LQ Newbie
 
Registered: Mar 2008
Location: Seguin, Texas
Distribution: Red Hat 9
Posts: 10

Original Poster
Rep: Reputation: 0
Parsing text string into array in bash scripot

Sorry for the lack of information all. I'm running on CentOS 6.4 64-bit. The source text string looks like

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399510800000,"abcdefg" :0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399514400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwx yz":0},{"date":1399518000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399521600000,"a bcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399525200000,"abcdefg":0,"hijklmn":0,"opqrstu" :0,"vwxyz":0},{"date":1399528800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":139953240 0000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399536000000,"abcdefg":0,"hijklmn":0,"o pqrstu":0,"vwxyz":0},{"date":1399539600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":13 99543200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399546800000,"abcdefg":0,"hijklm n":0,"opqrstu":0,"vwxyz":0},{"date":1399550400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"d ate":1399554000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399557600000,"abcdefg":0, "hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399561200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz" :0},{"date":1399564800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399568400000,"abcd efg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399572000000,"abcdefg":0,"hijklmn":0,"opqrstu":0, "vwxyz":0},{"date":1399575600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":139957920000 0,"abcdefg":1220,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":1399582800000,"abcdefg":0,"hijklmn":0,"o pqrstu":0,"vwxyz":0},{"date":1399586400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},{"date":13 99590000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0}]

but it's all one continuous string containing no newline characters. I need to put it into an array with each of the array elements looking like:

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399510800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399514400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399518000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399521600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399525200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399528800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399532400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399536000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399539600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399543200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399546800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399550400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399554000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399557600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399561200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399564800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399568400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399572000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399575600000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399579200000,"abcdefg":1220,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399582800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399586400000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399590000000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0}]

The leading and trailing braces ([]) can be removed.

The reason I wanted to place it into an array is because I have to perform some post-processing on various fields of each entry. Let me know if additional information is necessary.

Thank you
 
Old 05-22-2014, 10:37 AM   #8
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,316

Rep: Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878
Quote:
Originally Posted by rmori View Post
Sorry for the lack of information all. I'm running on CentOS 6.4 64-bit. The source text string looks like
but it's all one continuous string containing no newline characters. I need to put it into an array with each of the array elements looking like:

[{"date":1399507200000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},
{"date":1399510800000,"abcdefg":0,"hijklmn":0,"opqrstu":0,"vwxyz":0},

The leading and trailing braces ([]) can be removed.
Makes sense now. You can do this either by using eval, like this:
Code:
ResultVariable=`eval cat test.txt | sed 's/\},/\n/g' | sed 's/\(\[\|\]\)//g'`
...which will shovel the result of that command-string (which splits the line as you want, and removes the brackets), into the ResultVariable variable. Read it from there. Or you can shovel the input text file through to a temp file, and read IT line-by-line, processing it as needed...should be easy to split the variables, since the data appears to be delimited by commas, and begins with a colon. Note that the code above is untested, and bash isn't my strong suit...your mileage may vary, and it may need some tweaking. The sed statements work, though.
 
1 members found this post helpful.
Old 05-22-2014, 10:49 AM   #9
rmori
LQ Newbie
 
Registered: Mar 2008
Location: Seguin, Texas
Distribution: Red Hat 9
Posts: 10

Original Poster
Rep: Reputation: 0
The '\n' character is not valid in my sed substitution string. Does it actually work for you?
 
Old 05-22-2014, 11:02 AM   #10
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,316

Rep: Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878
Quote:
Originally Posted by rmori View Post
The '\n' character is not valid in my sed substitution string. Does it actually work for you?
Yes, it does. The "\n" is a newline. You might have to escape it with another backslash (as said, untested in bash script), but it does work from the command line.
 
Old 05-22-2014, 11:20 AM   #11
rmori
LQ Newbie
 
Registered: Mar 2008
Location: Seguin, Texas
Distribution: Red Hat 9
Posts: 10

Original Poster
Rep: Reputation: 0
Correct - it works from command line, but not from within a bash script. It actually inserts a space. When I escape it with another backslash, it inserts the '\n' string.

In a script, if I use

newvar=`cat test.txt |sed 's/\},/&\n/g'`

it will insert a newline as I wanted.

Thank you for your help.

Last edited by rmori; 05-22-2014 at 11:34 AM.
 
Old 05-22-2014, 11:35 AM   #12
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,316

Rep: Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878Reputation: 3878
Quote:
Originally Posted by rmori View Post
Correct - it works from command line, but not from within a bash script. It actually inserts a space. When I escape it with another backslash, it inserts the '\n' string.
Try either:
Code:
\'$'\n
or
Code:
\o12
...as the replacement strings.
 
Old 05-22-2014, 01:34 PM   #13
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,424

Rep: Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823
Using just bash you could:
Code:
read line <file

IFS='}' read -a arr <<<"${line//[][{]/}"

echo ${#arr[@]}
echo ${arr[0]#,}
echo ${arr[-1]#,}
 
1 members found this post helpful.
  


Reply

Tags
array, bash scripting, curl


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script- text file manip- delete everything before/after string justin99 Programming 9 11-20-2014 04:16 AM
[SOLVED] Bash script time comparison with one being a text string NetEng1 Linux - Software 4 09-05-2013 11:26 PM
Bash script to remove string from all text files, recursively iacchi Programming 9 04-27-2012 06:22 AM
How to remove string in the text file ? Bash script dlugasx Linux - Server 9 06-05-2009 12:40 PM
Bash script text line into an array toolshed Programming 1 06-13-2005 06:49 PM


All times are GMT -5. The time now is 02:58 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration