LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Bash script to merge files together (given as a comma separated string) (http://www.linuxquestions.org/questions/linux-newbie-8/bash-script-to-merge-files-together-given-as-a-comma-separated-string-905144/)

DomeKor 09-26-2011 08:00 PM

Bash script to merge files together (given as a comma separated string)
 
Hello everyone, I'm sorry if this sounds trivial!

I have a $file variable that contains a comma separated list of files

e.g.
'../file1,/somedir/anotherfile.txt,./file'

and I'd like to merge them into a single file, say in ./tempmerged
Having the files appended sequentially is fine.
How can I do this?

Thanks a lot!

grail 09-26-2011 09:43 PM

I am not sure I understand ... you want to merge a variable into a file?

Or are you asking how to remove the commas so the variable contains a single path?

DomeKor 09-26-2011 10:18 PM

Sorry I'll reformulate the question.

Imagine you have this variable (It comes from the arguments with a getopts)

files = 'path/to/file1,path/to/file2,path/to/file3'



file1 contains the following lines:
line1
line2

file2 contains the following lines:
line3
line4

file3 contains the following lines:
line5
line6


I'd like to obtain a new file that contains the following lines
line1
line2
line3
line4
line5
line6

derstephen 09-27-2011 12:03 AM

Hi,

I hope I understood your question:

Code:

#!/bin/bash
arr=$(echo $1 | tr "," "\n")
cat $arr > tempmerged

You basically replace the commas with newlines, create an array from the resulting substrings and pass them all as filenames into cat.

By the way, this took me longer to figure out than I care to admit.

dudeman41465 09-27-2011 12:21 AM

Quote:

Originally Posted by derstephen (Post 4483163)
Hi,

I hope I understood your question:

Code:

#!/bin/bash
arr=$(echo $1 | tr "," "\n")
cat $arr > tempmerged

You basically replace the commas with newlines, create an array from the resulting substrings and pass them all as filenames into cat.

By the way, this took me longer to figure out than I care to admit.

I guess I'm not the only person that actually spends their own personal time learning how to do things just so they can help other people, now I don't feel like such a loser, lol.

derstephen 09-27-2011 12:30 AM

Quote:

Originally Posted by dudeman41465 (Post 4483169)
I guess I'm not the only person that actually spends their own personal time learning how to do things just so they can help other people, now I don't feel like such a loser, lol.

Yeah, I need every motivation I can think of to learn pretty much anything, so really I'm only helping people for my own personal gain.

grail 09-27-2011 12:31 AM

Well as it is already in a variable and assuming bash:
Code:

cat ${file//,/ } > tempmerged

DomeKor 09-27-2011 03:55 PM

Thanks a lot for the solutions.

I'm particularly fascinated by this

Quote:

cat ${file//,/ } > tempmerged
I really can't see the rationale behind.

Learning how to do things to help other people is awesome. It's still something new you learn.

derstephen 09-27-2011 06:53 PM

^^^
I would also like to know how that works. I've heard of bracket expansion of variables but I'm not sure about the stuff inside it. A sed expression perhaps?

David the H. 09-27-2011 06:59 PM

Grail's solution is very simply an application of parameter expansion. It replaces all of the commas in the string with spaces, allowing the shell to word-split the result into individual filenames.

Notice that this is one of the rare instances where you do not want to quote the variable, otherwise no word-splitting would occur. And of course if the filenames already have spaces, then we'll have to use a different technique (see below).

Incidentally, when working with lists of things, like filenames, it's usually recommended to use arrays instead. That way each element in the list can be accessed individually, as well as collectively.

http://mywiki.wooledge.org/BashGuide/Arrays

Ideally you should rewrite whatever it is that generates that variable so that it sets an array instead. If you can't do that, then you can split it into an array easily, using the same basic technique that grail provided.
Code:

$ string='file1.txt,file2.txt,file3.txt,file4.txt'

$ list=( ${string//,/ } )

$ echo "[${list[0]}] [${list[1]}] [${list[2]}] [${list[3]}]"
[file1.txt] [file2.txt] [file3.txt] [file4.txt]

$ cat "${list[@]}"

To split the string on commas if the filenames themselves have spaces in them, you'll have to change the IFS environment variable instead, which defines which characters the shell uses for word-splitting.
Code:

$ string='file 1.txt,file 2.txt,file 3.txt,file 4.txt'

$ oldIFS=$IFS        #store the original value (space,tab,newline by default)
$ IFS=','        #tell the shell to split only on commas

$ list=( $string )

$ IFS=$oldIFS        #return IFS to the original value

$ echo "[${list[0]}] [${list[1]}] [${list[2]}] [${list[3]}]"
[file 1.txt] [file 2.txt] [file 3.txt] [file 4.txt]

$ cat "${list[@]}"


When you get a chance, read through the entire BashGuide above. It will teach you all the basic concepts you need to know for good scripting.

grail 09-27-2011 11:29 PM

@David - thanks for helping with the explanation :)


All times are GMT -5. The time now is 10:17 AM.