LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 09-26-2011, 08:00 PM   #1
DomeKor
LQ Newbie
 
Registered: Sep 2011
Posts: 3

Rep: Reputation: Disabled
Bash script to merge files together (given as a comma separated string)


Hello everyone, I'm sorry if this sounds trivial!

I have a $file variable that contains a comma separated list of files

e.g.
'../file1,/somedir/anotherfile.txt,./file'

and I'd like to merge them into a single file, say in ./tempmerged
Having the files appended sequentially is fine.
How can I do this?

Thanks a lot!
 
Old 09-26-2011, 09:43 PM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,413

Rep: Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874
I am not sure I understand ... you want to merge a variable into a file?

Or are you asking how to remove the commas so the variable contains a single path?
 
Old 09-26-2011, 10:18 PM   #3
DomeKor
LQ Newbie
 
Registered: Sep 2011
Posts: 3

Original Poster
Rep: Reputation: Disabled
Sorry I'll reformulate the question.

Imagine you have this variable (It comes from the arguments with a getopts)

files = 'path/to/file1,path/to/file2,path/to/file3'



file1 contains the following lines:
line1
line2

file2 contains the following lines:
line3
line4

file3 contains the following lines:
line5
line6


I'd like to obtain a new file that contains the following lines
line1
line2
line3
line4
line5
line6
 
Old 09-27-2011, 12:03 AM   #4
derstephen
Member
 
Registered: Sep 2011
Location: Northern Colorado
Distribution: Slackware64
Posts: 79

Rep: Reputation: Disabled
Hi,

I hope I understood your question:

Code:
#!/bin/bash
arr=$(echo $1 | tr "," "\n")
cat $arr > tempmerged
You basically replace the commas with newlines, create an array from the resulting substrings and pass them all as filenames into cat.

By the way, this took me longer to figure out than I care to admit.

Last edited by derstephen; 09-27-2011 at 12:05 AM.
 
Old 09-27-2011, 12:21 AM   #5
dudeman41465
Member
 
Registered: Jun 2005
Location: Kentucky
Distribution: Ubuntu
Posts: 794

Rep: Reputation: 56
Quote:
Originally Posted by derstephen View Post
Hi,

I hope I understood your question:

Code:
#!/bin/bash
arr=$(echo $1 | tr "," "\n")
cat $arr > tempmerged
You basically replace the commas with newlines, create an array from the resulting substrings and pass them all as filenames into cat.

By the way, this took me longer to figure out than I care to admit.
I guess I'm not the only person that actually spends their own personal time learning how to do things just so they can help other people, now I don't feel like such a loser, lol.
 
Old 09-27-2011, 12:30 AM   #6
derstephen
Member
 
Registered: Sep 2011
Location: Northern Colorado
Distribution: Slackware64
Posts: 79

Rep: Reputation: Disabled
Quote:
Originally Posted by dudeman41465 View Post
I guess I'm not the only person that actually spends their own personal time learning how to do things just so they can help other people, now I don't feel like such a loser, lol.
Yeah, I need every motivation I can think of to learn pretty much anything, so really I'm only helping people for my own personal gain.
 
Old 09-27-2011, 12:31 AM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,413

Rep: Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874
Well as it is already in a variable and assuming bash:
Code:
cat ${file//,/ } > tempmerged
 
Old 09-27-2011, 03:55 PM   #8
DomeKor
LQ Newbie
 
Registered: Sep 2011
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks a lot for the solutions.

I'm particularly fascinated by this

Quote:
cat ${file//,/ } > tempmerged
I really can't see the rationale behind.

Learning how to do things to help other people is awesome. It's still something new you learn.
 
Old 09-27-2011, 06:53 PM   #9
derstephen
Member
 
Registered: Sep 2011
Location: Northern Colorado
Distribution: Slackware64
Posts: 79

Rep: Reputation: Disabled
^^^
I would also like to know how that works. I've heard of bracket expansion of variables but I'm not sure about the stuff inside it. A sed expression perhaps?
 
Old 09-27-2011, 06:59 PM   #10
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Grail's solution is very simply an application of parameter expansion. It replaces all of the commas in the string with spaces, allowing the shell to word-split the result into individual filenames.

Notice that this is one of the rare instances where you do not want to quote the variable, otherwise no word-splitting would occur. And of course if the filenames already have spaces, then we'll have to use a different technique (see below).

Incidentally, when working with lists of things, like filenames, it's usually recommended to use arrays instead. That way each element in the list can be accessed individually, as well as collectively.

http://mywiki.wooledge.org/BashGuide/Arrays

Ideally you should rewrite whatever it is that generates that variable so that it sets an array instead. If you can't do that, then you can split it into an array easily, using the same basic technique that grail provided.
Code:
$ string='file1.txt,file2.txt,file3.txt,file4.txt'

$ list=( ${string//,/ } )

$ echo "[${list[0]}] [${list[1]}] [${list[2]}] [${list[3]}]"
[file1.txt] [file2.txt] [file3.txt] [file4.txt]

$ cat "${list[@]}"
To split the string on commas if the filenames themselves have spaces in them, you'll have to change the IFS environment variable instead, which defines which characters the shell uses for word-splitting.
Code:
$ string='file 1.txt,file 2.txt,file 3.txt,file 4.txt'

$ oldIFS=$IFS	#store the original value (space,tab,newline by default)
$ IFS=','	#tell the shell to split only on commas

$ list=( $string )

$ IFS=$oldIFS	#return IFS to the original value

$ echo "[${list[0]}] [${list[1]}] [${list[2]}] [${list[3]}]"
[file 1.txt] [file 2.txt] [file 3.txt] [file 4.txt]

$ cat "${list[@]}"

When you get a chance, read through the entire BashGuide above. It will teach you all the basic concepts you need to know for good scripting.

Last edited by David the H.; 09-27-2011 at 07:25 PM. Reason: minor rewording and code modification
 
Old 09-27-2011, 11:29 PM   #11
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,413

Rep: Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874Reputation: 1874
@David - thanks for helping with the explanation
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reading comma separated variable into other variables in shell script suryaemlinux Programming 5 12-22-2010 07:38 PM
[SOLVED] Randomize the comma separated string in shell mariakumar Linux - General 13 10-11-2010 12:32 AM
How to delete Comma in a comma separated file with double quotes as quote character pklcnu Linux - Newbie 2 03-24-2009 05:50 PM
help with comma separated values and what should be a simple script. zaber Programming 10 03-06-2008 12:58 PM
bash syntax: looping through a comma-separated list David the H. Linux - General 10 09-06-2007 10:23 AM


All times are GMT -5. The time now is 01:49 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration