LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-16-2014, 10:00 AM   #1
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Rep: Reputation: Disabled
Need a BASH for loop for processing many files and naming the output files


From another thread, I learned that I can extract the last line of all of my huge .csv files using something like this

tail -n 1 *U.csv

I can send it to a file using Linux redirection

tail -n 1 *U.csv >LC.csv

But I have 300 of these files, and I need 300 output files, each with a different name from the input file. If the input file is named, say, N=1024_Beta=03.00.U.csv, then I would like the output file to be named N=1024_Beta=03.00.LC.csv. That is, if I split the file using periods as a delimiter, the second to last section, which is currently a U, should be replaced with an LC (for "last condition"). All of the files I need to process end with U.csv (exactly like that, capitalization and everything).

I've got a start, but its merely putting the LC at the end of each (and I'm not sure it'll work the way I think it will)
Code:
for fn in *.U.csv;
do
  tail -n 1 $fn >$fn.LC;
done
Thanks for help.
 
Old 04-16-2014, 10:19 AM   #2
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,027

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
Quote:
Originally Posted by Jeff9 View Post
From another thread, I learned that I can extract the last line of all of my huge .csv files using something like this

tail -n 1 *U.csv

I can send it to a file using Linux redirection

tail -n 1 *U.csv >LC.csv

But I have 300 of these files, and I need 300 output files, each with a different name from the input file. If the input file is named, say, N=1024_Beta=03.00.U.csv, then I would like the output file to be named N=1024_Beta=03.00.LC.csv. That is, if I split the file using periods as a delimiter, the second to last section, which is currently a U, should be replaced with an LC (for "last condition"). All of the files I need to process end with U.csv (exactly like that, capitalization and everything).

I've got a start, but its merely putting the LC at the end of each (and I'm not sure it'll work the way I think it will)
Code:
for fn in *.U.csv;
do
  tail -n 1 $fn >$fn.LC;
done
Thanks for help.
i think this is safe enough to test without harming the input files (maybe make backups just in case so you can experiment a little without fear of breaking anything).

this is a good first attempt. here are a couple of options:
there is a sed command which can do pattern substitutions:
Code:
echo N=1024_Beta=03.00.U.csv | sed s/.U.csv/.LC.csv/
or there is an awk command that can replace feilds from row based input:
Code:
echo N=1024_Beta=03.00.U.csv | awk -F . '{$OFS="."} {$(NF-1) = "lc"} {print $0}'
happy hacking.
 
1 members found this post helpful.
Old 04-16-2014, 10:21 AM   #3
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 8,121

Rep: Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270
you may try to use:
Code:
fn=asdf.U.csv
f=${fn%.U.csv}.LC.csv
echo $f
 
2 members found this post helpful.
Old 04-16-2014, 10:30 AM   #4
allend
Senior Member
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 4,431

Rep: Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350Reputation: 1350
What you want to do can be done using a bash parameter expansion. Rather than actually write a script for you, see whether this code gives you a tip.
Code:
#!/bin/bash

fn="N=1024_Beta=03.00.U.csv"

prefix=${fn%U.csv}
suffix="LC.csv"
nfn=$prefix$suffix

echo "$prefix"
echo "$suffix"
echo "$nfn"
 
1 members found this post helpful.
Old 04-16-2014, 11:22 AM   #5
Philip Lacroix
Member
 
Registered: Jun 2012
Distribution: Slackware{,-current}, OpenBSD
Posts: 361

Rep: Reputation: 377Reputation: 377Reputation: 377Reputation: 377
Another way, although less compact than parameter expansion:

Code:
for fn_U in *.U.csv ;
do
  fn_LC="$(basename "${fn_U}" .U.csv).LC.csv"
  tail -n 1 "${fn_U}" > "${fn_LC}"
done
 
1 members found this post helpful.
Old 04-16-2014, 11:44 AM   #6
Shavais
LQ Newbie
 
Registered: Jan 2014
Posts: 2

Rep: Reputation: Disabled
Here is a bash script I wrote to try to do what you're asking for. It's very bash-specific, a bit over generalized, and could be reduced a bit, but hopefully it is illustrative.

Code:
#!/bin/bash
for fn in *U.csv; do
    IFS='.' pathparts=($fn)                     # split the path into an array using dot as a delimiter
    numparts=${#pathparts[@]}                   # get the length of the array
    extension=${pathparts[${numparts}-1]}       # get the last element of the array (the file extension; unnecessary in this case)
    unset pathparts[${numparts}-1]              # eliminate the last element from the array (chop off the extension)
    unset pathparts[${numparts}-2]              # eliminate the 2nd to last element (chop off the 'U')
    IFS='.' basename="${pathparts[*]}"          # join the remaining array back to a string to get the first part of the new name
    newname="${basename}.LC.${extension}"       # combine the first part with the last part to get the new name
    tail -n 1 $fn > $newname
done

Last edited by Shavais; 04-16-2014 at 11:46 AM.
 
1 members found this post helpful.
Old 04-16-2014, 12:33 PM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,252

Rep: Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685
geez ... couldn't we just use substitution:
Code:
tail -n 1 "$fn" > "${fn/U/LC}"
 
3 members found this post helpful.
Old 04-16-2014, 01:22 PM   #8
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
For my own future reference
Code:
for fn in *.U.csv;
do
  tail -n 1 "$fn" > "${fn/U/LC}";
done
was easiest to implement (allend's was easiest to understand).

TY for all good responses.
 
Old 04-16-2014, 02:19 PM   #9
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,252

Rep: Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685
Please remember to mark as SOLVED once you have a solution
 
Old 04-17-2014, 11:56 AM   #10
Shavais
LQ Newbie
 
Registered: Jan 2014
Posts: 2

Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
geez ... couldn't we just use substitution:
Code:
tail -n 1 "$fn" > "${fn/U/LC}"
Substitution.. basename.. I should post really bad ideas more often!

Whenever I think my shell knowledge has reached a level of maturity, I discover I'm still a n00b. In my defense, I wasn't really trying to minimize or optimize, I was just exploring how to do some common programming things, like split a string into an array, reference elements of an array, eliminate elements of an array, and rejoin an array back into a string. But as you point out, there are often much shorter paths to victory.
 
Old 05-01-2014, 09:06 PM   #11
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
geez ... couldn't we just use substitution:
Code:
tail -n 1 "$fn" > "${fn/U/LC}"
How would you modify this be to match five straight digits and replace them with five zeros?
 
Old 05-01-2014, 11:29 PM   #12
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,252

Rep: Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685
Unfortunately in bash it is a little more cumbersome for such a solution:
Code:
${fn/[0-9][0-9][0-9][0-9][0-9]/00000}
 
1 members found this post helpful.
Old 05-02-2014, 01:27 PM   #13
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
Unfortunately in bash it is a little more cumbersome for such a solution:
Code:
${fn/[0-9][0-9][0-9][0-9][0-9]/00000}
TY. Worked as expected. Is this "parameter expansion"?
 
Old 05-02-2014, 01:36 PM   #14
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,252

Rep: Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685
parameter substitution
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Re-naming and using user input files in bash script sready Programming 3 02-06-2012 12:37 AM
Bash - cron processing of files in directory but leave already processed untouched ygzo Programming 4 01-24-2012 08:00 AM
[SOLVED] Recursively naming files in a directory after processing them in a "for" loop geodave0110 Programming 6 11-14-2011 01:06 PM
bash programming - rename files in a loop turki_00 Programming 10 03-31-2011 10:41 PM
Trouble with making a bash script to read in different files and rename output files. rystke Linux - Software 1 05-07-2009 09:00 AM


All times are GMT -5. The time now is 06:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration