LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-11-2012, 02:52 AM   #1
khandu
Member
 
Registered: Sep 2003
Posts: 79

Rep: Reputation: 0
Question how to automate file processing in bash?


Ok

The heading might not have made sense..

Here is what I need to do

There is a perl script which takes input of a .txt file and outputs the result to STDOUT!!!

So in essence we run it as on bash prompt

Code:
./script.pl INPUTFILE1.txt > INPUTFILE1_RESULT.txt
./script.pl INPUTFILE2.txt > INPUTFILE2_RESULT.txt
...
...
./script.pl INPUTFILEn.txt > INPUTFILEn_RESULT.txt
Now there are over 500 txt files I need to run this script for.. each should generate a result.txt file (it can contain the orignal filename for easy identification)

How do I do this??


Also to take it to next level..

lets say there are multiple folders which have the INPUTFILE.txt.. it possible to do this same thing as above recursively and create a new output folder (if does not exist) as well for each folder processed with _RESULT in end name of folder.. ofcourse don't want to complicate things .. I can manually go into them and point the result to a manually created folder..

Again solution to 1st problem itself will be great

if it makes any difference, the code will run in RHEL..

I cannot modify the .pl script.. so just want to modify the execution style on bash

Cheers
 
Old 06-11-2012, 03:14 AM   #2
business_kid
Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 6,297

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
NO bash expert here, so fix the syntax:

for i in /wherever do;
./script.pl $i > $i.results; # probably need some brackets around '$i'
next i;
 
1 members found this post helpful.
Old 06-11-2012, 03:14 AM   #3
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,641

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Code:
for f in `cat inputfilelist`
do
script $f > $f.out
done
I would try something like this, but it depends on the inputfilelist, the filenames may not contain whitespace.
 
1 members found this post helpful.
Old 06-11-2012, 05:15 AM   #4
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,485

Rep: Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890Reputation: 1890
Would not the best option be to simply extend the perl script to cope with the new requirement?
 
Old 06-11-2012, 05:40 AM   #5
414N
Member
 
Registered: Sep 2011
Location: Italy
Distribution: Slackware
Posts: 608

Rep: Reputation: 179Reputation: 179
Code:
#!/bin/sh
# Executes script.pl over all the files specified on the CLI, preserving
# the file extensions of the original files

for FILE in "$@"
do
  FILEEXT=`echo "$FILE" | rev | cut -d. -f1 | rev`
  script.pl "$FILE" > "`basename "$FILE" $FILEEXT`-RESULT.$FILEEXT
done
 
1 members found this post helpful.
Old 06-11-2012, 05:49 AM   #6
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,641

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
oh, no! see bash variable substitution:
Code:
fileext=${FILE##*.}
basename=${FILE%.*}
 
Old 06-11-2012, 05:51 AM   #7
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Using Bash:
Code:
find DIR(s)... -name '*.txt' -print0 | while read -rd "" source ; do
   target="${source%.*}_RESULT.txt"
   ./script.pl "$source" > "$target"
done
Above, find will emit the files to work on using ASCII NULs as separators, handling even the weirdest file names correctly. This one looks for all files ending with .txt in DIR(s)... and their subdirectories.

The read -rd "" uses the Bash read built-in to read them into variable source one by one in the while .. ; do ... done loop.

target gets set to the value of source but with everything after the final . replaced with _RESULT.txt.

If you want to skip any existing result files, use
Code:
    [[ -e "$target" ]] || ./script.pl "$source" > "$target"
instead. The test is for existence, but || is else/otherwise, so the script is only run if $target does not exist yet. There is a short interval in between, when someone else could create $target. If that is a problem, use set -C to tell Bash to not redirect into existing files.
 
1 members found this post helpful.
Old 06-11-2012, 09:29 AM   #8
414N
Member
 
Registered: Sep 2011
Location: Italy
Distribution: Slackware
Posts: 608

Rep: Reputation: 179Reputation: 179
Quote:
Originally Posted by pan64 View Post
oh, no! see bash variable substitution:
Code:
fileext=${FILE##*.}
basename=${FILE%.*}
Thanks for the enhancement. I still need to grasp bash variable substitution
 
Old 06-11-2012, 10:40 AM   #9
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by grail View Post
Would not the best option be to simply extend the perl script to cope with the new requirement?
Depends on how you define 'best', I suppose. My logic is that when a tool that does something very well already exits, I want to use that. In this case, that tool is find. Using the existing perl script also adheres to this principle. It is known to do one thing well, so use it that way, and build around it.

Nominal Animal demonstrates the power of find very nicely, and also demonstrates and explains a well structured way to approach the problem: iteration over a file set, especially with recursion, suggests using find. The output of find is a list, which feeds the while loop, so there is the iteration part of the problem solved. Once that part of the problem is addressed, he assembles a couple of key variables that are the arguments to the perl script. Finally, that last thing in each iteration is to invoke the perl script with arguments that are variables with nice human-readable names.

There is more to learn from Nominal's code than just how to solve the problem. It is a good demonstration of how structure the solution.
--- rod.
 
  


Reply

Tags
automate, bash, file, perl, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Using BASH to automate data processing and table generation. marcusrex Programming 19 04-13-2012 09:20 AM
[SOLVED] reading unix file permissions into a bash array for processing dazdaz Programming 11 12-06-2010 03:21 PM
LXer: Speaking Unix, Part 6: Automate, automate, automate! LXer Syndicated Linux News 0 01-04-2007 09:54 AM
Automate Samba logon with a bash script? achtung_linux Linux - Networking 6 10-22-2006 02:36 AM
BASH: how to automate deletion of oldest folder big_mike_jones Programming 18 11-22-2005 02:59 PM


All times are GMT -5. The time now is 01:01 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration