LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 02-19-2010, 03:52 AM   #1
SilversleevesX
Member
 
Registered: May 2009
Posts: 181
Blog Entries: 9

Rep: Reputation: 15
BASH script with sed file "inside"?


Continuing the script file project discussed here, I would like to know if it were possible to 'embed' the "sed" file -- or at least the same data -- kindly created by jschiwal, into the script itself with an eye towards a careful (or quick-&-dirty, using shc-3.8.6) conversion to an executable for the command line.

I await your answers and advice.

BZT
PS: I have the last e-book/PDF edition made of the ABS Guide, and hadn't someone (Andreas Huggel, author of exiv2) just the other day suggest that I pick up the O'Reilly bash Cookbook instead, I'd be reading it cover to exit command. That opinions about both are plentiful I have no doubt. This postscript presumes to head off those inclined to offer such advice.

Last edited by SilversleevesX; 02-19-2010 at 03:53 AM.
 
Old 02-19-2010, 05:18 AM   #2
deadeyes
Member
 
Registered: Aug 2006
Posts: 605

Rep: Reputation: 79
I think HERE documents is what you need:
http://tldp.org/LDP/abs/html/here-docs.html
 
Old 02-19-2010, 01:00 PM   #3
SilversleevesX
Member
 
Registered: May 2009
Posts: 181
Blog Entries: 9

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by deadeyes View Post
I think HERE documents is what you need:
http://tldp.org/LDP/abs/html/here-docs.html
Thanks deadeyes.

I'm trying to get my mind around HERE documents, particularly as they would apply to the functions in the script. Maybe I'm mis-perceiving something about the original script? Another pair of eyes and a more experienced mind and hand are asked for now.

The script as written towards the end of that other thread:
Code:
#!/bin/bash

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f rename.sed descriptions >add_extensions.sh
Here's what I think is going on:
Line 1: The find command is set to look for files in the same folder the user is "in" on the command line, such files to be regular and have no period separating one part of the filename from the other. The result of this search is redirected through xargs to a file named descriptions.
Line 2: The sed command is reading in the file rename.sed and the file descriptions and, following the syntax of the first is applying it to the contents of the second and then creating a shell script called add_extensions.sh which the user can execute later on the files in that directory.

Got it right so far, have I?

Okay. Here's a modification of that script I attempted after a few readings, and re-readings, of the Here documents page on tldp.org (and a few others, including a jump from a function-by-function breakdown of sed found on grymoire.com)
Code:
find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f <<rename descriptions >add_extensions.sh

<<rename
/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p
/ISO Media, MPEG v4/s/\(^[^:]*\):.*$/mv "\1" "\1.mp4"/p
/MPEG ADTS, layer III/s/\(^[^:]*\):.*$/mv "\1" "\1.mp3"/p
rename
The modifcation returns an error such as
Code:
sed: file descriptions line 1: unknown command: `.'
...as well as, predictably, a zero-bytes add_extensions.sh.

As I say, I'm not "all here" with Here yet.

BZT

Last edited by SilversleevesX; 02-19-2010 at 01:03 PM.
 
Old 02-20-2010, 04:36 AM   #4
deadeyes
Member
 
Registered: Aug 2006
Posts: 605

Rep: Reputation: 79
Quote:
Originally Posted by SilversleevesX View Post
Thanks deadeyes.

I'm trying to get my mind around HERE documents, particularly as they would apply to the functions in the script. Maybe I'm mis-perceiving something about the original script? Another pair of eyes and a more experienced mind and hand are asked for now.

The script as written towards the end of that other thread:
Code:
#!/bin/bash

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f rename.sed descriptions >add_extensions.sh
Here's what I think is going on:
Line 1: The find command is set to look for files in the same folder the user is "in" on the command line, such files to be regular and have no period separating one part of the filename from the other. The result of this search is redirected through xargs to a file named descriptions.
Line 2: The sed command is reading in the file rename.sed and the file descriptions and, following the syntax of the first is applying it to the contents of the second and then creating a shell script called add_extensions.sh which the user can execute later on the files in that directory.

Got it right so far, have I?
Almost there
The find command lists all files with a dot (.) in them.
I cannot find -not in the man page so dont know if it is multiple options or just the word not.
-print0 prints all find files on the same line if I recall correctly. xargs will split up this list of files in sublists. This is because command can only have a limited amount of arguments. xargs will then execute the command "file" for each sublist. THe result is written to subscription.
The second line will read the file "descriptions" and will apply the rules defined in "rename.sed".
-f gives a file.
With HERE documents you can send data from the file as it was from standard input, for example:
Code:
cat afile | wc -l
Code:
wc -l <<afile
> test
> test2
> test3
> afile
3
Do note that the bigger then chars are printed by bash.

And I am not sure if that is possible for the rules of the sed command. You can use it for the file that should be processed.
So you might use:
sed -e 'firstrule' -e 'secondrule' ...
This might soon look ugly.

What I wonder is why you want this included? Maybe it is easier just to supply the rules file with it.

Quote:
Okay. Here's a modification of that script I attempted after a few readings, and re-readings, of the Here documents page on tldp.org (and a few others, including a jump from a function-by-function breakdown of sed found on grymoire.com)
Code:
find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f <<rename descriptions >add_extensions.sh

<<rename
/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p
/ISO Media, MPEG v4/s/\(^[^:]*\):.*$/mv "\1" "\1.mp4"/p
/MPEG ADTS, layer III/s/\(^[^:]*\):.*$/mv "\1" "\1.mp3"/p
rename
The modifcation returns an error such as
Code:
sed: file descriptions line 1: unknown command: `.'
...as well as, predictably, a zero-bytes add_extensions.sh.

As I say, I'm not "all here" with Here yet.

BZT
I think you need something like:
Code:
sed -e '/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
So this means if JPEG image data was printed by the file command it will substitute the filename with "mv filename.jpg"

So look at the explanation above
have fun!
 
Old 02-22-2010, 06:16 AM   #5
SilversleevesX
Member
 
Registered: May 2009
Posts: 181
Blog Entries: 9

Original Poster
Rep: Reputation: 15
Thanks for the continued help and advice. deadeyes.

Quote:
Originally Posted by deadeyes View Post
With HERE documents you can send data from the file as it was from standard input, for example:
Code:
cat afile | wc -l
Code:
wc -l <<afile
> test
> test2
> test3
> afile
3
Do note that the bigger then chars are printed by bash.
Why are we counting the number of rules? I made up a short script, borrowing "rules" from the bigger sed file in that other thread (addext.sed by name), and whether a slight tweak of the sed command you suggested here...

Code:
sed -e '/image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
... is above or below the rules I inserted...
Code:
>  /image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p;n}
>  /image\/png/{ s/\(^[^:]*\):.*$/mv "\1" "\1.png"/p;n}
>  /image\/gif/{ s/\(^[^:]*\):.*$/mv "\1" "\1.gif"/p;n}
>  /image\/tiff/{ s/\(^[^:]*\):.*$/mv "\1" "\1.tif"/p;n}
>  afile
...the number returned from wc -l is, predictably, always higher than the last time I run it. At one stage of making this scratch script, it had no sed line at all, and I got the usual returns for a file -i command (the way the mime types I'm using in the rules are written seems to require it), and with the sed line included, below the rules, the only return is the number from wc -l, with no effect on the files themselves. Nor any generation of an add_extensions.sh file. Seems I'm missing some code here.

Quote:
Originally Posted by "deadeyes" View Post
What I wonder is why you want this included? Maybe it is easier just to supply the rules file with it.
Perhaps you're right. A jumble-solving Python script I have needs the dictionary.txt file in plain text. I found mine by copping the american-english.txt file from somewhere in the deep recesses of my Ubuntu laptop's GNOME install. And even the old fortune-cookie executables need to read properly-formatted quote files. I just wasn't sure how it might compile without the rules right there inside it.

Guess I'm off to another subforum (Programming) with this? Or would you suggest I work on it some more as a shell script?

BZT

Last edited by SilversleevesX; 02-22-2010 at 06:20 AM.
 
Old 02-22-2010, 03:11 PM   #6
deadeyes
Member
 
Registered: Aug 2006
Posts: 605

Rep: Reputation: 79
Quote:
Originally Posted by SilversleevesX View Post
Thanks for the continued help and advice. deadeyes.
You're welcome

Quote:
Why are we counting the number of rules?
I just wanted to demonstrate the use of HERE documents
And showing that in your case it does not work .

Quote:
I made up a short script, borrowing "rules" from the bigger sed file in that other thread (addext.sed by name), and whether a slight tweak of the sed command you suggested here...

Code:
sed -e '/image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
... is above or below the rules I inserted...
You put before an example script. This should replace the existing sed line and rules file. This is a way of including the rules into the script without using HERE documents.
Do note that this is less readable.

Quote:
Code:
>  /image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p;n}
>  /image\/png/{ s/\(^[^:]*\):.*$/mv "\1" "\1.png"/p;n}
>  /image\/gif/{ s/\(^[^:]*\):.*$/mv "\1" "\1.gif"/p;n}
>  /image\/tiff/{ s/\(^[^:]*\):.*$/mv "\1" "\1.tif"/p;n}
>  afile
...the number returned from wc -l is, predictably, always higher than the last time I run it. At one stage of making this scratch script, it had no sed line at all, and I got the usual returns for a file -i command (the way the mime types I'm using in the rules are written seems to require it), and with the sed line included, below the rules, the only return is the number from wc -l, with no effect on the files themselves. Nor any generation of an add_extensions.sh file. Seems I'm missing some code here.
Sorry, just ignore that example... I put you on the wrong track there.
I wanted to demonstrate using HERE documents as standard input.

Quote:
Perhaps you're right. A jumble-solving Python script I have needs the dictionary.txt file in plain text. I found mine by copping the american-english.txt file from somewhere in the deep recesses of my Ubuntu laptop's GNOME install. And even the old fortune-cookie executables need to read properly-formatted quote files. I just wasn't sure how it might compile without the rules right there inside it.

Guess I'm off to another subforum (Programming) with this? Or would you suggest I work on it some more as a shell script?

BZT
I would suggest you ask yourself who you want to serve with this script. What you want to accomplish with it.
If you just want to include the rules I would suggest you keep it as it is. It is probably not worth spending your time on.
And if it works, I would start rewriting the code in python... but that is your free choice of course
 
Old 02-25-2010, 10:31 PM   #7
SilversleevesX
Member
 
Registered: May 2009
Posts: 181
Blog Entries: 9

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by deadeyes
I would suggest you ask yourself who you want to serve with this script. What you want to accomplish with it.
If you just want to include the rules I would suggest you keep it as it is. It is probably not worth spending your time on.
And if it works, I would start rewriting the code in python... but that is your free choice of course[/i]
Python seems the way to go, once I have the thing working the way I want. (The rest of this post is me thinking out loud.)

My own inclination is to include in the sed file the mimetypes for more or less common file types downloaded from the Web in Linux and Windows. I wasn't considering anything but the graphics file formats, for which I've developed the habit of thinking of (and thus typing in terminals, consoles and other CLI utilities) as names without file extensions.

In different situations, with the patterns I've come up with for my personal cataloging and perusal of JPEGS etc, it's often enough to just remember what goes with or before what: is this picture from site ABC, DEF or JWMRAF? Paradoxically, when configuring an OS, I often insist on said system displaying the file extensions whenever applicable or available. This is where Windows spoils the lot, imo. Without the ability to "take a peek" inside a file, to gather the tiniest bit of header information (even OS X 10.4 Tiger UB did this on Motorola Macs), it's left up to applications to gather this information. Two do it very well: XnView and IrfanView. But not everyone has those, and rather than put them through the trouble of finding and installing them, I wanted to both be courteous and head off comments such as "Your pictures don't open in anything I have installed."

And in case one might think I'm unreasonably inclined to be courteous to Windoze users, let me quote a limerick of my own creation (circa 1999-2000):

Quote:
A rich man who hailed from Seattle
Wrote Win98 to do battle.
So Mac users parry
With those too unwary
To know that it's all just for cattle.
BZT

Last edited by SilversleevesX; 02-25-2010 at 10:32 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script: using "select" to show multi-word options? (like "option 1"/"o zidane_tribal Programming 6 03-21-2013 10:35 AM
sed - use sed to replace multiple instances from "x" to "y" in a line mrodmac Linux - General 4 02-02-2010 11:37 AM
bash question about sed '"$i"!d' listofusers haliban Programming 6 09-25-2008 06:42 PM
How to write a bash script to replace all "KH" to "K" in file ABC??? cqmyg5 Slackware 4 07-24-2007 09:00 AM
bash "unexpected end of file" script error Runge_Kutta Linux - General 6 05-23-2007 03:36 PM


All times are GMT -5. The time now is 01:57 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration