LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   BASH script with sed file "inside"? (https://www.linuxquestions.org/questions/linux-newbie-8/bash-script-with-sed-file-inside-790143/)

SilversleevesX 02-19-2010 03:52 AM

BASH script with sed file "inside"?
 
Continuing the script file project discussed here, I would like to know if it were possible to 'embed' the "sed" file -- or at least the same data -- kindly created by jschiwal, into the script itself with an eye towards a careful (or quick-&-dirty, using shc-3.8.6) conversion to an executable for the command line.

I await your answers and advice.

BZT
PS: I have the last e-book/PDF edition made of the ABS Guide, and hadn't someone (Andreas Huggel, author of exiv2) just the other day suggest that I pick up the O'Reilly bash Cookbook instead, I'd be reading it cover to exit command. That opinions about both are plentiful I have no doubt. This postscript presumes to head off those inclined to offer such advice.

deadeyes 02-19-2010 05:18 AM

I think HERE documents is what you need:
http://tldp.org/LDP/abs/html/here-docs.html

SilversleevesX 02-19-2010 01:00 PM

Quote:

Originally Posted by deadeyes (Post 3869214)
I think HERE documents is what you need:
http://tldp.org/LDP/abs/html/here-docs.html

Thanks deadeyes.

I'm trying to get my mind around HERE documents, particularly as they would apply to the functions in the script. Maybe I'm mis-perceiving something about the original script? Another pair of eyes and a more experienced mind and hand are asked for now.

The script as written towards the end of that other thread:
Code:

#!/bin/bash

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f rename.sed descriptions >add_extensions.sh

Here's what I think is going on:
Line 1: The find command is set to look for files in the same folder the user is "in" on the command line, such files to be regular and have no period separating one part of the filename from the other. The result of this search is redirected through xargs to a file named descriptions.
Line 2: The sed command is reading in the file rename.sed and the file descriptions and, following the syntax of the first is applying it to the contents of the second and then creating a shell script called add_extensions.sh which the user can execute later on the files in that directory.

Got it right so far, have I?

Okay. Here's a modification of that script I attempted after a few readings, and re-readings, of the Here documents page on tldp.org (and a few others, including a jump from a function-by-function breakdown of sed found on grymoire.com)
Code:

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f <<rename descriptions >add_extensions.sh

<<rename
/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p
/ISO Media, MPEG v4/s/\(^[^:]*\):.*$/mv "\1" "\1.mp4"/p
/MPEG ADTS, layer III/s/\(^[^:]*\):.*$/mv "\1" "\1.mp3"/p
rename

The modifcation returns an error such as
Code:

sed: file descriptions line 1: unknown command: `.'
...as well as, predictably, a zero-bytes add_extensions.sh.

As I say, I'm not "all here" with Here yet.

BZT

deadeyes 02-20-2010 04:36 AM

Quote:

Originally Posted by SilversleevesX (Post 3869670)
Thanks deadeyes.

I'm trying to get my mind around HERE documents, particularly as they would apply to the functions in the script. Maybe I'm mis-perceiving something about the original script? Another pair of eyes and a more experienced mind and hand are asked for now.

The script as written towards the end of that other thread:
Code:

#!/bin/bash

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f rename.sed descriptions >add_extensions.sh

Here's what I think is going on:
Line 1: The find command is set to look for files in the same folder the user is "in" on the command line, such files to be regular and have no period separating one part of the filename from the other. The result of this search is redirected through xargs to a file named descriptions.
Line 2: The sed command is reading in the file rename.sed and the file descriptions and, following the syntax of the first is applying it to the contents of the second and then creating a shell script called add_extensions.sh which the user can execute later on the files in that directory.

Got it right so far, have I?

Almost there :)
The find command lists all files with a dot (.) in them.
I cannot find -not in the man page so dont know if it is multiple options or just the word not.
-print0 prints all find files on the same line if I recall correctly. xargs will split up this list of files in sublists. This is because command can only have a limited amount of arguments. xargs will then execute the command "file" for each sublist. THe result is written to subscription.
The second line will read the file "descriptions" and will apply the rules defined in "rename.sed".
-f gives a file.
With HERE documents you can send data from the file as it was from standard input, for example:
Code:

cat afile | wc -l
Code:

wc -l <<afile
> test
> test2
> test3
> afile
3

Do note that the bigger then chars are printed by bash.

And I am not sure if that is possible for the rules of the sed command. You can use it for the file that should be processed.
So you might use:
sed -e 'firstrule' -e 'secondrule' ...
This might soon look ugly.

What I wonder is why you want this included? Maybe it is easier just to supply the rules file with it.

Quote:

Okay. Here's a modification of that script I attempted after a few readings, and re-readings, of the Here documents page on tldp.org (and a few others, including a jump from a function-by-function breakdown of sed found on grymoire.com)
Code:

find . -type f -not -name "*.*" -print0 | xargs -0 file >descriptions
sed -f <<rename descriptions >add_extensions.sh

<<rename
/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p
/ISO Media, MPEG v4/s/\(^[^:]*\):.*$/mv "\1" "\1.mp4"/p
/MPEG ADTS, layer III/s/\(^[^:]*\):.*$/mv "\1" "\1.mp3"/p
rename

The modifcation returns an error such as
Code:

sed: file descriptions line 1: unknown command: `.'
...as well as, predictably, a zero-bytes add_extensions.sh.

As I say, I'm not "all here" with Here yet.

BZT
I think you need something like:
Code:

sed -e '/JPEG image data/s/\(.*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
So this means if JPEG image data was printed by the file command it will substitute the filename with "mv filename.jpg"

So look at the explanation above :)
have fun!

SilversleevesX 02-22-2010 06:16 AM

Thanks for the continued help and advice. deadeyes.

Quote:

Originally Posted by deadeyes (Post 3870244)
With HERE documents you can send data from the file as it was from standard input, for example:
Code:

cat afile | wc -l
Code:

wc -l <<afile
> test
> test2
> test3
> afile
3

Do note that the bigger then chars are printed by bash.

Why are we counting the number of rules? I made up a short script, borrowing "rules" from the bigger sed file in that other thread (addext.sed by name), and whether a slight tweak of the sed command you suggested here...

Code:

sed -e '/image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
... is above or below the rules I inserted...
Code:

>  /image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p;n}
>  /image\/png/{ s/\(^[^:]*\):.*$/mv "\1" "\1.png"/p;n}
>  /image\/gif/{ s/\(^[^:]*\):.*$/mv "\1" "\1.gif"/p;n}
>  /image\/tiff/{ s/\(^[^:]*\):.*$/mv "\1" "\1.tif"/p;n}
>  afile

...the number returned from wc -l is, predictably, always higher than the last time I run it. At one stage of making this scratch script, it had no sed line at all, and I got the usual returns for a file -i command (the way the mime types I'm using in the rules are written seems to require it), and with the sed line included, below the rules, the only return is the number from wc -l, with no effect on the files themselves. Nor any generation of an add_extensions.sh file. Seems I'm missing some code here.

Quote:

Originally Posted by "deadeyes" (Post 3870244)
What I wonder is why you want this included? Maybe it is easier just to supply the rules file with it.

Perhaps you're right. A jumble-solving Python script I have needs the dictionary.txt file in plain text. I found mine by copping the american-english.txt file from somewhere in the deep recesses of my Ubuntu laptop's GNOME install. And even the old fortune-cookie executables need to read properly-formatted quote files. I just wasn't sure how it might compile without the rules right there inside it.

Guess I'm off to another subforum (Programming) with this? Or would you suggest I work on it some more as a shell script?

BZT

deadeyes 02-22-2010 03:11 PM

Quote:

Originally Posted by SilversleevesX (Post 3872447)
Thanks for the continued help and advice. deadeyes.

You're welcome :)

Quote:

Why are we counting the number of rules?
I just wanted to demonstrate the use of HERE documents :)
And showing that in your case it does not work .

Quote:

I made up a short script, borrowing "rules" from the bigger sed file in that other thread (addext.sed by name), and whether a slight tweak of the sed command you suggested here...

Code:

sed -e '/image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p' -e ... descriptions > add_extensions.sh
... is above or below the rules I inserted...
You put before an example script. This should replace the existing sed line and rules file. This is a way of including the rules into the script without using HERE documents.
Do note that this is less readable.

Quote:

Code:

>  /image\/jpeg/{ s/\(^[^:]*\):.*$/mv "\1" "\1.jpg"/p;n}
>  /image\/png/{ s/\(^[^:]*\):.*$/mv "\1" "\1.png"/p;n}
>  /image\/gif/{ s/\(^[^:]*\):.*$/mv "\1" "\1.gif"/p;n}
>  /image\/tiff/{ s/\(^[^:]*\):.*$/mv "\1" "\1.tif"/p;n}
>  afile

...the number returned from wc -l is, predictably, always higher than the last time I run it. At one stage of making this scratch script, it had no sed line at all, and I got the usual returns for a file -i command (the way the mime types I'm using in the rules are written seems to require it), and with the sed line included, below the rules, the only return is the number from wc -l, with no effect on the files themselves. Nor any generation of an add_extensions.sh file. Seems I'm missing some code here.
Sorry, just ignore that example... I put you on the wrong track there.
I wanted to demonstrate using HERE documents as standard input.

Quote:

Perhaps you're right. A jumble-solving Python script I have needs the dictionary.txt file in plain text. I found mine by copping the american-english.txt file from somewhere in the deep recesses of my Ubuntu laptop's GNOME install. And even the old fortune-cookie executables need to read properly-formatted quote files. I just wasn't sure how it might compile without the rules right there inside it.

Guess I'm off to another subforum (Programming) with this? Or would you suggest I work on it some more as a shell script?

BZT
I would suggest you ask yourself who you want to serve with this script. What you want to accomplish with it.
If you just want to include the rules I would suggest you keep it as it is. It is probably not worth spending your time on.
And if it works, I would start rewriting the code in python... but that is your free choice of course :)

SilversleevesX 02-25-2010 10:31 PM

Quote:

Originally Posted by deadeyes
I would suggest you ask yourself who you want to serve with this script. What you want to accomplish with it.
If you just want to include the rules I would suggest you keep it as it is. It is probably not worth spending your time on.
And if it works, I would start rewriting the code in python... but that is your free choice of course[/i]

Python seems the way to go, once I have the thing working the way I want. (The rest of this post is me thinking out loud.)

My own inclination is to include in the sed file the mimetypes for more or less common file types downloaded from the Web in Linux and Windows. I wasn't considering anything but the graphics file formats, for which I've developed the habit of thinking of (and thus typing in terminals, consoles and other CLI utilities) as names without file extensions.

In different situations, with the patterns I've come up with for my personal cataloging and perusal of JPEGS etc, it's often enough to just remember what goes with or before what: is this picture from site ABC, DEF or JWMRAF? Paradoxically, when configuring an OS, I often insist on said system displaying the file extensions whenever applicable or available. This is where Windows spoils the lot, imo. Without the ability to "take a peek" inside a file, to gather the tiniest bit of header information (even OS X 10.4 Tiger UB did this on Motorola Macs), it's left up to applications to gather this information. Two do it very well: XnView and IrfanView. But not everyone has those, and rather than put them through the trouble of finding and installing them, I wanted to both be courteous and head off comments such as "Your pictures don't open in anything I have installed."

And in case one might think I'm unreasonably inclined to be courteous to Windoze users, let me quote a limerick of my own creation (circa 1999-2000):

Quote:

A rich man who hailed from Seattle
Wrote Win98 to do battle.
So Mac users parry
With those too unwary
To know that it's all just for cattle.

BZT


All times are GMT -5. The time now is 03:29 AM.