bash scripting for html generation

Tinkster · 10-22-2006, 08:21 PM

Ok ... and my apologies. I can't have tested that properly, must have
been very tired ;}

Try this version (which also should take care of the spaces):

Code:

BEGIN{
  print "<html>"
  print "<body>"
  print "<center>"
  print "<ul>"
}
{
  print "<li><a href=\""gensub(/\.mp3/ , ".html" , g )"\">"$0"</a>"
  # and the other files ... 
  target = gensub(/\.mp3/ , ".html" , g )
  print "<html>" > target
  print "<body>" > target
  print "<embed src=\""$0"\" autostart=\"true\">" > target
  print "</ul>" > target
  print "</center>" > target
  print "</body>" > target
  print "</html>" > target
  close( target )
}
END{
  print "</ul>"
  print "</center>"
  print "</body>"
  print "</html>"
}

Note: I don't think that apache likes files with spaces in their names,
but I leave that for you to find out and fix ;}

To create the index-file ...
awk -f index.html <filelist> > index.html

Or alternatively add a "> index.html" to all lines with
a print statement that doesn't go to target. ;}

Cheers,
Tink

muha · 10-23-2006, 03:44 AM

Quote:

Originally Posted by Tinkster

To create the index-file ...
awk -f index.html <filelist> > index.html

Shouldn't the awk scriptname come behind the -f ? Like:

Code:

awk -f script.awk 1.mp3 2.mp3 > index.html

daveoily · 10-23-2006, 08:49 AM

I seem to think you may be right about apache not liking spaces, but that might have been fixed, for now I have problems with an error message. Error messages are scary when you start looking at this stuff, it's probably simple but here it is anyhow, I called the code latest.awk, error reads...

dave@ubuntu:~/projects/scripts/awk_stuff$ awk -f latest.awk file_list > index.html
awk: latest.awk:1: BEGIN{
' in expression:1: ^ invalid char '

At line 1 as well! It looks the same as it always has done to me, what could possibly be wrong with that?

muha · 10-23-2006, 09:18 AM

What is the command you use? This:

Code:

awk -f latest.awk file_list > index.html

What does the file file_list look like?

daveoily · 10-23-2006, 09:28 AM

Yep, that's what I used.

It's a list of mp3's from the directory cut and pasted from the ls command. Sample to follow...

01___DOWNTOWN.mp3
03_SOMETHIN_STUPID.mp3
04 - Getting Old.mp3
06_STRAITEWN_UP_AND_FLY_RIG.mp3
14 - First 2 Years of Marriage.mp3
15___OYXGEN.mp3
16 - Nervous Breakdown.mp3
17.Macc Lads-Julie the Schooly.mp3
1stp Klosr (The Humble Brothers ft. Jonathan Davis).mp3
34 macc ladds - lads from mac.mp3
'39.mp3
Accelerator.mp3
After September 11th.mp3
Afx237 V7.mp3
_ Aggressive.mp3
Agua.mp3
Aim _ Cold Water Music.mp3
Airport Security.mp3
Allegretto-Quintet In E Flat, Kv452, III. Allegretto.mp3
_ Allegro (1).mp3
_ Allegro.mp3
All things dull and ugly.mp3
Always look on the bright side of life.mp3
[ always outnumbered never outgunned ].mp3
Always The Sun.mp3
Amazing.mp3
American Life.mp3

it starts looking better from here on in, the first files are my poorly named downloaded ones, the rest are rips

but it's a long list, so you see why I'm reluctant to do it by hand?

muha · 10-23-2006, 09:30 AM

Thanks, you want shorten your post a bit ...

Tinksters script works over here for me:

Code:

$ cat file_list
01___DOWNTOWN.mp3
01___HERE_WITH_ME.mp3
01-Way Out.mp3
02_BEAUTY_ON_THE_FIRE_.mp3
$ awk -f latest.awk file_list > index.html
$ cat index.html
<html>
<body>
<center>
<ul>
<li><a href="01___DOWNTOWN.html">01___DOWNTOWN.mp3</a>
<li><a href="01___HERE_WITH_ME.html">01___HERE_WITH_ME.mp3</a>
<li><a href="01-Way Out.html">01-Way Out.mp3</a>
<li><a href="02_BEAUTY_ON_THE_FIRE_.html">02_BEAUTY_ON_THE_FIRE_.mp3</a>
</ul>
</center>
</body>
</html>

Could you also try that short filelist?
I think the issue might be the weird characters later on in the file.
Save a backup of the file_list first ...
Hmm, you're long filelist works for me as well, so then the problem might be in the file: latest.awk

daveoily · 10-23-2006, 09:47 AM

ok, trimmed it down, just wanted to make the point, it's a long list...

So you think I might have cut and pasted it wrong?? :S I wouldn't put it past myself, I'll check.

Nope, I get the same error message

when I open it in gedit, the first print is written in black as opposed to pink which the rest of them are.

I have to put it onto a usb memory stick to transfer it from a windoze box where I am, no networking possible and my ubuntu laptop's modem is not working

could something be going wrong copying it over??

unSpawn · 10-23-2006, 10:04 AM

awk: latest.awk:1: BEGIN{
' in expression:1: ^ invalid char '
Means some chars are not escaped that should ('39.mp3). A quick check should show what chars need replacing/removing:

Code:

find /mp3/dir -type f -iname \*.mp3 | while read l; do 
[ -n "${l//[ a-z0-9._-]/}" ] && echo "\"${l}\": ${l//[ a-z0-9._-]/}"
done

Replace "/mp3/dir" with the full path leading to the directory that contains your mp3 files.
The result from your posted list shows:

Code:

"1stp Klosr (The Humble Brothers ft. Jonathan Davis).mp3": ()
"'39.mp3": '
"Allegretto-Quintet In E Flat, Kv452, III. Allegretto.mp3": ,,
"_ Allegro (1).mp3": ()
"[ always outnumbered never outgunned ].mp3": []

poorly named downloaded ones
Before generating a filelist I would rename files to something like "(album)artist_album_track_title" using mp3info (CLI) or your favourite GUI MP3 tag editing tool (also see: Freshmeat and Sourceforge). Not only is it a chance to weed out unwanted characters but it will also make the list better usable (readable).

daveoily · 10-23-2006, 10:12 AM

might those symbols have ocourred when i trimmed some of the file names out to make my post shorter?

unSpawn · 10-23-2006, 10:15 AM

might those symbols have ocurred when I trimmed some of the file names out to make my post shorter?
Unlikely.

daveoily · 10-23-2006, 10:36 AM

Unspawn, I'm having trouble getting your code going, I end up with an empty command prompt that I can't break out of, go easy on me eh? I'm very new to this...

Anyhoo, the script worked from a file_list containing all the characters that your script deems illegal, so perhaps awk can handle them anyway? (for instance one earlier script produced a file called "[always outnumbered never outgunned].html" it also has produced files containing the characters ' , ( ) _ and - )

I totally take on board the advice regarding mp3info, although I can only find an rpm (I managed to get that going using alien, then downgraded mp3info, so I had it already and didn't realise! that'll teach me!), anyone know a similar one in a deb?

muha · 10-23-2006, 10:54 AM

Quote:

Originally Posted by daveoily

Unspawn, I'm having trouble getting your code going, I end up with an empty command prompt that I can't break out of, go easy on me eh? I'm very new to this...

It was missing a ; in front of done.
Normally the syntax is while ... ; do ... ; done
Also , end each line with a bashslash (and enter) to continue on the next line with the command:

Code:

find /mp3/dir -type f -iname \*.mp3 | while read l; do \
[ -n "${l//[ a-z0-9._-]/}" ] && echo "\"${l}\": ${l//[ a-z0-9._-]/}"; \
done

Except for the last line.

This one outputs all files though, so i included the / as ok character in the fileformat:
You have to escape it so it is \/

Code:

find /mp3/dir -type f -iname \*.mp3 | while read l; do \
[ -n "${l//[ a-z0-9._-\/]/}" ] && echo "\"${l}\": ${l//[ a-z0-9._-\/]/}"; \
done

This only ouputs files with weird characters.
/edit: forgot the \/ in the end part as well ... last command is now updated.

daveoily · 10-23-2006, 11:34 AM

Thanks very much for all the help guys, Tinkster, I got your code going eventually, I don't get why it didn't work on my previous trys, but I deleted a few spaces just before the first print statement and everything works just fine!

I don't understand it at all, but it is definately most useful

My final code looks like this...

Code:

BEGIN{
print "<html>"
  print "<body>"
  print "<center>"
  print "<ul>"
}
{
  print "<li><a href=\""gensub(/\.mp3/ , ".html" , g )"\">"$0"</a>"
  # and the other files ... 
  target = gensub(/\.mp3/ , ".html" , g )
  print "<html>" > target
  print "<body>" > target
  print "<center>" > target
  print "<a href=\""$0"\">"$0"</a>" > target
  print "</br>" > target
  print "</br>" > target
  print "<embed src=\""$0"\" autostart=\"true\">" > target
  print "</ul>" > target
  print "</center>" > target
  print "</body>" > target
  print "</html>" > target
  close( target )
}
END{
  print "</ul>"
  print "</center>"
  print "</body>"
  print "</html>"
}

It just comes out a little more like my original page

Tinkster · 10-23-2006, 12:31 PM

It does two different things in one go.

Awk is a text-processor that operates line(record)-oriented (by default, you
can change that, though, and make a record spawn several lines). The BEGIN
and END statements are executed before and after the processing of the file
(or STDIN) is done.

The BEGIN{ } just creates and outputs a generic HTML header (which we
redirect from the prompt) into the index.html.

The main body reads your file line by line.

This one outputs the lines for the index-file, gensub() searches each
line passed to it for \.mp3 and replaces is with \.html.

Code:

print "<li><a href=\""gensub(/\.mp3/ , ".html" , g )"\">"$0"</a>"

The next lot creates the individual files, by outputing a complete
(very basic) HTML file for a line into a file named after the target.

Btw, I just realised that the line for the index could be made shorter
and execute faster by only using the gensub once

since it does the
same on both ...

Code:

  target = gensub(/\.mp3/ , ".html" , g )
  print "<li><a href=\""target"\">"$0"</a>"
  # and the other files ... 
  print "<html>" > target
  print "<body>" > target
  print "<embed src=\""$0"\" autostart=\"true\">" > target
  print "</ul>" > target
  print "</center>" > target
  print "</body>" > target
  print "</html>" > target
  close( target )

Cheers,
Tink

daveoily · 10-07-2007, 04:30 PM

many thanks for the help, I think i pm'd tinkster in particular to thank him, I used the script successfully for some time, but now I seem to have problems....

well i changed a few things and perhaps it's my fault (as is usually the case, but you guys sorted me out last time, so I thought i'd try asking the same crowd

).

anyway, my code listing looks like this these days....

Code:

BEGIN{
print "<html>"
  print "<body>"
  print "<link rel=\"stylesheet\" type=\"text/css\" href=\"../1.css\">"
  print "<center>"
  print "<ul>"
}
{
  print "<li><a href=\""gensub(/\.mp3/ , ".html" , g )"\" target=\"rhs\">"$0"</a>"
  # and the other files ... 
  target = gensub(/\.mp3/ , ".html" , g )
  print "<html>" > target
  print "<body>" > target
  print "<link rel=\"stylesheet\" type=\"text/css\" href=\"../1.css\">" > target
  print "<center>" > target
  print "<a href=\"../music/"$0"\">"$0"</a>" > target
  print "</br>" > target
  print "</br>" > target
  print "<embed src=\""$0"\" autostart=\"true\">" > target
  print "</ul>" > target
  print "</center>" > target
  print "</body>" > target
  print "</html>" > target
  close( target )
}
END{
  print "</ul>"
  print "</center>"
  print "</body>"
  print "</html>"
}

As you can see, I added a couple of little changes to get the file to pick up and use a css script in every file, and the download link now leads to another file on the server to pick up a full bitrate file ( i use 32kbps for streaming, and whatever i have lying around for the download link

)

But it seems to give up after doing a lot of files now

it didn't used to do this! could someone help me find out where I've screwed it up this time please?

I used cygwin on xp (oh for shame!), but when I try and use the same listing on my ubuntu machine awk doesen't even get past the first line!

I'll run it in a moment, the results will turn up at http://daveoily.dyndns.org/oldstreams/music.htm

I use frames to put the html files in their places, well, have a poke around, and maybe a listen to a track or two, let me know where you think I might have gone wrong please.

I guess the bit where things go wrong is where awk encounters the file name millie small - my boy lollipop.mp3

it gets as far as writing "mill" at the bottom of the track listing... why would it stop there?

thanks in arrears and advance

dave