script to extract blocks of text from many files.

gruessle · 10-18-2007, 04:43 PM

Hi

I have about 2000 html files with blocks of info which I need to extract.

I am thinking bash but I am more fluent in php so that would be better.

The file names are pro1.html pro2.html ... pro1980.html

So what I need is a script that:
1) opens each file
2) Try to find a start string, skip file if start string does not exist.
4) finds a end string
5) copies/prints everything in between to another file.

So that all the selected text ends up in one single long file.

Can you get me started on this?

Code:

#!/bin/bash

start_string = "<table>"
end_string = "</table>"
selected_text = ''

Proud · 10-18-2007, 05:21 PM

Is it XHTML or HTML4? Is it properly formed (i.e. matching tags&closing tags, nested correctly) XML? Do you know about XSLT?

I'm more of a Python or Java man myself, but I'm a little rusty&tired atm. It could probably be done with a grep statement of course

As you say, if you're writing the file searching yourself, you need to:
create an empty output file buffer
list the files
open them one at a time
start from the beginning, treat the file contents as a long string, look for the start string
if you find it, look from that place onward for the end string
if you find it, what's in between is what to append to the output file buffer OR copy what's after the start string until you maybe find the end string
close input file and move on to next file until none left
write output file

You may want to learn an efficient string matching algorithm if you're DIYing this.

Tinkster · 10-18-2007, 05:29 PM

Don't have files to test with handy, but this *should* work

Code:

awk '/<table>/,/<\/table>/' *htm > tables

Cheers,
Tink

gruessle · 10-19-2007, 01:47 AM

I have already got this done but I am still going to test your awk line because this would make a lot of things easier

Thanks

yongitz · 10-19-2007, 02:31 AM

another simple solution here, though it has not been tested..

Code:

for file in *.html
do 
sed -n '/start_string/,/end_string/p' $file  >> outputfile
done