How to parse 2GB file w linux commands

agtlewis · 10-11-2005, 06:00 PM

Hello,

I have a 2gb RDF feed which contains lots of urls. Is there any way for me to use a command prompt to extract all of the urls?

At the very least I would like a way to delete every line in the file that does not contain a URL, and then write a php script to extract the urls and insert them into a mysql database.

Any help/advice appreciated.

Thanks.

Tinkster · 10-11-2005, 06:33 PM

Not knowing the internal sturcture of the RDF thingamabob
I can't give you a turn-key answer. Chances are sed or
awk are the right tools for the job, though.

Cheers,
Tink

agtlewis · 10-11-2005, 06:43 PM

Hi,

Here is a snippet of the document. I will research the commands you noted.

Code:

<Topic r:id="Top/Arts/Movies/Titles/1">
  <catid>54803</catid>
</Topic>

<Topic r:id="Top/Arts/Movies/Titles/1/10_Rillington_Place">
  <catid>205108</catid>
  <link r:resource="http://www.britishhorrorfilms.co.uk/rillington.shtml"/>
  <link r:resource="http://www.shoestring.org/mmi_revs/10-rillington-place.html"/>
  <link r:resource="http://www.tvguide.com/movies/database/ShowMovie.asp?MI=22983"/>
  <link r:resource="http://us.imdb.com/title/tt0066730/"/>
</Topic>

<ExternalPage about="http://www.britishhorrorfilms.co.uk/rillington.shtml">
  <d:Title>British Horror Films: 10 Rillington Place</d:Title>
  <d:Description>Review which looks at plot especially the shocking features of it.</d:Description>
  <topic>Top/Arts/Movies/Titles/1/10_Rillington_Place</topic>
</ExternalPage>

Tinkster · 10-11-2005, 06:56 PM

If that's all something like

Code:

awk -F"\"" '$2 ~ "http" {print $2 }' rdf.txt > url.txt

may be all you need.

[edit]
If it works, can you please time the execution and post
back how long it took? :}
[/edit]

Cheers,
Tink

agtlewis · 10-11-2005, 07:53 PM

Tinkster,

Thank you very much!! The command you gave me was able to parse the file and correctly extract all of the url's.

As for the time involved, I'm not sure if that thing is written in pure assembly or what, but it only took 4 minutes and 45 seconds on a pentium 4 running at 2.0ghz with 512mb ram.

Tinkster · 10-11-2005, 08:06 PM

Heh - glad I could help.

awk is written in C, and if you find perl overkill (or the task
isn't complex enough for a perl-script) awk is an excellent
(and very fast and efficient) tool for the job. The speed
doesn't surprise me. :}

Cheers,
Tink