parsing script

sthompson · 06-08-2008, 08:40 PM

ok i am trying to parse a .xml file every couple mins i would need to be able to do it constantly and have the parse be very specific to mabey a couple of lines it is for a live audio stream and i'm compeletly lost at this i don't know where to begin any help suggesttions i can run with or anything would be helpful.

emi_ramo · 06-09-2008, 04:05 AM

What do you mean with 'constantly'? What is exactly the service you're trying to serve?

sthompson · 06-09-2008, 06:14 AM

the script would have to run constanly i would want it to run 24/7 and if possible have it start a new log file every day or once a week or something like that but the script would never be stopped. and i am trying to parse a list of ip adresses for stats on a mount point from an icecast stream. there is an xml page that comes free but it does not give past history of clients only current clients so i want to parse it to make a log.

emi_ramo · 06-09-2008, 06:28 AM

What about using crontab?

as a user with read access to logs and write access to stats, run:

Code:

crontab -e

For a complete description, see manual page. With that, you will be able to run commands periodically, with a user specified time interval.

sthompson · 06-09-2008, 07:19 AM

the crontab would allow me to run the command every set time is that right? what about the parsing itself?

emi_ramo · 06-09-2008, 07:31 AM

crontab is an ancient app from UNIX systems. It runs commands periodically; user or sysadmin sets its intervals. Each user has it's own crontab table. You tell crontab at which time/times to run an specified program (your script, for example), which days, months, years, ...

Syntax is a little bit complicated because it lets you a very high range of possibilities.

Just google a little bit and you'll find it out, sure.

For the parsing, you can do a every-minute parsing. Or, if it's a better idea, parse the XML every time it changes. Think, you cannot parse it constantly. At least, you must wait until a parsing process to begin another one. It's very CPU expensive to parse anything constantly, even when XML file is little, because it ends fast, and fast it rebegins...

Think if you REALLY need to change stats every microsecond or not...

sthompson · 06-09-2008, 07:40 AM

no not that offten i said constently cause i can't think of what it would be called but i want it at max every min but what would a parse script look like for an xml file.

p_s_shah · 06-09-2008, 09:42 AM

I am not an expert, but you can try to parse XML using perl module XML::Rules.
Link: http://search.cpan.org/~jenda/XML-Ru...b/XML/Rules.pm

Prepare your script and schedule it using Cronjob.

sthompson · 06-09-2008, 11:45 AM

what...? that just lost me

emi_ramo · 06-09-2008, 12:01 PM

I think p_s_shah told is that if you need to parse an XML file to do your stats, you can use an already done XML parser, written on perl and named XML::Rules, downloadable from http://search.cpan.org/~jenda/XML-Ru...b/XML/Rules.pm . Once you've got your program/script working, you put it into your crontab every minute.

Your script crontab entry should look something as:
* * * * * your_xml_parser.pl

And your stats rotater (at 3:10 morning, every day):
10 3 * * * your_rotater

And your stats rotater (at 3:10 morning, every sunday):
10 3 * * 7 your_rotater

archtoad6 · 06-09-2008, 05:27 PM

What are you parsing for?

sthompson · 06-09-2008, 06:15 PM

i'm am parsing for a list of clients on a streaming mount point that does a live update and i want to be able to see stats on it for like 2 in the morning but i don't want to pay for a service that is stupid and give way to much i just want what i get for free an ip so i can get a location of listener and how many people conected and a rough ideal of how long. thats it.

chrism01 · 06-09-2008, 08:01 PM

If you want to parse it that often, don't use cron, its a waste of resources to keep starting a new process.
Write a Perl daemon and use http://search.cpan.org/~mgrabnar/Fil...0.99.3/Tail.pm instead.

archtoad6 · 06-10-2008, 08:46 AM

I meant what is the format of the string(s) you are parsing for, as in what does the search regex need to look like.
Sorry for the ambiguity.

Would a background bash script w/ a sleep command be as light as a Perl Daemon?

I'm thinking something as simple as:

Code:

while :
do
   grep -o <regex> <infile>  >> <outfile>
   sleep 5m
done

Since I don't know what you're looking for, I don't know if this approach would work.

emi_ramo · 06-10-2008, 09:39 AM

That will be worst than a crontab entry, for sure. If the problem is to launch new processes, bash scripting is far away from the solution. A daemon is a program that stays "doing nothing" until it needs to do something. In C, it should be something as:

Code:

first = t = time(NULL);
while(1)
{
   if((t-first) >= interval)
   {
      first = t = time(NULL);
      do_stats();
   }
   else
      t = time(NULL);
}

or just (for a minimal CPU usage but less accurate)

Code:

while(1)
{
   do_stats();
   sleep(interval * 1000);
}

and run this program backgrounded (with '&' at the end of the execution line, or some other fork solution).