LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Desktop
User Name
Password
Linux - Desktop This forum is for the discussion of all Linux Software used in a desktop context.

Notices



Reply
 
Search this Thread
Old 02-07-2007, 04:51 AM   #1
tomwgf
LQ Newbie
 
Registered: Feb 2007
Posts: 4

Rep: Reputation: 0
How to write a script to change web log files?


hi
i have fasthost linux server web log files that i need to change to make them acceptable to Funnelweb.
I can change each line by hand but i need to write a script so i can make these changes automatically - but i don't know where to begin - i have never written one before.

the line looks like this:
floppyrecords.co.uk: [01/Nov/2006:00:06:55 +0000] 64.12.186.235 - - "GET /floppyrecords-free-mp3s/tantrum/maisies/8-tantrum-filmsongpart2.mp3 HTTP/1.1" 200 12913 "a url but blocked as my first posting" "asterias/2.0"

i need to change it to this:
64.12.186.235 - - [01/Nov/2006:00:06:55 +0000] "GET /floppyrecords-free-mp3s/tantrum/maisies/8-tantrum-filmsongpart2.mp3 HTTP/1.1" 200 12913 "a url but blocked as my first posting" "asterias/2.0"

ie delete:
floppyrecords.co.uk:

and move
[01/Nov/2006:00:06:55 +0000] or[*] perhaps??
to after:
64.12.186.235 - - or *.*.*.* - - perhaps??

thanks
tom
 
Old 02-07-2007, 06:02 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

First take a look at the way the webserver creates the logfile (which fields are printed). Apache has a lot of option to format the log(s) the way you want to, although not everything is possible.

If that is not an option, try the following sed one-liner:

sed 's/.*: \(\[.*\]\) \(.* - -\)\(.*\)/\2 \1 \3/' infile

You state that you don not have any experience with scripting (and probably one-liners), so here's a little breakdown of the sed command command:

With sed it is possible to change/rearrange content of a file (in normal use: one line at the time). This would look like this: sed 's/this/THAT/g' infile. This will change all instances of this into THAT. The g at the end makes sure it is done on the whole line, not just the first hit.

Regular expressions (regexp) and backreferencing is also possible:

.*: \(\[.*\]\) \(.* - -\)\(.*\) => this will split the line in 4 parts. Taking your example as infile it will brake that line up into the following pieces:
1) floppyrecords.co.uk: (including the space after the : )
2) [01/Nov/2006:00:06:55 +0000]
3) 64.12.186.235 - -
4) 'the rest of the line'

Parts 2, 3 and 4 have \( and \) around them, these are used so you can 'backreference' them (use them in the replace part of the statement). The replace part (\2 \1 \3) will discard part 1 and rearrange parts 2, 3 and 4.

The given sed statement does not save the output (you need to check for correctness first). If all is ok, add the -ibak option. This will make the changes in place and creates a backup of the original file.

Final command would look like this:

sed -ibak 's/.*: \(\[.*\]\) \(.* - -\)\(.*\)/\2 \1 \3/' infile

Hope this helps.
 
Old 02-07-2007, 06:09 AM   #3
tomwgf
LQ Newbie
 
Registered: Feb 2007
Posts: 4

Original Poster
Rep: Reputation: 0
hi drunna
thanks for this - i don't have access to the fasthost servers (and i have asked them in the past about this).
So i'll have a go at the script.
do i replace infile with the file name or do i add that after infile (or have i completely misunderstood this, you do mean in a terminal don't you?)

tom
 
Old 02-07-2007, 06:18 AM   #4
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

Quote:
thanks for this
You're welcome

Quote:
do i replace infile with the file name or do i add that after infile
infile is the name of the file you want/need to change.

And you need to do this from a terminal.

Hope this clears things up.
 
Old 02-07-2007, 09:16 AM   #5
tomwgf
LQ Newbie
 
Registered: Feb 2007
Posts: 4

Original Poster
Rep: Reputation: 0
hi drunna
that worked - thank you, a problem that has bugged me for over 5 years!
interestingly (for me only perhaps) the resulting logs didn't work in Funnelweb linux commandline - a segmentation fault. But did work for 2 out of 3 months in the windows version program - but without listing the referrals.

so i will now try and understand your script.

could there be hidden tabs in the logs that have to be accurately copied? how would you see them?

but thank you very much.
tom
 
Old 02-07-2007, 09:50 AM   #6
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

Quote:
could there be hidden tabs in the logs that have to be accurately copied? how would you see them?
That could be.

You could use the od command to check: head -5 infile | od -t c

For example, I created a file (infile) with 2 lines:
Code:
 $ cat infile 
space: (space)
tab:    (tab)
Using the od command on that infile gives the following:
Code:
$ head -2 infile | od -t c
0000000   s   p   a   c   e   :       (   s   p   a   c   e   )  \n   t
0000020   a   b   :  \t   (   t   a   b   )  \n
0000032
A tab shows up as \t and a space is just that (an empty spot). The \n indicates an end of line.

I also use the head -2 infile part to only show the first 2 lines (could be of use if the infile is rather large)

Depending on where the tab is present, you might have to edit the sed command to include this tab. But this is only needed if the tab character is in the first part of the line (In this part: floppyrecords.co.uk: [01/Nov/2006:00:06:55 +0000] 64.12.186.235 - -). Everything after and including "GET....... is taken care of in the sed statement.

If anything is still unclear, just ask

Hope this helps.
 
Old 02-07-2007, 10:38 AM   #7
tomwgf
LQ Newbie
 
Registered: Feb 2007
Posts: 4

Original Poster
Rep: Reputation: 0
hi druuna
i ran head -5 infile | od -t c on both before and after sed files but there were no tabs. so its probably not that.
but thank you very much anyway, its very good to learn these things.
thanks for all your help.
tom
 
  


Reply

Tags
convert, edit, file, logs, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
howto write a script to echo files with spaces in the name dsids Linux - Newbie 1 11-28-2006 08:18 AM
how to get a script to write to a log paul_mat Linux - Software 8 05-26-2006 07:15 AM
Help! Clear web.cache and log files on server DogByte Linux - Newbie 1 09-20-2005 07:51 PM
how to write a script to move files from one directory to another myeire Linux - Newbie 7 02-25-2005 11:01 AM
Change path of log files linuxboy69 Linux - Software 4 12-23-2003 02:17 PM


All times are GMT -5. The time now is 01:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration