remove the top of a file:

trscookie · 01-23-2008, 08:25 AM

Hello all:

I have a 400Mb log file that was created in Win$hite and I want to remove the top X amount of lines:

I have tried things like

Code:

tail -2000 filename.log > OP.file

However I get these control characters in the OP.file that I don't seem to get the the filename.log. I originally thought that this was a windows to Linux issue so I have issued the dos2unix command but it does not work.

I have also tried:

Code:

head -2000 filename.log > OP.file

And that works fine!? Am I missing something?

Thanks, trscookie.

colucix · 01-23-2008, 09:10 AM

What happens if you try...?

Code:

sed '1,2000d' filename.log > OP.file

trscookie · 01-23-2008, 09:18 AM

oh, will give that a try, I have noticed also:

Code:

$ od -x tmpfile2 | head
0000000 3500 3100 3400 3800 2000 7300 6c00 4900
0000020 4600 4400 4500 7600 7400 2e00 6500 7800

^ double byte characters any Idea on how to convert to single?

so I can cat the file but not vi/nano the file.

colucix · 01-23-2008, 10:03 AM

Quote:

Originally Posted by trscookie

^ double byte characters any Idea on how to convert to single?

I don't really understand what do you mean. The od command dumps the content of a file based on the type of representation you specify. What is the output of

Code:

od -c filename | head

to dump as one-byte ascii characters?

trscookie · 01-23-2008, 10:09 AM

Code:

$ od -c drwtsn32.log | head
0000000 377 376  \r  \0  \n  \0   M  \0   i  \0   c  \0   r  \0   o  \0
0000020   s  \0   o  \0   f  \0   t  \0      \0   (  \0   R  \0   )  \0
0000040      \0   D  \0   r  \0   W  \0   t  \0   s  \0   n  \0   3  \0
0000060   2  \0  \r  \0  \n  \0   C  \0   o  \0   p  \0   y  \0   r  \0
0000100   i  \0   g  \0   h  \0   t  \0      \0   (  \0   C  \0   )  \0
0000120      \0   1  \0   9  \0   8  \0   5  \0   -  \0   2  \0   0  \0
0000140   0  \0   2  \0      \0   M  \0   i  \0   c  \0   r  \0   o  \0
0000160   s  \0   o  \0   f  \0   t  \0      \0   C  \0   o  \0   r  \0
0000200   p  \0   .  \0      \0   A  \0   l  \0   l  \0      \0   r  \0
0000220   i  \0   g  \0   h  \0   t  \0   s  \0      \0   r  \0   e  \0

If you look you can see the work M i c r o s o f t starting at line 140, it is the output of a log file generated on a windows pc, thanks trscookie.

bigearsbilly · 01-23-2008, 11:11 AM

it looks like they've padded each char with a NULL byte,
tykes.

trscookie · 01-23-2008, 11:15 AM

what would be the best way to remove the padding of a 400Mb file?!

bigearsbilly · 01-23-2008, 11:21 AM

try this first off

Code:

#include <stdio.h>


int main(void)
{

    int ch;

    while((ch = getchar()) != EOF) {
        if(ch) putchar(ch);
    }

}

colucix · 01-23-2008, 11:41 AM

Ok, thanks. I see now. Looking at the output of "od -c" I'm puzzled about the first two bytes. I don't know what they mean, but maybe they cause the problem (when missing) you reported in your first post.

It's hard for me to figure out a solution without managing the files by myself. Anyway, regarding the question "on how to convert to single" you can try to simply substitute the NULL character with nothing, using sed:

Code:

sed 's/\x00//g' filename > new_file

then something similar to strip the carriage returns "\r"... but as I told it's hard for me to predict the result.

colucix · 01-23-2008, 11:43 AM

Uh oh.. sorry. I went away for awhile before posting and didn't see the replay from bigearsbilly. Try the C solution, first (thanks, billy).

bigearsbilly · 01-23-2008, 12:05 PM

Quote:

I'm puzzled about the first two bytes.

well, maybe that and the null bytes is simply a pathetic microsoft attempt at obfuscation, we all
know how they hate users looking at anything.

I'll be interested to see what it is,
(I don't have any DOS OS's myself).

trscookie · 01-24-2008, 03:18 AM

Thanks everybody for the help, that little C program works a charm!

Thank you, trscookie