LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   "hexinject -s | script_with_gawk > file" won't output anything into "file" (http://www.linuxquestions.org/questions/linux-newbie-8/hexinject-s-%7C-script_with_gawk-file-wont-output-anything-into-file-930123/)

Heraton 02-18-2012 09:02 PM

"hexinject -s | script_with_gawk > file" won't output anything into "file"
 
Hello everybody!

I know this is not exactly the kind of question a newbie should ask, but let me explain what this is all about:

I am teaching network administration fundamentals for a living and am preparing a lesson about network protocols and their flaws. For methodical reasons (to keep it simple and visual) I want to translate some programs I once wrote with C into bash scripts. My intention is to use hexinject to replace some of the socket I/O, because this way I can show my audience the data in a somewhat readable way. That said, I avow myself to using Backtrack 5 R1 and playing the network security game...

What I want to ask is purely about bash, gawk, pipelining and hexinject as the source of data, so I guess its okay to ask:

The script "isDTP" is the following
Code:

root@bt:~/scriptor# cat isDTP
#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
 && $4=="CC" && $5=="CC"  && $6=="CC")
{
        #test if it is OUI Cisco and Type DTP
        if($18==00 && $19==00 && $20=="0C" \
        && $21==20 && $22==04)
                print $0;
}
}'

I know I should test for SNAP as LLC protocol field, but I was too busy dealing with other problems. I'll deal with this one later.

The "testfile" is containing some DTP frames and some STP frames for stuffing. These dumps are output of hexinjext -s > testfile...

If I cat the testfile into the script, everything works as expected, making me believe the script is working well.
Quote:

root@bt:~/scriptor# cat testfile | ./isDTP > lala
root@bt:~/scriptor# cat lala
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
Piping hexinject -s into my filter skript is working fine too:
Quote:

root@bt:~/scriptor# hexinject -s | ./isDTP
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
^C
But once I put together the pieces, everything goes wrong:
Quote:

root@bt:~/scriptor# hexinject -s | ./isDTP > alal
^C
root@bt:~/scriptor# cat alal
root@bt:~/scriptor#
Nothing will ever show up in the file "alal". I don't expect privileges to be a problem as I am running root (only when using BT, I assure you).

So it seems to me hexinject -s and my filter script are compatible
and
it seems to me, my filter script and redirection to file are compatible.

Now I'm out of ideas what is going wrong with my code...


For suggestions or solutions I would be grateful.

Regards, Heraton

uhelp 02-18-2012 09:45 PM

Jsut a qustion of proper grouping, I think.
Try this:

Code:

( hexinject -s | ./isDTP  ) > alal

Heraton 02-18-2012 10:09 PM

Hello, uhelp. I tried your suggestion, but with no success. Thanks anyway.

Regards, Heraton

catkin 02-19-2012 01:17 AM

I suspect that what is happening is ...

Unlike cat testfile, hexinject -s outputs a stream of hex with no terminating line end. That is why the tests using hexinject -s must be terminated using Ctrl+C. Awk processes its input line-by-line so, when reading from hexinject -s it never does read a whole line.

When Ctrl+C is used, bash dismantles the pipeline and, in the failing test case, the output redirection to alal. The exact sequence of dismantling is not defined in the GNU bash reference section on signals so the next part of this theory is speculative.

Presuming that bash created the processes in the pipleline as a process group, it sends SIGINT to them simultaneously. What happens next depends on how SIGINT handling is set up in hexinject and awk. Judging from the hexinject -s | ./isDTP test case, awk is able to process the data received. Perhaps its "read line" terminates when input from hexinject is closed. The critical difference is what happens to the awk standard output, written to either the terminal buffer or to write buffer for alal. Presumably in the first case the buffer contents are used and in the second case they are discarded.

So -- what to do? awk is a record-orientated program, not suited to reading streams of data without record separators (by default line ends). Presumably the packet payloads can contain arbitrary data so the only way to parse the data is by counting ... ?

Heraton 02-19-2012 07:40 AM

still in doubt..
 
Dear catkin!

Although your reply testifies to your deep insight in that topic, I am still not convinced, maybe because of some misconception I still have. In that case, I would highly appreciate to be corrected. Anyway here is my unqualified attempt of a rebuttal:


As far as I know the output of hexinject is containing new line characters. Each packet is printed on a single line separated from the others by a line feed. What makes me think this is the following:

testfile is looking like this:
Code:

01 00 0C CC CC CD 00 1D E5 B0 AD 08 00 32 AA AA 03 00 00 0C 01 0B 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 02 00 01
01 80 C2 00 00 00 00 1D E5 B0 AD 08 00 26 42 42 03 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CD 00 1D E5 B0 AD 08 00 32 AA AA 03 00 00 0C 01 0B 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 02 00 01
01 80 C2 00 00 00 00 1D E5 B0 AD 08 00 26 42 42 03 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00

I know this is resulting in ugly scroll bars, but formatting might be important...

A hexdump of this file (I used xxd) is looking like that:
Code:

root@bt:~# xxd testfile
0000000: 3031 2030 3020 3043 2043 4320 4343 2043  01 00 0C CC CC C
0000010: 4420 3030 2031 4420 4535 2042 3020 4144  D 00 1D E5 B0 AD
0000020: 2030 3820 3030 2033 3220 4141 2041 4120  08 00 32 AA AA
0000030: 3033 2030 3020 3030 2030 4320 3031 2030  03 00 00 0C 01 0
0000040: 4220 3030 2030 3020 3030 2030 3020 3030  B 00 00 00 00 00
0000050: 2038 3020 3031 2030 3020 3144 2045 3520  80 01 00 1D E5
0000060: 4230 2041 4420 3030 2030 3020 3030 2030  B0 AD 00 00 00 0
0000070: 3020 3030 2038 3020 3031 2030 3020 3144  0 00 80 01 00 1D
0000080: 2045 3520 4230 2041 4420 3030 2038 3020  E5 B0 AD 00 80
0000090: 3038 2030 3020 3030 2031 3420 3030 2030  08 00 00 14 00 0
00000a0: 3220 3030 2030 4620 3030 2030 3020 3030  2 00 0F 00 00 00
00000b0: 2030 3020 3030 2030 3220 3030 2030 310a  00 00 02 00 01.
00000c0: 3031 2038 3020 4332 2030 3020 3030 2030  01 80 C2 00 00 0
00000d0: 3020 3030 2031 4420 4535 2042 3020 4144  0 00 1D E5 B0 AD
00000e0: 2030 3820 3030 2032 3620 3432 2034 3220  08 00 26 42 42
00000f0: 3033 2030 3020 3030 2030 3020 3030 2030  03 00 00 00 00 0
0000100: 3020 3830 2030 3120 3030 2031 4420 4535  0 80 01 00 1D E5
0000110: 2042 3020 4144 2030 3020 3030 2030 3020  B0 AD 00 00 00
0000120: 3030 2030 3020 3830 2030 3120 3030 2031  00 00 80 01 00 1
0000130: 4420 4535 2042 3020 4144 2030 3020 3830  D E5 B0 AD 00 80
0000140: 2030 3820 3030 2030 3020 3134 2030 3020  08 00 00 14 00
0000150: 3032 2030 3020 3046 2030 3020 3030 2030  02 00 0F 00 00 0
0000160: 3020 3030 2030 3020 3030 2030 3020 3030  0 00 00 00 00 00
0000170: 2030 300a 3031 2030 3020 3043 2043 4320  00.01 00 0C CC
0000180: 4343 2043 4420 3030 2031 4420 4535 2042  CC CD 00 1D E5 B
0000190: 3020 4144 2030 3820 3030 2033 3220 4141  0 AD 08 00 32 AA
00001a0: 2041 4120 3033 2030 3020 3030 2030 4320  AA 03 00 00 0C
00001b0: 3031 2030 4220 3030 2030 3020 3030 2030  01 0B 00 00 00 0
00001c0: 3020 3030 2038 3020 3031 2030 3020 3144  0 00 80 01 00 1D
00001d0: 2045 3520 4230 2041 4420 3030 2030 3020  E5 B0 AD 00 00
00001e0: 3030 2030 3020 3030 2038 3020 3031 2030  00 00 00 80 01 0
00001f0: 3020 3144 2045 3520 4230 2041 4420 3030  0 1D E5 B0 AD 00
0000200: 2038 3020 3038 2030 3020 3030 2031 3420  80 08 00 00 14
0000210: 3030 2030 3220 3030 2030 4620 3030 2030  00 02 00 0F 00 0
0000220: 3020 3030 2030 3020 3030 2030 3220 3030  0 00 00 00 02 00
0000230: 2030 310a 3031 2038 3020 4332 2030 3020  01.01 80 C2 00
0000240: 3030 2030 3020 3030 2031 4420 4535 2042  00 00 00 1D E5 B
0000250: 3020 4144 2030 3820 3030 2032 3620 3432  0 AD 08 00 26 42
0000260: 2034 3220 3033 2030 3020 3030 2030 3020  42 03 00 00 00
0000270: 3030 2030 3020 3830 2030 3120 3030 2031  00 00 80 01 00 1
0000280: 4420 4535 2042 3020 4144 2030 3020 3030  D E5 B0 AD 00 00
0000290: 2030 3020 3030 2030 3020 3830 2030 3120  00 00 00 80 01
00002a0: 3030 2031 4420 4535 2042 3020 4144 2030  00 1D E5 B0 AD 0
00002b0: 3020 3830 2030 3820 3030 2030 3020 3134  0 80 08 00 00 14
00002c0: 2030 3020 3032 2030 3020 3046 2030 3020  00 02 00 0F 00
00002d0: 3030 2030 3020 3030 2030 3020 3030 2030  00 00 00 00 00 0
00002e0: 3020 3030 2030 300a 3031 2030 3020 3043  0 00 00.01 00 0C
00002f0: 2043 4320 4343 2043 4320 3030 2031 4420  CC CC CC 00 1D
0000300: 4535 2042 3020 4144 2030 3820 3030 2032  E5 B0 AD 08 00 2
0000310: 3220 4141 2041 4120 3033 2030 3020 3030  2 AA AA 03 00 00
0000320: 2030 4320 3230 2030 3420 3031 2030 3020  0C 20 04 01 00
0000330: 3031 2030 3020 3035 2030 3020 3030 2030  01 00 05 00 00 0
0000340: 3220 3030 2030 3520 3833 2030 3020 3033  2 00 05 83 00 03
0000350: 2030 3020 3035 2041 3520 3030 2030 3420  00 05 A5 00 04
0000360: 3030 2030 4120 3030 2031 4420 4535 2042  00 0A 00 1D E5 B
0000370: 3020 4144 2030 3820 3030 2030 3020 3030  0 AD 08 00 00 00
0000380: 2030 3020 3030 2030 3020 3030 2030 3020  00 00 00 00 00
0000390: 3030 2030 3020 3030 2030 300a 3031 2030  00 00 00 00.01 0
00003a0: 3020 3043 2043 4320 4343 2043 4320 3030  0 0C CC CC CC 00
00003b0: 2031 4420 4535 2042 3220 3745 2030 3720  1D E5 B2 7E 07
00003c0: 3030 2032 3220 4141 2041 4120 3033 2030  00 22 AA AA 03 0
00003d0: 3020 3030 2030 4320 3230 2030 3420 3031  0 00 0C 20 04 01
00003e0: 2030 3020 3031 2030 3020 3035 2030 3020  00 01 00 05 00
00003f0: 3030 2030 3220 3030 2030 3520 3833 2030  00 02 00 05 83 0
0000400: 3020 3033 2030 3020 3035 2041 3520 3030  0 03 00 05 A5 00
0000410: 2030 3420 3030 2030 4120 3030 2031 4420  04 00 0A 00 1D
0000420: 4535 2042 3220 3745 2030 3720 3030 2030  E5 B2 7E 07 00 0
0000430: 3020 3030 2030 3020 3030 2030 3020 3030  0 00 00 00 00 00
0000440: 2030 3020 3030 2030 3020 3030 2030 300a  00 00 00 00 00.

When I run
Code:

hexinject -s | xxd
I get the same result, including the 0x0a bytes.

For what it's worth, I see 6 LF charaters (0a, right?), which is exactly the number of lines in that file. So unless there is happening some magic I assume hexinject is outputting LFs.

Another point I would like to bring up is, that pipelining hexinject -s into the script via
Code:

hexinject -s | ./isDTP
workes well without the redirection to the file. The results you saw in my quoting were just in time, so each time hexinject was sniffing a DTP frame, split seconds later the console was updated and the output was shown. If this output was processed while dismantling, wouldn't that mean it only appears the second I Ctrl+C ?

Something I found right now while testing is, that
Code:

hexinject -s | xxd > somefile
performs as intended. Might be worth to notice, although xxd should be stream oriented istead of record based...

Something not working is
Code:

hexinject -s | ./isDTP | xxd
So I might guess something is wrong with my gawk print statement. Is seems to work only, if the results are going to straight to console...

For now I am out of ideas. Would still appreciate any help.


Regards, Heraton

uhelp 02-19-2012 07:58 AM

Sounds like a buffering issue within pipes.
Try "stdbuf"
A GNU tool to disable this buffering.

Heraton 02-19-2012 08:16 AM

Hi!

stdbuf seems to be in coreutils on my Linux Mint box. On BT 5 R1 it is not. At least it can't find the command although coreutils is installed properly.

Suggestions?

Heraton

catkin 02-19-2012 11:04 AM

Hello Heraton :)

Yes -- line end is 0xa.

Thanks for explaining that you saw output before using Ctrl+C and for the hexinject -s | ./isDTP | xxd test.

The gawk print statement working only when writing to terminal would not make sense but that's what the evidence suggests ... ?

Musing out loud, no clear insight yet ...

If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload? Do you have any such DTP packets?

stdout is buffered. What happens if you let hexinject -s | ./isDTP > alal run long enough for the output to fill the buffer and it to be flushed (IDK the buffer size)? Or route it via stderr so it's not buffered: hexinject -s | ./isDTP 2>alal >&2 (I am not certain that has the desired effect).

Grasping at straws here: is alal really empty or does it just show as empty at the command prompt? What size does ls show it? Does od or xxd show more than cat?

Heraton 02-19-2012 11:54 AM

here we go...
 
Quote:

If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload?
hexinject is returning pure ascii unless used with the -r switch for raw. So basically it reads raw data from the nic and then does the same to the data hexdump would do. The result is hex in pure ascii. This way, there are no LFs in it's output (except for those delimiting frames).


When you came up with the idea to start a real long test on
Code:

hexinject -s | ./isDTP > alal
I set up a script firing a DTP frame per second at my BackTrack box and ran this about 5 minutes. Buffering seems to be the problem as a ls -l resulted in
Code:

-rwxr-xr-x  1 root root 61440 2012-02-19 18:32 alal
This file is about 300 lines long which fits perfectly and contains exactly the data I was longing for. Running the same command about 20 seconds will result in an empty file. All ls -l, cat, od and xxd agree about it being empty...


This makes it very likely to be a buffering issue. Unfortunately
Code:

hexinject -s | ./isDTP 2>alal >&2
is not solving the issue either, even given 20 seconds. Waiting for that long is inacceptable for my project, as you can imagine.

Anyway, where getting close.


Thanks a lot,

Heraton

chrism01 02-19-2012 07:14 PM

There's some good tricks to unbuffer here http://mywiki.wooledge.org/BashFAQ/009
HTH :)

Heraton 02-20-2012 12:02 PM

You made my day :-)
 
Dear chrism01!

I can hardly express how happy I am right now. Thank you very much for that advice.

The problem was indeed gawk buffering its output. As BackTrack 5 R1 gawk is not supporting
Code:

-W interactive
the use of
Code:

fflush()
did the trick.


The improved script looks like that and works like a charm:
Code:

#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
 && $4=="CC" && $5=="CC"  && $6=="CC")
{
        #test if it is OUI Cisco and Type DTP
        if($18==00 && $19==00 && $20=="0C" \
        && $21==20 && $22==04)
                print $0;
                fflush();
}
}'

Thumbs up for LinuxQuestions once again!


Regards,

Heraton

catkin 02-20-2012 01:26 PM

Glad you found a solution Heraton and thanks for sharing it :)


All times are GMT -5. The time now is 09:25 PM.