LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 02-18-2012, 09:02 PM   #1
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Rep: Reputation: 3
Question "hexinject -s | script_with_gawk > file" won't output anything into "file"


Hello everybody!

I know this is not exactly the kind of question a newbie should ask, but let me explain what this is all about:

I am teaching network administration fundamentals for a living and am preparing a lesson about network protocols and their flaws. For methodical reasons (to keep it simple and visual) I want to translate some programs I once wrote with C into bash scripts. My intention is to use hexinject to replace some of the socket I/O, because this way I can show my audience the data in a somewhat readable way. That said, I avow myself to using Backtrack 5 R1 and playing the network security game...

What I want to ask is purely about bash, gawk, pipelining and hexinject as the source of data, so I guess its okay to ask:

The script "isDTP" is the following
Code:
root@bt:~/scriptor# cat isDTP 
#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
 && $4=="CC" && $5=="CC"  && $6=="CC")
{
	#test if it is OUI Cisco and Type DTP
	if($18==00 && $19==00 && $20=="0C" \
        && $21==20 && $22==04)
		print $0;
}
}'
I know I should test for SNAP as LLC protocol field, but I was too busy dealing with other problems. I'll deal with this one later.

The "testfile" is containing some DTP frames and some STP frames for stuffing. These dumps are output of hexinjext -s > testfile...

If I cat the testfile into the script, everything works as expected, making me believe the script is working well.
Quote:
root@bt:~/scriptor# cat testfile | ./isDTP > lala
root@bt:~/scriptor# cat lala
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
Piping hexinject -s into my filter skript is working fine too:
Quote:
root@bt:~/scriptor# hexinject -s | ./isDTP
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
^C
But once I put together the pieces, everything goes wrong:
Quote:
root@bt:~/scriptor# hexinject -s | ./isDTP > alal
^C
root@bt:~/scriptor# cat alal
root@bt:~/scriptor#
Nothing will ever show up in the file "alal". I don't expect privileges to be a problem as I am running root (only when using BT, I assure you).

So it seems to me hexinject -s and my filter script are compatible
and
it seems to me, my filter script and redirection to file are compatible.

Now I'm out of ideas what is going wrong with my code...


For suggestions or solutions I would be grateful.

Regards, Heraton
 
Old 02-18-2012, 09:45 PM   #2
uhelp
Member
 
Registered: Nov 2011
Location: Germany, Bavaria, Nueremberg area
Distribution: openSUSE, Debian, LFS
Posts: 205

Rep: Reputation: 43
Jsut a qustion of proper grouping, I think.
Try this:

Code:
( hexinject -s | ./isDTP  ) > alal
 
Old 02-18-2012, 10:09 PM   #3
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Original Poster
Rep: Reputation: 3
Smile

Hello, uhelp. I tried your suggestion, but with no success. Thanks anyway.

Regards, Heraton
 
Old 02-19-2012, 01:17 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,544
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
I suspect that what is happening is ...

Unlike cat testfile, hexinject -s outputs a stream of hex with no terminating line end. That is why the tests using hexinject -s must be terminated using Ctrl+C. Awk processes its input line-by-line so, when reading from hexinject -s it never does read a whole line.

When Ctrl+C is used, bash dismantles the pipeline and, in the failing test case, the output redirection to alal. The exact sequence of dismantling is not defined in the GNU bash reference section on signals so the next part of this theory is speculative.

Presuming that bash created the processes in the pipleline as a process group, it sends SIGINT to them simultaneously. What happens next depends on how SIGINT handling is set up in hexinject and awk. Judging from the hexinject -s | ./isDTP test case, awk is able to process the data received. Perhaps its "read line" terminates when input from hexinject is closed. The critical difference is what happens to the awk standard output, written to either the terminal buffer or to write buffer for alal. Presumably in the first case the buffer contents are used and in the second case they are discarded.

So -- what to do? awk is a record-orientated program, not suited to reading streams of data without record separators (by default line ends). Presumably the packet payloads can contain arbitrary data so the only way to parse the data is by counting ... ?
 
Old 02-19-2012, 07:40 AM   #5
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Original Poster
Rep: Reputation: 3
Question still in doubt..

Dear catkin!

Although your reply testifies to your deep insight in that topic, I am still not convinced, maybe because of some misconception I still have. In that case, I would highly appreciate to be corrected. Anyway here is my unqualified attempt of a rebuttal:


As far as I know the output of hexinject is containing new line characters. Each packet is printed on a single line separated from the others by a line feed. What makes me think this is the following:

testfile is looking like this:
Code:
01 00 0C CC CC CD 00 1D E5 B0 AD 08 00 32 AA AA 03 00 00 0C 01 0B 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 02 00 01
01 80 C2 00 00 00 00 1D E5 B0 AD 08 00 26 42 42 03 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CD 00 1D E5 B0 AD 08 00 32 AA AA 03 00 00 0C 01 0B 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 02 00 01
01 80 C2 00 00 00 00 1D E5 B0 AD 08 00 26 42 42 03 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 00 00 00 00 80 01 00 1D E5 B0 AD 00 80 08 00 00 14 00 02 00 0F 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 0C CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
I know this is resulting in ugly scroll bars, but formatting might be important...

A hexdump of this file (I used xxd) is looking like that:
Code:
root@bt:~# xxd testfile 
0000000: 3031 2030 3020 3043 2043 4320 4343 2043  01 00 0C CC CC C
0000010: 4420 3030 2031 4420 4535 2042 3020 4144  D 00 1D E5 B0 AD
0000020: 2030 3820 3030 2033 3220 4141 2041 4120   08 00 32 AA AA 
0000030: 3033 2030 3020 3030 2030 4320 3031 2030  03 00 00 0C 01 0
0000040: 4220 3030 2030 3020 3030 2030 3020 3030  B 00 00 00 00 00
0000050: 2038 3020 3031 2030 3020 3144 2045 3520   80 01 00 1D E5 
0000060: 4230 2041 4420 3030 2030 3020 3030 2030  B0 AD 00 00 00 0
0000070: 3020 3030 2038 3020 3031 2030 3020 3144  0 00 80 01 00 1D
0000080: 2045 3520 4230 2041 4420 3030 2038 3020   E5 B0 AD 00 80 
0000090: 3038 2030 3020 3030 2031 3420 3030 2030  08 00 00 14 00 0
00000a0: 3220 3030 2030 4620 3030 2030 3020 3030  2 00 0F 00 00 00
00000b0: 2030 3020 3030 2030 3220 3030 2030 310a   00 00 02 00 01.
00000c0: 3031 2038 3020 4332 2030 3020 3030 2030  01 80 C2 00 00 0
00000d0: 3020 3030 2031 4420 4535 2042 3020 4144  0 00 1D E5 B0 AD
00000e0: 2030 3820 3030 2032 3620 3432 2034 3220   08 00 26 42 42 
00000f0: 3033 2030 3020 3030 2030 3020 3030 2030  03 00 00 00 00 0
0000100: 3020 3830 2030 3120 3030 2031 4420 4535  0 80 01 00 1D E5
0000110: 2042 3020 4144 2030 3020 3030 2030 3020   B0 AD 00 00 00 
0000120: 3030 2030 3020 3830 2030 3120 3030 2031  00 00 80 01 00 1
0000130: 4420 4535 2042 3020 4144 2030 3020 3830  D E5 B0 AD 00 80
0000140: 2030 3820 3030 2030 3020 3134 2030 3020   08 00 00 14 00 
0000150: 3032 2030 3020 3046 2030 3020 3030 2030  02 00 0F 00 00 0
0000160: 3020 3030 2030 3020 3030 2030 3020 3030  0 00 00 00 00 00
0000170: 2030 300a 3031 2030 3020 3043 2043 4320   00.01 00 0C CC 
0000180: 4343 2043 4420 3030 2031 4420 4535 2042  CC CD 00 1D E5 B
0000190: 3020 4144 2030 3820 3030 2033 3220 4141  0 AD 08 00 32 AA
00001a0: 2041 4120 3033 2030 3020 3030 2030 4320   AA 03 00 00 0C 
00001b0: 3031 2030 4220 3030 2030 3020 3030 2030  01 0B 00 00 00 0
00001c0: 3020 3030 2038 3020 3031 2030 3020 3144  0 00 80 01 00 1D
00001d0: 2045 3520 4230 2041 4420 3030 2030 3020   E5 B0 AD 00 00 
00001e0: 3030 2030 3020 3030 2038 3020 3031 2030  00 00 00 80 01 0
00001f0: 3020 3144 2045 3520 4230 2041 4420 3030  0 1D E5 B0 AD 00
0000200: 2038 3020 3038 2030 3020 3030 2031 3420   80 08 00 00 14 
0000210: 3030 2030 3220 3030 2030 4620 3030 2030  00 02 00 0F 00 0
0000220: 3020 3030 2030 3020 3030 2030 3220 3030  0 00 00 00 02 00
0000230: 2030 310a 3031 2038 3020 4332 2030 3020   01.01 80 C2 00 
0000240: 3030 2030 3020 3030 2031 4420 4535 2042  00 00 00 1D E5 B
0000250: 3020 4144 2030 3820 3030 2032 3620 3432  0 AD 08 00 26 42
0000260: 2034 3220 3033 2030 3020 3030 2030 3020   42 03 00 00 00 
0000270: 3030 2030 3020 3830 2030 3120 3030 2031  00 00 80 01 00 1
0000280: 4420 4535 2042 3020 4144 2030 3020 3030  D E5 B0 AD 00 00
0000290: 2030 3020 3030 2030 3020 3830 2030 3120   00 00 00 80 01 
00002a0: 3030 2031 4420 4535 2042 3020 4144 2030  00 1D E5 B0 AD 0
00002b0: 3020 3830 2030 3820 3030 2030 3020 3134  0 80 08 00 00 14
00002c0: 2030 3020 3032 2030 3020 3046 2030 3020   00 02 00 0F 00 
00002d0: 3030 2030 3020 3030 2030 3020 3030 2030  00 00 00 00 00 0
00002e0: 3020 3030 2030 300a 3031 2030 3020 3043  0 00 00.01 00 0C
00002f0: 2043 4320 4343 2043 4320 3030 2031 4420   CC CC CC 00 1D 
0000300: 4535 2042 3020 4144 2030 3820 3030 2032  E5 B0 AD 08 00 2
0000310: 3220 4141 2041 4120 3033 2030 3020 3030  2 AA AA 03 00 00
0000320: 2030 4320 3230 2030 3420 3031 2030 3020   0C 20 04 01 00 
0000330: 3031 2030 3020 3035 2030 3020 3030 2030  01 00 05 00 00 0
0000340: 3220 3030 2030 3520 3833 2030 3020 3033  2 00 05 83 00 03
0000350: 2030 3020 3035 2041 3520 3030 2030 3420   00 05 A5 00 04 
0000360: 3030 2030 4120 3030 2031 4420 4535 2042  00 0A 00 1D E5 B
0000370: 3020 4144 2030 3820 3030 2030 3020 3030  0 AD 08 00 00 00
0000380: 2030 3020 3030 2030 3020 3030 2030 3020   00 00 00 00 00 
0000390: 3030 2030 3020 3030 2030 300a 3031 2030  00 00 00 00.01 0
00003a0: 3020 3043 2043 4320 4343 2043 4320 3030  0 0C CC CC CC 00
00003b0: 2031 4420 4535 2042 3220 3745 2030 3720   1D E5 B2 7E 07 
00003c0: 3030 2032 3220 4141 2041 4120 3033 2030  00 22 AA AA 03 0
00003d0: 3020 3030 2030 4320 3230 2030 3420 3031  0 00 0C 20 04 01
00003e0: 2030 3020 3031 2030 3020 3035 2030 3020   00 01 00 05 00 
00003f0: 3030 2030 3220 3030 2030 3520 3833 2030  00 02 00 05 83 0
0000400: 3020 3033 2030 3020 3035 2041 3520 3030  0 03 00 05 A5 00
0000410: 2030 3420 3030 2030 4120 3030 2031 4420   04 00 0A 00 1D 
0000420: 4535 2042 3220 3745 2030 3720 3030 2030  E5 B2 7E 07 00 0
0000430: 3020 3030 2030 3020 3030 2030 3020 3030  0 00 00 00 00 00
0000440: 2030 3020 3030 2030 3020 3030 2030 300a   00 00 00 00 00.
When I run
Code:
hexinject -s | xxd
I get the same result, including the 0x0a bytes.

For what it's worth, I see 6 LF charaters (0a, right?), which is exactly the number of lines in that file. So unless there is happening some magic I assume hexinject is outputting LFs.

Another point I would like to bring up is, that pipelining hexinject -s into the script via
Code:
hexinject -s | ./isDTP
workes well without the redirection to the file. The results you saw in my quoting were just in time, so each time hexinject was sniffing a DTP frame, split seconds later the console was updated and the output was shown. If this output was processed while dismantling, wouldn't that mean it only appears the second I Ctrl+C ?

Something I found right now while testing is, that
Code:
hexinject -s | xxd > somefile
performs as intended. Might be worth to notice, although xxd should be stream oriented istead of record based...

Something not working is
Code:
hexinject -s | ./isDTP | xxd
So I might guess something is wrong with my gawk print statement. Is seems to work only, if the results are going to straight to console...

For now I am out of ideas. Would still appreciate any help.


Regards, Heraton
 
Old 02-19-2012, 07:58 AM   #6
uhelp
Member
 
Registered: Nov 2011
Location: Germany, Bavaria, Nueremberg area
Distribution: openSUSE, Debian, LFS
Posts: 205

Rep: Reputation: 43
Sounds like a buffering issue within pipes.
Try "stdbuf"
A GNU tool to disable this buffering.
 
Old 02-19-2012, 08:16 AM   #7
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Original Poster
Rep: Reputation: 3
Hi!

stdbuf seems to be in coreutils on my Linux Mint box. On BT 5 R1 it is not. At least it can't find the command although coreutils is installed properly.

Suggestions?

Heraton
 
Old 02-19-2012, 11:04 AM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,544
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Hello Heraton

Yes -- line end is 0xa.

Thanks for explaining that you saw output before using Ctrl+C and for the hexinject -s | ./isDTP | xxd test.

The gawk print statement working only when writing to terminal would not make sense but that's what the evidence suggests ... ?

Musing out loud, no clear insight yet ...

If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload? Do you have any such DTP packets?

stdout is buffered. What happens if you let hexinject -s | ./isDTP > alal run long enough for the output to fill the buffer and it to be flushed (IDK the buffer size)? Or route it via stderr so it's not buffered: hexinject -s | ./isDTP 2>alal >&2 (I am not certain that has the desired effect).

Grasping at straws here: is alal really empty or does it just show as empty at the command prompt? What size does ls show it? Does od or xxd show more than cat?
 
1 members found this post helpful.
Old 02-19-2012, 11:54 AM   #9
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Original Poster
Rep: Reputation: 3
Lightbulb here we go...

Quote:
If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload?
hexinject is returning pure ascii unless used with the -r switch for raw. So basically it reads raw data from the nic and then does the same to the data hexdump would do. The result is hex in pure ascii. This way, there are no LFs in it's output (except for those delimiting frames).


When you came up with the idea to start a real long test on
Code:
hexinject -s | ./isDTP > alal
I set up a script firing a DTP frame per second at my BackTrack box and ran this about 5 minutes. Buffering seems to be the problem as a ls -l resulted in
Code:
-rwxr-xr-x  1 root root 61440 2012-02-19 18:32 alal
This file is about 300 lines long which fits perfectly and contains exactly the data I was longing for. Running the same command about 20 seconds will result in an empty file. All ls -l, cat, od and xxd agree about it being empty...


This makes it very likely to be a buffering issue. Unfortunately
Code:
hexinject -s | ./isDTP 2>alal >&2
is not solving the issue either, even given 20 seconds. Waiting for that long is inacceptable for my project, as you can imagine.

Anyway, where getting close.


Thanks a lot,

Heraton
 
Old 02-19-2012, 07:14 PM   #10
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
There's some good tricks to unbuffer here http://mywiki.wooledge.org/BashFAQ/009
HTH
 
2 members found this post helpful.
Old 02-20-2012, 12:02 PM   #11
Heraton
Member
 
Registered: Apr 2011
Location: Germany
Distribution: Mint 10, openSuSE
Posts: 58

Original Poster
Rep: Reputation: 3
Thumbs up You made my day :-)

Dear chrism01!

I can hardly express how happy I am right now. Thank you very much for that advice.

The problem was indeed gawk buffering its output. As BackTrack 5 R1 gawk is not supporting
Code:
-W interactive
the use of
Code:
fflush()
did the trick.


The improved script looks like that and works like a charm:
Code:
#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
 && $4=="CC" && $5=="CC"  && $6=="CC")
{
	#test if it is OUI Cisco and Type DTP
	if($18==00 && $19==00 && $20=="0C" \
        && $21==20 && $22==04)
		print $0;
		fflush();
}
}'
Thumbs up for LinuxQuestions once again!


Regards,

Heraton

Last edited by Heraton; 02-20-2012 at 12:04 PM. Reason: typo
 
Old 02-20-2012, 01:26 PM   #12
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,544
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Glad you found a solution Heraton and thanks for sharing it
 
  


Reply

Tags
file, gawk, pipe, redirection


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how can I "cat" or "grep" a file to ignore lines starting with "#" ??? callagga Linux - Newbie 7 08-16-2013 06:58 AM
unpredictable "delete" "move to trash" or "cut" file menu option dorianrenato Linux - General 3 11-28-2011 06:41 PM
[SOLVED] Errors executing shell script: "command not found" and "no such file or directory" eko000 Linux - Newbie 1 01-14-2011 07:54 AM
"bad tty" & "file descriptor error" while building RPM on F13 gosssamer Linux - Software 7 12-28-2010 05:02 PM
Hostname defined in file "network" changes after change to file "hosts" qwkfish Linux - Networking 2 10-14-2009 04:07 PM


All times are GMT -5. The time now is 11:48 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration