Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I know this is not exactly the kind of question a newbie should ask, but let me explain what this is all about:
I am teaching network administration fundamentals for a living and am preparing a lesson about network protocols and their flaws. For methodical reasons (to keep it simple and visual) I want to translate some programs I once wrote with C into bash scripts. My intention is to use hexinject to replace some of the socket I/O, because this way I can show my audience the data in a somewhat readable way. That said, I avow myself to using Backtrack 5 R1 and playing the network security game...
What I want to ask is purely about bash, gawk, pipelining and hexinject as the source of data, so I guess its okay to ask:
The script "isDTP" is the following
Code:
root@bt:~/scriptor# cat isDTP
#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
&& $4=="CC" && $5=="CC" && $6=="CC")
{
#test if it is OUI Cisco and Type DTP
if($18==00 && $19==00 && $20=="0C" \
&& $21==20 && $22==04)
print $0;
}
}'
I know I should test for SNAP as LLC protocol field, but I was too busy dealing with other problems. I'll deal with this one later.
The "testfile" is containing some DTP frames and some STP frames for stuffing. These dumps are output of hexinjext -s > testfile...
If I cat the testfile into the script, everything works as expected, making me believe the script is working well.
Quote:
root@bt:~/scriptor# cat testfile | ./isDTP > lala
root@bt:~/scriptor# cat lala
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
Piping hexinject -s into my filter skript is working fine too:
Quote:
root@bt:~/scriptor# hexinject -s | ./isDTP
01 00 1 CC CC CC 00 1D E5 B0 AD 08 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B0 AD 08 00 00 00 00 00 00 00 00 00 00 00 00
01 00 1 CC CC CC 00 1D E5 B2 7E 07 00 22 AA AA 03 00 00 0C 20 04 01 00 01 00 05 00 00 02 00 05 83 00 03 00 05 A5 00 04 00 0A 00 1D E5 B2 7E 07 00 00 00 00 00 00 00 00 00 00 00 00
^C
But once I put together the pieces, everything goes wrong:
Quote:
root@bt:~/scriptor# hexinject -s | ./isDTP > alal
^C
root@bt:~/scriptor# cat alal
root@bt:~/scriptor#
Nothing will ever show up in the file "alal". I don't expect privileges to be a problem as I am running root (only when using BT, I assure you).
So it seems to me hexinject -s and my filter script are compatible
and
it seems to me, my filter script and redirection to file are compatible.
Now I'm out of ideas what is going wrong with my code...
Unlike cat testfile, hexinject -s outputs a stream of hex with no terminating line end. That is why the tests using hexinject -s must be terminated using Ctrl+C. Awk processes its input line-by-line so, when reading from hexinject -s it never does read a whole line.
When Ctrl+C is used, bash dismantles the pipeline and, in the failing test case, the output redirection to alal. The exact sequence of dismantling is not defined in the GNU bash reference section on signals so the next part of this theory is speculative.
Presuming that bash created the processes in the pipleline as a process group, it sends SIGINT to them simultaneously. What happens next depends on how SIGINT handling is set up in hexinject and awk. Judging from the hexinject -s | ./isDTP test case, awk is able to process the data received. Perhaps its "read line" terminates when input from hexinject is closed. The critical difference is what happens to the awk standard output, written to either the terminal buffer or to write buffer for alal. Presumably in the first case the buffer contents are used and in the second case they are discarded.
So -- what to do? awk is a record-orientated program, not suited to reading streams of data without record separators (by default line ends). Presumably the packet payloads can contain arbitrary data so the only way to parse the data is by counting ... ?
Although your reply testifies to your deep insight in that topic, I am still not convinced, maybe because of some misconception I still have. In that case, I would highly appreciate to be corrected. Anyway here is my unqualified attempt of a rebuttal:
As far as I know the output of hexinject is containing new line characters. Each packet is printed on a single line separated from the others by a line feed. What makes me think this is the following:
For what it's worth, I see 6 LF charaters (0a, right?), which is exactly the number of lines in that file. So unless there is happening some magic I assume hexinject is outputting LFs.
Another point I would like to bring up is, that pipelining hexinject -s into the script via
Code:
hexinject -s | ./isDTP
workes well without the redirection to the file. The results you saw in my quoting were just in time, so each time hexinject was sniffing a DTP frame, split seconds later the console was updated and the output was shown. If this output was processed while dismantling, wouldn't that mean it only appears the second I Ctrl+C ?
Something I found right now while testing is, that
Code:
hexinject -s | xxd > somefile
performs as intended. Might be worth to notice, although xxd should be stream oriented istead of record based...
Something not working is
Code:
hexinject -s | ./isDTP | xxd
So I might guess something is wrong with my gawk print statement. Is seems to work only, if the results are going to straight to console...
For now I am out of ideas. Would still appreciate any help.
stdbuf seems to be in coreutils on my Linux Mint box. On BT 5 R1 it is not. At least it can't find the command although coreutils is installed properly.
Thanks for explaining that you saw output before using Ctrl+C and for the hexinject -s | ./isDTP | xxd test.
The gawk print statement working only when writing to terminal would not make sense but that's what the evidence suggests ... ?
Musing out loud, no clear insight yet ...
If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload? Do you have any such DTP packets?
stdout is buffered. What happens if you let hexinject -s | ./isDTP > alal run long enough for the output to fill the buffer and it to be flushed (IDK the buffer size)? Or route it via stderr so it's not buffered: hexinject -s | ./isDTP 2>alal >&2 (I am not certain that has the desired effect).
Grasping at straws here: is alal really empty or does it just show as empty at the command prompt? What size does ls show it? Does od or xxd show more than cat?
If hexinject does write one packet per line, how can the reader of its output distinguish between a line end inserted by hexinject as a packet boundary and one from the packet payload?
hexinject is returning pure ascii unless used with the -r switch for raw. So basically it reads raw data from the nic and then does the same to the data hexdump would do. The result is hex in pure ascii. This way, there are no LFs in it's output (except for those delimiting frames).
When you came up with the idea to start a real long test on
Code:
hexinject -s | ./isDTP > alal
I set up a script firing a DTP frame per second at my BackTrack box and ran this about 5 minutes. Buffering seems to be the problem as a ls -l resulted in
Code:
-rwxr-xr-x 1 root root 61440 2012-02-19 18:32 alal
This file is about 300 lines long which fits perfectly and contains exactly the data I was longing for. Running the same command about 20 seconds will result in an empty file. All ls -l, cat, od and xxd agree about it being empty...
This makes it very likely to be a buffering issue. Unfortunately
Code:
hexinject -s | ./isDTP 2>alal >&2
is not solving the issue either, even given 20 seconds. Waiting for that long is inacceptable for my project, as you can imagine.
I can hardly express how happy I am right now. Thank you very much for that advice.
The problem was indeed gawk buffering its output. As BackTrack 5 R1 gawk is not supporting
Code:
-W interactive
the use of
Code:
fflush()
did the trick.
The improved script looks like that and works like a charm:
Code:
#!/bin/bash
awk '{
#test if it is multicast for DTP and others
if($1==01 && $2==00 && $3="0C" \
&& $4=="CC" && $5=="CC" && $6=="CC")
{
#test if it is OUI Cisco and Type DTP
if($18==00 && $19==00 && $20=="0C" \
&& $21==20 && $22==04)
print $0;
fflush();
}
}'
Thumbs up for LinuxQuestions once again!
Regards,
Heraton
Last edited by Heraton; 02-20-2012 at 12:04 PM.
Reason: typo
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
"hexinject -s | script_with_gawk > file" won't output anything into "file"
here we go...
You made my day :-)
