Help processing ngrep data needed!

GaWdLy · 10-08-2016, 10:46 AM

Hello all!

I am running a raspberry pi for some data collection and processing (a weatherstation), and in my shift to a homegrown software that runs on Linux, I've run into an issue with the data.

Since there's no easy way with my station to just send the data to another server, the packets must be sniffed out using tcpdump or ngrep, and then ncatted over to another local server port using nc like this:

ngrep -l -q -d eth0 '0004a369e0d6'|nc localhost 9999

In practice, here's the output:

~~~
ngrep -l -q -d eth0 '0004a369e0d6'

T 192.168.1.73:2079 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=292&wg=0.9&ws=0.0&ch=1&p=1

T 192.168.1.73:2080 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1006&ptr=0&wfor=0&p=1

T 192.168.1.73:2081 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=365&ch=1&p=1
~~~

But the initial data sent by the pipe breaks it:

~~~
<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: Bad request version ('(192.168.1.0/255.255.255.0)').
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>
~~~

Seems like the interface and match lines, as well as the 'T 192.168.1.73:2688 -> 54.228.205.188:80 [AP]' would break the software.

So my question is this: how would I remove that line? I don't see any options to do so within ngrep's documentation. So I was thinking possibly of piping the ngrep output into an intermediary step and having that line stripped from the output.

THe issue is that while I can implement it fairly easy, I guess I just need a bit of help with that intermediary step.

Let me know if you have ideas or questions.

Thanks!

GaWdLy · 10-08-2016, 10:51 AM

Put more concisely:

How to I take an active stream of data-say from an open ngrep command-and have all the nonessential data stripped as it makes its way to nc?

Turbocapitalist · 10-08-2016, 11:05 AM

"awk" or, maybe, "sed" might work. You show the data you have above. Can you show what it should look like by the time it gets piped to "nc" ?

GaWdLy · 10-08-2016, 02:09 PM

Quote:

Originally Posted by Turbocapitalist

"awk" or, maybe, "sed" might work. You show the data you have above. Can you show what it should look like by the time it gets piped to "nc" ?

Thanks. Yes, awk or sed would seem to be the right tools, but I don't know how to use them...I need training wheels

Here is the data:

~~~
# ngrep -l -q -d eth0 '0004a369e0d6'
interface: eth0 (192.168.1.0/255.255.255.0)
match: 0004a369e0d6

T 192.168.1.73:2688 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1007&ptr=1&wfor=0&p=1

T 192.168.1.73:2689 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.1&ch=3&p=1

T 192.168.1.73:2690 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=270&wg=0.9&ws=0.0&ch=1&p=1

T 192.168.1.73:2691 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=368&ch=1&p=1

T 192.168.1.73:2692 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.3&ch=3&p=1

T 192.168.1.73:2693 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=247&wg=0.9&ws=0.0&ch=1&p=1
~~~

Here is what the tool expects to see:

~~~
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1007&ptr=1&wfor=0&p=1
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.1&ch=3&p=1
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=270&wg=0.9&ws=0.0&ch=1&p=1
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=368&ch=1&p=1
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.3&ch=3&p=1
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=247&wg=0.9&ws=0.0&ch=1&p=1
~~~

So there are some extraneous lines to be removed.

Turbocapitalist · 10-08-2016, 02:20 PM

I'm not familiar enough with "ngrep", so maybe there is a way to whittle down the output on that end. Failing that, one can always tack on an extra "grep" using another pipe

Code:

ngrep -l -q -d eth0 '0004a369e0d6' | grep -E '^mac=' | nc localhost 9999

Check the manual page for "grep" about the -E and the pattern ^mac

Turbocapitalist · 10-09-2016, 12:04 AM

When you write the word data, I interpreted your question as wanting the text created by "ngrep"

Does the receiving program process text or raw packets?

Turbocapitalist · 10-09-2016, 09:54 PM

If you want raw packets from a specific MAC address using "ngrep" then you can specify using the ether host primitive

Code:

ngrep -l -q -w - -i eth0 ether host 00:04:a3:69:e0:d6 | nc localhost 9999

It looks like "ngrep" takes the same primitives as "tcpdump" or at least it takes most of them.