LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-08-2016, 10:46 AM   #1
GaWdLy
Member
 
Registered: Feb 2013
Location: San Jose, CA
Distribution: RHEL/CentOS/Fedora
Posts: 457

Rep: Reputation: Disabled
Help processing ngrep data needed!


Hello all!

I am running a raspberry pi for some data collection and processing (a weatherstation), and in my shift to a homegrown software that runs on Linux, I've run into an issue with the data.

Since there's no easy way with my station to just send the data to another server, the packets must be sniffed out using tcpdump or ngrep, and then ncatted over to another local server port using nc like this:

ngrep -l -q -d eth0 '0004a369e0d6'|nc localhost 9999

In practice, here's the output:

~~~
ngrep -l -q -d eth0 '0004a369e0d6'

T 192.168.1.73:2079 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=292&wg=0.9&ws=0.0&ch=1&p=1

T 192.168.1.73:2080 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1006&ptr=0&wfor=0&p=1

T 192.168.1.73:2081 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=365&ch=1&p=1
~~~

But the initial data sent by the pipe breaks it:

~~~
<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: Bad request version ('(192.168.1.0/255.255.255.0)').
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>
~~~

Seems like the interface and match lines, as well as the 'T 192.168.1.73:2688 -> 54.228.205.188:80 [AP]' would break the software.

So my question is this: how would I remove that line? I don't see any options to do so within ngrep's documentation. So I was thinking possibly of piping the ngrep output into an intermediary step and having that line stripped from the output.

THe issue is that while I can implement it fairly easy, I guess I just need a bit of help with that intermediary step.

Let me know if you have ideas or questions.

Thanks!

Last edited by GaWdLy; 10-08-2016 at 10:51 AM.
 
Old 10-08-2016, 10:51 AM   #2
GaWdLy
Member
 
Registered: Feb 2013
Location: San Jose, CA
Distribution: RHEL/CentOS/Fedora
Posts: 457

Original Poster
Rep: Reputation: Disabled
Put more concisely:

How to I take an active stream of data-say from an open ngrep command-and have all the nonessential data stripped as it makes its way to nc?
 
Old 10-08-2016, 11:05 AM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,854
Blog Entries: 3

Rep: Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040
"awk" or, maybe, "sed" might work. You show the data you have above. Can you show what it should look like by the time it gets piped to "nc" ?
 
Old 10-08-2016, 02:09 PM   #4
GaWdLy
Member
 
Registered: Feb 2013
Location: San Jose, CA
Distribution: RHEL/CentOS/Fedora
Posts: 457

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
"awk" or, maybe, "sed" might work. You show the data you have above. Can you show what it should look like by the time it gets piped to "nc" ?
Thanks. Yes, awk or sed would seem to be the right tools, but I don't know how to use them...I need training wheels

Here is the data:

~~~
# ngrep -l -q -d eth0 '0004a369e0d6'
interface: eth0 (192.168.1.0/255.255.255.0)
match: 0004a369e0d6

T 192.168.1.73:2688 -> 54.228.205.188:80 [AP]
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1007&ptr=1&wfor=0&p=1

T 192.168.1.73:2689 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.1&ch=3&p=1

T 192.168.1.73:2690 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=270&wg=0.9&ws=0.0&ch=1&p=1

T 192.168.1.73:2691 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=368&ch=1&p=1

T 192.168.1.73:2692 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.3&ch=3&p=1

T 192.168.1.73:2693 -> 54.228.205.96:80 [AP]
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=247&wg=0.9&ws=0.0&ch=1&p=1
~~~

Here is what the tool expects to see:

~~~
mac=0004a369e0d6&id=c2&pv=0&lb=0&ac=0&reg=1009&lost=0000&baro=1007&ptr=1&wfor=0&p=1
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.1&ch=3&p=1
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=270&wg=0.9&ws=0.0&ch=1&p=1
mac=0004a369e0d6&id=8e&rid=63&pwr=0&or=0&uvh=0&uv=368&ch=1&p=1
mac=0004a369e0d6&id=84&rid=e6&pwr=0&htr=2&cz=0&oh=56&ttr=1&ot=14.3&ch=3&p=1
mac=0004a369e0d6&id=90&rid=d0&pwr=0&gw=0&av=0&wd=247&wg=0.9&ws=0.0&ch=1&p=1
~~~

So there are some extraneous lines to be removed.
 
Old 10-08-2016, 02:20 PM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,854
Blog Entries: 3

Rep: Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040
I'm not familiar enough with "ngrep", so maybe there is a way to whittle down the output on that end. Failing that, one can always tack on an extra "grep" using another pipe

Code:
ngrep -l -q -d eth0 '0004a369e0d6' | grep -E '^mac=' | nc localhost 9999
Check the manual page for "grep" about the -E and the pattern ^mac
 
Old 10-09-2016, 12:04 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,854
Blog Entries: 3

Rep: Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040
When you write the word data, I interpreted your question as wanting the text created by "ngrep"

Does the receiving program process text or raw packets?
 
Old 10-09-2016, 09:54 PM   #7
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,854
Blog Entries: 3

Rep: Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040Reputation: 3040
If you want raw packets from a specific MAC address using "ngrep" then you can specify using the ether host primitive

Code:
ngrep -l -q -w - -i eth0 ether host 00:04:a3:69:e0:d6 | nc localhost 9999
It looks like "ngrep" takes the same primitives as "tcpdump" or at least it takes most of them.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Data Processing / Data Mining? Anobodyinok Linux - Software 4 07-18-2016 08:54 AM
Pipelining data of find command in an array after processing the contents of the data AshishJogeshwar Linux - Software 5 06-10-2010 01:52 AM
Pipelining data of find command in an array after processing the contents of the data AshishJogeshwar Linux - Software 0 06-07-2010 06:15 AM
Processing data from a 'foreign' database with mysql, or tools to pre-process data. linker3000 Linux - Software 1 08-14-2007 08:36 PM
Svc needed for speech processing Stephanie Linux - General 1 08-05-2003 09:27 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration