LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-21-2017, 11:59 AM   #1
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Rep: Reputation: Disabled
Data Manipulation from LAN Printer


Hello everyone! First post on LQ and it is pretty specific issue that I need help with. I was not sure which category it would fall under so I just picked General.

I just want to start by saying that I have basic understanding of BASH and Linux in general. Also beginner knowledge in Python (this is relevant for later in the discussion). So, with that being said, here is my issue. Bear with me as it may be pretty lengthy. Thanks, in advance for reading through my post.

I work for a video surveillance company and we use a 3rd party security software to view our cameras. We have asked the software developers (international) for help and they pretty much said there is nothing they can do. So, I have to take this on with my own hands and hopefully with the help of the community here at LQ.

The software is capable of doing POS integration where POS transaction data can be overlayed over the video so our clients can compare the video to the transaction data. This works through a TCP connection where a virtual network printer (generic/text, assigned the IP of the NVR on a specific port of our chosing) is setup on the registers and when the receipt is printed, the same data is sent to our NVR.

The problem that we are having is that the data is sent as one large packet which causes our software to only pickup the last 8 lines of the receipt (which is usually store info, etc.). Is there any way that I can develop a bash script or python program to take the incoming TCP data (just the data, not headers, flags, etc.) and send it to another TCP or UDP port, one line at a time (40 characters). This needs to happen as fast as possible because if it is late, the transaction will not correspond to the recorded video.

Here are some things that I have to take into account for any resolution. We can support multiple registers so I have to make something that can utilize multiple TCP connections. If I have to work with files, I am limited to only 64GB of storage space so I would want to overwrite files or use the same file over and over for each transaction. I am not allowed to put any software on the POS register. Everything has to be done on our units, after the data is sent (liability). We are working on CentOS 6.7 on a custom 4.7.2 kernel. If anyone can point me in a direction of how I could go about doing this, I would be truly grateful. Even if anyone has any "brainstorm" ideas, I am all ears. As I mentioned, I am new to Python but am familiar and would be more than willing to research/self-teach if I could get pointed in the right direction.

I hope that my explanation was clear and I thank everyone who can give a helping hand. Please let me know if anyone needs more specific info regarding any of this.
 
Old 12-21-2017, 04:30 PM   #2
rtmistler
Moderator
 
Registered: Mar 2011
Location: MA, USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 7,214
Blog Entries: 12

Rep: Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656
Not sure you can really do much here. First I'm trying to envision a system diagram. I can, but I'm trying to put boxes of functionality at each point and also areas where you can't touch software or things. Seems to be that I'd be drawing a bunch of Point of Sale terminals accompanied by cameras, and then circling them entirely to say that about all you can do is move the camera angles a bit.

The fact that the data comes in a big chunk and something can't deal with it, is beyond amazing. How much data is it and what is the data transfer conduit? Wireless allows us to view live video, and multiple streams. So sending one picture accompanied by standard POS accounting data doesn't seem to be too taxing.

I'm guessing that this team of developers in another country just don't know how to do it, you do or don't have a support contract with them, and either case, what you're asking for may be non-standard, the functions you're getting may be because you've coupled two or three systems together.

No amount of scripting in bash or python seems to be able to help you here. Seems as if you need to diagnose and fix or improve your network transport capabilities.

How about trying to give us more information about the amount of data per transaction, on average and where exactly the problem is.

In answer to your question about breaking down data and sending it in chunks. You can try to find the data which is normally sent one way, and alternatively send it using a program or script, and then you can send it in these smaller chunks. Meanwhile you run up against your performance requirement of getting it there soon enough to sync it up with your transactions. Well, all the data should be timestamped from the same clock source. If you're using lucky circumstance to correlate two or more sets of data, there's always a chance that they will drift apart, and Murphy's Law tells me that they will drift apart at exactly the point where you needed to verify a transaction.
 
Old 12-21-2017, 05:44 PM   #3
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
Hi rtmistler! I appreciate your response. For the data size and medium, we are looking at KBs of data per sent transaction (guessing 5-10KB). It is just ascii text based data that are systems can interpret. Just as much data as you would assume from a standard walmart receipt or the like. We are connected via LAN to the clients network. Are systems have two NICs so that we can have our network cameras on a stand-alone network connected just to the NVR (much easier bandwidth management on their LAN). So all in all, our system receives the data fine. I can see everything sent in the data list that it presents in the software. The issue is that it will not scroll in the overlay on the video (which can be used in criminal cases). Our customers have even stooped to taking videos with their phones of our camera system to present as evidence. This is not how it should be and I am the one left with figuring this out (we are an extremely small company, me being the IT manager, building the systems, testing new equipment, etc.) with no help from the developers. I am sure they use different POS solutions and don't have this issue.

Here is what I have found from TCP data tests. If I send a line of data (~40 characters) one second at a time, it scrolls fine and I can see each line appear on the recorded video. Even at 4 lines per second, I can see 4 lines at a time appear. So I have figured out that it has to do with the time-stamp on the lines of data. If I send all data at once (each line with same time-stamp), I only get the last 8. The software is only designed to have a max of 8 lines of text on the screen at one time.

I was thinking somewhere along the lines of a script/program that could save the text data from the TCP transmission to a file. Then the program could send the data (at the slower pace) to it's loopback address on which ever port is specified in the software and it would pick that up. It listens on all adapters for that specific port (even loopback). If this is even possible, I wouldn't even know where to start, that's the problem. Thanks, again!

Last edited by xX_Kryptos_Xx; 12-21-2017 at 05:50 PM.
 
Old 12-21-2017, 05:53 PM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 17,720

Rep: Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381Reputation: 2381
I agree that you have not provided enough information about the system and the individual hardware boxes.

Quote:
The software is capable of doing POS integration where POS transaction data can be overlayed over the video so our clients can compare the video to the transaction data. This works through a TCP connection where a virtual network printer (generic/text, assigned the IP of the NVR on a specific port of our chosing) is setup on the registers and when the receipt is printed, the same data is sent to our NVR.
I am not up on the current technology for POS equipment but is there a separate box for text insertion before the NVR? There has to be something that combines the video stream from the cameras and the text from the virtual printer.
 
Old 12-21-2017, 08:33 PM   #5
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
Hi michaelk! The text insertion is done by the software itself. It listens on a port setup in the software (one port per register). As soon as the POS integration function is enabled and the ports are set, it always listens to that port for incoming connections. So the virtual printer is setup as network printer using the IP address of the NVR and the port for that particular register. Each register has one camera pointing at it so each viewing window can have that register's data shown on the screen. Please see attached screenshot (from developers manual). You can see the data list on the left (which works fine), but the text on the screens (to the right) do not show properly, as described in earlier post. I need that data to show correctly, on the image, because this is the only way that it is presentable if our clients backup video with POS data. They do not have access to the data list when saving video, if that makes sense. Thank you for your response!
Attached Thumbnails
Click image for larger version

Name:	POS Example.jpg
Views:	12
Size:	252.5 KB
ID:	26579  
 
Old 12-22-2017, 07:22 AM   #6
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,032

Rep: Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762Reputation: 1762
Your screenshot suggests that the cameras are recording at one frame per second. When a transaction receipt is generated, that is sent to a server under your control, Rather than trying to capture the receipt line items on the video, why not just capture a single transaction identifier such as the print queue job number? This could then be married with a copy of the print queue data.
 
Old 12-22-2017, 09:44 AM   #7
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
Hi allend! Thank you for your response. The cameras run at 15 FPS. We have to use the overlay on the video so it can be presented as evidence in case of an investigation. It gives concrete proof if fraud has occurred. Plus the data that is captured is searchable in the software. So if our clients search for a product number or for a "return", it will show all instances of that product number or "return" that has been captured. They then can double-click on that line and play the video that corresponds to that particular transaction. So, I need all the text from the receipts.
 
Old 12-22-2017, 06:10 PM   #8
rtmistler
Moderator
 
Registered: Mar 2011
Location: MA, USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 7,214
Blog Entries: 12

Rep: Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656Reputation: 2656
The impression I get here is that this is not a network data throughput problem, but instead an application software problem.

You've tested that you can pass data. It is unclear to me what entity rejects too rapid data. Some application which correlates the receipt information to the camera image?

You either have to continue along the lines you're following, and decimate the data to get what you need affixed to an image or video file.

Or you have to fix whatever software or other thing which is rejecting the too much, or too rapid data.

I think the first option is what you're trying and seems to be your best option. You have tested sending data already. What challenges are you facing where you need to find and send the relevant data from those receipts?
 
Old 12-22-2017, 06:48 PM   #9
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
Yes, that is correct rtmistler. I am stuck on any software correction at this point as the developers pretty much said "there is no issue with receiving the data".

Dissecting the data is pretty much exactly what I want to do. Either use the entire data chunk and push it to the software in a "slower" manner (one line or so many characters at a time) or be able to pick and choose what data I want sent. The problem with the second option is that there are quite a few POS companies that our clients use and they have different receipt layouts. I am not sure how I could dissect each one. So that is where I am stuck. I don't have enough programming knowledge to just dive right in using Python or the like. I have tried Google'ing this stuff and I can't really get any concrete info on how to manipulate the text data. I have looked into Pcap captures (tcpdump, save to a file, pull from file, etc.). My issue with this is that there is too much "non-readable" data that I have to cut out or I don't know how to say "start from this line, push the data, stop at this line". I just want the text data (as you see on the video screens from my screenshot) and was hoping there was something simple (doesn't look like it is going to be the case at all). Since this is not going to be simple, I would just like some information (sites, examples, programming examples, etc.) that anyone can think of that could possibly pull this feat off. I really appreciate your time and others the same. Thanks!
 
Old 01-02-2018, 04:35 PM   #10
xX_Kryptos_Xx
LQ Newbie
 
Registered: Dec 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
Just bumping the thread to see if there are any more ideas for my issue. Thanks!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
data manipulation in linux: how to make columns from rows in linux Rozak Linux - Newbie 3 09-12-2015 03:07 AM
Data manipulation with awk chrisF682 Programming 5 09-25-2011 03:17 PM
24 bytes of data manipulation utkarshrawat Programming 6 07-13-2010 09:24 AM
Data manipulation Yogiz Linux - Newbie 2 12-11-2007 09:00 PM
help with data manipulation SeT Programming 2 10-20-2004 08:32 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration