Design of serial port logging program and then another to store in database
 

Hi,

I would like to pay someone to make for me a program to log a serial port and another program to then put relevant data into a database.

Can I ask what the hangups will be? Assume the serial logger will be on 24/7.

The initial logger should timestamp the serial port data.

Given that the program that does the filtering of the data and places it in the database will be slow it will not keep up with the logger and may lag behind by some time how do I ask for the solution?

Do the logger and the filtering programs share the same file?
Should the logger push its output into smaller size-limited files? Or perhaps change files within a short period of time? (say 100ms) The filtering program could then be using the older file ...

The filtering program needs to mark an entry into the logger file to record a successful database transaction I would think.

I'm concerned that the data to be logged should have unfettered access to its initial log file. If it is timestamped then the database stores the timestamp.

This is a design question.

I hope it makes some sense.

Cheers,
Steve

Many things to consider.

How many bytes are going to be sent from your device. What is the baud rate?
What is the expected recording rate as in samples per second?
How important is time latency?
Is the program going to request data or is going to be continuous?
What are you going to filter and why do you need a database?

Fixed sized files can be good because if the computer or logging program crashes you only loose the last opened file.

Hmmm.

The nature of the comms is that it is sporadic. The comms is initiated by both ends so comms being sent or received needs to be initially captured. The comms is in a range of 10 through 20 bytes at a time which is relatively small packets. It is important for the comms to be recorded quickly and that they be recorded with timestamping. Measures to ensure speed of recording and protective measures in case of some sort of failure (power, dislodged cable etc,.) are what I'm trying to reasonably define so that I can advertise on something like Freelancer.

Comms need to contain certain identifying features to help validate the data. So the filtering is removing the 'extraneous' information from th e comms frame to record only the actual data. A bit like taking the letter from the envelope. Not all comms seen on the port is valid. There will be broken frames and other 'noise'.

In terms of speed, I'd think say 20 interactions of 20 byte packages per second would be a solid start. So 400 bytes per second.

Comms in and out need to be monitored although some of the comms may not be recorded in the longer term. So if you can imagine a situation whereby comms goes out requesting something and expecting a certain reply. If that reply comes in then there would be little need to keep some of the data in the longer term. If the reply doesn't come in then a record of the attempt to communicate needs to be kept. All of this is further downstream of my immediate requirements though. For now, I need the ability to record all comms in a secure manner.

I hope this further clarifies my situation.

If I can be more precise in my initial job requirements this is the aim. Should I have a set of files through which comms is written? eg. say five(5) 1k files for the data sent and five(5) 1k files for data received?

Thanks for the reply,