Data Comparison and Synchronization Algorithm

ravishchugh · 02-01-2016, 02:46 AM

Hi all,

I have been tasked to look for ways to synchronize data being received in main and standby systems. I have 3 sensors which are pushing the data packets in main and standby PCs in serial format in a one-way transmission. The data stream is being fed to both the main and standby systems are are supposed to be identical.

I have to write an algorithm in C to compare the data being received in the main and standby PCs and check for any differences in the two streams(They are supposed to be exactly the same).

Also, how do i go about the taking over mechanism of standby PC in case of failure in main pC?

Regards

Ramurd · 02-01-2016, 04:59 AM

Did you work something out yourself already?
You should start drawing it out; What do you have already? How would you move forward? What are the challenges, and how do you think to overcome them?

This pretty much sounds like a homework assignment (being the specific language you have to use and the information provided)

smallpond · 02-01-2016, 12:50 PM

Although it seems like overkill, it is better to use existing clustering tools than to try to put something ad-hoc together yourself. You would have to find and debug all the race conditions at startup and shutdown, for example.

http://clusterlabs.org/doc/en-US/Pac..._from_Scratch/

ravishchugh · 02-10-2016, 04:42 AM

Hi Ramurd

I had worked on the prequel of this algorithm wherein i was tasked to develop an algorithm to sniff a data packet and extract the timestamp and other info out of it. This extracted data from the packets will be now fed to another system in a serial format (ethernet) and my task now is to check whether the data being received in these 2 systems (after splitting) is identical or not. Also, the data going out of the systems after processing is identical or not.

I have not worked on this kind of assignment before and looking for suggestions to start sir. I have to look for ways to :-

1. Split the data in 2 identical streams.
2. Check the data at output of the processing programs in both main and standby systems and ensure that the two outputs are identical.
3. Look for ways for auto-takeover of standby system in case of main system failure.
4. look for ways of standby system synchronisation with main system in case of its restarting.

I hope to receive the right direction from you sir.

Regards

ravishchugh · 02-10-2016, 04:43 AM

Quote:

Originally Posted by smallpond

Although it seems like overkill, it is better to use existing clustering tools than to try to put something ad-hoc together yourself. You would have to find and debug all the race conditions at startup and shutdown, for example.

http://clusterlabs.org/doc/en-US/Pac..._from_Scratch/

Thanx smallpond,

Ill definately have a look at the link provided and get back with my doubts

Regards

sundialsvcs · 02-10-2016, 08:06 AM

I agree that this smells very strongly of "[do not do a ...] thing already done," but one technique that is commonly used in data communication is Cyclic Redundancy Check (CRC). As bytes come in, you can count them and update a CRC. Separately remember what the CRC value was at the latest, say, 100th packet; keep a rolling history of those. Those tallies should be the same.

It's also useful, and almost always done, to calculate a CRC as part of each packet that is sent or received, so that a message can be recognized as garbled as soon as it arrives. (It is possible to correct some errors in this way.)

Again, though ... "you're not the first person on Planet Earth to have tackled this problem, so don't waste much time trying to invent a new one." Instead, look for existing solutions, which might well be more thorough and complete than any you have yet thought of.

ravishchugh · 02-15-2016, 01:16 AM

Quote:

Originally Posted by sundialsvcs

I agree that this smells very strongly of "[do not do a ...] thing already done," but one technique that is commonly used in data communication is Cyclic Redundancy Check (CRC). As bytes come in, you can count them and update a CRC. Separately remember what the CRC value was at the latest, say, 100th packet; keep a rolling history of those. Those tallies should be the same.

It's also useful, and almost always done, to calculate a CRC as part of each packet that is sent or received, so that a message can be recognized as garbled as soon as it arrives. (It is possible to correct some errors in this way.)

Again, though ... "you're not the first person on Planet Earth to have tackled this problem, so don't waste much time trying to invent a new one." Instead, look for existing solutions, which might well be more thorough and complete than any you have yet thought of.

Thanx Sundialsvcs....I have never used CRC before and need some clarity on its implementation sir.

I have a custom packet coming from a sensor. Its configuration is :-

Code:

Byte          Parameter           Reference

1-2           5Ah A5h             Header
3             48h                 NUMDATA
4             02h                 IDENT

5-9          STATUS 1-5

10-12        Time            

13-15        Data1

16-18        Data2

19-21        Data3

22           XXh                  CHECKBYTE

23           AAh                  Terminator

This data packet will be stripped and Data 1-3 will be fed to me for CRC in a serial manner.
How do i go about performing CRC on this data stream?

How do i go about mirroring the data coming to main system into my standby system (hot Standby)?

I need the CRC or any other system to check whether the outputs of the main and standby systems are same or not.

Regards

Ramurd · 02-15-2016, 01:33 AM

so you get a char[22] (one char is one byte, and you start at 0);

Most likely byte 22 is a CRC; See what kind of correlation the data has with the first 21 bytes.
And then try to find out on the interwebs what a CRC is, there's so much documentation in that regard, that I'll not reinvent that and write it down here.