LinuxQuestions.org - very weird situations with SSD drives

I have run to a very weired-fustrated situation and i started a topic on CentOS however i think this is quite interesting wridle so i though to share it here and listen what you guys think.

I did solve the issues but funny thing is that the cause of the problem is still "undiscovered". How is that possible...keep reading.

I am facing a very "nasty" situation where it is very tough to conclude / figure out where the problem might be coming from.

I have setup two centos 5.9 systems which do clustering. The way i did it was with 30GB SSD drives where each disk of each system is:

hda--
-------hda1=/boot
-------hda2=/
-------hda3= cryptsetup partition -> drbd

MOBOS:
a) branchA
b) MakerX

the hda3 partition is unlocked on startup and then it is used as a drbd block device which is synchronized with the peer system.

The problem is that i get various error messages on 2-3 different installs (either imaged or fresh):

1) I/O errors
2) DRQ errors
3) Some times data on the hda3 partition are scrambled (ascii chars appear like binary)
4) EXT3-fs error (device dm-0): ext3_lookup: unlinked inode 944717 in dir #944705

In the beginning i thought it could be a compatibility issue between the SSD and the MOBO. I changed the mobo (from branchB to makerX) and imaged the SSD disk from a backup image i had which was specifically for MakerX.

All was working well until i made a reboot and then i noticed again that some files on the hda3 partition were not as they should. I decided to delete all files from the relevant partition and copy them back but when i deleted the files i then received the following error:

Code:

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 3681081083, count = 1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 2237382943, count = 1                  

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 1422606189, count = 1                  

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 3681081083, count = 1                  

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 3991989911, count = 1

and then it mounts the file system as read-only

I am having hard time even to narrow down the possibilities which could cause the problem.
I tried different motherboard with different SSD
I tried same motherboard with three different different SSDs

(mobos are embedded systems)

but the errors are still hitting on random moments...

On my last try i got again the below errors:

Code:

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 4040611822, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 3524023294, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 1865901477, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 1529808743, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 359069044, co1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 828308971, co1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 2137651154, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 4169087681, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 4161535557, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 3657360678, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 1241984481, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 1787500438, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 2587318750, c1

EXT3-fs error (device dm-0): ext3_free_blocks: Freeing blocks not in datazone - block = 4244106120, c1

EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted                          

EXT3-fs error (device dm-0) in ext3_truncate: Journal has aborted                                    

EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted                          

EXT3-fs error (device dm-0) in ext3_orphan_del: Journal has aborted                                  

EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted                          

EXT3-fs error (device dm-0) in ext3_delete_inode: Journal has aborted                                

__journal_remove_journal_head: freeing b_committed_data                                              

__journal_remove_journal_head: freeing b_committed_data

I will replay below on this topic what i did and sorted my issues but the cause of the problem remain unknow...at least to me