LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-25-2007, 02:25 AM   #1
ravindranvasudevan
LQ Newbie
 
Registered: Oct 2007
Posts: 1

Rep: Reputation: 0
Sqlite Database image malformed


Hi All,

We have an application written in c++ that gets connected to a web service gets the data and populates the sqlite db.This is a daemon which runs continuosuly fetching data and writing in to the tables.

We have a problem when our application runs for around 3-4 days our database gets corrupted.We are getting the following error

SQL Error: Database Disk Image malformed

We are not able to find any help from the websites.Kindly help us in this.


Thanks,
Ravi
 
Old 10-26-2007, 01:25 PM   #2
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 65
Well, it sounds like you need to do some tricky debugging of your C++ application.

Unless the program is really trivial and you can post the sourcecode here, I don't really see how we can help except to tell you to debug it.
 
Old 10-26-2007, 05:54 PM   #3
PAix
Member
 
Registered: Jul 2007
Location: United Kingdom, W Mids
Distribution: SUSE 11.0 as of Nov 2008
Posts: 195

Rep: Reputation: 40
Hands high in the air, I haven't the foggiest, haven't used sqlite and don't write C++, so I'm a wast of time I guess?
From what you say, I note the following points:
1. It works fine for 3 or 4 days, so it can't be all bad.
2. I get the impression that this has happened several times, so what have you got to do to recover the database.
3. Silly to ask, but what does "SQL error: Database Disk Imaged Malformed" actually mean?
4. Is any database reorganisation going on or indexes being re-written during the running of your C++ program.
5. How many other users are there of this database?
6. How large is this database?
7. Why do I ask so many questions. [[simple, when you have a problem you get to ask a single question, but when providing answesr you get to ask several. This is why I try to supply answers rather than to have problems.]]
Quote:
From the Sqlite Wiki: This simple design is achieved by locking the entire database file at the beginning of a transaction.
Quote:
Several computer processes or threads may access the same database without problems. Several read accesses can be satisfied in parallel. A write access can only be satisfied if no other accesses are currently being serviced, otherwise the write access fails with an error code (or can automatically be retried until a configurable timeout expires). This concurrent access situation would change when dealing with temporary tables.
In my experience, and I don't know if Sqlite falls into this category, I have come across a number of small databases developed into systems and farmed out to multiple users only to suffer from frequent corruption of table indexes that needed to be rebuilt to get things up and working again. Causes of failure probably ranged from locking contention and people switching off PC's that seemed to be hung, lack of error handling and similar. Basically these databases were designed for the desktop, deployed as multi-user and proved not to be of industrial strength for the job in hand. Are indexes cached on remote systems and disenfranchised by periodic table reorgs, optimisations etc?
I ask all these questions because from the tone of your question, I feel your database is in production and just appears to be screwing up from time to time on a regular basis which you have identified as every three or four days, but I feel you recover it ok and don't have to re-build it, so it can't be totally catastrophic.
Same days of the week?
Same sort of time of day?
Fails at 10:30 in the morning or after lunch just when people are getting busy on the system?
Nothing to do with unclosed files causing a running out of available file handles every few days, down to a programming oversight?
Tell me more - your readers are intreagued - and someone doubtless has a good suggestion; hopefully I have made it already. Good Luck Ravi.

PAix
 
Old 10-29-2007, 04:26 AM   #4
ravindranvasudevan
LQ Newbie
 
Registered: Oct 2007
Posts: 1

Original Poster
Rep: Reputation: 0
Thanks for the reply.

The database does not contain any indexes.
Only one user (the daemon as such) is using the particular database.
We couldn’t recover the database (3 out of 5 tables). Actually we didn’t try to recover it as the data as such is not critical.
"SQL error: Database Disk Imaged Malformed" means the data in the tables are not available or the table is corrupted.
Size is small as it has 7 columns that stores integers and less than 1000 records should be there by the time of corruption.
There is no specific time pattern for the malformation except it takes 3-4 days of time length to get malformed.
There are 8 other databases which do not get affected, but only this specific database tied with the particular c++ application gets malformed. So general power surge or other reasons are not valid.
We are not sure what to look for while debugging the application. We checked for memory leaks using valgrind and cleared all possibilities.

Ravi
 
Old 03-04-2009, 05:47 PM   #5
donnied
Member
 
Registered: Oct 2006
Distribution: Debian x64
Posts: 198

Rep: Reputation: 30
This sounds similar to a problem I'm having. I was recommended the link:
http://www.sqlite.org/atomiccommit.html
but I'm not sure if that answers my question.
Did you find an answer to your question?
 
Old 03-09-2009, 06:34 AM   #6
AnanthaP
Member
 
Registered: Jul 2004
Location: Chennai, India
Posts: 952

Rep: Reputation: 217Reputation: 217Reputation: 217
Options.
(1) Check the application (web app) that writes to the data base. (Not your sqllite db). Is it getting corrupted?

(2) Everyday change the database you create to DB_yyyymmdd.db - say, and whenever you want to aggregate data, sequentially read in the files (using attach_database and detach_database respectively).

(3) See the "vaccum" statement. (It is like a house keeping routine that compresses the data base (but there should be no active transactions.

(4) If the c++ program uses the `system` command to populate the SQL, check the commit and close data base states.

End
 
Old 08-10-2009, 03:37 AM   #7
kotao_78
LQ Newbie
 
Registered: Sep 2006
Posts: 19

Rep: Reputation: 0
Can we fix malformed database file?

I mean if there any method to fix malformed database?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
zen-updater Error: database disk image is malformed mahantya SUSE / openSUSE 2 03-29-2007 06:29 AM
upload of image in database (php+mysql) Pravab Programming 5 11-08-2006 11:05 AM
LXer: MedPix medical image database uses healthy dose of FOSS LXer Syndicated Linux News 0 09-21-2006 04:21 AM
SQLite database security - doesn't exist? vharishankar Linux - Security 2 11-30-2005 08:40 PM
SQLite database backup ddpicard Linux - Software 2 11-01-2004 06:54 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 02:47 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration