Data Storage Question
 

I need to implement a database system, either using an existing library, or with my bare hands, that does the following:

Can store records of arbitrary length
Can increase/decrease the length of these records
Can insert/delete/retrieve records given a user-defined key

Whats my best option for fastest performance? At the moment all i can think of is just storing each record as a file on a ReiserFS file system. This is a quick and dirty solution, my database will (hopefully) run into the millions so I need something thats fast and reliable.

Edit: Don't be afraid to point me in the direction of some really complicated theoretical webpage thats full of equations ant the like, as long as its rlevant
All suggestions welcome

Of course, a full featured SQL-database would be the most flexible. The best-known open-source options are MySQL and PostgreSQL. Arguably, MySQL is more widely used, has more information available on the internet and is easier to use, than PostgreSQL. On the other side, PostgreSQL is probably more reliable/robust.

For local-disk database-libraries (no direct networking support, very fast, but more difficult to use), Berkeley DB and CDB come to mind. CDB has a 4Gb limit per database, and only source-distribution is allowed, but can be expected to be robust and fast.

Just files on you ReiserFS, would be a bad idea, if you're serious about lots of data and still having good performance.

From the information you posted, and assuming you're not afraid of some bare-metal programming, I'd think my choice would be Berkeley DB.

I had a look at the Berkeley DB API, and the only database type that allows for arbritrary-length records is RECNO. However this is not suited to my needs because the keys with RECNO are generated by Berkeley, whereas I want to use a 64-bit hash as a key.

No offence, but I want to steer clear of connecting to MySQL or Postgre Sql because theres a major overhead involved there. also to the best of my knowledge, MySQL does not allow arbritrary record lengths.

come on!!! help me out

Please do not bump your posts before 24 hours have
passed. Your problem is no more important than any-
one elses, and you're not paying for support here,
either ...


Cheers,
Tink

What about SQLLite?