[SOLVED] Creating a simple database with libdb

RandomTroll · 09-14-2017, 06:22 PM

I want to take 2 lists of names (about 1M each), pick out the names in the second list that aren't in the first list. When
the lists were small, I read names in the second then did a
grep on each on the first. That now takes about 10 hours. A
database seems like a better choice. I found dbopen,
but it's out-of-date; its man page reads

Quote:

This page documents interfaces provided in glibc up
until version 2.1. Since version 2.2, glibc no longer
provides these interfaces. Probably, you are looking for the
APIs provided by the libdb library instead.

I have all the db packages installed but can't find man pages
for the API;

Code:

apropos db

doesn't return anything
relevant. The contents of the db packages don't have man pages.
The /usr/doc pages for the packages refer me to
Oracle's homes for the packages, which are voluminous and
opaque. Is there a simple tutorial for the person who doesn't
want to become an expert?

norobro · 09-14-2017, 08:40 PM

I recommend using Sqlite. The command line shell is simple to use. https://sqlite.org/cli.html

You can create two tables and import your files. Not knowing how your files are laid out, you could do something like this:

Code:

sqlite> create table first_table(firstname, lastname);
sqlite> .separator " "   # not neccessary if delimiter is a comma
sqlite> .import firstfile first_table
sqlite> create table second_table(firstname, lastname);
sqlite> .separator " "   
sqlite> .import secondfile second_table
sqlite> .save combined.db

RandomTroll · 09-17-2017, 09:38 AM

Thanks! That was the ticket. As soon as I figured out

Code:

select name from first_table where name not in second_table

it worked in seconds.