Why don't you structure your filesystem so that you don't have millions of files in a directory? For example, having each object contained in it's own subdirectory. Filesystems usually employ hashing on the filename to speed up lookups but you are pushing that.
There are some embedded devices that use sqllite in lieu of a filesystem. This might be useful as you would control what you use as an index which could have a relationship to how the data is indexed in your application.
The inode number is given on the filesystem along with the directory entry. The filename is looked up to reveal the inode number, and then the inode structure itself is looked up. The fopen() and lopen() functions use the pathname while the open() function uses the file description (as in 0 for stdin, 1 for stdout, 2 for stderr) which means the the file was already open. So at one point or another the inode number was first located by looking up the pathname.
Also keep in mind that there are different filesystems which can be structured differently. Accessing the inode structures themselves would be performed by the kernel in the filesystem module.
One thing to consider is to have the information contained in a file where you use your "complex" indexing to directly access the structure instead of using a directory lookup.
Another thing to consider is to create a subdirectory structure which reflects your indexing. The indexing could then determine which subdirectory to search in.
Last edited by jschiwal; 04-24-2008 at 03:05 AM.
Reason: added info.
|