filesystems & encryptions

We are going to have a large flat file containing details of members, this file is constantly updated by members (changes of address, new members etc) and we want the data to be encrypted. Conventional encryption doesn't seem suitable since the whole file would need decryption-make changes-encryption for every change.

Questions:
1) Is it a reasonable solution to create a lot of small files holding just the details of one or a few members?
2) How can I calculate or know the space that is going to be occupied by an encrypted file for a given method/algorithm?
3) Is one encryption software better suited than others for what we want to do (we're contemplating using dm-crypt)?
4) It looks like the filesystem of choice would be XFS. Any comment?

Thank you for your help.

The Linux kernel can encrypt an entire file system. It can be an actual device, or just a regular file that you've mkfs'ed a file system on. Take a look at `man losetup`. You also need to make sure that you have loopback and cryptoloop support either built into your kernel or built as modules.
ta0kira

Thanks to ta0kira.
The question we have is that it is impractical to decrypt a large file, then make changes to it then re-encrypt it, therefore is it practical to make a lot of small files holding the details of one or a few members so that the size of the file to be decrypted, changed and re-encrypted is as small as possible.

Sounds like you have control over the file format, which helps out a lot. There are several ways around processing the whole file for every change. You could "containerize" the file by putting separately encoded sections within a single file for each user. You could also have a separate file for each user. I don't think it makes that big of a difference, except separate files might be easier to work with in your program, but then you have a whole bunch of files instead of just one.

What type of cypher do you use (block or stream)? The kernel uses block ciphers, and when you have an encoded file system it just applies the cipher to the part of the file system being changed.

That brings me to another idea; you could have an encrypted disk image file (which allows you to transport it as one file) with separate files for each user. All you would have to do for this is mount the disk image with encryption on, then read/write like it was a normal disk. The kernel would take care of all of the encryption/decryption and would only process the part of the image being changed. This would require you use a built in cipher, or if you have your own cipher you would need to make a kernel module out of it (only if it's a block cipher). I can help you out if you want to do that.

Oh, to answer your questions:
1) Yes, that sounds like a good idea.

2) If it's a block cipher, you need to find out how big the block size in bytes. You then round the size of your file to the nearest block size. If the method adds a header to the file (such as a checksum), you need to add that size also. If it compresses the file, you really need to wait until it's done to see what size it is.

3) I don't like dm-crypt all that much. It's too complex for everyday use. losetup gets the job done for me because it's very simple (I hear it has several bugs though). A lot of people use GPG, but I don't know how you interface software with it.

4) I don't know about XFS. I use ext3 and ReiserFS because they journal, and therefore recover extremely fast after a hard shutdown.

ta0kira

This is volunteer work for a Not-For-Profit. I have a free hand and have to set it all up as best I can.

Since you OK the idea, it looks like very small files are the way to go as the existing routines would interface very well with this system (they're written in assembly, I only have a very basic knowledge of C). We don't need portability.

We can choose any cypher we want, the more secure the better, I only had dm-crypt in mind because it's the only one for which I found some sort of user manual on line. The encryption is intended to protect the data if the machine holding it is stolen or mis-appropriated, what we envisage is a system by which the key would be stored in a file in memory (ramdisk or similar) and is lost when power is turned off.

From your explanation it seems we can use a kernel based block cypher. We wouldn't compress the data unless it reinforces the protection.

If I understood what I read correctly, losetup is loop-AES, from which comes cryptoloop from which comes dm-crypt. Loop-AES is better than either and I remember reading its implementation has a lot of bugs but those wouldn't affect us because of the way we would use it although I can't remember the details.

XFS also journal and is reported to be particularly good with lots of small files. I have my distro set up with ext3 which is not supposed to be safe for encryption and I'll probably set up Debian as the OS on the production machine.

I don't know what an encrypted disk image file is but your explanation seems to indicate it might be what we need provided it doesn't imply encrypting a full hard disk.

I'd be very grateful if you could make a few recommendations.
Thank you very much.

...ext3 which is not supposed to be safe for encryption...

Yes, I forgot about this. You should probably use ext2.

If you want to make an encrypted disk image, do this:

1) Make sure you `modprobe loop` and `modprobe cryptoloop` unless you have these built in to your kernel.

2) Create a random disk image:

> dd if=/dev/urandom of=my.img bs=1K count=1024

This will make a file "my.img" with 1024 blocks of 1K each (1M) with random data.

3) modprobe your ciper (I'd suggest aes or blowfish)

4) Loop the random image:

> losetup -e aes /dev/loop0 my.img

You will be asked for a password. You may need to specify aes-256 instead of just aes.

5) Create your file system on the disk image:

> mkfs -t ext2 /dev/loop0

6) Check the file system:

> fsck /dev/loop0

7) Mount the loop:

> mount /dev/loop0 /mnt/hd

You now have an encrypted disk image that you can mount and use just like a hard drive. To disconnect the image file, do this:

> umount /dev/loop0
> losetup -d /dev/loop0

To connect it later, do this:

> losetup -e aes /dev/loop0 my.img
> mount /dev/loop0 /mnt/hd

You might also be able to bypass the losetup part for remounting if your mount supports the encryption option:

> mount -o encryption=aes my.img /mnt/hd

ta0kira

I got as far as the password. After entering this, I get a message:
ioctl: LOOP_SET_STATUS: Invalid argument
I have compiled the kernel with AES_586, blowfish and others in the kernel (option= Y). I suppose that would not make any difference. I can't figure out what the invalid argument could be.
Can you help again?
Thank you very much for the script.

Type `cat /proc/crypto` and see if aes shows up. I actually had to upgrade to kernel 2.6 to get this to work. Did you get any error messages when modprobing loop or cryptoloop? Look at lsmod and, unless you have them built in to your kernel (not default), then you should see loop, cryptoloop, and aes. If they all show up and you still get the error, AFAIK there is no easy fix except to upgrade kernels. Some people use dm-crypt instead, but that is quite complex and I'm not entirely sure how/if you can mount a disk image with it. The stuff I showed you is standard Linux stuff, which is why I like it; it should supposedly work on all Linux systems.
ta0kira

cat /proc/crypto shows all the ciphers including aes and blowfish etc, all in the kernel.
I got no error modprobing loop and cryptoloop, they both show with lsmod

I specially installed a second kernel (2.6.12.3) for this so if you've got another suggestion, I'll try it.

Originally posted by rblampain cat /proc/crypto shows all the ciphers including aes and blowfish etc, all in the kernel. I got no error modprobing loop and cryptoloop, they both show with lsmod I specially installed a second kernel (2.6.12.3) for this so if you've got another suggestion, I'll try it.

From: Linux Encryption HOWTO: FAQ

Cipher xyz does not work. It says ``unsupported cipher xyz'' / ``ioctl: LOOP_SET_STATUS: Invalid argument'' although I have compiled it into the kernel. Not for all ciphers does a user-space setup tool currently exist. Please see section Patching the util-linux source for a list of working ciphers or try them all in turn.

4.2 Patching the util-linux source After re-compiling the kernel and rebooting it, you need to patch some of its helper programs, namely losetup and/or mount. The patch used resides in the Documentation/crypto hierarchy of the kernel source tree. losetup and mount need to be patched if you want to use any of the following ciphers: * AES aka Rijndael * Twofish * Serpent * MARS * RC6 * DFC * Blowfish * IDEA * 3DES * RC5 (missing test vectors) Note that this collection is compiled from my own personal tests and is only valid for patch-int-2.2.17.7. As it is one short-term goal to unify the conceptional design of the crypto modules (i.e. all will use the loop_gen abstraction someday), the size of this list is likely to monotonically increase as new releases come out. In order to patch the mount utilities to support encrypted filesystems over the loop device, you need to fetch a recent source tree of the util-linux package, e.g. from ftp.kernel.org/pub/linux/utils/util-linux/ or oneof it's mirrors. As usual, you issue the following commands to apply the patch: user$ tar xfzv util-linux*.tar.gz user$ cd util-linux* user$ patch -p1 < /usr/src/linux/Documentation/crypto/util-linux*.patch It should apply cleanly if you use the patch only on util-linux trees that have a higher version number than the patch file shows. Next you have to make sure that only the mount utilities are built. You can get in all sorts of troubles if you mess around with init or the password authentication programs. You can render your computer unusable if you do not make sure that you only compile mount and losetup!. In order to avoid this mess, you need to install the executable files yourself, so you can gain control over what is being installed. But that is not hard to do: After paching the util-linux source tree as described above, issue the following commands in the util-linux* directory. Make sure you have not currently mounted any filesystems using the loop device. user$ ./configure user$ make -C lib setproctitle.o # mount depends on this user$ cd mount user$ make losetup mount umount root# for i in losetup mount umount; do root> j=$(locate bin/$i) root> cp $j{,.old} && chattr +i $j.old root> cp $i $j root> done If the make step fails, check that /usr/include/linux and /usr/include/asm resp. are symlinks to include/linux and include/asm-arch resp. Some distributions (e.g. Debian) only have a copy of the kernel include there. If you change the kernel, they become outdated. Change the permissons and ownership of the new binaries according to your distribution's policy (check the old binaries for the right settings).

Good luck,
GH

Thanks to freegianghu, this is a lot of enlightenment and a beautiful quote. In my case, this will definitely put encryption back on an unused testing machine.

And thanks to taOkira for the personal guidance.