Disk file systems
 

I am trying to understand what a disk file system is and how it interacts with an operating system.
Here is my take on it so far, please correct or comment as you see fit.

A disk file system describes the physical method in which data is written to the magnetic media (hard disk).
Windows uses a system called NTFS and Linux uses a system called ext3.
The two systems are analogous to VHS versus Beta video tapes, they do the same thing in different and incompatible ways.
A hard disk must be formatted by the OS installation program before the OS is installed. This entails something being done magnetically to the entire portion of the disk being formatted. For example Windows will first format the entire disk (or designated portion) as NTFS, making it unusable in any way by a Linux OS. Conversely Linux will first format the entire disk (or designated portion) as ext3, making it unusable in any way by a Windows OS.
Thus if you have Windows and Linux on one hard disk, you will have two partitions (more than two I realize, but I will keep it conceptual here) each of which has been physically (magnetically) altered in different ways during the formatting process.
BIOS is the central controller that can "see" all disk file system formatting types. The BIOS now knows that areas AAA111 to JJJ999 of the hard disk are NTFS and that areas KKK111 to ZZZ999 are ext3.

What file format is the boot sector that can boot into either the ext3 Linux OS partition or the NTFS Windows OS partition?

Maybe much of the above makes no sense, but I have to start somewhere. I find almost all writing on this subject I have found so far impossible to conceptualize.

http://en.wikipedia.org/wiki/File_system

Thank me later :)

Just my opinion:
No. Physically writing to a medium like a hard disk (ie putting down the magnetic 1's and 0's on the surface of the disk) is hardware related. It has nothing to do with the logical structure that an OS puts at the beginning of the disk so that it can find & use the files it puts on the disk later on.
In other words: a file system is a logical system (ie lookup tables in FAT, binary lookup tree, etc) that the OS uses to "organize" the contents of the disk. Nothing to do with physically writing on the medium.

Actually, Windows supports FAT16, FAT32 & NTFS (and in the future, some others too), Linux supports a wider variety (ext2, ext3, reiser, fat, ... even to some extent NTFS as well).

I wouldn't compare them to VHS vs Betas, but I agree that they do the same thing in different and incompatible ways.

Think of it this way:
Delete (all) files => keeps file organization intact, just tells the OS that it can start recycling the data bytes on the medium it used to hold for the deleted files/folders.
Format filesystem => can erase all files (ie overwriting the entire surface with 0's or a random sequence), but doesn't have to. It does reset the logical organization of the disk to it's most basic form, thus making it impossible for the OS to find any of the files/folders previously stored.
When you "reset" the disk organization, you can of course install a different type of organization (ie another filesystem) over the old one in the process.

The physical writing/reading of the disk is done by the hardware in the same way, regardless of the logical structure (filesystem) implemented. So no "magnetic" alterations, except at the beginning of the disk (where the OS stores the filesystem's structure - ie the lookup tables to find the files on disk).

Linux & Windows can see the same types of filesystems. As said, Linux supports FAT filesystems (also used by Windows) and to some extent also NTFS filesystems. Windows, on the other hand, doesn't really support the filesystems that were "invented" for Linux/Unix.
A file access happens like this, in general:
Computer wants to look for a file => OS looks this up in the filesystem (ie the lookup table), thus locates the file on the disk => OS calls disk driver to read the necessary hardware bytes (ie magnetic/optical bytes on the medium's surface).
The lookup in the filesystem depends on the filesystem, the call to the hardware is handled by the hardware driver and is thus not filesystem related.
Drivers are however implemented differently on Windows & Linux.

BIOS is the central controller that can see any hardware you attach to your motherboard. It does not have any clue regarding the filesystem used.

The boot sector has a specific format.

Thanks a lot. I will have another go. Please correct and comment if you would be so kind.

In a standardized area of any hard disk is an area that acts as an index for the entire disk. In this index, resides information that describes what areas of the disk use what file system. The various operating systems on the disk constantly refer to this index so they know where to write and read data.

The boot sector of all disks is formatted in the same way. It is a universal format. The details of the boot sector are also kept in the "index" I refer to above. The boot sector houses such applications as grub, lilo and whatever the application that boots windows is called.

The data is written to the magnetic media in the same physical way by the hardware for all file systems, but the way the data (1s & 0s) are arranged on the disk varies between file systems. I am guessing there must be some difference in the way data is stored between the different file systems.

You're welcome.

I'm no expert in this matter, but yep, there's a thing called a "partition table" too. Per partition (logical part of the disk), you can have only 1 file system.
OS's usually lookup stuff in the file systems' indexes. The partition table is only used when re-partitioning (ie using stuff like Partition Magic, fdisk, etc).

Can't tell if the boot sector is in the same format as the partition table. But don't worry about this. The boot sector just tells the BIOS where it can find your boot loader (grub, lilo, NTldr on Win,...). The boot loader "knows" the disks/partitions and how to boot the OS's installed on them.

I wouldn't go as far as to say that it's written in the same way for ALL file systems. The hardware read/write depends on the device type (and the driver).
ie 1's and 0's are written differently to CD-ROMs (optical) than hard drives (magnetic).
Both devices will have a "file system" (ie a lookup table of where to find files).

There are still a few things not covered here (ie disk geometry, arrangement of files in "blocks", etc), but I have a feeling you're starting to get an idea.

The analogy is a little flawed, but you'll get an idea if you consider it this way:
in any office, you're bound to find a bunch of file cabinets. Each of these cabinets will hold an index of it's contents. Each file in the cabinet can be a single piece of paper or a folder that contains other files.
The similarities with your computer are obvious:
office = computer
file cabinet = hard disk
index in cabinet = file system
files/folders = ...
and if we take this a few steps further:
-the files can be cut up into small pieces (disk blocks), ie if the dimensions of the cabinet are small,
-the files/folders can be separated by dividers (ie disk partitions)
-it's likely you'll need some file lookup software (OS) because you'll soon get tired of going through the cabinet file-by-file or piece-by-piece...

Thanks again. I am starting to be able to visualize what is going on (baby steps).
What initially prompted my thoughts was the concept of dual boot machines, ie having 2 different operating systems located on one disk (namely Linux & Windows), and the ins and outs of making it work.
So it seems that when you turn on the PC, the BIOS first looks at the boot sector. This is where the application that provides the GUI allowing various boot loaders to be selected for execution is stored. In a single OS disk, the boot sector would automatically redirect execution to the one and only boot loader. If the boot sector contains a program that launches a "boot loader choice" GUI, once the user makes the choice, execution passes to the chosen boot loader and the relevant OS starts.
I remember having problems getting a dual OS disk to work. I eventually went for 2 separate hard disks. I now choose OS by modifying BIOS at startup to reflect which hard disk to boot from, and restarting the PC. When I tried to do it all on one hard disk, Windows seemed to interfere with the process of allowing me to choose OS. I think Windows may have rewritten something in the boot sector each time the PC was started. Is this possible?

You're starting to get the picture. Just keep in mind that there is usually only 1 boot loader (grub or lilo if installed by Linux or the NT loader if installed by Windows).

In my humble opinion, the proper bootup sequence is something like:
-BIOS looks for hard drive (or boots from floppy/CD instead);
-if hard disk is found, BIOS locates the boot loader via the boot sector;
-the boot loader shows the typical menu where you can select the OS to boot;
-you select an OS and, if the boot loader is properly configured, it'll boot it.

Installing windows or Linux can overwrite the boot loader in some cases. Linux installs however usually detect other OSs already present and configure the OS select menu for you. Windows does no such thing.
So: first do the Windows install (will install NT loader), then add the Linux. It'll see the Windows (and/or other Linuxes), install grub or lilo as new boot loader, and configure them to boot all the OSs found.
Linux then installs itself typically in a separate partition, but it can also be on a separate physical hard disk (if you have more than one).

Thanks, this is good information.
It seems that when you install Linux on a hard disk that already has Windows on it, it detects the presence of the NT loader, gets rid of it and replaces it with Grub or Lilo. Grub or Lilo can emulate the function of the removed NT loader as well as providing a boot loader for the Linux OS.

Yep, that sounds about right.
Just keep in mind that the NT loader also can boot Linux, it's just not as handy/flexible as grub or lilo.
And besides, it's probably already tricky for Linux to detect the presence of other OSs on the system during install, let alone that it would have to try to configure NT of the detected Windows, in order for it to boot Linux. For the Linux installation, it's just less hassle to overwrite the existing boot loader and provide the new boot loader (grub/lilo) with sufficient configuration to boot all the other, detected OSs.

So, in short, both Linux style & Windows style boot loaders can do the work, lilo/grub is simply installed "for convenience".

Not exactly--the partition table is written when repartitioning, but it is accessed (read) every time the computer is booted.

It would be more precise to say "When you install Linux on a hard disk that already has Windows on it, the installation program detects the presence of Windows and overwrites the previous boot loader stored in the master boot record (MBR) with Grub or Lilo." Technically, I should say "...optionally overwrites..." because you can choose whether to keep the old boot loader or switch to the new one. It's best to switch, of course.

Every hard drive has a MBR. You can think of it as a tiny, automatic partition that you can neither destroy nor reallocate. It's still there even if you zero-fill the hard drive (also known as low-level formatting). What's more confusing, some modern BIOSs allow you to choose which hard drive's MBR you want to boot from. You will most likely want to avoid confusion and only install a boot loader to the master drive on the first controller (hda). Fortunately, this is the default, so nothing to worry about.

Thanks again. My current system has 2 hard drives, one for Windows (Primary HD) and one for Linux (Secondary HD). When starting up, if I want to change OS from the one used in the previous session, I access BIOS settings during startup (is this the correct term?) and change the hard disk to be booted from. I then save the BIOS change and restart.

Does this mean both hard drives have a boot loader?

I recently realized that when I boot from the Linux hard drive, a GUI appears during boot sequence that gives me a few options. Most of them are related to my Mandrake 10 Linux OS I believe (failsafe and a few others whose names I don't understand), but it also provides Windows as an option. I found this strange as Windows is on another hard disk. But selecting Windows here does work, booting Windows. How did the Linux OS I installed know there was a Windows OS on another hard disk and then configure the bootloader on the Linux hard disk to boot Windows? Pretty clever.

The reverse does not work, when I boot from the Windows hard disk, Windows boots with no option to boot the Linux OS on the secondary hard disk. Why is this the case? Could I add this functionality to the Windows boot sequence?

Yep, both disks must have a boot sector, otherwise you couldn't boot from them (by specifying that you want to do so in your BIOS). Note that your BIOS will usually allow you to specify a "sequence" of devices to boot from. ie if booting from the first device fails, it automatically tries the 2nd one, etc.

Your Linux offers the boot options because your 2nd hard drive seems to have grub or lilo installed. When you installed Linux, the Windows was detected (all your hard drives are checked) correctly and grub/lilo was configured to boot it for you. Neat, he?

Windows, as always, isn't known for it's ability to work well with other OSs in the same computer. So it's quite likely that it didn't detect your Linux (if it was installed earlier on your computer, that is) during the Windows install. And in any event, Windows will most likely NOT try to configure it's boot loader for booting other OSs. You could try configuring the Win boot loader yourself to accomplish this. When you do so, you'll also get a menu from the Windows boot loader when you boot from your first hard drive (with Windows on it).

In short, avoid having to enter your BIOS too often. I'd recommend setting the BIOS to boot from the secondary (Linux) disk first and if that fails, revert to the primary (Windows). Then you'll always get the boot menu to select the OS you want to boot. In your grub/lilo config, you can then make changes you need (ie eliminate some boot options you don't want to see, makeing one of the options the default, specifying the time-out period for booting the default if the user doesn't make any OS selection, etc).

Thanks, new doors are opening all over the place here, this is great. Would the following description of the heirarchy of disk organization be correct?

Hard Disk contains ONE Boot Sector and MANY Partitions.
Boot Sector contains ONE Master Boot Record and ONE Partition Table.
Partitions contain MANY Files.
Master Boot Record contains ONE Boot Loader.
Partition Table contains MANY references to addresses on Hard Disk.

Is the file system which a partition uses defined merely by references in the partition table or by the way in which that partition is physically prepared?
For example, does the partition table hold data that says sectors ADC243 – > FGR563 inclusive comprise partition hd1a, and that this partition uses the ext3 file system?
And/Or are sectors ADC243 – > FGR563 inclusive somehow physically prepared when partition hd1a is created by an operating system or partitioning application?

A crude analogy would be mowing of the grass on a football field. The surface of a new unused hard disk being analogous to the field being un-mown. When you create an ext3 partition on the hard disk this is analogous to mowing the field in concentric circles and making a note that you did so in the groundskeepers diary (partition table). When you create a NTFS partition on the hard disk this is analogous to mowing the field in a criss-cross pattern and making a note that you did so in the groundskeepers diary (partition table).

Does this analogy make sense?

As far as my knowledge goes (again, I'm no expert):
1. One boot sector: OK, many partitions: not necessarily. Hard disks usually contain only 1 partition, unless you re-partition them yourself.
2. It can be that some of this stuff is actually duplicated on the disk, just for safety. For proper operation, you'll only need one functional copy though.
3. Very true.
4. Sounds right.
5. The partition table only contains entries regarding the installed partitions. So, if you have only 1 partition, it'll contain only 1 entry, probably. The file system itself (ie the FAT tables for FAT filesystems,
the inode/superblock structure of ext2/ext3 filesystems, etc, will hold many references to parts of the disk, namely the parts where each file can be found.

Think of the file system as the organizational structure, used to store files on disk and to find the stored files afterwards. Different filesystems have different "structures", so they'll be stored differently on the disk. I suggest you start off with reading up on "superblocks", "inodes", "FAT tables", ...
Since the different file systems store a different structure on disk, yes, they are "physically" different.
Those structures are typically placed on the same or similar physical locations (ie start of the disk, right after the boot sector and stuff), though.

That sounds about right to me. Re-partitioning can change the disk parts covered by any partition and often requires reformatting. Reformatting can install a new type of file system (ie a new file organizational structure), thus erasing all references to any existing files.

The concentric/criss-cross patterns may be somewhat off, but the rest of the analogy sounds right to me.