I have several questions about disk partitions, that'll help me understand this area that's new to me. Hopefully some of the Linux guru's out there could help to clarify:

1) Can you ever create an extended partition if you have used less than three primary partitions? (i.e. 1 or 2 primary and then add an extended?)

2) Can Linux manipulate (e.g. write bytes to, read bytes from) unpartitioned space, or is only partitioned space visible to Linux?

3) Can you ever create 4 primary/extended partitions that don't take up the entire disk? Why might anyone ever do this? Isn't that space lost to any OS that ever tries to use the disk until someone makes the remaining space viewable to the OS?

4) Is the way that a disk is formatted with its partitions completely independent of (i.e. standard across) which tool or OS does it, and which filesystem goes on top of it?

5) Are there any performance or other differences between making a partition primary vs logical, or can they be considered identical for all purposes? (Or are there cases when you should definitely make it one or the other - e.g. production environment, database server, etc?)

6) Can there ever be more than one extended partition on a disk? (I've read the term nested partitions a lot - does this mean that an extended can contain other extended partitions?)

It's a lot of questions, but it'll definitely be some good learning for me. Thanks.

Yes, but the device nodes will still be starting at /dev/*d4 for any extended (logical) partitions.
No - the space needs to have a filesystem present as when data is written, it is not passed to the data blocks directly. All data flow is either passed to, or intercepted by, the filesystem, which then shuffles the data around accordingly.
Yes, yes you can. The unformatted space will just be as though it doesn't exist (unless you run Windows, which will set up a "drive" for it, but the moment you try to access it, will attempt to format it with a filesystem). You may want to do this if you need to leave space for another operating system that does not have publicly-available tools that work under the primary OS (ie. ZFS for Solaris).
The tool used to format the partition will be from the developer and/or maintainer of the filesystem. In the case of a filesystem like NTFS or HPFS, the tools are built into the OS that primarily uses them. While NTFS formatting tools do exist for Linux, they aren't "official." The whole idea of third-party filesystem tools is that they try and imitate the original as best as possible without adding features the original doesn't have.
While I don't know for sure if there are any performance differences, I highly doubt there would be. A partition is a partition is a partition. The whole "primary" vs. "extended/logical" forum is probably a convention left over from around the time they discovered dirt, and had some completely solidified reason as to why it was this way. If this is the case, the reason has long since been lost, and yet still maintained due to the infrastructure around it (think meta-headers in files - the POSIX method - versus the extension method inherent in DOS). DOS started the whole file extension scheme so as to know filetype A is run by program A. Whereas the POSIX method relies on file encoding and meta-information in the file headers (think MIME types). The only reason the DOS-method of file naming still exists - IMHO - is because it makes it marvelously easy to parse long lists of files.
Primary partitions exist in numbers 1 through 3. Extended always resides in partition 4. Anything with a parition number > 4 is a subset - "nested" as you put it - within partition 4. For a better idea of how this works, try installing FreeBSD (or any SVR4 UNIX for that matter). They get you working around the whole idea between "partitions" and "slices."

In Windows you will only ever see C:, D:, ..., Z:. In Linux you will see /dev/{h,s}d{a,b,c,d}N. But in an SVR4 UNIX, you will see /dev/hd{a,b,c,d}NsM.

Thanks a lot for your answers - some follow-up questions below:

So if I have this right, Linux will be able to view that space is available, but cannot read/write it? It would have to do the following:
- format it with a filesystem
- mount it
- now it can read/write?

New question:

If you need a filesystem to write to a disk, how do tools like fdisk work, which write bytes directly to a disk without any knowledge or use of filesystems?

Could you expand on this, I can't understand the example of when you'd not partition all the space - can't OSs not read unpartitioned space? So the other OS (Solaris, like you described) would still be out of luck? I'm a bit confused - it's all new to me.

Because disks partitioned using one tool (e.g. fdisk) on one OS (e.g. Linux) can be recognized, used, and manipulated by another OS (e.g. Windows), does this mean that the protocol for partitioning a disk (e.g. all that stuff about MBRs, partition sectors, etc) is a standard that relates to all disks, regardless of in what environment they'll be used?

Thanks.

Yes - but you will have to partition the space first using gparted, cfdisk, fdisk, or the like so Linux knows to create a node to that space in /dev.
When you parition a drive, the first X sectors of a drive a reserved for a partition table. Fdisk and the like do nothing more than read and write partition tables which, at the very least, contain the total size of the drive, start and end sectors for each partition, and the filesystem which they are inteded for regardless of whether or not they've been formatted yet. In fdisk and cfdisk, you can see code lists of all the different types of partition types. Any filesystem you would find in use on a Linux install (ext2, ext3, reiserfs, xfs, etc.) are all classified under the "Linux" and "Linux swap" codes (82 and 83, respectively, I believe).
I can only expand on this slightly as I don't have an intrinsic knowledge on the inner workings of Windows (dim the lights, add dark, sinister chanting and steady drum beats). It appears as though Windows will read the partition table and create a drive for the unpartitioned space. It won't format the space until you try to access it through the drive link it provides. However, Windows cannot (and will not) read non-FAT or non-NTFS partitions and will not set up drive letters for them.

Essentially, to access a paritioned and formatted drive, the OS needs to have - at the very least - installed drivers that provide read capabilities for that filesystem.

In the Linux world (since it is the most pertinent to know about), Linux will NOT attempt to create half-assed drive links (in Linux terms, "nodes") to unpartitioned spaces on a hard drive; it will leave well enough alone. It will only recognize drives which have been paritioned with a particular filesystem code as described in the parition table.

As an additional note, you don't need to format the drive for Linux to recognize it, it just needs to have an appropriate filesystem code in the partition table. After every parition table modification, you must restart your computer for Linux to read the partition table, and create appropriate nodes.
Yes. Reading and writing parition tables is standardized. However, an OS cannot read and/or write to a partition unless it knows how to, through drivers and such.

NOTE: MBR (Master Boot Records) have little, to nothing, to do with this discussion. I'm not slamming down, I'm just letting you know. But yes, MBRs are standardized as well. While writing to the MBR isn't standardized, HOW it is written to, is. Reading the MBR (at least from the standpoint of your computer booting up) is handled - practically - exclusively by the BIOS.

Just a couple of hair-splitting nuances:

Using low-level tools like dd you can read and write anywhere on the disk---even if it has no partitions.

I like to think of extended/logical partitions as a linked list. What actually happens is that the first logical requires the first extended to be created. (Many tools do this automatically.) Think of the extended as a pointer to the first logical. The logical partition contains a "boot sector" with space for the same 4-entry partition table that lives in the MBR. When the 2nd logical is created, an extended partition is created in that 1st boot sector---pointing to logical #2......etc.

Can you use all four entries in these "logical" partition tables? No way to do it that I have seen.

Reading your responses, I realize that things may be more complicated than I thought, with there being a difference between partitioning, formatting, and adding a filesystem to a disk! Could somebody please elaborate on these differences as it relates to this discussion?

Based on my understanding, I would guess that the dd tool can write to a disk where none of the above 3 things have been performed, but only parts of the disk that have been partitioned (though not necessarily formatted or made to contain filesystem) are visible to Linux?

Just when I thought I had it...

So do tools like fdisk that write out partition tables build upon the dd tool (or use the same underlying system calls, if I've used the term correctly, as dd)?

You've got me on the last one---One sure way to find out would be to get the source code for fdisk. Great project for a student: Recreate fdisk using BASH + all the built-ins (which includes dd)

Anything which is a filesystem function will only work in areas of the disk that have a filesystem--this is not unique to Linux. So, what is the definition of "filesystem function"? I would guess anything that can operate on filenames and directory names.

The dd command could read/write the bare space, placing anything in that space. You could, in fact, use dd to construct your own "file system" on otherwise bare and unformatted HD space. Wouldn't be efficient and would have no error tolerance, but you could do it. Actually, dd can read/write to any part of the disk, regardless of whether or not that space is partitioned and in use.

dd, fdisk, and every other tool that talks to a disk, communicates through a device driver which is a piece of code that translates a set of universal Linux system calls (such as open, close, read, write, seek) into device specific commands.

fdisk accesses the disk through this driver and implements a set of rules that exist throughout the x86 world for historical reasons, to build a disk partition structure that is recognizable by a large number of different operating systems. This need not be done; this built-in structure could be completely ignored by any OS but the ability of that OS to interoperate with - well - with a lot of things - would be greatly impaired. You can think of a partition as being a "logical disk", in that the system will treat it pretty much like a separate disk, while it is in fact only a part of a disk.

The file system also accesses the hard drive through this driver, working within one partition, but the file system has built a complete structure on the partition which enables files to be stored, found, and maintained. Basically, the filesystem defines and implements a substantial set of rules by which data is placed in the partition, while the dd command just puts anything you tell it to put anyplace you tell it to put it.

So, dd ignores all the file system and partitioning rules. This makes it very powerful, very useful, and very dangerous.

Yes. An extended partition uses only one entry in a partition table to point at an partition table entry in the first "logical" partition. That partition table entry can be unused, or it can point at the second "logical" partition. In this manner, you can create arbitrary number of partitions, limited only by the amount of disk space you have.

Only partitioned space is visible to Linux. Linux reads the partition table to get the disk addresses it uses to access space on the drives. It will not access access unallocated space or partitions it does not recognize unless an error occurs or you install a custom device driver that allows direct access to the hard drive interface.

The exception to this (as other posters have mentioned) is the fact that a hard drive looks like a file to the operating system (e.g. /dev/hda). You can open that file and seek to any position in it and read and write it. The tool most commonly used to do this type of operation is "dd". However, you could write a C program or even a Perl script to do the same thing. In all cases, you risk data loss.

a) Yes
b) One example: Some OEMs use this technique to hide system restoration information from the operating system so that it does not get corrupted by the user.
c) Yes

I do not know the answer to this question. I do know that an NTFS partition formatted using Linux works under Windows, but Windows often adds some extra housekeeping files to the disk (e.g. "System Volume Information" and "RECYCLER" directories).

There should be no performance differences. The operating system most likely reads the partition table information (including the partition tables of extended partitions) and saves the information in a cache. Once it has this information, it can calculate drive addresses without incurring any extra overhead no matter what the partition type is.

No. In my previous response, I was incorrect. You only need one extended partition. Inside of that, you can create an arbitrary number of "logical" partitions.

ok, patitioning relates to the harddrive, physical layout...

formatting is the way that an OS sets the flags for its filesystem to use that partition...

filesystem is the method that files are located on space and indexed...

these are the simple answers, and can be elaborated upon in more detail but then it becomes more complicated...

dd is a tool for low level work on a drive, and unless you have lots of drive knowledge, dont touch...
lowlevel is the actual viewing of data on the drive, irrelevant of drive fs and is how you can recover data thats erased(thats why high security organisations rewrite over deleted data several times, so it cant be recovered)...a lowlevel format will destroy any data on the surface of a drive, this is not recommended as it can sometimes even destroy the drive it self...

fdisk sets partition tables so that an OS can format that space for its own filesystem...
dd is another app that performs a task at a lower level of the drive...

I don't think so---the "secondary" partition table has the standard four entries, but only two get used. I don't know any partitioning tool that will use all 4.

(In the output of fdisk -l, you'll never see more than one extended partition.)

Actually, I can see where this would be extremely efficient (no slack space*).

Leave a chunk of disk unpartitioned and use dd to read and write from it.

You tell dd where & how much to read (Sectors? We don't need no stinkin' sectors!)

Nice way to keep private things private...

As for error tolerance, it's no worse than losing a sector in almost any other filesystem. Once it's gone it's gone.


Yup.


*http://en.wikipedia.org/wiki/Slack_space

I apologize for posting an erroneous reply.

pixellany is correct. The extended partition table entry is used to point to the first "logical" partition. The first sector (512 bytes) of the first logical partition is used to point to the next logical partition. In this manner, you can create an arbitrary number of partitions. I just verified this using fdisk on an IDE drive.