Quote:
Originally Posted by itsallgood
I am confused about the definition of a "filesystem".
I am understanding the basic components of a Linux filesystem and that it includes folders like /boot, /dev, /usr, /var, etc...
|
Indeed. Data 'on disk' is stored in blocks and there has to be some convention that somehow relates a collection of bits to some sort of meaning.
A filesystem is not quite what you think it is. The stuff about directories is a filesystem hierarchy. There are conventions for that and they evolve slightly over time, but, in principle, all you need is a root directory, and some way of assigning other data blocks to things like metadata and data blocks within files and to subdirectories.
That would allow you to build any hierarchy that you'd like, but, for it to have the conventional meaning, files have to appear in the hierarchy where you'd expect them to. This can be done on top of any supported filesystem, and really, from most points of view, you can't actually see which file system underlies the hierarchy without a bit of digging.
Quote:
Originally Posted by itsallgood
Am understanding they run on "nodes" which are separate parts of a working cluster. How do those work and what is the difference? Do they run on top of the Linux filesystem I am thinking of or are they completely different? Do these both have the /boot, /opt, etc subdirectories or am I completely off here?
|
In allocating data to files, nodes are very widely used. Mostly, you don't have to worry about nodes at all, unless something goes wrong (ie, file recovery, or something has gone wrong with the allocation process and you haven't allocated enough).
In principle, at least, it is not necessary to have one file system for your Linux box you could have your / partition on, say, ext4, your /home on F2FS, /boot on ext2, /opt on BTRFS, etc. It would be a bit eccentric, but the system would do it. All the system has to know is which filesystem to mount where (from /etc/fstab) and that information to be correct, and that will do it. Of course, there are some situations in which some filesystems give different performance from others, but you'd expect that.
Quote:
Originally Posted by itsallgood
To rephrase, do these HDFS and GPFS systems replace the basic Linux filesystem I am thinking of or are they mounted separately somehow? Not really sure how to phrase my question...
|
Well, mostly you won't have to bother about those two. The Hadoop Distributed File System must have some advantages for use with Hadoop, which does have uncommon data use patterns (I suspect, probably to with the 'distributed' word), but it doesn't see much general purpose usage. In any case, you could have a perfectly 'normal' (whatever that means) linux setup that had a HDFS partition mounted with some arbitrary name (say, data1, for example) and everything would work normally, except that you'd have this extra bit of the filesystem called /data1 which would be appropriate for your hadoop data.
Of course, and I need to emphasise this
just because you could, doesn't mean that you should. If it turns out that you have mentioned HDFS for some particular reason (such as, err, Hadoop, probably), then there will probably be some kind of guide or tutorial that you should read, and that will give you more information. Or, in the case of GPFS, something about the cluster of machines that you are setting up will have pushed you in that direction, and again some kind of usage guide will be available.
Both of the filesystems that you mention seem concerned with high performance in very particular circumstances and won't be of interest to most people most of the time. Do you have some particular requirements?
Another comment is that while things should be clearer once you have read that tldp document, it isn't very up to date, so don't think that it covers the latest developments.
