Hello people!

Server farms like Google and other big ones have tonns and tonns of HDDs working together and they have many HDDs die per day. If I'm not mistaken, IT wokers there just pull out dead HDDs from server racks and replace them with new ones on the fly, with no need to turn the servers off. Apparently they don't even care about on which disks their data is stored. It's like RAID but much more sophisticated than, say, RAID5 or RAID6 and one piece of data is stored on several HDDs spreaded all over server farm which may cover several huge buildings and even network of several buildings... At least it's how I understand it. Is it so? And if yes — what software do they use for this approach (to detect dead HDDs, their location and replace them on the fly)?

I believe you're broadly correct, but that they store the same info in more than one place and no-one worries if a fact does go missing; google search is not supposed to be definitive, much less ACID compliant.
The google spider bot will find it again ....

Google apps (docs, email etc) are different.

There's any num of ways of checking for bad disks; usually via snmp and some systems will notify you when a disk dies.

They probably don't bother for this app, but you can get HW RAID that allows hot swapping of disks and the OS never even notices

1 members found this post helpful.

The disk must support hotplug, sata disk are supposed to be hotpluggable. The disk subsystem also must support hotplug. Mdraid also supports hotplug. For the os to support hotplug raid is required.

Yes, and when you say there's one piece of data spread out over multiple drives...that IS what RAID is. And they don't use software (per se), to do this, they use SAN's. The 'disk' is presented to a server via a fiber channel host-bus adapter (HBA). What that disk is, depends on how the SAN administrator presents it. It could be one part of one disk, one whole disk, or 20 whole disks, split into four array's of 5. The operating system will 'see' one disk. That's it. All the hot-plug and failover happens in the SAN....the OS never knows.

You can buy cheap (relatively), hardware RAID systems, and define an array with a hot-spare drive, and the OS will notice that a drive has failed, but the system will keep running. You can then. swap out the failed drive, and it will rebuild the array, and put the spare back into its previous state.

1 members found this post helpful.

Google servers according to wikipedia
And if you want to have a look at google datacenters

In terms of detecting when a hard drive might fail, the utilities in the smartmontools package can be used, in particular the online self-test commands.

There are also distributed file systems (e.g. PVFS, Lustre) that automatically distribute data and metadata over multiple servers in a cluster. Google is using something like this, but they've written their own (if you search for GoogleFS, you should be able to find some high level descriptions of it).

Thanks for information.
There were some news about Google using ext4 (probably with their own modifications). They switched from ext2 without using ext3.

And, besides, I'm not only talking about Google. What about Twitter, Tumblr, Facebook, MySpace, Blekko, Wikipedia, Linkedin, PayPal, IBM, Intel, AMD...?

I'd think more like ZFS was used.