LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Linux Answers > Hardware
User Name
Password

Notices


By sindimo at 2009-10-29 14:48
Author: Mohamad Sindi
Email: sindimo@ieee.org


__________________________________________________________________________________________________

This is a detailed procedure of how to install a cluster of diskless Linux nodes. The procedure if fully done via command line tools so it is convenient to script and use for large scale clusters. My operating system of choice was RedHat EL 5.3 but it should work with other distributions as well. RedHat's documentation doesn't seem to have any procedure on doing this using command line, they only illustrate it using a GUI tool which is not convenient at all to use when you have a large number of nodes in a diskless cluster in an enterprise enviornment, so I thought it would be useful to share this with everyone, I hope you find it useful.
__________________________________________________________________________________________________


Table of Contents:

1. Introduction
2. Setting up the Server Node
3. Setting the Diskless Environment via Command Line
4. Booting the Diskless Clients
5. Modifying the Golden Image Node


1. Introduction


This is a step by step procedure to set up a cluster of diskless compute nodes along with a node acting as the server for the diskless environment. The operating system of choice was RedHat EL 5.3 using an x86_64 IBM Infiniband cluster. We cover how to do it in specific details via RedHat’s netboot. We will use the command line version of it since the GUI version might not be an efficient method to use when setting a large number of diskless nodes.

Types of nodes that are used for this demonstration:
• node002 (Image server – IP 10.8.226.72)
• node003 (Diskless client – IP 10.8.226.73)
• node004 (Golden image node installed on disk – IP 10.8.226.74)


2. Setting up the Server Node


First of all, install the server node node002 with all RedHat packages for simplicity (use “@Everything” in kickstart packages section) and disable firewall and selinux.

Install one of the client nodes, e.g. node004, normally on disk and include whatever settings and software you want to have on it, this will be the golden source for our diskless image.

It is very important to make sure that the “busybox-anaconda” package is installed on the client golden image node and on the server node since it’s needed for diskless environments:
[root@node002 ~]# rpm -qa | grep busybox-anaconda
busybox-anaconda-1.2.0-4.el5
[root@node004 ~]# rpm -qa | grep busybox-anaconda
busybox-anaconda-1.2.0-4.el5


Create the root diskless directory (this will be shared with all diskless clients in Read Only mode):
[root@node002 ~]# mkdir -p /diskless/i386/RHEL5-AS/root

Copy your golden image (node004) to the server (make sure to exclude /proc and /sys):
[root@node002 ~]# rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' node004:/ /diskless/i386/RHEL5-AS/root/

Verify it has been copied:
[root@node002 ~]# du -hs /diskless/i386/RHEL5-AS/root
1.6G /diskless/i386/RHEL5-AS/root


Make sure tftp is running on the server:
[root@node002 ~]# chkconfig xinetd start
[root@node002 ~]# chkconfig tftp on
[root@node002 ~]# chkconfig --list | grep tftp
tftp: on


Setup your DHCP configuration file, I used the below settings (obviously modify it with your specific IPs, MACs, PXE booting NIC, hostnames, DNS, etc…):

[root@node002 ~]# cat /etc/dhcpd.conf

Code:
authoritative;
ddns-update-style none;

allow bootp;
allow booting;

shared-network eth0 {

subnet 10.8.226.0 netmask 255.255.255.0 {
        option routers		10.8.226.250; # --- default gateway
	option subnet-mask		255.255.255.0;
        option nis-domain	        "<YOUR-DOMAIN>";
	option domain-name		"<YOUR-SEARCH-DOMAIN>";
	option domain-name-servers	<YOUR-DNS-IP>;
	option time-offset		-18000;	# Eastern Standard Time
	default-lease-time 		21600;
	max-lease-time 			43200;
        next-server 		10.8.226.72; #<--- TFTP server IP here

	host node003 {
		hardware ethernet 00:21:5E:40:64:5C;
		fixed-address 10.8.226.73;
        	              filename "linux-install/pxelinux.0";
        		}

	host node004 {
		hardware ethernet 00:21:5E:40:63:C8;
		fixed-address 10.8.226.74;
        	              	filename "linux-install/pxelinux.0";
	             }
   }
}
Make sure the DHCP is running:
[root@node002 ~]# chkconfig dhcpd on
[root@node002 ~]# service dhcpd start
[root@node002 ~]# service dhcpd status
dhcpd (pid 28250) is running...


Set up NFS (notice the read-only and read-write settings):
[root@node002 ~]# cat /etc/exports
/diskless/i386/RHEL5-AS/root *(ro,sync,no_root_squash)
/diskless/i386/RHEL5-AS/snapshot *(rw,sync,no_root_squash)
[root@node002 ~]# chkconfig nfs on
[root@node002 ~]# service nfs start
[root@node002 ~]# showmount -e node002

Export list for node002:
/diskless/i386/RHEL5-AS/root *
/diskless/i386/RHEL5-AS/snapshot *



3. Setting the Diskless Environment via Command Line


Now we need to add clients to the diskless environment and create a snapshot image for each diskless client. The system-config-netboot is a RedHat GUI utility to do so. However, what if you have a cluster with hundreds of nodes and you want to add them to the diskless environment, it would be a hassle to use the GUI to do so. There are 2 commands that come with the system-config-netbook package which are called “pxeos” and “pxeboot”. The “pxeos” enables you to create a new operating system under /tftpboot and places the vmlinux and initrd images there. The “pxeboot” enables you to add new hosts to the diskless environment. The snapshot image for each diskless client contains data that the client has read-write access to, for example /var/log/messages and so on.

Below is how to add a new O.S. called “rhel5compute” using “pxeos”:
[root@node002 ~]# pxeos -a -i rhel5compute -p NFS -D 1 -s 10.8.36.26 -L /diskless/i386/RHEL5-AS rhel5compute
Kernel not specified, using 2.6.18-128.el5


To verify that the O.S. was created:
[root@ node002 ~]# pxeos -l
rhel5compute
Description: rhel5compute
Protocol: NFS
isDiskless: True
Server: 10.8.36.26
Location: /diskless/i386/RHEL5-AS


[root@ node002 ~]# ls /tftpboot/linux-install/rhel5compute/
initrd.img vmlinuz


Options for “pxeos”:
-a: Adds new O.S.
-i: Description of O.S.
-p: Protocol to use
-D: If set to 1, then it indicates it’s diskless
-s: NFS server
-L: Location of your O.S., this should contain the vmlinuz and initrd somewhere in it.

The “pxeboot” command adds clients to the diskless environment and creates the HEX file for the host in the /tftpboot directory and it also creates the snapshot directory for the individual nodes being added. You can script it to do all of this easily, below is an example for a 128 node cluster.

First thing create a file with the IPs and hostnames that need to be added, you can use the entries found in your /etc/hosts file if they exist:
[root@node128 ~]# cat nodes

Code:
10.8.136.1	node001
10.8.136.2	node002
10.8.136.3	node003
………………………
10.8.136.128	node128
Now use the script below to add all 128 nodes to your diskless environment:
[root@node128 ~]# cat add-pxe-hosts

Code:
#!/bin/bash
#sindimo

#This script will generate the HEX files in /tftpboot as well as the individual snapshoot directories for nodes. The nodes file has 2 columns, first one is the IP and the second is the hostname

while read line
do 
ip=`echo $line | gawk -F" " '{ print $1 }'`
host=`echo $line | gawk -F" " '{ print $2 }'`

pxeboot -a -O rhel5compute -r 28753 -S $host -e eth0 -N my_nis_domain -s console=ttyS0,115200n8r     $ip

done < nodes
Options for “pxeboot”:
-a: Adds new host
-O: Name of O.S. (You should have already created this with the system-config-netboot or pxeos)
-r: Size of ram disk
-S: Snapshot name of the diskless client.
-e: The interface through which the pxe boot will take place.
-N: NIS domain
-s: Serial console settings to view remote consoles during boot up

Now verify that all 128 hex files for tftpboot were created:
[root@node128 ~]# ls /tftpboot/linux-install/pxelinux.cfg/ | grep 0A | wc -l
128


An example of the content of one of the HEX files for node node001:
[root@node128 ~]# cat /tftpboot/linux-install/pxelinux.cfg/0A088801

Code:
default rhel5compute

label rhel5compute
    kernel rhel5compute/vmlinuz
    append console=ttyS0,9600n8 initrd=rhel5compute/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=10.8.226.72:/diskless/i386/RHEL5-AS ramdisk_size=28753 ETHERNET=eth0 SNAPSHOT=node001 NISDOMAIN=my_nis_domain

Verify that all 128 snapshots were created:
[root@node128 ~]# ls /diskless/i386/RHEL5-AS/snapshot/ | grep node | wc -l
128


Each snapshot image should have a unique name since its data is specific to that client and the client has read-write access to it.
[root@node002 ~]# du -hs /diskless/i386/RHEL5-AS/snapshot/node003
80M /diskless/i386/RHEL5-AS/snapshot/node003
[root@node002 ~]# ls /diskless/i386/RHEL5-AS/snapshot/node003
boot etc home lib root var


Verify that the snapshot directory contains a text file called “files” which lists the files that can be mounted in read-write mode. Do not modify the “files” file manually, if you need to add more read-write files in your diskless system, create a new file called “files.custom” with the same format as “files” and list the desired files in it:

[root@node002 snapshot]# cat /diskless/i386/RHEL5-AS/snapshot/files

Code:
#
# This file contains the list of files/directories to be stored in the 
# snapshot directory for each diskless client.  Please do not edit this file,
# as Red Hat will be updating it with each release. If you wish to 
# add files please create a files.custom in this directory and add entries to it.
#
/boot/kernel.h
/etc/resolv.conf
/etc/resolv.conf.predhclient
/etc/yp.conf
/etc/yp.conf.predhclient
/etc/ntp.conf
/etc/ntp.conf.predhclient
/etc/ntp/step-tickers
/etc/ntp/step-tickers.predhclient
/etc/adjtime
/etc/cups/certs/
/etc/reader.conf
/etc/smartd.conf
/etc/printcap
/etc/aliases.db
/etc/modules.conf
/etc/fstab
/etc/sysconfig/hwconf
/etc/X11/XF86Config
/etc/X11/XF86Config.backup
/etc/X11/xorg.conf
/etc/X11/xorg.conf.backup
/etc/ssh/
/etc/selinux/
/etc/blkid/
/var/empty/sshd/
/var/gdm/
/var/lib/nfs/
/var/lib/random-seed
/var/lib/xkb/
/var/tmp/
/var/lock/
/var/run/
/var/spool/
/var/log/
/var/lib/logrotate.status
/var/cache/man/
/var/lib/misc/
/var/lib/mrtg/
/var/lib/dhcp/
/var/lib/dhclient/
/var/lib/ntp/
/var/lib/setroubleshoot/
/var/yp/binding/
/root/
/home/
Verify that the kernel images have been created:
[root@node002 ~]# ls /tftpboot/linux-install/rhel5compute
initrd.img vmlinuz



4. Booting the Diskless Clients


At this stage the server is ready. Now we need to boot the clients into their diskless environment. Make sure that your client’s BIOS and NIC cards support PXE boot and that that they are setup in the BIOS to PXE boot via eth0 since our DHCP was configured with the MACs of eth0.

You can verify that the tftpboot file for the diskless client is ready by checking the tftpboot directory and file below for node003:
[root@node002 ~]# cat /tftpboot/linux-install/pxelinux.cfg/0A08E249

Code:
default rhel5compute

label rhel5compute
    kernel rhel5compute/vmlinuz
    append  initrd=rhel5compute/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=10.8.226.72:/diskless/i386/RHEL5-AS ramdisk_size=28760 ETHERNET=eth0 SNAPSHOT=node003 NISDOMAIN=my_nis_domain
When a diskless client boots, the DHCP server will assign it an IP and the vmlinux and initrd.img images will be loaded via tftp to the memory of the diskless client and the client will boot up giving you the login prompt. After you log in you can verify that it is running from memory and not from disk (/dev/sda) by running commands such as “df” and “fdisk” and verifying your /etc/fstab file which indicates a diskless file system.


5. Modifying the Golden Image Node

If later on you need to install more packages on these nodes, you only need to install it on the golden image node then sync your root directory on the server. Another example if you need to update the root password on all nodes, you just need to change it on the golden image and sync.

For example let’s say you want to install the “dstat” package on all nodes.

Go to the golden image node node004 which is running on normal disk and install it there:
[root@node004 ~]# yum install dstat

Now on your NFS server sync again with the golden image:
[root@node002 ~]# rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' node004:/ /diskless/i386/RHEL5-AS/root/

Now the diskless client (node003) should be updated on the fly with no need for a reboot. This node has 12GB of physical RAM and only around 250MB seems to be used by the diskless environment:
[root@node003 ~]# dstat -m
------memory-usage-----
used buff cach free
131M 256k 116M 11G



I hope you find this useful :)

Mohamad Sindi
sindimo@ieee.org


  



All times are GMT -5. The time now is 10:16 AM.

Main Menu
Advertisement
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration