By jeremy at 2007-08-05 21:21
Remote Administration with DSH
Previous "Tech Support" columns introduced both Perlbal and Pound, two packages to load balance traffic across multiple machines. Dividing load among several servers provides better performance and increased reliability. However, more machines translates to more tedium. Previously simple actions - such as restarting a service - becomes complex.
There are multiple solutions to managing a collection of machines. One common solution is to write a custom script for each task. A more innovative solution is dsh, the "distributed shell." dsh is a wrapper implementation to execute remote shell commands on multiple machines. dsh abstracts the problem of multiple machines and offers a general solution.
dsh is provided under the terms of the GNU Public License and can be downloaded from the dsh home page at http://www.netfort.gr.jp/~dancer/software/dsh.html.en. Before you install, download, build, and install libdshconfig, which is also available from the dsh home page.
To build and install dsh, run the standard commands:
dsh has three system-wide configuration files, and you can extend those with personal settings.
$ sudo make install
*/usr/local/etc/machines.list and $HOME/.dsh/machines.list enumerate the machines to operate upon if the -a ("all machines") option is specified.
*/usr/local/etc/group/groupname and $HOME/.dsh/group/ groupname define the group named groupname. To operate on a group of machines, specify -g groupname.
*/usr/local/etc/dsh.conf and $HOME/.dsh/dsh.conf set parameters for dsh.
man dsh.conf describes all of the available configuration options. Both the machines.list and the dsh group files are a list of machines, one machine name per line.
For the best security, always set remoteshell=ssh and ensure that SSH keys provide ready and automatic access between all of the machines. Depending on your environment, the remoteshellopt option can also be extremely useful. You can use remoteshellopt to specify an alternate port number if your machines do not run SSH on port 22.
You’re now ready to use dsh. To get the output of the w command from all of the machines listed in machines.list, run:
If you'd like the name of the machine prepended to the output, add the -M option, as in dsh -M -a w.
While it should immediately be clear how useful dsh can be in a distributed environment, the machine group feature really increases the utility's flexibility. You can create a distinct group for your Web cluster, another for your database cluster, and yet another for your compute servers. Machines can be grouped arbitrarily to let you execute commands on a subset of your infrastructure.
For example, if you create a Web group, you can restart Apache on all machines with:
You can execute commands in a more ad-hoc fashion with -m (lowercase). For instance, the next command runs last on the machines test1 and production1:
$ sudo dsh -g web '/usr/local/apache/bin/apachectl restart'
You can even point dsh to an arbitrary file with a list of machines names with the -f flag. As you can see, while dsh is very simplistic, it was built with flexibility in mind.
$ dsh -m test1 -m production1 last
Finally, if you're concerned about overwhelming your machinery with commands or want to progress more slowly, try -F to limit the number of concurrent commands. In most cases, you can ignore such a governor, but if you use dsh on a huge number of machines, it's a good idea to set the limit somewhere between 100 and 200.
dsh is a simple mechanism to execute the same command on any number of remote machines. Its group feature makes it flexible enough for almost any environment.