Configuring Distributed shell with SSH for AIX software Management

System administrators of large AIX installations very often find them in situations where they want to execute operating system commands in parallel on multiple systems. Imagine about a situation where a system administrator has to manage 100-200 AIX based servers which are widely spread across multiple sites. One morning when he comes in office and as soon as he starts his bright and sunny morning with a hot cup of coffee, planning about coming weekend, he receives an emergency call from his immediate boss to prepare a list of all such AIX servers which have not been installed with a security related APAR (which has to be installed immediately as per IBM instructions). Now his day starts with a laborious task; he has to telnet (or Ssh) to each and every AIX server and check the availability of the specified APAR on all servers, one by one. If not present, he also has to apply this specific APAR on the systems.

We can not deny that such tasks are essential part of system admin roles and jobs, but some times, such jobs become a great burden for system administrators especially in scenarios where installation base is very large. IBM luckily has now included some very useful commands in AIX 5L, which can help system administrators in such scenarios. In this article, I will cover these relatively new and less commonly used (but very useful) commands and their configuration using Ssh as way of passing data between systems. Later in my following article, I will show how dsh and dshbak commands can play a vital role in making AIX day to day system administration tasks a bit easier.

Distributed Shell and its configuration with SSH

Dsh (distributed shell) is a tool which was initially part of PSSP component for SP/2 systems. However, with the introduction of CSM client software with AIX 5L, now it has become a standard for AIX 5L. Both dsh and dshbak commands are now available with AIX 5.2 base operating system also. You however have to install following csm related filesets (including csm.dsh) to get these commands available on AIX 5.2 systems.

Following are the csm related filesets, which will be required for availability of dsh and dshbak commands.

bkmecomm[/home/root] # lslpp -l csm*

Fileset Level State Description

----------------------------------------------------------------------------

csm.client 1.4.1.0 COMMITTED Cluster Systems Management

Client

csm.core 1.4.1.0 COMMITTED Cluster Systems Management

Core

csm.diagnostics 1.4.1.0 COMMITTED Cluster Systems Management

Probe Manager / Diagnostics

csm.dsh 1.4.1.0 COMMITTED Cluster Systems Management Dsh

csm.gui.dcem 1.4.1.0 COMMITTED Distributed Command Execution

Manager Runtime Environment

csm.gui.websm 1.4.1.0 COMMITTED CSM Graphical User Interface

csm.msg.EN_US.core 1.4.0.0 COMMITTED CSM Core Func Msgs - U.S.

English (UTF)

csm.msg.en_US.core 1.4.0.0 COMMITTED CSM Core Func Msgs - U.S.

English

You can easily verify that csm.dsh is actually the fileset containing dsh and dshbak command. All these CSM related filesets can be found in AIX media pack CDs.

# whereis dsh

dsh: /usr/bin/dsh

#lslpp –w /usr/bin/dsh

lslpp -w /usr/bin/dsh

File Fileset Type

----------------------------------------------------------------------------

/usr/bin/dsh csm.dsh Symlink

Now next step would be addition of directory containing dsh and dshbak commands in current path. I did it temporarily using PATH environment variable, you can alternatively edit /etc/environment or .profile to get it done permanently

#export PATH=$PATH:/usr/bin

You have to specify which nodes, you want to add in your dsh management domain .In my case, I need that dsh should be able to execute commands on all of my AIX based database and application servers. As a practical solution, it is also better to have only one node as the management server for all of remaining servers in server Farm. So I selected my AIX 5.2 based “bkbweb” server as management server. On bkbweb, I installed all csm related filesets and added dsh and dshbak commands path in root’s profile. I also created a file called serverlist and putted names of all nodes (which are required to be managed from this bkbweb server using dsh).It is important, off course, that you should be resolving all node names successfully from this management server.

The contents of serverlist file are nothing but simply containing names of all nodes, one name per line.

---------------------------------------------------------------------------------------

bkbdb

bkbapp

bkbqdb

bkbsapp

---------------------------------------------------------------------------------------

After that, I exported the environment variable “DSH_LIST” in root’s .profile

export DSH_LIST=/home/root/serverlist

I then verified this environment variable existence in root user environment by using following command

root@bkbweb-/# env | grep DSH

DSH_LIST=/home/root/serverlist

Next important step would be the specification of way of communication used by dsh command between nodes. The dsh program uses a remote shell of your choice to issue remote commands to the managed nodes which means that in our case root user on bkbweb server should be able to execute rsh to all nodes ( specified in serverlist file ) successfully otherwise dsh will not work properly and you will get error ( something like following)

#/home/root dsh date

dsh: 2617-009 bkbdb remote shell had exit code 1

As rsh is considered as a security loophole for many corporate and therefore not acceptable, I decided to use Ssh as communication infrastructure between nodes for dsh. For this to work, I installed openssh on all of my AIX nodes and started sshd daemon on these AIX nodes.

On bkbweb server, I generated pair of public/private keys using following command:

#ssh-keygen –t dsa –b 2048

This command will generate public and private keys in /home/root/.ssh directory. For sake of simplicity, I did not use any passphrase while storing these keys in the files

( id_dsa and id_dsa.pub).

I then copied public file from bkbweb server to all AIX nodes using scp command as follows:

#scp /home/root/.ssh/id_dsa.pub root@bkbdb/home/root/.ssh/

#scp /home/root/.ssh/id_dsa.pub root@bkbapp/home/root/.ssh/

Now, at bkbdb and bkbapp nodes, I putted these public key files in authorized_keys file

# cd /home/root/.ssh

# cat id_dsa.pub >> authorized_keys

Ssh connectivity with DSA authentication (from bkbweb server to bkbdb and bkbapp) should work now and you will be able to able to login from bkbweb to bkbdb and bkbapp nodes without any password prompt.

Final step would be configuring dsh to use this fully functional Ssh setup. This can be done easily be the use of an environment variable “DSH_REMOTE_CMD”.

I exported this environment variable in root user profile at bkbweb server as follows

#export DSH_REMOTE_CMD=/usr/local/bin/ssh

Now test date command with dsh, which will execute date command simultaneously on all servers specified in serverlist file and will return following output

root@bkbweb-/home/root# dsh date

bkbapp: Mon 15 Jan 11:15:22 2007

bkbdb: Mon 15 Jan 11:15:24 2007

bkbqdb: Mon 15 Jan 11:15:27 2007

bkbsapp: Mon 15 Jan 11:15:22 2007

You can also now use dshbak command in conjunction with dshbak command. This “dshbak” command group all those nodes for which dsh command output is same. For example, if we execute same above command with dshbak –c option , output would like as follows:

root@bkbweb-/home/root#dsh date | dshbak –c

HOSTS ------------------------------------------------------------------------

bkbapp, bkbsapp

-------------------------------------------------------------------------------

Mon 15 Jan 11:17:34 2007

HOSTS-------------------------------------------------------------------

bkbdb

Mon 15 Jan 11:17:36 2007

HOSTS----------------------------------------------------------------

bkbqdb

Mon 15 Jan 11:17:39 2007

Sample Software Maintenance Scripts using DSH

You can make use of distributed shell implementation for many administration tasks across server farm. Below are some sample scripts, which can be proved very helpful for system administrators in AIX software maintenance across server farm.

For instance, combination of dsh and “oslevel –r” command could be used in finding all those servers which are below ML05 for AIX 5.2. I have written a small and simple shell script for this purpose which infact uses already established Ssh based dsh command setup.

-----------------------------------------------------------------

#mlfind.sh

#Created by Khurram Shiraz on 15 Jan 2006

# Help system administrators to find all those servers which are not currently at any

# specified ML- Example Usage : mlfind.sh 5200-06

-------------------------------------------------------------

#!/bin/ksh

dsh "oslevel -r" > /tmp/wrkfile # using dsh to get required output from all servers

while read HOSTNAME ML

do

HOSTNAME=$(echo $HOSTNAME | sed s/\://g) # Removing colon (:) from host names

if [ $ML -eq "$1" ]

then

echo

echo $HOSTNAME has specified ML installed

else

echo

echo $HOSTNAME does not have specified ML installed

fi

done < /tmp/wrkfile

exit 0

-----------------------------------------------------------------------------------

And for system administrators, who want to check presence of any critical fix in hundred of AIX servers, the same script can be modified slightly as follows:

-----------------------------------------------------------------

#fixfind.sh

#Created by Khurram Shiraz on 15 Jan 2006

# Help system administrators to find all those servers which are not currently applied #with a specified patch: example usage fixfind.sh IY43265

-------------------------------------------------------------

#!/bin/ksh

dsh "instfix -ik $1" 1> /tmp/wrkfile 2>/tmp/wrkfile

while read HOSTNAME FIXRESP

do

HOSTNAME=$(echo $HOSTNAME | sed s/\://g)

echo $FIXRESP | grep -E "Not|no" 1>/dev/null

if [[ $? -eq 0 ]]

then

echo $HOSTNAME does not have speceified FIX installed

echo $HOSTNAME > /tmp/servers_no_patch

else

echo $HOSTNAME have this specified FIX installed

fi

done < /tmp/wrkfile

exit 0

---------------------------------------------------------------------

If you execute fixfind.sh with the FIX number, you will get some output like following:

root@bkbweb-/home/root# ./fixfind.sh IY54515

bkbapp does not have speceified FIX installed

bkbdb does not have speceified FIX installed

bkbweb have this specified FIX installed

Now it is time to see how we can use our dsh setup to apply specific APARs on all those nodes which don’t have that APAR installed. From the execution result of last script (fixfind.sh), we have list of all those servers which don’t have this APAR( in /tmp/servers_no_patch file). So, first step would be downloading this APAR and then transferring related filesets to a filesystem (/swexport) on our management server (bkbweb).Then export this filesystem ( I am assuming that NFS setup on bkbweb is already working fine ) by executing

# /usr/sbin/mknfsexp -d '/swexport' -t 'rw' -c 'bkbdb bkbapp' '-B'

Also modify servers list by pointing to only those servers which don’t have patch installed. Name of these servers are present in /tmp/servers_no _patch file.

#export DSH_LIST=/tmp/servers_no_patch

Now mount the /exportsw filesystem on all nodes simultaneously

# dsh mount bkbweb:/swexport  /mnt

Finally, apply this patch to all these nodes

# dsh instfix –k -d /mnt

After successful completion, unmount and unexport the filesystem

# dsh unmount /mnt

And on management server (bkbweb)

#exportfs -u /swexport

In summary, if you have server farms to manage, distributed shell from IBM is a gift for you. No doubt, you have to implement dsh (along with Ssh for better security) across whole server farm for the first time , but once it is installed and configured properly it can make your life bit easier. Every evening, you can be at home in time, rather than sitting in office for late timings and preparing comparison reports between hundred of servers for submission to your management. Thanks Dsh!!!!

About Author: Khurram Shiraz is senior system Administrator at KMEFIC, Kuwait. In his eight years of IT experience, he worked mainly with IBM technologies and products especially AIX, HACMP Clustering, Tivoli and IBM SAN/ NAS Storage. He also has worked with IBM Integrated Technology Services group. His area of expertise includes design and implementation of high availability and DR solutions based on pSeries, Linux and windows infrastructure. He can be reached at aix_tiger@yahoo.com.

Note: This article was published in AIX Update, Xephon Inc ( A Print Magazine which was used to publish monthly from US)