When one of my customer called me that his AIX server hanging on AIX Boot on LED 581 , I was of the instantaneous point of view that problem is just related to Network configuration of server.
When i was driving to customer site , i was thinking that i will come back to office within 1 hour and enjoy lunch at office.But when i reached at customer site , i realized that problem seems huge as there are almost 4 VIO servers and two dedicated AIX boxes which were having almost 30 minutes hang at 581 LED during boot.There was some POWER related issue at their data center and therfore their all network equipments as well as servers faced POWER failure.
AIX servers were completing boot process , but never giving a login prompt.
VIO servers were also having successful boot , but giving some abnormal behavior.
We decided to not to start AIX client Lpars ( using VIO servers ) until we know the main reason for 581.
While searching on internet , i could not find any details about LED except that this LED code has something to do with network configuration and DNS settings.
While inuring with customer , we realized that they do have some issues with their core switch, DHCP services, DNS services and their PDC. Their windows based PDC was also crashed and was facing some problem related to network adapter driver corruption.
While wasting almost 3-4 hours at customer site , and multiple restart , i was quite sure that we have no other option except wait till their all network related issues resolve.
After 4 hours they were successful in bringing their PDC ( DHCP/DNS) up and as soon as DNS server started pinging, my all AIX Lpars, servers and VIO servers stopped hanging at 581 ( without doing any thing on AIX).
Lesson learnt:If you AIX servers are DNS clients then you are dependent on DNS server. If DNS server is down and you try to reboot AIX or VIOS then you may face issue while rebooting AIX ( LED 581).Better to put two DNS servers entry in AIX or at least put hosts=local,bind4 entry in /etc/netsvc.conf file.
No comments:
Post a Comment