Looking into Middleware server logs made simple

Middleware Admin means the day runs with Emergency calls or On-calls, supporting team task always under high pressure cooker -- many of them feel tension tension due to heat in the head many got into hair-fall!!! ...

Its a challenging task, only those who can handle the jobs under pressure can do this with tricks. But When you don't have calls you might think to reduce your stress in the work with smart automation scripting to make life more easier.

Best Practices for WebLogic Logs

If the environment is shorter let say it is size of 10 - 20 managed servers, then the best idea is to implement is that all these managed server logs, JMS stores must shared a common mount point. When application support team run into issues, this will give you the flexibility to search the logs easily and faster way to get rid of issues with this unique log location.

In most of your project environments you might find the multiple logs such as application reated, JMS message stores or some transactional logs. In some critical situations, where you might need to look-up for a particular phrase of text in the WebLogic managed server generated logs. If you already follows best practices for collecting logs all together. It is easy to search in the sub sequant sub directories.

#FileName: CheckLogs.sh
#This script is used to find the given pharse in all logs available in the machine

INST_HOME="/home/wlusr/instances"
INSTANCES=`ls $INST_HOME|grep c`
phrase=$1
for instance in $INSTANCES
do
        echo 'Checking for the :' $instance
        cd $INST_HOME/$instance/logs
        egrep -i $phrase c*log*
        egrep -i $phrase c*out
done

If you keep above script in the general accessible path, such a way that it can be run from any directory on your machine. The Best practices for every WebLogic Middleware server running machines uses JAVA_HOME in their PATH. The best option to store this script is $JAVA_HOME/bin you can run from any path in your user.

Suppose you want to find all the WebLogic Servers which got RUNNING state on a machine. The search pattern you can give as bea-000360.

$ logverify bea-000360
Checking for the : cmgdserver01
 
      

Multi server code Backup

If you digest Operating Systems basics or SHELL basics, you might think much effectively and you can make  more optimized ways to leverage your work, you can design your script with enhanced ability with productivity. Once you taste that flavor in my scripts, you can feel that and understand that why did I told those exaggerated words. Let me share my experience on this.

In every J2EE production environment is going to have an application code (most of the developer say it the composition of 'dist' and 'properties'), which could be no-stage mode deployment style. Where you need to have a copy of code on every WebLogic server running machine. For every production release there could be success or fail. A wise WebLogic Administrator always takes the backup of the existing code and then go further to do fresh code deployment.

According to the best practices of WebLogic deployment process, First time you will do the sample test on the production environment by making single site available for test. That is if your application having EJB-Tier, Web-tier then, start each one server in the tier. Test the basic flow test, which makes you know wheather your new code works or not. Most of the cases the code come to production move is already checked on Staging environment which is the simulated environment of production environment.
It is always good to do double check your work before you do something on Production environments.

Here the important point is that to overcome failure situation, when a new changes in the code fails , such cases alternate solution is to revert back the old code.

How to do this?
Think!! Think wisely, Think for optimizing, Think for better outputs...

We can do simple copy command but what makes error-less work
Take backup with date and time stamp.
There could be multiple application codes on the same machine.
By accomplishing these above task with the following script on a machine.

clear
if [ $# -gt 0 ]; then
 today=`date "+%b%d_%y"`
 APPNAME=$1
 SRC=$HOME/$APPNAME
 BKP=$HOME/backups/$APPNAME
 mkdir -p $BKP
 BKP=$HOME/backups/"$APPNAME"_"$today"
 echo $SRC
 echo $BKP
  
 echo '1. Code backup [dist]only'
 echo '2. Full backup [includes dist, properties]'
 read -p "Please enter your option [1/2]:"  opt
       if [ $opt -eq 1 ] || [ $opt -eq 2 ]; then 
  case "$opt" in
  
 1)  echo "copying dist ..."
  cp -rp $SRC/dist $BKP
     ;;
 2)  echo  "copying all including properties..."
  cp -rp $SRC/dist $BKP
  cp -rp $SRC/properties $BKP
     ;;
 esac
        else
                echo "  invalid option "
        fi
else
    echo "Please enter application name in command line arguments"
fi

Multiple site backup
The above backup is required for every site (multiple machines). If you have already used ssh key generated and all sites are accessible without password. In Ramayan Ram's single arrow hits seven trees in a row. Similarly I want to make the backup script run on one machine and hit on every machine where the application is running. But to make this idea come true I took the help of LinkedIn discussion. Mr. John suggested to keep CMD and execute ssh command.

clear
today=`date "+%b%d_%y"`
if [ $# -gt 0 ]
then
        APPNAME=$1
        SRC=$HOME/$APPNAME
        BKP=$HOME/backups/"$APPNAME"_"$today"
        echo '1. Code backup only'
        echo '2. Full backup [includes properties]'
        read -p "Please enter your option [1/2]:" opt
        if [ $opt -eq 1 ] || [ $opt -eq 2 ]
        then
                CMD="mkdir -p $BKP"
                case "$opt" in
                1 )
                        echo "Will copy only dist."
                        CMD="$CMD ;    cp -rp $SRC/dist $BKP ;     ls -lrtR $BKP ;   exit"
                        ;;
                  2 )
                        echo "Will copy dist and properties."
                        CMD="$CMD ;    cp -rp $SRC/dist $SRC/properties $BKP ;    ls -lrtR $BKP ;   exit"
                        ;;
                 esac
  # serverlist is the text file
                for i in `cat serverlist`
                do
                        echo "Connecting to $i."
                        ssh -a $i $CMD
                done
                echo "done !"
           else
                echo " invalid option "
          fi
else
        echo "Please enter application name in command line arguments"
fi

This script worked... awesome