Looking into Middleware server logs made simple

Emergency calls, On call supporting team task always tension tension...
Its a challenging task, only those who can handle the jobs under pressure can do. But When you don't have calls you might think wisely to reduce your stress in work with wise shell/PERL/any other scripting to make life more easier.

In most of your project environments you might find the multiple logs. In some critical situations you might need to look-up for a particular phrase in the WebLogic instances generated logs. If you already follows best practices for collecting logs all together. It is easy to search in the sub sequant sub directories.

#FileName: CheckLogs.sh
#This script is used to find the given pharse in all logs available in the machine

INST_HOME="/home/wlusr/instances"
INSTANCES=`ls $INST_HOME|grep c`
phrase=$1
for instance in $INSTANCES
do
        echo 'Checking for the :' $instance
        cd $INST_HOME/$instance/logs
        egrep -i $phrase c*log*
        egrep -i $phrase c*out
done

If you keep above script in the general accessible path, such a way that it can be run from any directory on your machine. The Best practices for every WebLogic Middleware server running machines uses JAVA_HOME in their PATH. The best option to store this script is $JAVA_HOME/bin you can run from any path in your user.

Suppose you want to find all the WebLogic Servers which got RUNNING state on a machine. The search pattern you can give as bea-000360.

$ logverify bea-000360
Checking for the : cmgdserver01
 
      

Multi server code Backup

If you digest Operating Systems basics or SHELL basics, you might think much effectively and you can make  more optimized ways to leverage your work, you can design your script with enhanced ability with productivity. Once you taste that flavor in my scripts, you can feel that and understand that why did I told those exaggerated words. Let me share my experience on this.

In every J2EE production environment is going to have an application code (most of the developer say it the composition of 'dist' and 'properties'), which could be no-stage mode deployment style. Where you need to have a copy of code on every WebLogic server running machine. For every production release there could be success or fail. A wise WebLogic Administrator always takes the backup of the existing code and then go further to do fresh code deployment.

According to the best practices of WebLogic deployment process, First time you will do the sample test on the production environment by making single site available for test. That is if your application having EJB-Tier, Web-tier then, start each one server in the tier. Test the basic flow test, which makes you know wheather your new code works or not. Most of the cases the code come to production move is already checked on Staging environment which is the simulated environment of production environment.
It is always good to do double check your work before you do something on Production environments.

Here the important point is that to overcome failure situation, when a new changes in the code fails , such cases alternate solution is to revert back the old code.

How to do this?
Think!! Think wisely, Think for optimizing, Think for better outputs...

We can do simple copy command but what makes error-less work
Take backup with date and time stamp.
There could be multiple application codes on the same machine.
By accomplishing these above task with the following script on a machine.

clear
if [ $# -gt 0 ]; then
 today=`date "+%b%d_%y"`
 APPNAME=$1
 SRC=$HOME/$APPNAME
 BKP=$HOME/backups/$APPNAME
 mkdir -p $BKP
 BKP=$HOME/backups/"$APPNAME"_"$today"
 echo $SRC
 echo $BKP
  
 echo '1. Code backup [dist]only'
 echo '2. Full backup [includes dist, properties]'
 read -p "Please enter your option [1/2]:"  opt
       if [ $opt -eq 1 ] || [ $opt -eq 2 ]; then 
  case "$opt" in
  
 1)  echo "copying dist ..."
  cp -rp $SRC/dist $BKP
     ;;
 2)  echo  "copying all including properties..."
  cp -rp $SRC/dist $BKP
  cp -rp $SRC/properties $BKP
     ;;
 esac
        else
                echo "  invalid option "
        fi
else
    echo "Please enter application name in command line arguments"
fi

Multiple site backup
The above backup is required for every site (multiple machines). If you have already used ssh key generated and all sites are accessible without password. In Ramayan Ram's single arrow hits seven trees in a row. Similarly I want to make the backup script run on one machine and hit on every machine where the application is running. But to make this idea come true I took the help of LinkedIn discussion. Mr. John suggested to keep CMD and execute ssh command.

clear
today=`date "+%b%d_%y"`
if [ $# -gt 0 ]
then
        APPNAME=$1
        SRC=$HOME/$APPNAME
        BKP=$HOME/backups/"$APPNAME"_"$today"
        echo '1. Code backup only'
        echo '2. Full backup [includes properties]'
        read -p "Please enter your option [1/2]:" opt
        if [ $opt -eq 1 ] || [ $opt -eq 2 ]
        then
                CMD="mkdir -p $BKP"
                case "$opt" in
                1 )
                        echo "Will copy only dist."
                        CMD="$CMD ;    cp -rp $SRC/dist $BKP ;     ls -lrtR $BKP ;   exit"
                        ;;
                  2 )
                        echo "Will copy dist and properties."
                        CMD="$CMD ;    cp -rp $SRC/dist $SRC/properties $BKP ;    ls -lrtR $BKP ;   exit"
                        ;;
                 esac
  # serverlist is the text file
                for i in `cat serverlist`
                do
                        echo "Connecting to $i."
                        ssh -a $i $CMD
                done
                echo "done !"
           else
                echo " invalid option "
          fi
else
        echo "Please enter application name in command line arguments"
fi

This script worked... awesome

Check system information for Capacity Planning

Working for migration projects requires lots of mesures about the current running environment and predictions, estimations about the newly, about to make environment. Thinking BIG!!! as architect for the enterprise applications, I started digging details of the system information of every machine I mean, every box and I need to prepare a table that will give all the details about the machine. This will make easy for deciding various things for WebLogic domain and its related environment.

The Capacity plan is critical where a WebLogic Admin need to be more smarter to make wise decisions on infrastructure. The mesures enables you to predicate proper estimations for your application environment. I had seen there is a legacy shell script which gives the required information for Web Admin. But it is not really robust and effective to give the desired outputs. I thought better to renovate it myself.  What do you say? Yes! right hmmm, I on top of it :).

First understood each line it was written in the script. then I refine those lines as per my thoughts and tried to get the effective output. bugging my friends if I m not able to get desired outputs.
One Generic factor is that, I want to make this script must executed on both Solaris and Redhat Linux operating environment. The script must give outcome following factors
1. CPU Architecture
2. Num CPUs and  CPU Speed
3. Operating System Release version
4. Memory Sized (RAM)
5. Server platform
6. Virtual IP Addresses



Hope you are excited to see what is that script ? Here it is...

#===================================
# FileName: CheckSystem.sh
#!/bin/bash
PATH=$PATH:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:/bin
export PATH
 
os=`uname`
case $os in
SunOS)
 nproc=`psrinfo |wc -l` ;
 prspeed=` psrinfo -v | grep -i Hz |awk '{print $6" " $7}'|uniq  `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' `
 mem=`prtconf |grep -i size`
 ;;
Linux)
 nproc=`cat /proc/cpuinfo |grep -i processor |wc -l`
 prspeed=`cat /proc/cpuinfo |grep -i model |grep -i hz |awk '{print $7 }'|uniq  `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' |cut -d ":" -f 2`
 mem=`cat /proc/meminfo |grep -i MemTotal`
 ;;
esac
hname=`hostname`

echo "Host name : $hname"
echo "Operating system is $os release `uname -r `"
echo "Number of CPU : $nproc with the speed of $prspeed"
echo "RAM Size: " $mem

for IPAddr in $inet_addrs
do
 if [ $IPAddr != "127.0.0.1" ]
 then 
  if [ $os = "Linux" ]
  then
   
   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  else
   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  fi
  if [ ! -z ${IPAddr_name} ]; then 
   echo $IPAddr === $IPAddr_name 
  fi 

 fi
done

After preparing my statistical table, reviewed and got an idea for include the disk space avaialble on every machine. That is the great result of good reviewer (my buddy did it!). So I need a disk size that I can get by df command but -lk is going to give in kilobytes, option -h works on Linux, Solaris 10 but summing up is issue so I took kilobytes and did convertions as follows:

df -lk | egrep -v "Filesystem|/proc|/dev/fd|swap|/home|/platform" | awk '
 { t += $2 }
 { u += $3}
 {GB2 = 1024*1024} END 
 { printf "%d of %d Gb in use.\n", u/GB2, t/GB2 }'

This command line excludes header, /proc, floppy, swap, user space and sun packages in /platform for Veritas.

35 of 130 Gb in use

Good Reference:

http://www.brandonhutchinson.com/Gathering_Solaris_system_statistics.html

Keep writing your ideas, suggestions in comments!!