JMS MessageBridgeRuntime monitoring

We were in the developing a orchestration application platform which requires multiple JMS Messaging bridges. We have configured near 10 Bridges to different external systems. When maanaged servers each one host a bridge it will be doubles the count that we cannot monitor on single screen in the  admin console. On the Weblogic Admin Console we can monitor the Bridge status using Bridges -> Monitoring tab.For testing we had started single managed server and checked the Bridge monitoring status.

 It could be either 'Forwarding Messages' or 'Fail to connect Source' or 'Fail to connect to Target'. The actual tragedy starts when it comes to production environment, where we have multiple managedservers in the cluster. Usually Bridges status matters when it connects to third party messaging providers such as MQ Series or something else. Assume that, there are 10 Bridges deployed to 10 managed servers on the WebLogic Console becomes very slow when you use 'Monitoring' tab to show 100 result rows from the JMX. Alternative is we must develop a Python script or something else. Searched in search engines but no luck, OTN forums have same discussions but using python there is no output or MessagingBridgeRuntime is not there on the MBean tree. This is due to bug in WebLogic 9.2 ver to WebLogic 10.3.6. It is fixed in WebLogic 12c. So previous version need to update the patch. Alternative to patch you can use weblogic.Admin utility. The following scripting trick works well to show below output.

Fetching RUNNING managed server list

The weblogic.Admin command can give you the CLUSTERSTATE which will give the all the managed servers status(such as RUNNING, SHUTDOWN, UNKNOWN, ADMIN, FAILED etc.,) in that cluster. To give more robustness to the script, extract the that running list of managed servers. So that we can get all live server bridges status. using our 'awk' seizer we can pick the managed server names which 'grep' RUNNING state.

managdserver=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

Now the script

#!/bin/ksh
#================================================
# This is MessageBridge Monitoring script
# Updated by: Pavan Devarakonda
#================================================
admpasswd=Secret
admURL= hostname.company.com
domain=mydomain

echo "Starting at:"
date # For trace the time spending for this monitoring script
managdservers=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

for mserver in managdservers
do
 echo "Managed Server :" $mserver
 for bridge in `grep MessagingBridge mydomain.properties| cut -d'=' -f2`
 do
  echo " "
  java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd GET -pretty -type MessagingBridgeRuntime -mbean $domain:ServerRuntime=$mserver,Name=$bridge,Type=MessagingBridgeRuntime,Location=$mserver -property Description -property Name -property State | grep -v MBeanName
  
 done
done
echo "Bridge Status completed "
date

You must create a properties from your domain as follows:

mydomain.properties
#===================
srvr_target1=ms1_mydomain
srvr_target2=ms2_mydomain
srvr_target3=ms3_mydomain

bridge1=MQMessagingBridge1
bridge2=JMSMessagingBridge2

How to execute this script?

To execute this script you must have weblogic.jar in the CLASSPATH and java must be in the PATH that is JAVA_HOME/bin must be appended in your PATH environment variable. To execute the above Kron shell script

$ ./bridgeMonitor.ksh

To know more on Bridges and how to configure with simple python script. you can find on WLSTbyExample blog link




Technorati claim code QK7QN6U9ZST6

RegEx for WebLogic Admin

Trick on grep

Normally every WebLogic user/Admin knows that "ps -ef|grep java" gives you the WebLogic servers process list in any UNIX or Linux environments. Most of the forums suggesting use "|grep -v" to exclude the "grep line". But, here I don't want to use one more pipe and grep -v option really makes expensive in CPU cycles. Searched for the optimistic solutions and finally found! Two choices using --exclude  and using regex.

The exclude options is not available on all UNIX systems. So I've chosen next option.

How do you avoid the grep line in the grep command output itself?
Here the trick is using the regular expression to workout this requirement. Use your searching pattern any one of the character in [] then it will automatically excludes the grep line from the output. actually it wouldn't get output itself!

bash-2.05$ ps -ef |grep jav[a]

weblogic   950   934  0   Feb 10 ?        0:06 /usr/local/bea/jdk142_11/bin/java -client -Xms32m -Xmx200m -Xverify

:none

Note here I have tried out this trick on Sun Solaris 9 machine. Write your experiments in the comment box, share it!

Mailing alerts for Administrator

As a Middleware Admin, we might need to check the mail service is working or not on the Linux machine or Soloris machine on which our Weblogic running. To monitor and send the traps for issues we must have mail service enabled. In Linux how to check mail service up? We have tried to test the sendmail command from the Linux machine. with the following command:

echo "This is the body."| mailx -s "mailx Test1" bhavanishekhar@gmail.com
echo "test" | mailx -s "test_sub" bhavanishekhar@gmail.com

When I tested with the above test message got the error saying as follows: postdrop: warning: unable to look up public/pickup: No such file or directory. There is no mail service is running you validate the same with the following: We have several options checking the processes list for the pattern 'mail'.

$  ps aux | grep mail |grep -v grep
$

or use other option is usually the mail service on Unix machine runs on port 25, so grep it from the network status.

$  netstat -nl|grep 25

Already working but not working then how to debug that wheather it is working or not? My next choice is checking the service.

$ svcs | grep -i sendmail
$ ps -ef |grep -i sendmail

Thanks for these valuable inputs from my mate Mangaleshwaran.

Keep osting us your experience on mailx or sendmail.

cheers!

Check URL using shell script

We had an issue on production environment. Suddenly one of the webserver got crashed that no one aware of it. That was happen in the Weekend there were no users accessing at that time. When the business hour started Monday morning customer reported to service desk that one of the URL is not working for them. We middleware support team check the URL and identified that the HTTP server not working. Next action items as you know taking the logs backup and starting the HTTP server done.
As part of issue management preventive action to this there could be a monitoring of the URLs. All the web servers hosted application, load balancer that is serving the application or Application server that hosted the web application must be checked randomly or every day before business hours starts.
To do this monitoring team started working on listing out all loadbalancer URL, primary, secondary webserver URLs and also Application server hosted URL lists and checking on the browser each URL. Here is the pain-area doing this in manually there could be
• Human errors, Re-testing same URL 
• Each URL checking takes almost 1 min
• Business have 10 domains then each domain have load balancer URL, primary, secondary webserver URL approx 30 min
Thought of the day! Why don’t we automate this task using UNIX shell scripting. My friend Pubba Praveen came up with idea using wget command.  Started brainstorming searching on internet for a scripts that could monitor the URL. Many forums, blogs, most of them are using curl command whereas Solaris box is not having this command.
Alternate to curl command we have powerful wget command on Solaris.
Challenges
1. While using wget it is downloading junk
2. We have URL list some of them are SSO enabled
The first challenge, can be fix with –spider option will avoid the downloading anything from the website.
The second challenge overcome by using --no-check-certificate option
Redirect output of command into a variable
Test the test url from the command line, that is giving output from HTTP header of that URL. From that we need only single line that shows HTTP status. The grep command will do this, but it cannot be applied directly to that command. Because the wget execution is unable to sending the response to STDOUT, rather it will send to STDERR. To fix this, redirect the STDERR to STDOUT and do this in background ( 2> &1).  You can do this by storing that into a file, but it is not optimum solution. To store it into a variable, So that grep command will be enabled to find the pattern ‘HTTP’.
Create a file with desired URL list - urls.txt
Prepared a bash script that will work as web crawler for your business to check the URL and tells you it’s host is OK or not.
Effective automation script made the script as follows:

#!/bin/bash
#=========================================================
# This script will check for the appliction url working or not
# Author: Pavan Devarakonda
# Date  : 5/29/2012
#=========================================================

# Function that check the URL returns HTTP Status Code
checkURL()
{
       # keep below 2 lines in one
       s=`(/usr/sfw/bin/wget -t 0 --spider --no-check-certificate $1) 2>&1 \
       grep HTTP| tail -1|cut -c 41-43`
       echo $s
}

#========== = = = main script = = = ===========
clear
for url in `cat urls.txt`
do
         HTTPCode=$(checkURL $url) # function call
         if [ $HTTPCode -ge 500 ]
         then
             status="DOWN"
             # mail if this have issue
         else
             status="UP"
         fi
         TIME=`date +%d-%m-%Y_%H.%M.%S`
         echo At $TIME the URL $url $HTTPCode $status
done

This script can be associated with schedule job
 1. crontab old way
2. autosys job scheduler
3. at command for trials
 
Now concern is when my webserver is down the script hangs..
Looking for help now... any comments are invited

Multi server code Backup

If you digest Operating Systems basics or SHELL basics, you might think much effectively and you can make  more optimized ways to leverage your work, you can design your script with enhanced ability with productivity. Once you taste that flavor in my scripts, you can feel that and understand that why did I told those exaggerated words. Let me share my experience on this.

In every J2EE production environment is going to have an application code (most of the developer say it the composition of 'dist' and 'properties'), which could be no-stage mode deployment style. Where you need to have a copy of code on every WebLogic server running machine. For every production release there could be success or fail. A wise WebLogic Administrator always takes the backup of the existing code and then go further to do fresh code deployment.

According to the best practices of WebLogic deployment process, First time you will do the sample test on the production environment by making single site available for test. That is if your application having EJB-Tier, Web-tier then, start each one server in the tier. Test the basic flow test, which makes you know wheather your new code works or not. Most of the cases the code come to production move is already checked on Staging environment which is the simulated environment of production environment.
It is always good to do double check your work before you do something on Production environments.

Here the important point is that to overcome failure situation, when a new changes in the code fails , such cases alternate solution is to revert back the old code.

How to do this?
Think!! Think wisely, Think for optimizing, Think for better outputs...

We can do simple copy command but what makes error-less work
Take backup with date and time stamp.
There could be multiple application codes on the same machine.
By accomplishing these above task with the following script on a machine.

clear
if [ $# -gt 0 ]; then
 today=`date "+%b%d_%y"`
 APPNAME=$1
 SRC=$HOME/$APPNAME
 BKP=$HOME/backups/$APPNAME
 mkdir -p $BKP
 BKP=$HOME/backups/"$APPNAME"_"$today"
 echo $SRC
 echo $BKP
  
 echo '1. Code backup [dist]only'
 echo '2. Full backup [includes dist, properties]'
 read -p "Please enter your option [1/2]:"  opt
       if [ $opt -eq 1 ] || [ $opt -eq 2 ]; then 
  case "$opt" in
  
 1)  echo "copying dist ..."
  cp -rp $SRC/dist $BKP
     ;;
 2)  echo  "copying all including properties..."
  cp -rp $SRC/dist $BKP
  cp -rp $SRC/properties $BKP
     ;;
 esac
        else
                echo "  invalid option "
        fi
else
    echo "Please enter application name in command line arguments"
fi

Multiple site backup
The above backup is required for every site (multiple machines). If you have already used ssh key generated and all sites are accessible without password. In Ramayan Ram's single arrow hits seven trees in a row. Similarly I want to make the backup script run on one machine and hit on every machine where the application is running. But to make this idea come true I took the help of LinkedIn discussion. Mr. John suggested to keep CMD and execute ssh command.

clear
today=`date "+%b%d_%y"`
if [ $# -gt 0 ]
then
        APPNAME=$1
        SRC=$HOME/$APPNAME
        BKP=$HOME/backups/"$APPNAME"_"$today"
        echo '1. Code backup only'
        echo '2. Full backup [includes properties]'
        read -p "Please enter your option [1/2]:" opt
        if [ $opt -eq 1 ] || [ $opt -eq 2 ]
        then
                CMD="mkdir -p $BKP"
                case "$opt" in
                1 )
                        echo "Will copy only dist."
                        CMD="$CMD ;    cp -rp $SRC/dist $BKP ;     ls -lrtR $BKP ;   exit"
                        ;;
                  2 )
                        echo "Will copy dist and properties."
                        CMD="$CMD ;    cp -rp $SRC/dist $SRC/properties $BKP ;    ls -lrtR $BKP ;   exit"
                        ;;
                 esac
  # serverlist is the text file
                for i in `cat serverlist`
                do
                        echo "Connecting to $i."
                        ssh -a $i $CMD
                done
                echo "done !"
           else
                echo " invalid option "
          fi
else
        echo "Please enter application name in command line arguments"
fi

This script worked... awesome