JMS MessageBridgeRuntime monitoring

We were in the developing a orchestration application platform which requires multiple JMS Messaging bridges. We have configured near 10 Bridges to different external systems. When maanaged servers each one host a bridge it will be doubles the count that we cannot monitor on single screen in the  admin console. On the Weblogic Admin Console we can monitor the Bridge status using Bridges -> Monitoring tab.For testing we had started single managed server and checked the Bridge monitoring status.

 It could be either 'Forwarding Messages' or 'Fail to connect Source' or 'Fail to connect to Target'. The actual tragedy starts when it comes to production environment, where we have multiple managedservers in the cluster. Usually Bridges status matters when it connects to third party messaging providers such as MQ Series or something else. Assume that, there are 10 Bridges deployed to 10 managed servers on the WebLogic Console becomes very slow when you use 'Monitoring' tab to show 100 result rows from the JMX. Alternative is we must develop a Python script or something else. Searched in search engines but no luck, OTN forums have same discussions but using python there is no output or MessagingBridgeRuntime is not there on the MBean tree. This is due to bug in WebLogic 9.2 ver to WebLogic 10.3.6. It is fixed in WebLogic 12c. So previous version need to update the patch. Alternative to patch you can use weblogic.Admin utility. The following scripting trick works well to show below output.

Fetching RUNNING managed server list

The weblogic.Admin command can give you the CLUSTERSTATE which will give the all the managed servers status(such as RUNNING, SHUTDOWN, UNKNOWN, ADMIN, FAILED etc.,) in that cluster. To give more robustness to the script, extract the that running list of managed servers. So that we can get all live server bridges status. using our 'awk' seizer we can pick the managed server names which 'grep' RUNNING state.

managdserver=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

Now the script

#!/bin/ksh
#================================================
# This is MessageBridge Monitoring script
# Updated by: Pavan Devarakonda
#================================================
admpasswd=Secret
admURL= hostname.company.com
domain=mydomain

echo "Starting at:"
date # For trace the time spending for this monitoring script
managdservers=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

for mserver in managdservers
do
 echo "Managed Server :" $mserver
 for bridge in `grep MessagingBridge mydomain.properties| cut -d'=' -f2`
 do
  echo " "
  java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd GET -pretty -type MessagingBridgeRuntime -mbean $domain:ServerRuntime=$mserver,Name=$bridge,Type=MessagingBridgeRuntime,Location=$mserver -property Description -property Name -property State | grep -v MBeanName
  
 done
done
echo "Bridge Status completed "
date

You must create a properties from your domain as follows:

mydomain.properties
#===================
srvr_target1=ms1_mydomain
srvr_target2=ms2_mydomain
srvr_target3=ms3_mydomain

bridge1=MQMessagingBridge1
bridge2=JMSMessagingBridge2

How to execute this script?

To execute this script you must have weblogic.jar in the CLASSPATH and java must be in the PATH that is JAVA_HOME/bin must be appended in your PATH environment variable. To execute the above Kron shell script

$ ./bridgeMonitor.ksh

To know more on Bridges and how to configure with simple python script. you can find on WLSTbyExample blog link




Technorati claim code QK7QN6U9ZST6

WebLogic Port mapping to Process ID

Hey dear WLA most of us come from Development environment to production environments. Hope the scienario thatI am going to discuss here is a common to everyone, who is working on development environment. Sometime or other you might felt I should have a handy script that could takes input as a WebLogic instance port which usually get from the applicaiton URL. The WebLogic port and list all the Process IDs which are associated with it on UNIX(here I got solution for Solaris) environment. Recently I found a Sun blog which is clearly discussed similar issue. I am re-compiling the same with customizing to our WebLogic Listen Port, WebLogic Server runs with a Java generated child Process ID. 'lsof' is the command you can execute and get this solved but most of the times it is a Sun third party downloads, that is not allowed in many Banking, Financial and Insurance Service organizations. And one more reason is small companies doesn't efford for third party tools.

Then, Your choice will be writing a handy script that will do the same task as lsof command helps to find the process id for a given WebLogic Listening port.

#!/bin/ksh 
 
pids=$(/usr/bin/ps -ef -o pid=)
 
if [ $# -eq 0 ]; then 
   read wlport?"Enter port you would like to know Java Process Id for: " 
else 
   wlport=$1 
fi 
 
for f in $pids 
do 
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $wlport$" 
   if [ $? -eq 0 ]; then 
        echo "===============***=============***==============="
        echo "ListenPort: $wlport is being used by Java PID:\c" 
        ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f 
        exit 0 # if you suspect more Weblogic instances with same listen port remove this
   fi 
done

Note: Save this script to commonly accessing user location name it as WLPort2Pid.ksh
Run with argument or without also it will work!! But you need to input the listening port that is must.
-->

Writing about this experiment is a new learning for me too.
Hope you enjoyed this one!! HAPPY SCRIPTING!!

Monitoring CPU Load Averages with Shell Script

Today I started re-inventing myself, started looking what all I did for me and my team to perform in better ways. Remembered those we used to open UNIX SSH windows to monitor How the CPU load average in each site. While doing this monitoring activity on site1, there might be possible that some other site reach to overload, which leads to uncontrol tendency to work. It was funny, my dear buddy named it as 'Barbar work!!' :)

After little R & D on Google/Internet found few suitable solutions. I had chosen 'uptime' command running with remote SSH connection in a loop. Adding more value to this sending a mail on the event of crossing the threshold value. This threshold will be vary depending upon the application and CPU power. Trail and error make you to identify what could be the threshold. Defining these threshold values Venu GopalRao helped a lot. Once script started working he was amazed and appreciated as well.

This script can be run forever with a specified time interval. You can use 'at' command or 'crontab' also for this task. I prepared a 'bash' script that could work for Solaris and also on Linux.

Before to this script we need to establish the password less connection to all the remote machines with 'key-gen' command. Public key authentication, which is the good choice password less connecting remote UNIX machines. Here, you can use any choice for encryption algorithms such as RSA, DSA etc.,


Customization/Cosmotics to this script
When you run this script at your prompt you can see the high load average server details in red color which makes sense to act up on that quicker. All server list I had kept in a plan text file and accessed it line by line as array for looping.

#!/bin/bash
#======================================================
# This script will check CPU Load, network ping status
# and also checks diskspace on every machine
#======================================================
RECIPIENTS="pavanwla@yahoo.co.in"
LOG=./load.log
 
check_load()
{
        loadnow=`echo $msg| cut -d, -f4 | cut -d: -f2 | cut -d. -f1`
        d=`echo $msg |awk '{print $((NF-1))}'`
        SD=`date "+%Y-%h-%d@%H:%M:%S"`
        echo $SD '****'
        if [ $loadnow -gt 14 ]; then
                echo -e ' \033[31m' $server ' ' $loadnow '\033[m'>>$LOG
                echo $SD $server ' ' $loadnow |mailx -s LOAD_WARN $RECPIENTS
        elif [ $loadnow -gt 19 ]; then
                echo -e ' \033[31m' $server ' ' $loadnow '\033[m'>>$LOG
                echo $SD $server ' ' $loadnow |mailx -s LOAD_CRITICAL $RECPIENTS
        else
                echo -e $server '\t' $loadnow '\t' $p '\t'$d >>$LOG
        fi
}

#==============================================================
#                 M A I N  S C R I P T
#==============================================================
if [ -f $LOG ]
then
        rm $LOG
fi
serlist=`cat prodServers.txt`
echo -e "========================================================">>$LOG
echo -e "  HOSTNAME  CPU Load     Network status       Disk Space">>$LOG
echo -e "========================================================">>$LOG
 
for server in $serlist
do
        echo 'connecitng to ' $server
        msg=`ssh $server "uptime; df -k /app|grep app |awk '{print \$5}'"`
        p=`ping -s $server 56 2 |grep loss | awk -F',' '{ print $3 }'`
        check_load
done
cat load.log

Please make sure that you must have prodServers.txt file in the same script path. Sample prodServers.txt file as follows:

myprod.server1.com
myprod.server2.com
...
myprod.server20.com

Upgrade Script
Adding more flavor to the load average script finding the disk space on every machine and also verifying network connectivity that ping response to every machine. Initially, I made it with two ssh commands one is for finding load average on each remote machine, other one is to check disk space on each machine. But it is not a good scripting way. With the help of the linkedin discussion I have updated it to single ssh command so that it will process faster by making less ssh sessions.

What is Next step??
If you find CPU load average is going above the threshold then you need to prepare yourself alert. Open that concern UNIX machine and find the causing process on that machine with 'top' command or 'prstat -L -a ' command options respective UNIX environment.

Take the Thread dumps of that culprit java process id of WebLogic instance. If CPU load reaching more than threshold then terminate that process/instance.
Analyze why that time CPU load gone high what thread were doing that time.

Note: This script created and executed on Solaris which remotely connects Linux and Solaris machines.

Good Forum Reference:
1. Linkedin Discussion
2. http://www.daniweb.com/forums/thread48764.html">Shell Script for Load monitoring!

Comments are most welcome!! HAPPY TO HELP!!

JVM monitoring with jstat Shell Script

The Oracle WebLogic Server instances are nothing but each one running on a single JVM, when this JVM crashes there could be one of the reasons as overworking of Garbage Collection or not working at all (waiting continuously).  

It is good practice that monitoring GC in JVM with detailed statistics will give you a clear idea to analysis at what circumstances happening wrong. And the best way to look in deeper levels of garbage collection also like Young Generation (Eden space, Survivor spaces S0, S1) Old Generation (tenured Generation), and Perm Generation (Statistic Objects/Classes).

JDK 1.5 and latest providing excellent JDK command utilities for interrogate the current running Java Process and look inside of JVM take snap with following:
1. jps (Java Process)
2. jstat (JVM status)

The 'jps' command with -lv options gives you complete detailed java process arguments for MEM settings and relavent WebLogic Server instances name. We have already discussed about this command utility in another post.

The jstat command with -gc option will produce the Garbage Collection of given java process id.

GC Monitoring shell script with jstat tool

I have modified according to my convenience and requirements as a shell script that will take the argument as WebLogic Server instance name and give us the output as statistics of Young, Old, Perm Generations Current, Used Memory details in MB conversion function.

##############################################################
# Purpose:  To display current, used spaces for an instance 
#   in the JVM Eden, Old, Perm sizes
# Author:  Pavan Devarakonda
##############################################################
# Function for converting the Kilo bytes to Mega Bytes.
toMb()
{
        echo 'scale=0;'$1'/1024' | bc; 
} 

#=============================================
# JVM status Function starts here
#=============================================
instJvm()
{
        DATA=`$JAVA_HOME/bin/jstat -gc $PID | grep -v S0C`
         
        EC=`echo $DATA | awk '{print $5}'`
        EU=`echo $DATA | awk '{print $6}'`
        OC=`echo $DATA | awk '{print $7}'`
        OU=`echo $DATA | awk '{print $8}'`
        PC=`echo $DATA | awk '{print $9}'`
        PU=`echo $DATA | awk '{print $10}'`
        echo -e  "$2\t|" `toMb $EC`"M   "`toMb $EU`"M\t| "`toMb $OC`"M   "`toMb $OU`"M\t| "`toMb $PC`"M   "`toMb $PU`"M"
}
#=============================================
# main starts here
#=============================================
 
 
echo -e "=============================================================="
echo -e " Instance\t |    Eden    \t|    Old Gen   \t|    Perm Gen"
echo -e "         \t |Current Used\t| Current Used \t| Current Used"
echo -e "=============================================================="
 
DOMAIN_HOME=$1

for i in ` ls $DOMAIN_HOME/servers/|grep -v bkp`
do 
        PID=`$JAVA_HOME/bin/jps -lv | grep $i | awk '{print $1}'`
        if [ "$PID" != ""  ]
        then    
                instJvm $PID $i
        fi      
done
echo -e "=============================================================="

Example execution:

  ./jstat_mon.sh  /u01/domains/apex_domain
  

The output as follows:

The jstat command using script

        

Taking Advantage of Naming Conventions

In most of the Middleware environments, the WebLogic domain will have admin servers, managed servers use naming convention construct that will have some common identity for each domain unique. In the above example, The domain uses naming convention with the letter 'c', This can be any word or letter that is used in your environment. To make your script you need to replace 'c' with your enviroment key word/letter. (Thanks Raman Jain for your valuable comment)

Let me show you the sample Output:

Looking ahead to your valuable comments and suggestions to improve our scripting tricks for better WebLogic Administration.

The jstat Command reference:

http://roundoverlinux.blogspot.in/2013/03/jstat-command-is-java-virtual-machine.html