Looking into Middleware server logs made simple

Middleware Admin means the day runs with Emergency calls or On-calls, supporting team task always under high pressure cooker -- many of them feel tension tension due to heat in the head many got into hair-fall!!! ...

Its a challenging task, only those who can handle the jobs under pressure can do this with tricks. But When you don't have calls you might think to reduce your stress in the work with smart automation scripting to make life more easier.

Best Practices for WebLogic Logs

If the environment is shorter let say it is size of 10 - 20 managed servers, then the best idea is to implement is that all these managed server logs, JMS stores must shared a common mount point. When application support team run into issues, this will give you the flexibility to search the logs easily and faster way to get rid of issues with this unique log location.

In most of your project environments you might find the multiple logs such as application reated, JMS message stores or some transactional logs. In some critical situations, where you might need to look-up for a particular phrase of text in the WebLogic managed server generated logs. If you already follows best practices for collecting logs all together. It is easy to search in the sub sequant sub directories.

#FileName: CheckLogs.sh
#This script is used to find the given pharse in all logs available in the machine

INST_HOME="/home/wlusr/instances"
INSTANCES=`ls $INST_HOME|grep c`
phrase=$1
for instance in $INSTANCES
do
        echo 'Checking for the :' $instance
        cd $INST_HOME/$instance/logs
        egrep -i $phrase c*log*
        egrep -i $phrase c*out
done

If you keep above script in the general accessible path, such a way that it can be run from any directory on your machine. The Best practices for every WebLogic Middleware server running machines uses JAVA_HOME in their PATH. The best option to store this script is $JAVA_HOME/bin you can run from any path in your user.

Suppose you want to find all the WebLogic Servers which got RUNNING state on a machine. The search pattern you can give as bea-000360.

$ logverify bea-000360
Checking for the : cmgdserver01
 
      

Heap dump on a Unix machine

After exploring the Jay/Faisal's blog on 'jmap' java utility usage and Heap dump. It is really great work by Jay/Faisal. I just thought that similar kind of experiment we did long back on UNIX machines. I am glad to sharing that with you guys.

Last year, we were struggling to overcome the OutOfMemoryError, which would effect the most of productive hours. In this assignment I need to figure out what process is causing the low memory in the environment is identified by searching a all log files in the machine. Assume that all the WebLogic instance log files are collected into common directory structure, each of them are stored respective instance named folder.

Script 1:

After identifying the impacted instances, I need to take the heap dump of that particular instance with corresponding process id.

#==============================================================
# File Name  : CheckLogs.sh
# Author  : Pavan Devarakonda
# Purpose  : Script for searching all WebLogic instances logs in a box
#==============================================================
instances=`ls /path/instances|grep web`
phrase=$1
date
for x in $instances
do
        echo 'Checking in the :' $x
        cd /path/instances/$x/logs
        egrep -i $phrase instance*log
        egrep -i $phrase instance*out
done
 
# Know the CPU load average at this time
date
uptime

-->

Script 2: Automatic script for Heap dump, here you need to provide the managed server name at the command-line argument. 

#!/bin/bash

#=======================================================
# Name    : InstanceHeapdump.sh
# Purpose : This script will takes instance name as input
# Takes the thread dump and also takes heap dump
#=======================================================
 
if [ "$1" = "" ]; then
        echo "Usage : $0 "
        exit
else
        instance="$1"
        user=$LOGNAME
        ppid=`ps -fu $user|grep $instance|grep startMan|grep -v grep|awk '{print $2}'`
        wpid=`ps -fu $user|grep $ppid|grep startWebLogic.sh|awk '{print $2}'`
        jpid=`ps -fu $user|grep $wpid|grep java|awk '{print $2}'`
        echo 'The Java PID:' $jpid
        kill -3 $jpid
        if [ $jpid = "" ]; then
                echo "Invalid instance input..."
                exit
        else
               jmap -dump:live,format=b,file=heap.bin $jpid
                mv heap.bin $instance_heap.bin
        fi
fi

This could give you one more way of finding a java process in UNIX machine. You can use jps command instead of three lines of awk filters. In this same script to make hold the java process not to crash, we can call a WLST script to suspend the instance and then you can happily take the heap dump.

What to do after heapdump creation?
Use jhat command to run the analyzer to show the outcome on a browser.
Follow the Faisal tips to find memory leaks, use eclipse MAT that is comfortable for your system.