Thread Dump Analysis

Dear WebLogic users/Administrator you might already visited many other sites that discussed about - how to take the thread dump. But here I am going to tell you about automation script makes more simplified way of working, with more information - almost 70% on the occasion of STUCK thread Thread dumps will reveals. In case of High CPU utilization situations, which method is badly working.

Step by Step analysis for Thread Dump

Thread dump troubleshooting from Jerry Chan

Step 1: Automated Thread Dump for WebLogic

Take the Thread dump for a managed server that have issues or slowness or latency etc. You just read the description and start applying to your test environment and then try out on your Live environments. Cheers!

#!/bin/bash

# This script will takes the managed server name as argument and it will work generic to Linux or SunOS or HP-UX.
# The only ps command options varies for each platform
# Kill -3 command used for every 10 seconds of interval. You can increase the number of dumps in the for loop

WL_SERVER=$1
OSENV=`uname`

   case $OSENV in
      Linux)WLSPID=`ps auxww|grep weblogic|grep -v grep|grep $WL_SERVER | awk '{print $2}'`;;
      SunOS)WLSPID=`/usr/ucb/ps -auxww|grep weblogi[c]|grep $WL_SERVER | awk '{print $2}'`;;
      HP-UX)WLSPID=`ps -efx|grep -v grep|grep weblogic|grep $WL_SERVER | awk '{print $2}'`;;
   esac
   if [ ! -z "$WLSPID" ]; then
      for i in 1 2 3 4 5
      do
         echo "Generating thread dump number $i in logs/${WL_SERVER}_stdout.log"
         jstack  $WLSPID >> /log/${WL_SERVER}_stdout.log
         sleep 10
      done
   else
      echo "Error!!! process id not found for $WL_SERVER"
   fi

Step 2: Start analizing Thread dump

Take the sed sword into your hands !! sed UNIX utility will give you the right way of using regular expressions to expose desired outputs. The following is more effective script for you, love it share it :)

# Script Name: ThreadAnalizer.sh
# Description: Pass the threa dump log file name as argument to this script it will show you the required lines that need to see 
# Regular expression pattern is applied for search in the log file.
clear

for i in `egrep 'trying to get lock|waiting to lock' $1 |  sed -n 's/.*[<@]\(.*\)[>[].*/\1/p' | sort | uniq`
do
 # Under that above line you will look thru the log for locked or holding lock
    sed -e '/./{H;$!d;}' -ne 'x;/ locked/{;/'${i}'/p;};/Holding lock/{;/'${i}'/p;}' $1
done

Look for more deeper with the handy utility that uses python. The following script can work to convert the given decimal number to hexadecimal number.

USAGE="Usage: $0 {number}"
if [ ! $# == 1 ]; then
  echo $USAGE
  exit
fi
 
python -c "print hex($1)"
exit 0

The following script can work to convert the given decimal number to hexadecimal number.

If you want to generate a separate thread dump file with prstat, pstack commands output it will be much wise. Everything in single file to analysis at the time of critical situations. Ofcourse it happen only in UNIX environments :).

WebLogic Application performance problems: Diagnosing

The following presentation will take you to understand with a clarity on OOME types and what are all the reasons. It also address the High CPU issues.

Java Application Problems: Diagnostics from Віталій Тимчишин

Write a comment what do you thing on this issues. Happy trouble shooting!!!

Heap dump on a Unix machine

After exploring the Jay/Faisal's blog on 'jmap' java utility usage and Heap dump. It is really great work by Jay/Faisal. I just thought that similar kind of experiment we did long back on UNIX machines. I am glad to sharing that with you guys.

Last year, we were struggling to overcome the OutOfMemoryError, which would effect the most of productive hours. In this assignment I need to figure out what process is causing the low memory in the environment is identified by searching a all log files in the machine. Assume that all the WebLogic instance log files are collected into common directory structure, each of them are stored respective instance named folder.

Script 1:

After identifying the impacted instances, I need to take the heap dump of that particular instance with corresponding process id.

#==============================================================
# File Name  : CheckLogs.sh
# Author  : Pavan Devarakonda
# Purpose  : Script for searching all WebLogic instances logs in a box
#==============================================================
instances=`ls /path/instances|grep web`
phrase=$1
date
for x in $instances
do
        echo 'Checking in the :' $x
        cd /path/instances/$x/logs
        egrep -i $phrase instance*log
        egrep -i $phrase instance*out
done
 
# Know the CPU load average at this time
date
uptime

-->

Script 2: Automatic script for Heap dump, here you need to provide the managed server name at the command-line argument. 

#!/bin/bash

#=======================================================
# Name    : InstanceHeapdump.sh
# Purpose : This script will takes instance name as input
# Takes the thread dump and also takes heap dump
#=======================================================
 
if [ "$1" = "" ]; then
        echo "Usage : $0 "
        exit
else
        instance="$1"
        user=$LOGNAME
        ppid=`ps -fu $user|grep $instance|grep startMan|grep -v grep|awk '{print $2}'`
        wpid=`ps -fu $user|grep $ppid|grep startWebLogic.sh|awk '{print $2}'`
        jpid=`ps -fu $user|grep $wpid|grep java|awk '{print $2}'`
        echo 'The Java PID:' $jpid
        kill -3 $jpid
        if [ $jpid = "" ]; then
                echo "Invalid instance input..."
                exit
        else
               jmap -dump:live,format=b,file=heap.bin $jpid
                mv heap.bin $instance_heap.bin
        fi
fi

This could give you one more way of finding a java process in UNIX machine. You can use jps command instead of three lines of awk filters. In this same script to make hold the java process not to crash, we can call a WLST script to suspend the instance and then you can happily take the heap dump.

What to do after heapdump creation?
Use jhat command to run the analyzer to show the outcome on a browser.
Follow the Faisal tips to find memory leaks, use eclipse MAT that is comfortable for your system.