Monday, March 24, 2014

Thread Dump Analysis

Dear WebLogic users/Administrator you might already visited many other sites that discussed about - how to take the thread dump. But here I am going to tell you about automation script makes more simplified way of working, with more information - almost 70% on the occasion of STUCK thread Thread dumps will reveals. In case of High CPU utilization situations, which method is badly working.

Step by Step analysis for Thread Dump

Step 1: Automated Thread Dump for WebLogic

Take the Thread dump for a managed server that have issues or slowness or latency etc. You just read the description and start applying to your test environment and then try out on your Live environments. Cheers!

# This script will takes the managed server name as argument and it will work generic to Linux or SunOS or HP-UX.
# The only ps command options varies for each platform
# Kill -3 command used for every 10 seconds of interval. You can increase the number of dumps in the for loop


   case $OSENV in
      Linux)WLSPID=`ps auxww|grep weblogic|grep -v grep|grep $WL_SERVER | awk '{print $2}'`;;
      SunOS)WLSPID=`/usr/ucb/ps -auxww|grep weblogi[c]|grep $WL_SERVER | awk '{print $2}'`;;
      HP-UX)WLSPID=`ps -efx|grep -v grep|grep weblogic|grep $WL_SERVER | awk '{print $2}'`;;
   if [ ! -z "$WLSPID" ]; then
      for i in 1 2 3 4 5
         echo "Generating thread dump number $i in logs/${WL_SERVER}_stdout.log"
         jstack  $WLSPID >> /log/${WL_SERVER}_stdout.log
         sleep 10
      echo "Error!!! process id not found for $WL_SERVER"

Step 2: Start analizing Thread dump

Take the sed sword into your hands !! sed UNIX utility will give you the right way of using regular expressions to expose desired outputs. The following is more effective script for you, love it share it :)
# Script Name:
# Description: Pass the threa dump log file name as argument to this script it will show you the required lines that need to see 
# Regular expression pattern is applied for search in the log file.

for i in `egrep 'trying to get lock|waiting to lock' $1 |  sed -n 's/.*[<@]\(.*\)[>[].*/\1/p' | sort | uniq`
 # Under that above line you will look thru the log for locked or holding lock
    sed -e '/./{H;$!d;}' -ne 'x;/ locked/{;/'${i}'/p;};/Holding lock/{;/'${i}'/p;}' $1
Look for more deeper with the handy utility that uses python. The following script can work to convert the given decimal number to hexadecimal number.
USAGE="Usage: $0 {number}"
if [ ! $# == 1 ]; then
  echo $USAGE
python -c "print hex($1)"
exit 0

The following script can work to convert the given decimal number to hexadecimal number.

If you want to generate a separate thread dump file with prstat, pstack commands output it will be much wise. Everything in single file to analysis at the time of critical situations. Ofcourse it happen only in UNIX environments :).

WebLogic Application performance problems: Diagnosing

The following presentation will take you to understand with a clarity on OOME types and what are all the reasons. It also address the High CPU issues.
Write a comment what do you thing on this issues. Happy trouble shooting!!!

No comments:

Post a Comment

Blurb about this blog

Blurb about this blog

Essential Middleware Administration takes in-depth look at the fundamental relationship between Middleware and Operating Environment such as Solaris or Linux, HP-UX. Scope of this blog is associated with beginner or an experienced Middleware Team members, Middleware developer, Middleware Architects, you will be able to apply any of these automation scripts which are takeaways, because they are generalized it is like ready to use. Most of the experimented scripts are implemented in production environments.
You have any ideas for Contributing to a Middleware Admin? mail to me