JMS MessageBridgeRuntime monitoring

We were in the developing a orchestration application platform which requires multiple JMS Messaging bridges. We have configured near 10 Bridges to different external systems. When maanaged servers each one host a bridge it will be doubles the count that we cannot monitor on single screen in the  admin console. On the Weblogic Admin Console we can monitor the Bridge status using Bridges -> Monitoring tab.For testing we had started single managed server and checked the Bridge monitoring status.

 It could be either 'Forwarding Messages' or 'Fail to connect Source' or 'Fail to connect to Target'. The actual tragedy starts when it comes to production environment, where we have multiple managedservers in the cluster. Usually Bridges status matters when it connects to third party messaging providers such as MQ Series or something else. Assume that, there are 10 Bridges deployed to 10 managed servers on the WebLogic Console becomes very slow when you use 'Monitoring' tab to show 100 result rows from the JMX. Alternative is we must develop a Python script or something else. Searched in search engines but no luck, OTN forums have same discussions but using python there is no output or MessagingBridgeRuntime is not there on the MBean tree. This is due to bug in WebLogic 9.2 ver to WebLogic 10.3.6. It is fixed in WebLogic 12c. So previous version need to update the patch. Alternative to patch you can use weblogic.Admin utility. The following scripting trick works well to show below output.

Fetching RUNNING managed server list

The weblogic.Admin command can give you the CLUSTERSTATE which will give the all the managed servers status(such as RUNNING, SHUTDOWN, UNKNOWN, ADMIN, FAILED etc.,) in that cluster. To give more robustness to the script, extract the that running list of managed servers. So that we can get all live server bridges status. using our 'awk' seizer we can pick the managed server names which 'grep' RUNNING state.

managdserver=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

Now the script

#!/bin/ksh
#================================================
# This is MessageBridge Monitoring script
# Updated by: Pavan Devarakonda
#================================================
admpasswd=Secret
admURL= hostname.company.com
domain=mydomain

echo "Starting at:"
date # For trace the time spending for this monitoring script
managdservers=`java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd  CLUSTERSTATE | grep managed |grep RUNN|awk -F'-' '{print $1}'`

for mserver in managdservers
do
 echo "Managed Server :" $mserver
 for bridge in `grep MessagingBridge mydomain.properties| cut -d'=' -f2`
 do
  echo " "
  java weblogic.Admin -url t3://$admURL -username weblogic -password $admpasswd GET -pretty -type MessagingBridgeRuntime -mbean $domain:ServerRuntime=$mserver,Name=$bridge,Type=MessagingBridgeRuntime,Location=$mserver -property Description -property Name -property State | grep -v MBeanName
  
 done
done
echo "Bridge Status completed "
date

You must create a properties from your domain as follows:

mydomain.properties
#===================
srvr_target1=ms1_mydomain
srvr_target2=ms2_mydomain
srvr_target3=ms3_mydomain

bridge1=MQMessagingBridge1
bridge2=JMSMessagingBridge2

How to execute this script?

To execute this script you must have weblogic.jar in the CLASSPATH and java must be in the PATH that is JAVA_HOME/bin must be appended in your PATH environment variable. To execute the above Kron shell script

$ ./bridgeMonitor.ksh

To know more on Bridges and how to configure with simple python script. you can find on WLSTbyExample blog link




Technorati claim code QK7QN6U9ZST6

Setting up a best login profile on UNIX

Usually when you are working on building the platform for any application, you might be given UNIX (Solaris/Linux/HP-UX/AIX) boxes and new generation with Virtual boxes. which contains nothing you need to do many installations and configurations. To make your administrative task easy and simple we need to setup a fine tuned profile file. This profile file will be holding the multiple environment variables those are reusable when you log-on to UNIX Kernel  Here you can define shell functions or you can define simple single lined aliases that makes lengthy commands set to simple trimmed single letter or meaningful words.

What is user SHELL and  its profile association?

In your UNIX Kernel when a new user login will check the /etc/passwd file which shell is assigned by the super-user. It will look for the system profile and assign the environment values, then on top of it user profile can be loaded. Hence the variable is defined in system profile can be overridden by user profile. Example PATH environment variable that can be over written by user defined profiles most usual thing it is. So careful while defining such variables. It could behave differently for Solaris same line cannot work on Linux or on AIX box.

Why do we need this to be customize User profile?

Any Middleware admin may involve install and configuring the Apache or Oracle WebCenter Sites or spaces or SOA, Hadoop or Build and deployment platform may requires ANT or Gradle, Jenkins or Hudson setups. Every component that interacts with Middleware runs on the same UNIX Kernel need customized variables that can be referred from any directory or path, so we need them as global variables, this you can acquire by defining them as environment variables in SHELL profile. UNIX Operating system provides all shells associated with a  specific default system provided and a user profile script and that is in the hidden file which prefixed with dot(.) so it looks like .profile in your home directory. You can view them by ls -la command.

Where can I find this profile script?

Usually when you login to UNIX Kernel, you will be landing to a directory that is designated to your user HOME directory. Suppose you already working on some other path, you can simply use cd  or cd ~ and press enter the OS will takes to home directory. Alternatively you can use system defined environment variable by giving cd $HOME.

How do I know I am in which SHELL?

First you need to know about your user login SHELL(default). Every user can have different choice of having a shell. Once you have assigned user or role for WebLogic environment setup then  how do you know what SHELL it is? use the following:

$echo $SHELL
/bin/sh

How many SHELL do my env have?
To know this you could do little experiment on your prompt with the following commands

$ type bash
/usr/bin/bash

Then you can look in that directory all shell names ends with 'sh', list them with small regular expression trick on your ls command. The main trick here is using 'anchor' symbols, the $ symbol indicates ending of the word pattern, and you can use for avoiding the .sh files in that directory character classes can be negated with the [^] syntax.

[WLA@server~]$ cat /etc/shells
/bin/sh
/bin/bash
/sbin/nologin
/bin/tcsh
/bin/csh
/bin/ksh
...

[WLA@server~]$ ls /usr/bin | grep [^.]sh$
bash
csh
hash
jsh
ksh
pbksh
pbsh
pfcsh
pfksh
pfsh
remsh
rksh
rsh
ssh
tcsh
zsh
[WLA@server~]$ ls /usr/bin |grep [^.]sh$ |wc -l
cat /etc/shells

How to set bash as login SHELL?

I feel the Bourne Again Shell (bash) is best suitable login shell for WebLogic environment setup.
Sample .bashrc script which will be executed automatically when you hit 'bash' command at your Linux command prompt.

export JAVA_HOME=/u01/app/oracle/jdk1.8.0_121
export PATH=$JAVA_HOME/bin:$PATH
export MW_HOME=/u01/app/oracle/fmw122120
export WL_HOME=$MW_HOME/wlserver
alias wlst="$MW_HOME/oracle_common/common/bin/wlst.sh"

echo ===================================

Now your immediate question might be 'How do you know which shell is assigned to your user profile?' Here is the Trick for you!!

$ echo $SHELL
/bin/sh

The login shell will be assigned as per the common profile set by Solaris/Linux Administrator. That can be changed by passwd -e or passwd -r if you must have root access then only you can do editing the passwd file and update your desired shell there. Some times it is rejects you and shouts 'permission denied'. Just smile on it you can proceed with the following trick!! Using exec command, that will replaces the existing shell with new shell that you want.

export SHELL=/usr/bin/bash
exec $SHELL -l

Shell replacement ksh to bash with exec

What is a process?

process is an entity in execution. The process can be created by fork, exec system calls. the exec system call replaces current process. In the above also happens same for login shell. Example  If your default login shell is given as KSH(/bin/ksh), but in your .profile given exec command to replaces the shell with BASH(/bin/bash).

So, now I am ready with sample .bash_profile for WebLogic 11g for Solaris 10

Implementing the best practices all together, to make this most generic profile. 

#==============================================
# Sample .bash_profile or .bashrc or .profile for ksh
# Middleware Admin  on Solaris 10 operating environment
# Updated : October 8, 2010
#==============================================
clear

USERNAME=$LOGNAME
HOSTNAME=`hostname`
#---------- echo Welcome message ---------------------
echo
echo Welcome $USERNAME. You are on Server: $HOSTNAME !!!
export JAVA_HOME=$HOME/Oracle/Middleware/jdk160_18
#---------------WebLogic Specific ---------------------------------------
export MW_HOME=$HOME/Oracle/Middleware/
export WL_HOME=$MW_HOME/wlserver_10.3

export ORACLE_HOME=/opt/oracle/product/9.2.0
export LD_LIBRARY_PATH=$WL_HOME/server/native/solaris/sparc:$WL_HOME/server/lib/solaris/oci920_8:$ORACLE_HOME/lib32:$LD_LIBRARY_PATH
 
export THREADS_FLAG=native
export PATH=$PATH:.:$JAVA_HOME/bin:/usr/bin:/usr/ucb:/etc:$WL_HOME/bin:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:$WL_HOME/server/bin

export CLASSPATH=$WL_HOME/server/lib/weblogic.jar:$JAVA_HOME/jre/lib/rt.jar
#---------------- End of WebLogic specific ---------------------------------
export MANPATH=$MANPATH:/usr/share/man
export EDITOR=vi
export PS1="[\u@\h \w]\$ "
 
alias cls="clear"
alias sl=ls
alias cd..="cd .."
alias cd...="cd ../.."
alias gi="grep -i"
alias l="ls -al|grep '^d'"
alias lm="ls -al | more"
alias h=history
alias hm="history | more"
alias nu="who|wc -l"                # nu - number of users
alias np="ps -ef|wc -l"             # np - number of processes running
alias p="ps -ef"
alias help=man
alias path="echo $PATH"
##### End of the sample .bash_profile for smart WLA #################
#===== Published on http://wlatricksntips.blogspot.com =============#

Importance of DOT(.) in PATH
I just started using the above .bash_profile but, something missing in it... when I wish to start my WebLogic server from domain home I need to give dot and forward slash (./). This looks somewhat unusual, when I add a DOT( .) in PATH variable made everything normal and I can start my WebLogic instances with my customized script from the prompt.

Power of PATH
More power to your user shell with adding /usr/sbin most of the network, system relavent commands available to you at command prompt.

The changes I made for the PATH line adding JAVA_HOME/bin your latest Java binaries will be useful at command prompt such as jps, jstat, jmap very useful when you deal with heap dumps and memory issues, $WL_HOME/server/bin consists of most of the WebLogic related configuration and maintenance commands available to you, it is little trick you need adding this in Solaris and Linux because it uses L-R precedence on Solaris for Linux R-L use appropriately.

export PATH=$PATH:.:$JAVA_HOME/bin:/usr/bin:/usr/ucb:/etc:$WL_HOME/bin:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:$WL_HOME/server/bin

Java want CLASSPATH
When there is a need of running ANT or build scripts you need runtime that is rt.jar and tools.jar in CLASSPATH this very often.

If you are using JRockit instead of SunJava then you don't need the rt.jar or tools.jar files in CLASSPATH. Other than this we have WebLogic specific utility commands that can be run from any where if you add weblogic.jar into the CLASSPATH. You can run the following command line utilities:

1. java weblogic.WLST
2. java weblogic.Admin
3. java weblogic.Deployer

Overcome Human Errors with Alias
Mistakes can be made by humans it is obvious thing. Mostly while working on UNIX environments while typing the command there could be mistype happens. Example very effective WebLogic troubleshooter in the pressure situations might mistype 'sl -l' instead of 'ls -l'. sometimes 'l s-lrt' these mistakes can be over come with proper identifying our typing styles :) define them as alias with correct command.

That looks pretty and easy to use. If you need any assistance write a comment or reach me on G+ or FB please don't shy!!  : )

Usage of umask

When your WebLogic server generating logs need common access to the other users who are just monitoring the system or auditing the system. Such users are not WebLogic admin user, even though they need to access your logs. For this privileges every WebLogic server logs folder you need to update using chmod command repeatedly. To resolve this best option is using umask.

Don't forget to write back your comments and queries.

Ubuntu profile setup in VM

Now a days Ubuntu became almost alternative for Windows OS with great UI features in a Linux flavors. In Ubuntu the default user shell is 'bash'. The order it will execute according to the system profile. The system contains bashrc. So you can place your environment variables int the .bashrc or else alternatively you can call the . ~/.bash_profile in the .bashrc

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export PATH=$PATH:$JAVA_HOME/bin
export WL_HOME=/home/pavanwla/Oracle/Middleware/Oracle_Home/wlserver
export CLASSPATH=$WL_HOME/server/lib/weblogic.jar:.

Please share your experiance with profile and without knowing about the profile. Lets have best profile for your WebLogic environments that you build. References
1. http://www.unixguide.net/unix/bash/A7.shtml
2. http://djcraven5.blogspot.com/2008/03/change-solaris-shell-to-bash.html
3. http://www.cyberciti.biz/faq/solaris-unix-change-default-shell/
4. http://www.ibm.com/developerworks/aix/library/au-speakingunix_commandline/?S_CMP=cn-a-aix&S_TACT=105AGX52
5. The umask command usage

Multi server code Backup

If you digest Operating Systems basics or SHELL basics, you might think much effectively and you can make  more optimized ways to leverage your work, you can design your script with enhanced ability with productivity. Once you taste that flavor in my scripts, you can feel that and understand that why did I told those exaggerated words. Let me share my experience on this.

In every J2EE production environment is going to have an application code (most of the developer say it the composition of 'dist' and 'properties'), which could be no-stage mode deployment style. Where you need to have a copy of code on every WebLogic server running machine. For every production release there could be success or fail. A wise WebLogic Administrator always takes the backup of the existing code and then go further to do fresh code deployment.

According to the best practices of WebLogic deployment process, First time you will do the sample test on the production environment by making single site available for test. That is if your application having EJB-Tier, Web-tier then, start each one server in the tier. Test the basic flow test, which makes you know wheather your new code works or not. Most of the cases the code come to production move is already checked on Staging environment which is the simulated environment of production environment.
It is always good to do double check your work before you do something on Production environments.

Here the important point is that to overcome failure situation, when a new changes in the code fails , such cases alternate solution is to revert back the old code.

How to do this?
Think!! Think wisely, Think for optimizing, Think for better outputs...

We can do simple copy command but what makes error-less work
Take backup with date and time stamp.
There could be multiple application codes on the same machine.
By accomplishing these above task with the following script on a machine.

clear
if [ $# -gt 0 ]; then
 today=`date "+%b%d_%y"`
 APPNAME=$1
 SRC=$HOME/$APPNAME
 BKP=$HOME/backups/$APPNAME
 mkdir -p $BKP
 BKP=$HOME/backups/"$APPNAME"_"$today"
 echo $SRC
 echo $BKP
  
 echo '1. Code backup [dist]only'
 echo '2. Full backup [includes dist, properties]'
 read -p "Please enter your option [1/2]:"  opt
       if [ $opt -eq 1 ] || [ $opt -eq 2 ]; then 
  case "$opt" in
  
 1)  echo "copying dist ..."
  cp -rp $SRC/dist $BKP
     ;;
 2)  echo  "copying all including properties..."
  cp -rp $SRC/dist $BKP
  cp -rp $SRC/properties $BKP
     ;;
 esac
        else
                echo "  invalid option "
        fi
else
    echo "Please enter application name in command line arguments"
fi

Multiple site backup
The above backup is required for every site (multiple machines). If you have already used ssh key generated and all sites are accessible without password. In Ramayan Ram's single arrow hits seven trees in a row. Similarly I want to make the backup script run on one machine and hit on every machine where the application is running. But to make this idea come true I took the help of LinkedIn discussion. Mr. John suggested to keep CMD and execute ssh command.

clear
today=`date "+%b%d_%y"`
if [ $# -gt 0 ]
then
        APPNAME=$1
        SRC=$HOME/$APPNAME
        BKP=$HOME/backups/"$APPNAME"_"$today"
        echo '1. Code backup only'
        echo '2. Full backup [includes properties]'
        read -p "Please enter your option [1/2]:" opt
        if [ $opt -eq 1 ] || [ $opt -eq 2 ]
        then
                CMD="mkdir -p $BKP"
                case "$opt" in
                1 )
                        echo "Will copy only dist."
                        CMD="$CMD ;    cp -rp $SRC/dist $BKP ;     ls -lrtR $BKP ;   exit"
                        ;;
                  2 )
                        echo "Will copy dist and properties."
                        CMD="$CMD ;    cp -rp $SRC/dist $SRC/properties $BKP ;    ls -lrtR $BKP ;   exit"
                        ;;
                 esac
  # serverlist is the text file
                for i in `cat serverlist`
                do
                        echo "Connecting to $i."
                        ssh -a $i $CMD
                done
                echo "done !"
           else
                echo " invalid option "
          fi
else
        echo "Please enter application name in command line arguments"
fi

This script worked... awesome

Best Practices for WebLogic Environment

Here I am jotting out few interesting Best practices for Oracle WebLogic environments, which I have experienced/encountered hurdles while preparing a WebLogic Domain. To Win this Running race you must overcome these hurdles, the best solutions is remembering all of them now I am sharing with you guys here:

1. Dedicated User and group
Oracle WebLogic installation on Solaris machine or Linux or a Windows machine, it is better to have a dedicated user and shared growup where you can install the Middleware components WebLogic, Coherence, WebCenter sites, Content Management etc. provide access to all  so that all other users need not to installing  for each new domain on the same machine.

  useradd [options] LOGIN

Some of important options are:

-d home directory
-s starting program (shell)
-p password
-g (primary group assigned to the users)
-G (Other groups the user belongs to)
-m (Create the user's home directory

My experiment:

useradd -g wladev -s /bin/bash -p xxxxxxxx -d /home/wladmin -m wladmin

Remember that, You can run above user creation command if you have root user access only. Double check the password working for the newly created user. Now a days Virtual Box users are becoming super user (root) just by sudo access. Switch to user (su - wladmin) will connect to the new user.
Change the user password from root user, using passwd wladmin command. On the root user it won't ask you previous password.

2. Using of sed for Migrations
When I worked for WebLogic 8.1 to WebLogic 9.2 migration for each instance wise configuring fresh properties updating took me around a week time for whole environment. Time changed and the requirements changed and this time WebLogic 11g migration from WebLogic 9.2 I have an idea to use stream editing option, and applied with proper regular expressions to finish my task. It worked for me perfectly it is awesome, whole updating done in half hour with small script that included sed in a for loop. I had experianced the fastness  with sed to change multiple lines search and replace in multiple files in the same machine

The following diagram will tells you how sed works on text patterns.

SED Script functionality

Learning SED scripting

for i in `ls instances|grep c`
do
cd $INST_HOME/$i/config
cp /oldinstance/$i/config/*.properties .
sed -e "s/$i-//g" \
      -e "s/$i\_//g" \
      -e "s/\_$i//g" \
      -e "s/-$i//g" \
      -e "s/wluser92/wluser11/g"  <  log4j.properties  >temp
mv temp log4j.properties

User per domain: If you are preparing a development environment then you can choose a user per domain it is the best way to avoid conflicts between developers code changes etc. Install new Oracle License and keep always a backup of old License.

3. Customizing your domain

AdminServer name, ListenAddress, ListenPort, Some times you might see errors saying that "Listen Port already in use", To avoiding port conflicts: Before assigning a port to your WebLogic instance better you check whether it is already in use or not? by using netstat command.
About Virtual IP issues.
i) WebLogic Server wise logs generation
WebLogic Server instance each one can generate separate server side STDOUT logs, STDERR logs as well as application logs. These logs must be collected in a separate mount point will make free for disk utilization memory problems. According to the application severity we can keep archiving the rotated logs on the disk. Most of the Admins, developers while doing troubleshooting for an issue they must revisit these logs and they must know from which server it was happening for this log4j provides more flexibility to digg/debug every Java package, class level, even method, line level too.

How to collect it?
To make this possible you need to enable your WebLogic server library path must pick the log4j-1.2.8.jar and the logging definitions in a separate file lets say it as log4j.properties file in the CLASSPATH.

Where to set?
Before weblogic.jar path or after? Oracle recommands application related jars and third-party jars must be set after weblogic.jar. So log4j.xx.jar must be in POST_CLASSPATH.

ii) Editing for all Domain Environments

a. JAVA_OPTIONS
b. USER_MEM_ARGS
c. JVM type
d. CLASSPATH (PRE/POST)
e. Native IO options
f. MuxerThread
g. SocketReaders

Changing JVM Hotspot Compiler

Editing common scripts impacts all the domains in that machine. If there is a need for the WebLogic server run with server JVM that will give more scope for In commEnv.sh script we are going to update .
Sample Example:

191    Sun)
192      JAVA_VM=-client
change to
191    Sun)
192      JAVA_VM=-server

d
4. Scalability for Domain
Adding servers to  or removing servers existing Cluster is nothing but scalability. While defining new Cluster identify proper multi-cast address suitable to your environment with multi-cast test. You must use Interface Address as the DNS/IP of the machine where the instance is configured in each server's Cluster tab. This will make you easy to run the clustered environment.

5. Effort saving means cost saving
While preparing your configuration keep focus on portable coding. I have been to many UNIX forums to find portable and flexible scripting. Here I am sharing for you. Efforts can be focused on the following things

Customized Stop All script for each domain 
Best option I found from stopManagedWebLogic.sh given by the Oracle. It is normal shell script it will invokes the Python script on the fly. I just replaced the 'Server' with 'Cluster' argument for shutdown WLST command and also called admin stop script in the bottom.

#!/bin/sh

DOMAIN_HOME="/home/domains/mydomain"

. ${DOMAIN_HOME}/bin/setDomainEnv.sh
ADMIN_URL="t3://my.adminhost.com:adminPort"

echo "wlsUserID ='username'" >>"shutdown.py" 
echo "wlsPassword = 'password'" >>"shutdown.py" 
echo "connect(${wlsuserID}, ${wlsPassword}, url='${ADMIN_URL}')" >>"shutdown.py" 
 
#=== fetching cluster list from the domain configuration repository =============
for cl in `grep -i clstr\<\/n ../config/config.xml|sed  's/.*\(.*\)<\/name>.*/\1/'
do
 echo "shutdown($cl','Cluster')" >>"shutdown.py" 
 echo "state($cl)">>"shutdown.py" 
done
echo "shutdown()" >>"shutdown.py" 
echo "exit()" >>"shutdown.py" 
 
echo "Stopping Weblogic Server..."
java -classpath ${FMWCONFIG_CLASSPATH} ${MEM_ARGS} ${JVM_D64} ${JAVA_OPTIONS} weblogic.WLST shutdown.py  2>&1 

echo "Done"

iv. Server Health checking script
Separate is a time, effort saving

6. Deployment Strategies
System resource deployments better you prepare your customized JDBC Data source configuration script using WLST. Avoid Start-up Classes configuration which will make dependable deployment, which leads you to not able to use side-by-side(SBS) deployment advantage.

7. WebLogic 9.x onward you have a flexibility to use Deployment plans for UAT, QA, Staging environments as same as production. This will reduce the problem of porting the code on one environment to other without any configuration changes.

8. OutOfMemoryError

Most of the production environments first hurdle is OOME, if it is occurring in your Web-tier environment then you can use JSP pre-compile.Prepare GC monitoring scripts

9. Now a days everywhere you can find Virtualization (SOLARIS ZONES), Cloud computing, Clustering (Database RAC), Grid concepts, Veritas Clustering in Disks/RAID etc. Have each topic in breif knowledge that could make you understand if anything goes wrong somewhere in the application environment you can easily figure it out.

10. Know about your production environment end to end how the data flows? Firewall, Load balancer software or hardware, proxy-plugin security aspects, Network connectivity, Net-backup locations etc.

Reference URL:
1. Best Virtual IP usage http://blogs.oracle.com/muralins/2007/08/ipmp_ip_multipath.html
http://www.eng.auburn.edu/~doug/howtos/multipathing.html
2. Linux Administration Commands http://www.faqs.org/docs/abs/HTML/system.html
3. WLST Configuration help http://wlstbyexamples.blogspot.com/
4. WebLogic Upgrade http://download-llnw.oracle.com/docs/cd/E13179_01/common/docs100/upgrade/comm_ref.html