Tuesday, November 30, 2010

Script for Bouncing a WebLogic instance

In the last post I was made deeper research on getting the process id of all WebLogic instances with a script. After looking to that script, My asked me 'Why don't you try for a script for bouncing a WebLogic instance?'. I thought that's really good idea, this makes WebLogic Admin life more easier, which is very much need for every WebLogic Admin. Let me get all the clues before putting the logic into the script.






The bounce script will take the input as WebLogic instance name.
Then the script should do search for the corresponding Java Process id of given WebLogic instance.
That Java Process ID can be used for finding:
1. Thread dump
2. Domain directory
3. Shutdown instance (kill the process)


Above 1, 3 are commonly used commands but 2nd one required when situation like this, on the same machine if there exists multiple WebLogic domain instances then how to find the WebLogic instance started from? Here, I need to find the origin directory of startManagedWebLogic.sh used which gives the WebLogic domain directory. How to resolve this? The solution is pwdx command in Solaris gives you this directory path by passing process id to it.

This solution gives me to start the WebLogic instance which is residing in irrespective WebLogic domain. That's all I need to make the bounce script.

The bounce script goes like this
#==========================================================
# This script you can keep in common accessing PATH
# such as in $JAVA_HOME/bin
# For WebLogic instance take ThreadDump, Shutdown, and Start
#===========================================================
flag=0
if [ "$1" = "" ]; then
        echo "Usage : $0 [instance]"
        exit
else
        instance="$1"
        pid=`jps -v|grep $instance|awk '{print $1}'`
        echo 'pid ' $pid
        domainPath=` pwdx $pid |awk '{print $2}'`
 
        kill -3 $pid
        sleep 3
        kill -3 $pid
        sleep 3
        kill -3 $pid

 #Now shutting down the instance here
        kill -9 $pid

 #verifying the instance is existance if not wait
        while : 
        do 
                jps -v |grep $instance
                if [ $? -eq 1 ]; then
                        echo 'kill success'
                        flag=`expr $flag + 1`
                        break;
                else 
                        echo 'kill failed... sleeping 1 sec'
                        sleep 1
                fi 
        done 

 #Once kill is success then you can proceed for Start it
 # Here you can call startManagedWebLogic.sh script
        if [ $flag -gt 0 ]; then
                . ${domainPath}/startinstance.sh $instance
        fi
fi

Recently one of my blog ->pingbox user asked me about pwdx command, does it works on HP-UX? I thought I should update this post with proper references.
There is compatibility list given for pwdx command:
pwdx compatibility

Alternate solution is cwd for Linux.
Please suggest your ideas, Keep commenting

Sunday, September 12, 2010

X11 Forwarding in SSH for Solaris and Linux

You might be wonder what is fun in this blog looking for X11 forwarding on Solaris 5.10!! You might be excited to see the beauty of colorful Oracle Fusion Middleware WebLogic 11g installation window when you executable binaries (wls1033_solaris32.bin), You might wish to see Oracle WebLogic domain configuration Wizard (config.sh), Oracle WebLogic Domain Template builder(config_builder.sh), BEA Smart Update (bsu.sh) windows or your Oracle Database 11g while you work on a remote UNIX machines.

After spending few hours on the Internet found many blogs written on their experiences, few document on websites, which illustrated about SSH, X11 forwarding in UNIX. I understand that what I need to do? To achieve my objective X11 Forwarding using SSH window, I must have a X emulating software installed on my desktop. you guys have this on your desktop?? If not install Hummingbird Connectivity (Xming is alternative)

My experimenting environment is Solaris 10, Oracle WebLogic, SSH (You can use Putty also), Hummingbird Connectivity (Xming is alternative)

Applicable to:
This you can use for any Java AWT or Swing programs in any UNIX based Operating environment.
You might download free Hummingbird Software also providing evaluation version who want to test their environment.

Here I am sharing with smart guys WLA/DBA my version of brain storming on this. When I first-time tried to invoke X window command it rejected and told as shown below:

No X11 DISPLAY variable was set, but this program performed an operation which requires it.

I had updated DISPLAY variable as follows

$ export DISPLAY=localhost:11.0

My first experiment with normal UNIX user
When I executed bsu.sh script I got the following in Normal UNIX user:

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class
sun.awt.X11GraphicsEnvironment
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
at sun.awt.X11.XToolkit.(XToolkit.java:89)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.Toolkit$2.run(Toolkit.java:834)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
at sun.swing.SwingUtilities2$AATextInfo.getAATextInfo(SwingUtilities2.java:126)
at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:1556)
at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:130)
at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1591)
at javax.swing.UIManager.setLookAndFeel(UIManager.java:541)
at javax.swing.UIManager.setLookAndFeel(UIManager.java:581)
at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1343)
at javax.swing.UIManager.initialize(UIManager.java:1432)
at javax.swing.UIManager.maybeInitialize(UIManager.java:1420)
at javax.swing.UIManager.getDefaults(UIManager.java:660)
at javax.swing.UIManager.put(UIManager.java:989)
at com.bea.plateng.common.ui.gui.GUIHelper.initPLAF(GUIHelper.java:69)
at com.bea.plateng.patch.gui.PatchGUIHelper.startGui(PatchGUIHelper.java:90)
at com.bea.plateng.patch.Patch.main(Patch.java:338)

My UNIX admin buddy told that dear, first try with xclock then go for other, then I thought Ohhh!! that's good idea, let me try that...

$ xclock
bash: xclock: command not found

Hmmmmmmm!!, what is this? Think, Think Think!!!, On a Solaris 5.10 you can find some of the commands in OpenSource folder, Let me try in that path this xclock on the following path.
/usr/openwin/bin/xclock

No luck buddy, same kind of error message as bsu.sh got above.

Now I realized and verified all those PATH, CLASSPATH are pointing to proper JAVA_HOME/bin, lib paths respectively, one of them is misconfigured reset to correct path.

One of Sun blog saying about Updating $Home/.ssh/config file as follows:

Host *
ForwardX11 yes
ForwardAgent no

After doing this I had ran the following:

$ ssh -X user@hostname

This is the command will activallte the X emulation on your UNIX machine with given SSH configuration. Actually it will refer to the default configuration /etc/.ssh/config but when you create new config file on your user home directory it will prefer to execute for your user settings. This command also updates .Xauthority file when you ssh -X command.

Verified every step again finally I got X window for xclock

Hey!! Cheerup on my face!!

Experiment 2: LDAP user logon with assume user name as 'pavanwla', switched to role 'wladmin'. I started trying on it for X11 forwarding. First Let me try, xclock on

-->

[wladmin@hostname~]$ /usr/openwin/bin/xclock
Error: Can't open display:

Ohh its same DISPLAY variable problem right? You remember it!!

[wladmin@hostname~]$ export DISPLAY=localhost:10.0
[wladmin@hostname~]$ echo $DISPLAY
localhost:10.0

Let me try now, what it will say...

[wladmin@hostname~]$ /usr/openwin/bin/xclock
X connection to localhost:10.0 broken (explicit kill or server shutdown)

Oh My Gash!!! what to do now????????????
After a while, I bugged again my buddy Mr. Unix Admin about this. He told that configurations, X emulating authoriy must match ldap user and role. Now my turn, first I tried to get xclock on ldap user, that confirms everything fine at ldap user level. This configuration, Xauthority I can use in role user too, then see what I did:

[wladmin@hostname~]$ scp pavanwla@hostname:/home/pavanwla/.ssh/config .ssh/
[wladmin@hostname~]$scp pavanwla@hostname:/home/pavanwla/.Xauthority .

Started ssh session with ssh -X command.

Finally, the climax came to end, xclock started. ahh!!
Reference:

https://cygwin.com/setup-x86_64.exe
http://sourceforge.net/projects/xming/files/Xming/6.9.0.31/Xming-6-9-0-31-setup.exe/download
http://download.oracle.com/docs/cd/E15051_01/common/smartupdate/guide/quickrefax.html

Monday, August 30, 2010

Configuring Plain NodeManger on Solaris

Today morning one of my buddy pinged me from my team, seeking help on an issue he is faceing in his newly  configured WL domain for his learnings.

Understanding the Issue
My buddy novice WLA is trying to start the WebLogic instance on a remote machine. He is working on Solaris 10 operating environment, WebLogic 9.2. I asked for Java version, He executed the following command:

$ java -version

I asked for jps command execution the system is refused and said that ' jps is bad cammand'. I understand that  JAVA_HOME is not set for his environment. asked for echo $PATH. It is showing Java Path. Ohh!! what is missing then??

$which java
this resulted that JAVA_HOME is NOT set to weblogic installation JDK, it was pointed to JAVA_HOME come with the Solaris installation.  I understand that could be one of the reason of not starting the server instance. Update the .profile file with  JAVA_HOME as required to set JDK residing in WebLogic installation path.

What is your NodeManager saying?
My buddy located the logs and found the following line
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE

To avoiding this Exception you must have certificate generated and installed as requested bye nodemanager. Why we need this SSL certificate? Open the WebLogic Admin console, found that he is working with SSL type of Node Manager. If you don't have secure certificates to configure you should not use SSL type nodemanager. I suggested better to use "Plain" type for the NodeManager configuration on the machine for learning purpose.  Change it in the admin console and save the changes.

Redefining NM properties

You need to update few NodeManager properies on every machine your domain is configured.

Machine wise changes
ListenAddress=yourmachineip
ListenPort=5557

Common "Plain" setting for all the machines involved in domain
SecureListener=false

Stop the Admin server and managed servers (if any running) stop the NodeManger in all machines the domain configured.


Now, First start the NodeManager in the which Admin server residing then start the Admin Server. After Admin Server starting all the domain configuration details will be captured by NodeManager on that machine and ready for communicate with remote NodeManagers. Next, Start the NodeManager on the remote Machine(s) . 

Finally, start the remote server using admin console.  to start your remote managed server make sure that your admin server, Node Managers on admin machine and remote machines must be running without any ERROR or WARN messages. Conclude your configuration is successful by checking the Node Manager logs, and on the admin console check the server state

Note that jps command will list all java processes which includes NodeManager process too.
$jps
12121 NodeManager
4544 Server

Further References for Secure NodeManager:
1. SSL Node Manager configuration
2. Fisal SSL

Friday, August 13, 2010

Best Practices for WebLogic Environment

Here I am jotting out few interesting Best practices for Oracle WebLogic environments, which I have experienced/encountered hurdles while preparing a WebLogic Domain. To Win this Running race you must overcome these hurdles, the best solutions is remembering all of them now I am sharing with you guys here:

1. Dedicated User and group
Oracle WebLogic installation on Solaris machine or Linux or a Windows machine, it is better to have a dedicated user and shared growup where you can install the Middleware components WebLogic, Coherence, WebCenter sites, Content Management etc. provide access to all  so that all other users need not to installing  for each new domain on the same machine.
  useradd [options] LOGIN

Some of important options are:

-d home directory
-s starting program (shell)
-p password
-g (primary group assigned to the users)
-G (Other groups the user belongs to)
-m (Create the user's home directory
My experiment:
useradd -g wladev -s /bin/bash -p xxxxxxxx -d /home/wladmin -m wladmin
Remember that, You can run above user creation command if you have root user access only. Double check the password working for the newly created user. Now a days Virtual Box users are becoming super user (root) just by sudo access. Switch to user (su - wladmin) will connect to the new user.
Change the user password from root user, using passwd wladmin command. On the root user it won't ask you previous password.

2. Using of sed for Migrations
When I worked for WebLogic 8.1 to WebLogic 9.2 migration for each instance wise configuring fresh properties updating took me around a week time for whole environment. Time changed and the requirements changed and this time WebLogic 11g migration from WebLogic 9.2 I have an idea to use stream editing option, and applied with proper regular expressions to finish my task. It worked for me perfectly it is awesome, whole updating done in half hour with small script that included sed in a for loop. I had experianced the fastness  with sed to change multiple lines search and replace in multiple files in the same machine

The following diagram will tells you how sed works on text patterns.
SED Script functionality

Learning SED scripting

for i in `ls instances|grep c`
do
cd $INST_HOME/$i/config
cp /oldinstance/$i/config/*.properties .
sed -e "s/$i-//g" \
      -e "s/$i\_//g" \
      -e "s/\_$i//g" \
      -e "s/-$i//g" \
      -e "s/wluser92/wluser11/g"  <  log4j.properties  >temp
mv temp log4j.properties


User per domain: If you are preparing a development environment then you can choose a user per domain it is the best way to avoid conflicts between developers code changes etc. Install new Oracle License and keep always a backup of old License.

3. Customizing your domain

AdminServer name, ListenAddress, ListenPort, Some times you might see errors saying that "Listen Port already in use", To avoiding port conflicts: Before assigning a port to your WebLogic instance better you check whether it is already in use or not? by using netstat command.
About Virtual IP issues.
i) WebLogic Server wise logs generation
WebLogic Server instance each one can generate separate server side STDOUT logs, STDERR logs as well as application logs. These logs must be collected in a separate mount point will make free for disk utilization memory problems. According to the application severity we can keep archiving the rotated logs on the disk. Most of the Admins, developers while doing troubleshooting for an issue they must revisit these logs and they must know from which server it was happening for this log4j provides more flexibility to digg/debug every Java package, class level, even method, line level too.

How to collect it?
To make this possible you need to enable your WebLogic server library path must pick the log4j-1.2.8.jar and the logging definitions in a separate file lets say it as log4j.properties file in the CLASSPATH.

Where to set?
Before weblogic.jar path or after? Oracle recommands application related jars and third-party jars must be set after weblogic.jar. So log4j.xx.jar must be in POST_CLASSPATH.

ii) Editing for all Domain Environments
a. JAVA_OPTIONS
b. USER_MEM_ARGS
c. JVM type
d. CLASSPATH (PRE/POST)
e. Native IO options
f. MuxerThread
g. SocketReaders

Changing JVM Hotspot Compiler

Editing common scripts impacts all the domains in that machine. If there is a need for the WebLogic server run with server JVM that will give more scope for In commEnv.sh script we are going to update .
Sample Example:
191    Sun)
192      JAVA_VM=-client
change to
191    Sun)
192      JAVA_VM=-server
d
4. Scalability for Domain
Adding servers to  or removing servers existing Cluster is nothing but scalability. While defining new Cluster identify proper multi-cast address suitable to your environment with multi-cast test. You must use Interface Address as the DNS/IP of the machine where the instance is configured in each server's Cluster tab. This will make you easy to run the clustered environment.

5. Effort saving means cost saving
While preparing your configuration keep focus on portable coding. I have been to many UNIX forums to find portable and flexible scripting. Here I am sharing for you. Efforts can be focused on the following things




Customized Stop All script for each domain 
Best option I found from stopManagedWebLogic.sh given by the Oracle. It is normal shell script it will invokes the Python script on the fly. I just replaced the 'Server' with 'Cluster' argument for shutdown WLST command and also called admin stop script in the bottom.


#!/bin/sh

DOMAIN_HOME="/home/domains/mydomain"

. ${DOMAIN_HOME}/bin/setDomainEnv.sh
ADMIN_URL="t3://my.adminhost.com:adminPort"

echo "wlsUserID ='username'" >>"shutdown.py" 
echo "wlsPassword = 'password'" >>"shutdown.py" 
echo "connect(${wlsuserID}, ${wlsPassword}, url='${ADMIN_URL}')" >>"shutdown.py" 
 
#=== fetching cluster list from the domain configuration repository =============
for cl in `grep -i clstr\<\/n ../config/config.xml|sed  's/.*\(.*\)<\/name>.*/\1/'
do
 echo "shutdown($cl','Cluster')" >>"shutdown.py" 
 echo "state($cl)">>"shutdown.py" 
done
echo "shutdown()" >>"shutdown.py" 
echo "exit()" >>"shutdown.py" 
 
echo "Stopping Weblogic Server..."
java -classpath ${FMWCONFIG_CLASSPATH} ${MEM_ARGS} ${JVM_D64} ${JAVA_OPTIONS} weblogic.WLST shutdown.py  2>&1 

echo "Done"

iv. Server Health checking script
Separate is a time, effort saving

6. Deployment Strategies
System resource deployments better you prepare your customized JDBC Data source configuration script using WLST. Avoid Start-up Classes configuration which will make dependable deployment, which leads you to not able to use side-by-side(SBS) deployment advantage.

7. WebLogic 9.x onward you have a flexibility to use Deployment plans for UAT, QA, Staging environments as same as production. This will reduce the problem of porting the code on one environment to other without any configuration changes.

8. OutOfMemoryError

Most of the production environments first hurdle is OOME, if it is occurring in your Web-tier environment then you can use JSP pre-compile.Prepare GC monitoring scripts

9. Now a days everywhere you can find Virtualization (SOLARIS ZONES), Cloud computing, Clustering (Database RAC), Grid concepts, Veritas Clustering in Disks/RAID etc. Have each topic in breif knowledge that could make you understand if anything goes wrong somewhere in the application environment you can easily figure it out.

10. Know about your production environment end to end how the data flows? Firewall, Load balancer software or hardware, proxy-plugin security aspects, Network connectivity, Net-backup locations etc.

Reference URL:
1. Best Virtual IP usage http://blogs.oracle.com/muralins/2007/08/ipmp_ip_multipath.html
http://www.eng.auburn.edu/~doug/howtos/multipathing.html
2. Linux Administration Commands http://www.faqs.org/docs/abs/HTML/system.html
3. WLST Configuration help http://wlstbyexamples.blogspot.com/
4. WebLogic Upgrade http://download-llnw.oracle.com/docs/cd/E13179_01/common/docs100/upgrade/comm_ref.html

Monday, August 9, 2010

Do you have alias in your profile??

Making mistake is human nature but, overcoming that mistake turning to desired outcome is very wise thing. You want to learn about that wise trick?


If yes follow me!!

In Unix platform there is a file called .profile (some Linux enviroments it is .bash_profile). Which executed automatically when you logon to your user, which inturn have the enviroment variables which are required to run your application on WebLogic. For WLA required to setup a standard .profile for his production enviroment that is for JAVA_HOME, WL_HOME are major some of the enviroments requires ORA_HOME in this file.

Defining JAVA_HOME, WL_HOME, etc is common thing for any WLA. But your my Smart WLA then you might applying a intersting UNIX command 'alias'. Most of the time we use 'ls' command for listing the files/folders. By mistake you might mistype as 'sl' instead of 'ls'. sometimes feeling hungry!! or thursty!! and typing command that may go wrong like 'cd..' or 'cd...'. Collect all these common mistypos and make use of 'alias' command then see!!

Though, You make mistake in typing it will give you desired command output. I like this command, since when I found it. This command makes magic, it is very interesting feature in UNIX. Just append to your .profile the following lines:
######WLA Smart idea for using alias #############################
alias cls=clear
alias sl=ls 
alias cd..="cd .."
alias cd...="cd ../.."
alias gi="grep -i"
alias l="ls -al"
alias lm="ls -al | more"
alias h=history
alias hm="history | more"

This is value added to your work, obviously it will reduce your time spending on working machine. Performance will be improved if you find more common mistakes in your enviroment and sortout with 'alias' the magical command, it will give you fruties results!!

Wednesday, August 4, 2010

WebLogic Port mapping to Process ID

Hey dear WLA most of us come from Development environment to production environments. Hope the scienario thatI am going to discuss here is a common to everyone, who is working on development environment. Sometime or other you might felt I should have a handy script that could takes input as a WebLogic instance port which usually get from the applicaiton URL. The WebLogic port and list all the Process IDs which are associated with it on UNIX(here I got solution for Solaris) environment. Recently I found a Sun blog which is clearly discussed similar issue. I am re-compiling the same with customizing to our WebLogic Listen Port, WebLogic Server runs with a Java generated child Process ID. 'lsof' is the command you can execute and get this solved but most of the times it is a Sun third party downloads, that is not allowed in many Banking, Financial and Insurance Service organizations. And one more reason is small companies doesn't efford for third party tools.

Then, Your choice will be writing a handy script that will do the same task as lsof command helps to find the process id for a given WebLogic Listening port.
#!/bin/ksh 
 
pids=$(/usr/bin/ps -ef -o pid=)
 
if [ $# -eq 0 ]; then 
   read wlport?"Enter port you would like to know Java Process Id for: " 
else 
   wlport=$1 
fi 
 
for f in $pids 
do 
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $wlport$" 
   if [ $? -eq 0 ]; then 
        echo "===============***=============***==============="
        echo "ListenPort: $wlport is being used by Java PID:\c" 
        ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f 
        exit 0 # if you suspect more Weblogic instances with same listen port remove this
   fi 
done

Note: Save this script to commonly accessing user location name it as WLPort2Pid.ksh
Run with argument or without also it will work!! But you need to input the listening port that is must.
-->

Writing about this experiment is a new learning for me too.
Hope you enjoyed this one!! HAPPY SCRIPTING!!

Sunday, July 18, 2010

Heap dump on a Unix machine

After exploring the Jay/Faisal's blog on 'jmap' java utility usage and Heap dump. It is really great work by Jay/Faisal. I just thought that similar kind of experiment we did long back on UNIX machines. I am glad to sharing that with you guys.

Last year, we were struggling to overcome the OutOfMemoryError, which would effect the most of productive hours. In this assignment I need to figure out what process is causing the low memory in the environment is identified by searching a all log files in the machine. Assume that all the WebLogic instance log files are collected into common directory structure, each of them are stored respective instance named folder.

Script 1:

After identifying the impacted instances, I need to take the heap dump of that particular instance with corresponding process id.


#==============================================================
# File Name  : CheckLogs.sh
# Author  : Pavan Devarakonda
# Purpose  : Script for searching all WebLogic instances logs in a box
#==============================================================
instances=`ls /path/instances|grep web`
phrase=$1
date
for x in $instances
do
        echo 'Checking in the :' $x
        cd /path/instances/$x/logs
        egrep -i $phrase instance*log
        egrep -i $phrase instance*out
done
 
# Know the CPU load average at this time
date
uptime
-->

Script 2: Automatic script for Heap dump, here you need to provide the managed server name at the command-line argument. 
#!/bin/bash

#=======================================================
# Name    : InstanceHeapdump.sh
# Purpose : This script will takes instance name as input
# Takes the thread dump and also takes heap dump
#=======================================================
 
if [ "$1" = "" ]; then
        echo "Usage : $0 "
        exit
else
        instance="$1"
        user=$LOGNAME
        ppid=`ps -fu $user|grep $instance|grep startMan|grep -v grep|awk '{print $2}'`
        wpid=`ps -fu $user|grep $ppid|grep startWebLogic.sh|awk '{print $2}'`
        jpid=`ps -fu $user|grep $wpid|grep java|awk '{print $2}'`
        echo 'The Java PID:' $jpid
        kill -3 $jpid
        if [ $jpid = "" ]; then
                echo "Invalid instance input..."
                exit
        else
               jmap -dump:live,format=b,file=heap.bin $jpid
                mv heap.bin $instance_heap.bin
        fi
fi

This could give you one more way of finding a java process in UNIX machine. You can use jps command instead of three lines of awk filters. In this same script to make hold the java process not to crash, we can call a WLST script to suspend the instance and then you can happily take the heap dump.

What to do after heapdump creation?
Use jhat command to run the analyzer to show the outcome on a browser.
Follow the Faisal tips to find memory leaks, use eclipse MAT that is comfortable for your system.

Monday, July 12, 2010

Clearing Cache for WebLogic instance

Hey smart WLA, Here I am with one more interesting story of WebLogic Administration "Clearing Cache". This is most often in development environments, where you need to clear the cache for new releases for Web-tier changes, XML loadings, JDBC connection changes, JMS issues etc.,.

Some of the great developers think like this "I can change minor things directly in the jsp files to test stuff"

What is actually WebLogic Cache?
Basically all the web-tier related files (.jsp .class, JSPCompiled files etc.,) get stored in ./wlnotdelete/app_yourapplicaiton directory. This is treated as cache whenever there is restart of a WebLogic instance happen then the WebLogic server will look-up for last serviced object status stored in the cache to service for any pending requests. Usually, when your EJB Classes need sessions, JMS object requires persistance, your web-tier may contain static contents then Cache will be used by WebLogic Application Server instance.

Why we need to remove Cache?
Whenever your application is accessed for the first time that fresh deployment of a new version, WebLogic server lookup in this directory, if there are older objects persists that will be conflict with new code objects. This is where the need of removal of cache arises.

Where there is a need of new version deployment we might need to clear the cache when the changes to the new version is not reflected. In the old version WebLogic 8.1 below we used to remove .wldonotdelete folder. In new version of WebLogic 9.x onwards removal of cache means deleting each server instance's tmp folder contains cache, stage folders or you can simply remove tmp folder too provided there should not be configuration changes happen to the server.

Generally for WebLogic 9.x and higher versions
WIN: C:\bea\user_projects\domains\yourdomain\servers\yourserver\tmp
UNIX: /bea/user_projects/domains/yourdomain/servers/yourserver/tmp

you can use the following commands to clear the cache:
WIN: rd C:\bea\user_projects\domains\yourdomain\servers\yourserver\tmp \s
UNIX: rm -rf /bea/user_projects/domains/yourdomain/servers\yourserver/tmp

Here I am removing all the subdirectories and files in the the given directory.

When to do this?
After Stopping the WebLogic server instance you can go for removal of cache.
Mostly, Spring framework users, struts framework users have this no changes reflected issue for their web applications.

Alternative solution is you can use 'stage' mode set to 'no_stage' deployment. when undeployed an applicaiton then WebLogic server itself removes the cache objects.

Wednesday, July 7, 2010

Copying to multiple remote machines

Here is another interesting story of WLA (of-course mine), When I visited US in 2007 there was lot of restrictions in work places. "Hey its production you know what happen if you touch it??" "Don't open this files", "Don't enter into that folders", it will be dangerous... I know that very well what is missing in the system, where it is required a change but my hands kept criss cross!!

Days passed I got opportunity to come again on long term. Now, the whole new System is going shapeup with my hands. The system is awaiting for me since long days. :) All those sparkling colorful ideas running around my mind, got chance to flow onto the system to form various automated scripts, which are having little in size with greater capabilities.

Whenever there is a application version release the archive files(.jar, .war, .ear) need to copied to all over the remote machines. In olden days we were using 'sftp' command and its related 'put', 'mput', 'get' and 'mget' commands to complete the task. Manually double checking wheather the copying is done correct or not, by verifying in each machine content by comparing the each file sizes. Here I found a flaw that there could be chance of human error. While understanding 'Six Sigma Course', where I learnt about human errors makes greater defect to many customer's business. To avoid this better option is automation of the task as possible as much.I remembered Mr. Neil Gogte words about "Cyber Coolie". The software engineer who works as per his contractor asked him to do only those things he will do. Never think other than the work which is assigned to him. My soul shouts out 'NO!!', I cannot be a Cyber coolie any more !!

My beautiful sparkling colorful ideas SSH password less connection to multiple remote machines, powerful bright idea of 'scp' command usage with verification option built within a script come out as a wonderful shell script, which had mighty power of built-in checking with no chances of human error. When I show the execution of this script to my teammates they are very happy and appreciated me. Many productive hours are saved though this activity was disturging to other regular job. The script made almost hands free task!! Finally, That's the way team turn happy ever by using the easy script.

Script is :
TADAA!!!!!!!!!!

# Define variables values
src=
target=
hostlist=
user=
Logfile=

#=== script logic starts here ====
if [ -d $src ]
then
echo "Code folder found, preparing to transfer\n"
while read server
do
result=scp -r $src $user@${server}:$target
if [ $result -eq 0 ]
then
echo $server transfer done >> $Logfile
else
echo $server transfer failed.
exit
fi
done < $hostlist
else
echo "Code folder \"$src\" not found\n"
fi

Tuesday, June 29, 2010

Monitoring CPU Load Averages with Shell Script

Today I started re-inventing myself, started looking what all I did for me and my team to perform in better ways. Remembered those we used to open UNIX SSH windows to monitor How the CPU load average in each site. While doing this monitoring activity on site1, there might be possible that some other site reach to overload, which leads to uncontrol tendency to work. It was funny, my dear buddy named it as 'Barbar work!!' :)

After little R & D on Google/Internet found few suitable solutions. I had chosen 'uptime' command running with remote SSH connection in a loop. Adding more value to this sending a mail on the event of crossing the threshold value. This threshold will be vary depending upon the application and CPU power. Trail and error make you to identify what could be the threshold. Defining these threshold values Venu GopalRao helped a lot. Once script started working he was amazed and appreciated as well.

This script can be run forever with a specified time interval. You can use 'at' command or 'crontab' also for this task. I prepared a 'bash' script that could work for Solaris and also on Linux.

Before to this script we need to establish the password less connection to all the remote machines with 'key-gen' command. Public key authentication, which is the good choice password less connecting remote UNIX machines. Here, you can use any choice for encryption algorithms such as RSA, DSA etc.,


Customization/Cosmotics to this script
When you run this script at your prompt you can see the high load average server details in red color which makes sense to act up on that quicker. All server list I had kept in a plan text file and accessed it line by line as array for looping.

#!/bin/bash
#======================================================
# This script will check CPU Load, network ping status
# and also checks diskspace on every machine
#======================================================
RECIPIENTS="pavanwla@yahoo.co.in"
LOG=./load.log
 
check_load()
{
        loadnow=`echo $msg| cut -d, -f4 | cut -d: -f2 | cut -d. -f1`
        d=`echo $msg |awk '{print $((NF-1))}'`
        SD=`date "+%Y-%h-%d@%H:%M:%S"`
        echo $SD '****'
        if [ $loadnow -gt 14 ]; then
                echo -e ' \033[31m' $server ' ' $loadnow '\033[m'>>$LOG
                echo $SD $server ' ' $loadnow |mailx -s LOAD_WARN $RECPIENTS
        elif [ $loadnow -gt 19 ]; then
                echo -e ' \033[31m' $server ' ' $loadnow '\033[m'>>$LOG
                echo $SD $server ' ' $loadnow |mailx -s LOAD_CRITICAL $RECPIENTS
        else
                echo -e $server '\t' $loadnow '\t' $p '\t'$d >>$LOG
        fi
}

#==============================================================
#                 M A I N  S C R I P T
#==============================================================
if [ -f $LOG ]
then
        rm $LOG
fi
serlist=`cat prodServers.txt`
echo -e "========================================================">>$LOG
echo -e "  HOSTNAME  CPU Load     Network status       Disk Space">>$LOG
echo -e "========================================================">>$LOG
 
for server in $serlist
do
        echo 'connecitng to ' $server
        msg=`ssh $server "uptime; df -k /app|grep app |awk '{print \$5}'"`
        p=`ping -s $server 56 2 |grep loss | awk -F',' '{ print $3 }'`
        check_load
done
cat load.log
Please make sure that you must have prodServers.txt file in the same script path. Sample prodServers.txt file as follows:
myprod.server1.com
myprod.server2.com
...
myprod.server20.com
Upgrade Script
Adding more flavor to the load average script finding the disk space on every machine and also verifying network connectivity that ping response to every machine. Initially, I made it with two ssh commands one is for finding load average on each remote machine, other one is to check disk space on each machine. But it is not a good scripting way. With the help of the linkedin discussion I have updated it to single ssh command so that it will process faster by making less ssh sessions.

What is Next step??
If you find CPU load average is going above the threshold then you need to prepare yourself alert. Open that concern UNIX machine and find the causing process on that machine with 'top' command or 'prstat -L -a ' command options respective UNIX environment.

Take the Thread dumps of that culprit java process id of WebLogic instance. If CPU load reaching more than threshold then terminate that process/instance.
Analyze why that time CPU load gone high what thread were doing that time.

Note: This script created and executed on Solaris which remotely connects Linux and Solaris machines.

Good Forum Reference:
1. Linkedin Discussion
2. http://www.daniweb.com/forums/thread48764.html">Shell Script for Load monitoring!

Comments are most welcome!! HAPPY TO HELP!!

Wednesday, June 16, 2010

Restricting the access from Apache HTTP Configuration

There is one common problem that most of the Web Admins encounter when configuring the Apache Web Server as proxy plug-in. The Java archive files such as .jar/.war/.ear files or any content of the Server accessing will be shown to everyone when that URL is published. Some situations demands some of the images/pictures display by your site must be restricted access of copying. The best way restrict this files from unauthorized use is by Apache File directive. This File directive can be configured in httpd.conf file in the Apache configuration.

By the way my friend Sarangapani already discussed about Apache installation & configuration, So I am not repeating again here.



In the same configuration file we can also restrict the access from specific IP address or DomainNames. Explore more about this topic


File Directive

A detail description about valnerabilities about Apache plugin is given in a separate blog. The link is here click

Friday, May 14, 2010

Java parameters for WebLogic Server

There are many burning issues on going WebLogic based environments due to this Java parameters

  • JAVA_OPTIONS, l
  • USER_MEM_ARGS, 
  • PRE_CLASSPATH, 
  • POST_CLASSPATH. 

Heap size in MEM


The USER_MEM_ARGS can be defined for WebLogic as follows as minimum requirement :

-Xms1024m The minimum size of JVM
-Xmx1024m The maximum size of JVM keeping the same as minimum will avoid number of GCs.
-XX:NewSize=512m The Young Generation (YG) space size
-XX:MaxNewSize=512m The maximum sise for Young Generation space
-XX:SurvivorRatio=6 There will be two (from, to) survivor spaces both will equal in space and one will be always empty for swapping which will be used after GC.

-Xnoclassgc

Perm space 

-XX:PermSize=128m The permanant space for static classes they will be loaded only once.
-XX:MaxPermSize=128m There could be chance of increase of classes as the application functionality increased widely.

Troubleshoot OOM errors
In production or Development environment if there is issue with memory we need to have a snapshot of how the JVM is occupied this we can get by command line with passing WebLogic process id. Other option is that we can have it in the JAVA_OPTION because it will capture the heap dump automatically as on when there is   memory issue.

-XX:+HeapDumpOnOutOfMemoryError This parameter will be used whenever OOME encountered by instances heap dump will be take automatically
-XX:+DisableExplicitGC This parameter will restrict the external Garbage collection programs from explicit

GC Algorithms

We have three options of GC algorithms Serial, Parallel, Concurrent Mark Sweep. Most of WebLogic production environments prefers CMS.

-XX:+UseConcMarkSweepGC The GC algorithm you can choose other ways as per your application need.
-verbose:gc The GC functionality will be traced with this parameter

While debugging the GC issue, you might need to know that how frequently GC happen for WebLogic instance. the following options is going to help you.

-XX:+PrintGCTimeStamps This will print the timestamp of minor GC happen

-XX:+PrintGCDetails This will write to GC logs about Eden, Young, Perm space details
-XX:+PrintGCApplicationStoppedTime Whenever Full GC happen there will be pause for the application execution that will be written to logs
-XX:+PrintHeapAtGC Current JVM space details after Full GC.

We can collect the GC logs in a separate file by this argument it takes absolute path.
-Xloggc:/$HOME/logs/instance_GCoutput.log GarbageCollection log path can be specified by this.

-Xint interpreter-only mode for JIT this is introduced in mixed mode JVMs.

References:

1. http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
2. IBM DeveloperWorks :
3. http://olex.openlogic.com/wazi/2009/how-to-fix-memory-leaks-in-java/
4. http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gblfj
5. JVM Options -Xint

Blurb about this blog

Blurb about this blog

Essential Middleware Administration takes in-depth look at the fundamental relationship between Middleware and Operating Environment such as Solaris or Linux, HP-UX. Scope of this blog is associated with beginner or an experienced Middleware Team members, Middleware developer, Middleware Architects, you will be able to apply any of these automation scripts which are takeaways, because they are generalized it is like ready to use. Most of the experimented scripts are implemented in production environments.
You have any ideas for Contributing to a Middleware Admin? mail to me wlatechtrainer@gmail.com
QK7QN6U9ZST6