X11 Forwarding in SSH for Solaris and Linux

You might be wonder what is fun in this blog looking for X11 forwarding on Solaris 5.10!! You might be excited to see the beauty of colorful Oracle Fusion Middleware WebLogic 11g installation window when you executable binaries (wls1033_solaris32.bin), You might wish to see Oracle WebLogic domain configuration Wizard (config.sh), Oracle WebLogic Domain Template builder(config_builder.sh), BEA Smart Update (bsu.sh) windows or your Oracle Database 11g while you work on a remote UNIX machines.

After spending few hours on the Internet found many blogs written on their experiences, few document on websites, which illustrated about SSH, X11 forwarding in UNIX. I understand that what I need to do? To achieve my objective X11 Forwarding using SSH window, I must have a X emulating software installed on my desktop. you guys have this on your desktop?? If not install Hummingbird Connectivity (Xming is alternative)

My experimenting environment is Solaris 10, Oracle WebLogic, SSH (You can use Putty also), Hummingbird Connectivity (Xming is alternative)

Applicable to:
This you can use for any Java AWT or Swing programs in any UNIX based Operating environment.
You might download free Hummingbird Software also providing evaluation version who want to test their environment.

Here I am sharing with smart guys WLA/DBA my version of brain storming on this. When I first-time tried to invoke X window command it rejected and told as shown below:

No X11 DISPLAY variable was set, but this program performed an operation which requires it.

I had updated DISPLAY variable as follows

$ export DISPLAY=localhost:11.0

My first experiment with normal UNIX user
When I executed bsu.sh script I got the following in Normal UNIX user:

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class
sun.awt.X11GraphicsEnvironment
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
at sun.awt.X11.XToolkit.(XToolkit.java:89)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at java.awt.Toolkit$2.run(Toolkit.java:834)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
at sun.swing.SwingUtilities2$AATextInfo.getAATextInfo(SwingUtilities2.java:126)
at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:1556)
at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:130)
at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1591)
at javax.swing.UIManager.setLookAndFeel(UIManager.java:541)
at javax.swing.UIManager.setLookAndFeel(UIManager.java:581)
at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1343)
at javax.swing.UIManager.initialize(UIManager.java:1432)
at javax.swing.UIManager.maybeInitialize(UIManager.java:1420)
at javax.swing.UIManager.getDefaults(UIManager.java:660)
at javax.swing.UIManager.put(UIManager.java:989)
at com.bea.plateng.common.ui.gui.GUIHelper.initPLAF(GUIHelper.java:69)
at com.bea.plateng.patch.gui.PatchGUIHelper.startGui(PatchGUIHelper.java:90)
at com.bea.plateng.patch.Patch.main(Patch.java:338)

My UNIX admin buddy told that dear, first try with xclock then go for other, then I thought Ohhh!! that's good idea, let me try that...

$ xclock
bash: xclock: command not found

Hmmmmmmm!!, what is this? Think, Think Think!!!, On a Solaris 5.10 you can find some of the commands in OpenSource folder, Let me try in that path this xclock on the following path.
/usr/openwin/bin/xclock

No luck buddy, same kind of error message as bsu.sh got above.

Now I realized and verified all those PATH, CLASSPATH are pointing to proper JAVA_HOME/bin, lib paths respectively, one of them is misconfigured reset to correct path.

One of Sun blog saying about Updating $Home/.ssh/config file as follows:

Host *
ForwardX11 yes
ForwardAgent no

After doing this I had ran the following:

$ ssh -X user@hostname

This is the command will activallte the X emulation on your UNIX machine with given SSH configuration. Actually it will refer to the default configuration /etc/.ssh/config but when you create new config file on your user home directory it will prefer to execute for your user settings. This command also updates .Xauthority file when you ssh -X command.

Verified every step again finally I got X window for xclock

Hey!! Cheerup on my face!!

Experiment 2: LDAP user logon with assume user name as 'pavanwla', switched to role 'wladmin'. I started trying on it for X11 forwarding. First Let me try, xclock on

-->

[wladmin@hostname~]$ /usr/openwin/bin/xclock
Error: Can't open display:

Ohh its same DISPLAY variable problem right? You remember it!!

[wladmin@hostname~]$ export DISPLAY=localhost:10.0
[wladmin@hostname~]$ echo $DISPLAY
localhost:10.0

Let me try now, what it will say...

[wladmin@hostname~]$ /usr/openwin/bin/xclock
X connection to localhost:10.0 broken (explicit kill or server shutdown)

Oh My Gash!!! what to do now????????????
After a while, I bugged again my buddy Mr. Unix Admin about this. He told that configurations, X emulating authoriy must match ldap user and role. Now my turn, first I tried to get xclock on ldap user, that confirms everything fine at ldap user level. This configuration, Xauthority I can use in role user too, then see what I did:

[wladmin@hostname~]$ scp pavanwla@hostname:/home/pavanwla/.ssh/config .ssh/
[wladmin@hostname~]$scp pavanwla@hostname:/home/pavanwla/.Xauthority .

Started ssh session with ssh -X command.

Finally, the climax came to end, xclock started. ahh!!
Reference:
https://cygwin.com/setup-x86_64.exe
http://sourceforge.net/projects/xming/files/Xming/6.9.0.31/Xming-6-9-0-31-setup.exe/download
http://download.oracle.com/docs/cd/E15051_01/common/smartupdate/guide/quickrefax.html

Configuring Plain NodeManger on Solaris

Today morning one of my buddy pinged me from my team, seeking help on an issue he is faceing in his newly  configured WL domain for his learnings.

Understanding the Issue
My buddy novice WLA is trying to start the WebLogic instance on a remote machine. He is working on Solaris 10 operating environment, WebLogic 9.2. I asked for Java version, He executed the following command:

$ java -version

I asked for jps command execution the system is refused and said that ' jps is bad cammand'. I understand that  JAVA_HOME is not set for his environment. asked for echo $PATH. It is showing Java Path. Ohh!! what is missing then??

$which java
this resulted that JAVA_HOME is NOT set to weblogic installation JDK, it was pointed to JAVA_HOME come with the Solaris installation.  I understand that could be one of the reason of not starting the server instance. Update the .profile file with  JAVA_HOME as required to set JDK residing in WebLogic installation path.

What is your NodeManager saying?
My buddy located the logs and found the following line
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE

To avoiding this Exception you must have certificate generated and installed as requested bye nodemanager. Why we need this SSL certificate? Open the WebLogic Admin console, found that he is working with SSL type of Node Manager. If you don't have secure certificates to configure you should not use SSL type nodemanager. I suggested better to use "Plain" type for the NodeManager configuration on the machine for learning purpose.  Change it in the admin console and save the changes.

Redefining NM properties

You need to update few NodeManager properies on every machine your domain is configured.

Machine wise changes
ListenAddress=yourmachineip
ListenPort=5557

Common "Plain" setting for all the machines involved in domain
SecureListener=false

Stop the Admin server and managed servers (if any running) stop the NodeManger in all machines the domain configured.


Now, First start the NodeManager in the which Admin server residing then start the Admin Server. After Admin Server starting all the domain configuration details will be captured by NodeManager on that machine and ready for communicate with remote NodeManagers. Next, Start the NodeManager on the remote Machine(s) . 

Finally, start the remote server using admin console.  to start your remote managed server make sure that your admin server, Node Managers on admin machine and remote machines must be running without any ERROR or WARN messages. Conclude your configuration is successful by checking the Node Manager logs, and on the admin console check the server state

Note that jps command will list all java processes which includes NodeManager process too.
$jps
12121 NodeManager
4544 Server

Further References for Secure NodeManager:
1. SSL Node Manager configuration
2. Fisal SSL

Best Practices for WebLogic Environment

Here I am jotting out few interesting Best practices for Oracle WebLogic environments, which I have experienced/encountered hurdles while preparing a WebLogic Domain. To Win this Running race you must overcome these hurdles, the best solutions is remembering all of them now I am sharing with you guys here:

1. Dedicated User and group
Oracle WebLogic installation on Solaris machine or Linux or a Windows machine, it is better to have a dedicated user and shared growup where you can install the Middleware components WebLogic, Coherence, WebCenter sites, Content Management etc. provide access to all  so that all other users need not to installing  for each new domain on the same machine.

  useradd [options] LOGIN

Some of important options are:

-d home directory
-s starting program (shell)
-p password
-g (primary group assigned to the users)
-G (Other groups the user belongs to)
-m (Create the user's home directory

My experiment:

useradd -g wladev -s /bin/bash -p xxxxxxxx -d /home/wladmin -m wladmin

Remember that, You can run above user creation command if you have root user access only. Double check the password working for the newly created user. Now a days Virtual Box users are becoming super user (root) just by sudo access. Switch to user (su - wladmin) will connect to the new user.
Change the user password from root user, using passwd wladmin command. On the root user it won't ask you previous password.

2. Using of sed for Migrations
When I worked for WebLogic 8.1 to WebLogic 9.2 migration for each instance wise configuring fresh properties updating took me around a week time for whole environment. Time changed and the requirements changed and this time WebLogic 11g migration from WebLogic 9.2 I have an idea to use stream editing option, and applied with proper regular expressions to finish my task. It worked for me perfectly it is awesome, whole updating done in half hour with small script that included sed in a for loop. I had experianced the fastness  with sed to change multiple lines search and replace in multiple files in the same machine

The following diagram will tells you how sed works on text patterns.

SED Script functionality

Learning SED scripting

for i in `ls instances|grep c`
do
cd $INST_HOME/$i/config
cp /oldinstance/$i/config/*.properties .
sed -e "s/$i-//g" \
      -e "s/$i\_//g" \
      -e "s/\_$i//g" \
      -e "s/-$i//g" \
      -e "s/wluser92/wluser11/g"  <  log4j.properties  >temp
mv temp log4j.properties

User per domain: If you are preparing a development environment then you can choose a user per domain it is the best way to avoid conflicts between developers code changes etc. Install new Oracle License and keep always a backup of old License.

3. Customizing your domain

AdminServer name, ListenAddress, ListenPort, Some times you might see errors saying that "Listen Port already in use", To avoiding port conflicts: Before assigning a port to your WebLogic instance better you check whether it is already in use or not? by using netstat command.
About Virtual IP issues.
i) WebLogic Server wise logs generation
WebLogic Server instance each one can generate separate server side STDOUT logs, STDERR logs as well as application logs. These logs must be collected in a separate mount point will make free for disk utilization memory problems. According to the application severity we can keep archiving the rotated logs on the disk. Most of the Admins, developers while doing troubleshooting for an issue they must revisit these logs and they must know from which server it was happening for this log4j provides more flexibility to digg/debug every Java package, class level, even method, line level too.

How to collect it?
To make this possible you need to enable your WebLogic server library path must pick the log4j-1.2.8.jar and the logging definitions in a separate file lets say it as log4j.properties file in the CLASSPATH.

Where to set?
Before weblogic.jar path or after? Oracle recommands application related jars and third-party jars must be set after weblogic.jar. So log4j.xx.jar must be in POST_CLASSPATH.

ii) Editing for all Domain Environments

a. JAVA_OPTIONS
b. USER_MEM_ARGS
c. JVM type
d. CLASSPATH (PRE/POST)
e. Native IO options
f. MuxerThread
g. SocketReaders

Changing JVM Hotspot Compiler

Editing common scripts impacts all the domains in that machine. If there is a need for the WebLogic server run with server JVM that will give more scope for In commEnv.sh script we are going to update .
Sample Example:

191    Sun)
192      JAVA_VM=-client
change to
191    Sun)
192      JAVA_VM=-server

d
4. Scalability for Domain
Adding servers to  or removing servers existing Cluster is nothing but scalability. While defining new Cluster identify proper multi-cast address suitable to your environment with multi-cast test. You must use Interface Address as the DNS/IP of the machine where the instance is configured in each server's Cluster tab. This will make you easy to run the clustered environment.

5. Effort saving means cost saving
While preparing your configuration keep focus on portable coding. I have been to many UNIX forums to find portable and flexible scripting. Here I am sharing for you. Efforts can be focused on the following things

Customized Stop All script for each domain 
Best option I found from stopManagedWebLogic.sh given by the Oracle. It is normal shell script it will invokes the Python script on the fly. I just replaced the 'Server' with 'Cluster' argument for shutdown WLST command and also called admin stop script in the bottom.

#!/bin/sh

DOMAIN_HOME="/home/domains/mydomain"

. ${DOMAIN_HOME}/bin/setDomainEnv.sh
ADMIN_URL="t3://my.adminhost.com:adminPort"

echo "wlsUserID ='username'" >>"shutdown.py" 
echo "wlsPassword = 'password'" >>"shutdown.py" 
echo "connect(${wlsuserID}, ${wlsPassword}, url='${ADMIN_URL}')" >>"shutdown.py" 
 
#=== fetching cluster list from the domain configuration repository =============
for cl in `grep -i clstr\<\/n ../config/config.xml|sed  's/.*\(.*\)<\/name>.*/\1/'
do
 echo "shutdown($cl','Cluster')" >>"shutdown.py" 
 echo "state($cl)">>"shutdown.py" 
done
echo "shutdown()" >>"shutdown.py" 
echo "exit()" >>"shutdown.py" 
 
echo "Stopping Weblogic Server..."
java -classpath ${FMWCONFIG_CLASSPATH} ${MEM_ARGS} ${JVM_D64} ${JAVA_OPTIONS} weblogic.WLST shutdown.py  2>&1 

echo "Done"

iv. Server Health checking script
Separate is a time, effort saving

6. Deployment Strategies
System resource deployments better you prepare your customized JDBC Data source configuration script using WLST. Avoid Start-up Classes configuration which will make dependable deployment, which leads you to not able to use side-by-side(SBS) deployment advantage.

7. WebLogic 9.x onward you have a flexibility to use Deployment plans for UAT, QA, Staging environments as same as production. This will reduce the problem of porting the code on one environment to other without any configuration changes.

8. OutOfMemoryError

Most of the production environments first hurdle is OOME, if it is occurring in your Web-tier environment then you can use JSP pre-compile.Prepare GC monitoring scripts

9. Now a days everywhere you can find Virtualization (SOLARIS ZONES), Cloud computing, Clustering (Database RAC), Grid concepts, Veritas Clustering in Disks/RAID etc. Have each topic in breif knowledge that could make you understand if anything goes wrong somewhere in the application environment you can easily figure it out.

10. Know about your production environment end to end how the data flows? Firewall, Load balancer software or hardware, proxy-plugin security aspects, Network connectivity, Net-backup locations etc.

Reference URL:
1. Best Virtual IP usage http://blogs.oracle.com/muralins/2007/08/ipmp_ip_multipath.html
http://www.eng.auburn.edu/~doug/howtos/multipathing.html
2. Linux Administration Commands http://www.faqs.org/docs/abs/HTML/system.html
3. WLST Configuration help http://wlstbyexamples.blogspot.com/
4. WebLogic Upgrade http://download-llnw.oracle.com/docs/cd/E13179_01/common/docs100/upgrade/comm_ref.html