Thursday, November 12, 2009

Steps to Troubleshoot Typical Java Problems

Steps To Troubleshoot Typical Java Problems


This section list some typical java problems and the actions we need to perform, the tools we need to use to collect more information and/or to analyse and debug the problem.


Hung, Deadlocked, or Looping Process

  • Print thread stack for all Java threads.
    • Control-\
  • Print thread trace for a process.
    • kill -QUIT pid
    • jstack -F pid (with -F option if pid does not respond)
  • Detect deadlocks.
    • JConsole tool, Threads tab
    • Print information on deadlocked threads: Control-\
    • Print lock information for a process: jstack -l pid
  • Get a heap histogram for a process.
    • Start Java process with -XX:+PrintClassHistogram, then Control-\
    • Start Java process with -XX:+PrintClassHistogram, then kill -QUIT pid
    • jmap -histo pid (with -F option if pid does not respond)
  • Dump Java heap for a process in binary format to file.
    • jmap -dump:format=b,file=filename pid (with -F option if pid does not respond)
  • Print shared object mappings for a process.
    • jmap pid
  • Print heap summary for a process.
    • jmap -heap pid
  • Print finalization information for a process.
    • jmap -finalizerinfo pid
  • Attach the command-line debugger to a process.
    • jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=pid
  • Attach the command-line debugger to a core file on the same machine.
    • jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=path,core=corefile
  • Attach the command-line debugger to a core file on a different machine.
    • On the machine with the core file: jsadebugd path corefile
      and on the machine with the debugger: jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine

Back to top

Post-mortem Diagnostics

  • Examine the fatal error log file. Default file name is hs_err_pidpid.log in the working-directory.
  • Create a heap dump.
    • HPROF: java -agentlib:hprof=file=file,format=b application; then Control-\
    • HPROF: java -agentlib:hprof=heap=dump application
    • JConsole tool, MBeans tab
    • Start VM with -XX:+HeapDumpOnOutOfMemoryError; if OutOfMemoryError is thrown, VM generates a heap dump.
  • Browse Java heap dump.
    • jhat heap-dump-file
  • Dump Java heap from core file in binary format to a file.
    • jmap -dump:format=b,file=filename corefile
  • Get a heap histogram from a core file.
    • jmap -histo corefile
  • Print shared object mappings from a core file.
    • jmap corefile
  • Print heap summary ffrom a core file.
    • jmap -heap corefile
  • Print finalization information from a core file.
    • jmap -finalizerinfo corefile
  • Print Java configuration information from a core file.
    • jinfo corefile
  • Print thread trace from a core file.
    • jstack corefile
  • Print lock information from a core file.
    • jstack -l corefile

Back to top

Monitoring

  • Print statistics on the class loader.
      jstat -classvmID
  • Print statistics on the compiler.
    • Compiler behavior: jstat -compiler vmID
    • Compilation method statistics: jstat -printcompilation vmID
  • Print statistics on garbage collection.
    • Summary of statistics: jstat -gcutil vmID
    • Summary of statistics, with causes: jstat -gccause vmID
    • Behavior of the gc heap: jstat -gc vmID
    • Capacities of all the generations: jstat -gccapacity vmID
    • Behavior of the new generation: jstat -gcnew vmID
    • Capacity of the new generation: jstat -gcnewcapacity vmID
    • Behavior of the old and permanent generations: jstat -gcold vmID
    • Capacity of the old generation: jstat -gcoldcapacity vmID
    • Capacity of the permanent generation: jstat -gcpermcapacity vmID
  • Monitor objects awaiting finalization
    • JConsole tool, Summary tab
    • jmap -finalizerinfo pid
    • getObjectPendingFinalizationCount method in java.lang.management.MemoryMXBean class
  • Monitor memory
    • Heap allocation profiles via HPROF: java -agentlib:hprof=heap=sites
    • JConsole tool, Memory tab
    • Control-\ prints generation information.
  • Monitor CPU usage.
    • By thread stack: java -agentlib:hprof=cpu=samples application
    • By method: java -agentlib:hprof=cpu=times application
    • JConsole tool, Overview and Summary tabs
  • Monitor thread activity: JConsole tool, Threads tab
  • Monitor class activity: JConsole tool, Classes tab

Back to top

Actions on a Remote Debug Server

First, attach the debug daemon jsadebugd, then execute the command.

  • Dump Java heap in binary format to a file: jmap -dump:format=b,file=filename hostID
  • Print shared object mappings: jmap hostID
  • Print heap summary : jmap -heap hostID
  • Print finalization information : jmap -finalizerinfo hostID
  • Print lock information : jstack -l hostID
  • Print thread trace : jstack hostID
  • Print Java configuration information: jinfo hostID

Back to top

Other Functions

  • Interface with the instrumented Java virtual machines.
    • Monitor for the creation and termination of instrumented VMs: jstatd daemon
    • List the instrumented VMs: jps
    • Provide interface between remote mointoring tools and local VMs: jstatd daemon
  • Print Java configuration information from a running process.
    • jinfo pid
  • Dynamically set, unset, or change the value of certain Java VM flags for a process.
    • jinfo -flag flag
  • Pass a Java VM flag to the virtual machine.
    • jhat -Jflag ...
    • jmap -Jflag ...
  • Print statistics of permanent generation of Java heap, by class loader.
    • jmap -permstat
  • Report on monitor contention.
    • java -agentlib:hprof=monitor=y application
  • Evaluate or execute a script in interactive or batch mode.
    • jrunscript
  • Interface dynamically with an MBean, via JConsole tool, MBean tab:
    • Show tree structure.
    • Set an attribute value.
    • Invoke an operation.
    • Subscribe to notification.

Find Swap usage on Solaris

In Solaris, swap and physical memory can be interchanged, that is physical memory can be
reserved for swap instead of using the swap device.

1. swap -lh
2. swap -sh
3. prtconf | grep size
4. ps -A -o vsz,rss | awk 'BEGIN {size = 0; rss = 0;} {size += $1; rss
+= $2} END{printf("Size = %d kb, RSS = %d kb\n", size, rss);}'

The first shows how much of the physical swap device has been used. The
second looks at how much swap is used and reserved, versus available.
The third is just a sanity check for the physical memory. The fourth
looks at all of the processes in the system and allows us to see the
total difference between the vsize and the rss.


Now to figure out per process usage of swap space, we can do the following


1) bash-3.00$ top -b -o size
load averages: 2.99, 2.94, 3.02; up 199+21:07:21 12:06:32
103 processes: 99 sleeping, 4 on cpu
CPU states: 88.7% idle, 9.0% user, 2.3% kernel, 0.0% iowait, 0.0% swap
Memory: 64G phys mem, 4834M free mem, 16G total swap, 16G free swap

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
12510 uxbea 57 33 4 2313M 963M sleep 199:53 0.08% java
10790 uxbea 208 59 4 2309M 1258M sleep 33.8H 0.01% java
17942 uxbea 153 22 0 2273M 1098M sleep 27:32 0.04% java
26082 uxbea 130 50 0 1981M 729M sleep 426:29 0.07% java
23365 uxbea 130 59 0 1976M 723M cpu/4 332:06 0.13% java
24662 uxbea 129 56 0 1963M 751M sleep 186:11 0.03% java
9353 uxbea 150 24 4 1778M 1365M sleep 182:53 1.81% java
18747 uxbea 130 1 0 1486M 610M sleep 84.2H 0.03% java
19334 uxbea 130 1 0 1441M 1063M cpu/20 33:17 3.18% java
25622 uxbea 125 53 0 1431M 583M sleep 183:50 0.04% java
7565 uxbea 97 54 0 1274M 673M sleep 38:27 0.02% java
24810 uxbea 285 53 0 1226M 1091M sleep 27.6H 0.09% java
17895 uxbea 230 1 4 1098M 820M sleep 25.0H 0.06% java
10901 uxbea 164 55 4 1066M 693M sleep 21:10 0.03% java
12056 uxbea 227 43 4 1061M 606M sleep 24.4H 0.08% java


looks like pid 12510 is using most virtual memory.. to figure out how much of i
is swap, use pmap's help

2) bash-3.00$ pmap -S 12510 | tail -1
total Kb 2368032 2022672

PID 12510 is using 2.02G of swap.






Friday, November 6, 2009

Built-in Variables for Shell Scripts

When doing scripting in shell, like bash, there are a few built-in variables that we can use to optimize our script. Below are a few useful ones:

  • $$ = The PID number of the process executing the shell.
  • $? = Exit status variable.
  • $0 = The name of the command you used to call a program.
  • $1 = The first argument on the command line.
  • $2 = The second argument on the command line.
  • $n = The nth argument on the command line. n = 0-9
  • $* = All the arguments on the command line.
  • $# = The number of command line arguments.

Tuesday, November 3, 2009

Solaris System Level Debugging using truss

What is truss?

From the Solaris man pages, "the truss utility executes the specified command and produces a trace of the system calls it performs, the signals it receives, and the machine faults it incurs. Each line of the trace output reports either the fault or signal name or the system call name with its arguments and return value(s)".

We will use truss with the following switches

  • -o produces an output file
  • -f follows all children created by fork() and vfork() and includes their signals,faults and system calls in the trace output. Normally, only the first level command or process is traced. When -f is specified, the process-id is included with each line of the trace output to indicate which process executed the system call or received the signal
  • -p indicates the pid which is being traced

To trace a PID issue the following command
bash#truss -vall -fall -p $pid -o /tmp/mailcheck.out

To trace a command issue the following command
bash#truss -vall -fall -o /tmp/commandtrace.out telnet


In the truss output the code ENOENT is shown very frequently; its meaning can be found in the system include file errno.h

bash#grep ENOENT /usr/include/sys/*
/usr/include/sys/errno.h:#define ENOENT 2 /* No such file or directory */

We therefore know that in the output of truss the ENOENT means "no such file or directory"

Sunday, November 1, 2009

Steps for Creating Vertical Websphere portal cluster

IBM recommends that you should user vertical clustering i.e. having more than one portal JVM's running on single machine to make full use of the resources. Also it looks like setting up vertical cluster is much easier than the horizontal cluster.

I did setup a vertical cluster on my VMWare by following these steps

Start the Deployment manager if it is not already started
Login into the WAS Admin Console of the deployment manager by going to https://localhost:9043/ibm/console URL.
In the WAS Admin Console Go to Server -> Clusters. And select the cluster in which you want to add new member


Click on the Cluster members, it will list out the members of that cluster. Click on New. It will open a Create additional cluster member page like this


On this page enter name of the new cluster and select the node where this vertical node should be added. Dont forget to check the Generate Unique HTTP ports check box. Click on next couple of pages and when you say finish it will create set of configuration files for the new server
Important Note :You must update the virtual host entries for the new port created when adding a cluster member. You can do this by updating the default_host virtual host in the administrative console and adding a new alias entry for the port number (use an asterisk [*] wildcard character for the host name).

Once the new vertical cluster member is created you can verify it by going to Cluster topology


Next step would be to configure dynamic cache so that the cache entries created on this newly created server get copied to other members of the cluster. You can do that by going to Application Server -> Newly created server -> Dynamic Cache service.
On this page check Enable cache replication check box, select name of your cluster as Full Group replication domain and set Replication Type as pull only


Next go to the wp_profile/ConfigEngine.sh directory on the target machine and execute ./ConfigEngine.sh cluster-node-config-vertical-cluster-setup -DServerName=WebSphere_Portal_V1 task to clean up the server-scoped resources, caches, and resource providers
Restart the newly added vertical cluster member
In the Admin Console change the value of WCM_HOST and WCM_PORT websphere variables scoped at the server level to point to the web server which will be used to serve WCM content
Re Synchronize changes from Deployment manager to the cluster members
Regenerate the Web Server plugin to include the newly added vertical cluster and copy the newly generated plugin-cfg.xml file to web server and restart web server so that it can start forwarding request to newly added vertical cluster member

No Downtime Production redeployment in WebLogic

Upgrading a running application in a J2EE production environment isn't easy. You either have to undeploy the old version of the application and deploy the new one—causing a temporary outage—or you may have to set up a redundant server/cluster to route the new requests.

BEA WebLogic Server 9.0 supports a production redeployment feature that provides a way to seamlessly upgrade an application in a production environment without affecting application availability. After redeploying a new version of the application, all new client connection requests go to the new application. The existing client connections continue to use the old application that will be undeployed/retired after all the existing connections are closed. The two application versions are completely isolated from each other and do not share any resources. Alternatively, the old version of the application can be retired by specifying a retire timeout for the application.

This article uses a sample application to demonstrate this functionality.

Requirements

Currently, WebLogic Server 9.0 supports this production redeployment feature only for Web application (WAR) modules and enterprise applications (EARs). All other types of archives (EJB JAR, JCA RAR, WebServices archives, JMS, or JDBC standalone modules) are not supported. EARs can contain all supported module types, except WebServices archives. Production redeployment only supports HTTP clients; Java clients are not supported. Attempting to perform production redeployment with an unsupported archive type will result in an error. To redeploy such modules, remove their version identifiers and explicitly redeploy the modules.

In addition, only versioned applications can be redeployed using this feature. A versioned application is an application that has an application archive version specified in the manifest of the application archive.

A deployed application must specify a version number before you can perform subsequent production redeployment operations on the application. In other words, you cannot deploy a non-versioned application and later perform production redeployment with a new version of the application.

WebLogic Server 9.0 can host a maximum of two different versions of an application at any one time. Also, when you redeploy a new version of an application, you cannot change the application's deployment targets, security model, or persistent store settings. To change any of the above features, you must first undeploy the active version of the application.

Application Version Information

The application version information can be specified in the MANIFEST.MF file's WebLogic-Application-Version property. The manifest is a special file that can contain information about the files packaged in a JAR file. By tailoring this "meta" information that the manifest contains, you enable the JAR file to serve a variety of purposes.

For example, an application archive whose application archive version is "v1" could have the following manifest content:

Manifest-Version: 1.0



Created-By: 1.4.1_05-b01 (Sun Microsystems Inc.)



WebLogic-Application-Version: v1

The application archive version is a string that can only contain the following characters: alphanumeric ("A"-"Z," "a"-"z," "0"-"9"), period ("."), underscore ("_"), and hyphen ("-"). The length of the application archive version should be less than 215 characters. Additionally, the application archive version string cannot be "." or "..". You can either specify a version for an application using the manifest file, or assign one when using the Deployer tool's -appversion option. The value specified in MANIFEST.MF file will take precedence over the -appversion value.

The version number is important when considering the redeployment of versionable applications. If a new application archive version is specified, WebLogic Server will perform production redeployment with version isolation. If the same application archive version is specified, WebLogic Server will perform in-place redeployment.

The Sample Versioned Application

The attached sample application (VersionedApp1) contains a Web application, which has three JSP files:

  • The versionedjsp.jsp file contains a simple print statement.
  • The invalidatesession.jsp contains a session.invalidate() command to invalidate all the sessions.
  • The timeoutsession.jsp file contains code to set the timeout value for the session.

The other application, VersionedApp2, contains the same set of files, but the versionedjsp.jsp file will print a different message.

We'll use these applications to demonstrate the versioning process.

Deploying the Application

The sample applications provided along with this article do not have version information in their manifest files. If you want to use production redeployment with an application that does not include a version string in the manifest file, the Deployer tool allows you to manually specify a unique version string using the -appversion option when deploying or redeploying an application. Run this command to deploy the application with a version of version1:

java weblogic.Deployer -adminurl http://localhost:8802

-username weblogic -password weblogic -name VersionedApp

-targets adminServer

-deploy -source C:/tmp/VersionedApp1 -appversion version1

Deployer is a Java-based deployment tool that provides a command-line interface to the WebLogic Server deployment API. Deployer is intended for administrators and developers who want to perform interactive, command line-based deployment operations.

Note that the version string specified with -appversion is applied only when the deployment source files do not specify a version string in MANIFEST.MF. For applications with version information in the manifest files, you need not provide the -appversion option.

You can also display version information for deployed applications from the command line using the Deployer -listapps command. So for example, after deploying the above application you can run this command to list the application:

java weblogic.Deployer -adminurl http://localhost:8802 

-user weblogic -password weblogic -listapps

Redeploying a New Version of the Application

Now that we've deployed the application, let's look at redeploying it. Since our deployment files do not contain version information in the manifest files, we perform redeploy with the -appversion option as mentioned above:

java weblogic.Deployer -adminurl http://localhost:8802

-username weblogic -password weblogic -name VersionedApp

-targets adminServer -redeploy -source

C:/tmp/VersionedApp2 -appversion version2

If you want to specify a fixed time period after which the older version of the application is undeployed (regardless of whether clients finish their work), use the -retiretimeout option with the -redeploy command. (-retiretimeout specifies the number of seconds after which the older version of the application is retired):

java weblogic.Deployer -adminurl http://localhost:8802

-username weblogic -password weblogic -name VersionedApp

-targets adminServer -redeploy -source

C:/tmp/VersionedApp2 -appversion version2 -retiretimeout 300

If WebLogic Server has not yet retired an application version, you can immediately undeploy the application version without waiting for retirement to complete. This may be necessary if, for example, an application remains in the retiring state with only one or two long-running client sessions that you do not want to preserve. To force the undeployment of a retiring version of an application, use the -undeploy command and specify the application version:

java weblogic.Deployer -adminurl http://localhost:8802

-username weblogic -password weblogic -name VersionedApp

-targets adminServer -undeploy -name VersionedApp

-appversion version1

If you do not explicitly specify an application version with the -appversion option, WebLogic Server undeploys the active version and all retired versions of the application.

Verifying the Deployment

After deploying the first version of the application, open a browser and invoke the versionedjsp.jsp:

http://localhost:8802/VersionedApp/versionedjsp.jsp

That should establish an HTTP session to the VersionedApp1 application. In the browser window we should see the "Output from VersionedApp1 JSP" message. After deploying the VersionedApp2 application, open another browser window and invoke the versionedjsp.jsp. Now we should see the "Output from VersionedApp2 JSP" message from the VersionedApp2 application. At this time, both the versions of application should be alive.

Now invoke the invalidatesession.jsp from the first browser window:

http://localhost:8802/VersionedApp/invalidatesession.jsp

This will invalidate all the established sessions to the VersionedApp1 application. Take a look at the server console window. The retirement process should have started now. Wait a few moments for the retirement process to complete and invoke versionedjsp.jsp from the first browser window:

http://localhost:8802/VersionedApp/versionedjsp.jsp

This time you should see the "Output from VersionedApp2 JSP" message from the VersionedApp2 application.

Rolling Back the Production Redeployment Process

Reversing the production redeployment process switches the state of the active and retiring applications and redirects new client connection requests accordingly. Reverting the production redeployment process may be necessary if you detect a problem with a newly deployed version of an application, and you want to stop clients from accessing it.

To roll back the production redeployment process, issue a second -redeploy command and specify the deployment source files for the older version:

java weblogic.Deployer -adminurl http://localhost:8802

-user weblogic -password weblogic -redeploy

-name VersionedApp C:/tmp/VersionedApp1

-retiretimeout 300

Conclusion

Production redeployment is a very powerful functionality. With this functionality, customers get the ability to roll out application upgrades in a production environment transparently, without disruption to clients. Production redeployment not only requires fewer hardware resources but also provides more flexibility and control of application availability. Administrators should definitely consider using this functionality in production environments, which will not only make their tasks easier, but also provide minimal disruption to the end user.