More on How to make Acegi work on Sun Application Server 8.x?

Applying the solution mentioned in my previous post IMHO: How to make Acegi work on Sun Application Server 8.x? has been proved working quite well. Well, until I encountered the same ClassCastException again. This time it was because the access was denied. Looking into the stack trace, I found that it was because Acegi's AccessDeniedHandlerImpl forwards the SecurityContextHolderAwareRequestWrapper to show the error page.

Similar to the solution in the previous post, I creates a NullPrincipalAccessDeniedHandlerImpl, which extends Acegi's AccessDeniedHandlerImpl but wrap the HttpServletRequest with the NullPrincipalHttpServletRequestWrapper.

Following is the source code of NullPrincipalAccessDeniedHandlerImpl:

package au.net.ozgwei.util.spring.security;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;

import org.acegisecurity.AccessDeniedException;
import org.acegisecurity.ui.AccessDeniedHandlerImpl;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import au.net.ozgwei.util.httpservlet.NullPrincipalHttpServletRequestWrapper;

/**
 * An Acegi AccessDeniedHandler implementation designed specific to get around
 * the bug in Sun Application Server 8.x where a custom security framework's
 * implementation of Principal is casted to Sun Application Server's own
 * implementation.
 *
 * @author Alex
 * @version 1.0
 */
public class NullPrincipalAccessDeniedHandlerImpl extends
   AccessDeniedHandlerImpl {

   @SuppressWarnings("unused")
   private static final Log log =
       LogFactory.getTrace(
       NullPrincipalAccessDeniedHandlerImpl.class);

   /**
    * Default no-arg constructor.
    */
   public NullPrincipalAccessDeniedHandlerImpl() {
       super();
   }

   @Override
   public void handle(ServletRequest aRequest, ServletResponse aResponse,
       AccessDeniedException aAccessDeniedException) throws IOException,
       ServletException {

       super.handle(new NullPrincipalHttpServletRequestWrapper(
           (HttpServletRequest)aRequest), aResponse, aAccessDeniedException);
   }

}

Of course, the Spring application context must be changed to replace the original AccessDeniedHandler implementation with this in the definition of the exceptionTranslationFilter bean:

<bean id="exceptionTranslationFilter"
   class="org.acegisecurity.ui.ExceptionTranslationFilter">
   <property name="authenticationEntryPoint">
       <ref local="authenticationProcessingFilterEntryPoint"/>
   </property>
   
   <property name="accessDeniedHandler">
       <bean  class="au.com.cardlink.common.util.spring.security.NullPrincipalAccessDeniedHandlerImpl">
           <property name="errorPage" value="/faces/ForbiddenAccess.jsp"/>
       </bean>
   </property>

</bean>

Now it works even if user's access to a protected URL is denied by Acegi.

How to make Acegi work on Sun Application Server 8.x?

We are currently developing an application that employs Acegi and runs on Sun Application Server 8 (Sun AS 8.x).

Being a mature and widely adopted security solution, we did not have many problems until we encountered the following puzzling exception:

[#|2008-01-18T16:58:28.984+1100|SEVERE|sun-appserver-pe8.2|javax.enterprise.system.container.web|_ThreadID=22;|ApplicationDispatcher[/express_portal] Servlet.service() for servlet jsp threw exception
java.lang.ClassCastException: org.acegisecurity.providers.UsernamePasswordAuthenticationToken
at com.sun.web.server.J2EEInstanceListener.handleBeforeEvent(J2EEInstanceListener.java:130)
at com.sun.web.server.J2EEInstanceListener.instanceEvent(J2EEInstanceListener.java:68)
at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:300)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:482)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:417)
at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:80)
at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:95)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313)
at org.acegisecurity.ui.AccessDeniedHandlerImpl.handle(AccessDeniedHandlerImpl.java:65)
at org.acegisecurity.ui.ExceptionTranslationFilter.handleException(ExceptionTranslationFilter.java:166)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:106)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:161)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:263)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:132)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:933)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:189)
at com.sun.enterprise.web.connector.grizzly.ProcessorTask.doProcess(ProcessorTask.java:604)
at com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:475)
at com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:371)
at com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:264)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:281)
at com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:83)
|#]

We traced the application server and found that it was likely caused by the Sun AS 8.x trying to cast the Acegi-implemented Principal to an internal Sun AS implementation.

We did some googling and found that Andrey Grebnev blogged about this two years ago, and he suggested a workaround by overriding the getUserPrincipal() method (of SecurityContextHolderAwareRequestWrapper) by always returning a null.

Because SecurityContextHolderAwareRequestWrapper and its subclasses are used internally by Acegi, his workaround implied changing the source code of Acegi, which we are reluctant to do.

Realizing that the it was only unsafe for the HttpServletRequest to return the Acegi-implemented Principal when the servlet filter chain has been executed and the control is handed over to the AS and the running application, we came up with a NullPrincipalFilter that wraps the incoming HttpServletRequest with a NullPrincipalHttpServletRequestWrapper, which returns null for getUserPrincipal(), and hands the control over to the AS. This filter should always be placed at the end of the filter proxy chain in Acegi. And of course, the application must not use HttpServletRequest's getUserPrincipal() method to retrieve the user principal, which is very easy to do, as it can invoke SecurityContextHolder.getContext().getAuthentication() to achieve the same goal, without coupling to the Servlet API.

The following is the source code of NullPrincipalFilter:

package au.net.ozgwei.util.httpservlet;

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
 * A filter designed specific to get around the bug in Sun Application Server
 * 8.x where a custom security framework's implementation of Principal is
 * casted to Sun Application Server's own implementation.
 *
 * @author Alex
 * @version 1.0
 */
public class NullPrincipalFilter implements Filter {

 @SuppressWarnings("unused")
 private static final Log log = LogFactory.getLog(NullPrincipalFilter.class);

 /* (non-Javadoc)
  * @see javax.servlet.Filter#destroy()
  */
 public void destroy() {
 }

 /* (non-Javadoc)
  * @see javax.servlet.Filter#doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain)
  */
 public void doFilter(ServletRequest aRequest, ServletResponse aResponse, FilterChain aFileterChain) throws IOException, ServletException {
     aFileterChain.doFilter(
         new NullPrincipalHttpServletRequestWrapper(
             (HttpServletRequest) aRequest), aResponse);
 }

 /* (non-Javadoc)
  * @see javax.servlet.Filter#init(javax.servlet.FilterConfig)
  */
 public void init(FilterConfig aArg0) throws ServletException {
 }

}

And the source code of NullPrincipalHttpServletRequestWrapper:

package au.net.ozgwei.util.httpservlet;

import java.security.Principal;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
 * A HttpServletRequestWrapper that always return null for Principal.
 *
 * @author Alex
 * @version 1.0
 */
public class NullPrincipalHttpServletRequestWrapper extends HttpServletRequestWrapper {

 @SuppressWarnings("unused")
 private static final Log log = LogFactory.getLog(
        NullPrincipalHttpServletRequestWrapper.class);

 public NullPrincipalHttpServletRequestWrapper(HttpServletRequest aReq) {
     super(aReq);
 }

 @Override
 public Principal getUserPrincipal() {
     return null;
 }

}

And a nullPrincipalFilter bean should be defined in the Spring application context and added to the end of the filterChainProxy, as following:

<bean class="org.acegisecurity.util.FilterChainProxy" id="filterChainProxy">
 <property name="filterInvocationDefinitionSource">
     <value>
         CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON
         PATTERN_TYPE_APACHE_ANT
         /**=concurrentSessionFilter,httpSessionContextIntegrationFilter,logoutFilter,authenticationProcessingFilter,securityContextHolderAwareRequestFilter,anonymousProcessingFilter,exceptionTranslationFilter,filterInvocationInterceptor,nullPrincipalFilter
     </value>
 </property>
</bean>

<bean class="au.net.ozgwei.util.httpservlet.NullPrincipalFilter" id="nullPrincipalFilter">

Finally, this bug has been fixed in GlassFish eventually.

classpath*:BeanRefFactory.xml not found when instantiating a singleton Spring application context inside EJB2.x running on Sun application server 8.2

For the project I'm currently working on, we use Spring's EJB support. In order to share application context among all EJB instances, SingletonBeanFactoryLocator is used to locate or load the shared application context. And we used the default "classpath*:beanRefFactory.xml" selector key.

However, when the EJB is deployed and invoked, a FatalBeanException is thrown stating

Unable to find resource for specified definition. Group resource name
[classpath*:beanRefFactory.xml], factory key [...]

Looking into the implementation of SingletonBeanFactoryLocator, we found that it delegates to PathMatchingResourcePatternResolver to load the XML file. If the resource starts with "classpath*:" prefix, it uses ClassLoader's getResources(String) method to load the resource; otherwise, a ClasspathResource is returned, which eventually uses ClassLoader's getResource(String) method to load the resource. Pay attention to the plural form of the method name.

In the base java.lang.ClassLoader, both getResource(String) and getResources(String) delegate to its parent first. If the resource is not found by the parent, it invokes findResource(String) and findResources(String) methods, both of which simply return null. The Javadoc of ClassLoader recommends that, for both finder methods:

Class loader implementations should override this method to specify where to
load resources from.

However, when tracing the execution in debug mode, we found that the EJBClassLoader used by Sun Application Server 8.2 only overrides findResource(String) method and searches for the resource in the expanded directory of the EJB Jar file; it does not override findResources(String), which remains returning null, against their own advice.

Now it's obvious, this is due to a defect in Sun Application Server 8.2. The simplest solution is to specify the selector key as "classpath:beanRefFactory.xml" instead of the default value.

Cryptic JTS5031 and JTS5068 errors on Sun Application Server 8.1 and 8.2

We encountered the following exceptions when testing our EJB 2.1 + Spring + Hibernate + Osworkflow (also using Hibernate) application.

When running on Sun Application Server 8.1 EE, the stack trace is listed below:

[#2007-10-25T10:50:34.347+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--Created new J2EETransaction, txId = 25#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;
In J2EETransactionManagerOpt.enlistResource, h=5 h.xares=com.sun.gjc.spi.XAResourceImpl@c21e52 h.alloc=com.sun.enterprise.resource.ConnectorAllocator@54d24d tx=J2EETransaction: txId=25 nonXAResource=null jtsTx=null localTxStatus=0 syncs=[]#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: begin#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Control object :com.sun.jts.CosTransactions.ControlImpl@162aeda corresponding to this transaction has been createdGTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistResource#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.enlistResource, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.404+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.405+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: delistResource#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13; ejbDestroyed: AccountProcessServiceBean; id: [B@7219a#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: rollback#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.409+1000SEVEREsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation.#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;context with empty container in ContainerSynchronization.afterCompletion#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.411+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5092:Exception occurred in postInvokeTx : [{0}]
javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation. vmcid: 0x0 minor code: 0 completed: No
        at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:295)
        at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1054)
        at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
        at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
        at $Proxy22.processUser(Unknown Source)
        at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)
...
[#2007-10-25T10:50:34.414+1000INFOsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5018: An exception was thrown during an ejb invocation on [UserProcessServiceBean]#]

When running on Sun Application Server 8.2 PE, the stack trace is listed below:

[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO  org.hibernate.impl.SessionFactoryObjectFactory  - Not binding factory to JNDI, no JNDI name configured
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO  org.hibernate.util.NamingHelper  - JNDI InitialContext properties:{}
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO  org.springframework.context.support.ClassPathXmlApplicationContext  - Bean 'siteManagerSessionFactory' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO  org.springframework.context.support.ClassPathXmlApplicationContext  - Bean 'org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory  - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1956ba5: defining beans [siteManager,siteRepository,siteAssembler,org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor,dataSource,transactionManager,siteManagerSessionFactory,eventLogService]; root of factory hierarchy
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO  org.springframework.transaction.jta.JtaTransactionManager  - Using JTA UserTransaction: com.sun.enterprise.distributedtx.UserTransactionImpl@ad9064
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO  org.springframework.transaction.jta.JtaTransactionManager  - Using JTA TransactionManager: com.sun.ejb.containers.PMTransactionManagerImpl@1eafdce
#]
[#2007-10-25T17:52:26.027+1000WARNINGsun-appserver-pe8.2javax.enterprise.system.core.transaction_ThreadID=15;JTS5068: Unexpected error occurred in rollback
java.lang.NullPointerException
 at com.sun.gjc.spi.ManagedConnection.transactionCompleted(ManagedConnection.java:429)
 at com.sun.gjc.spi.XAResourceImpl.rollback(XAResourceImpl.java:140)
 at com.sun.jts.jta.TransactionState.rollback(TransactionState.java:168)
 at com.sun.jts.jtsxa.OTSResourceImpl.rollback(OTSResourceImpl.java:271)
 at com.sun.jts.CosTransactions.RegisteredResources.distributeRollback(RegisteredResources.java:971)
 at com.sun.jts.CosTransactions.TopCoordinator.rollback(TopCoordinator.java:2240)
 at com.sun.jts.CosTransactions.CoordinatorTerm.rollback(CoordinatorTerm.java:504)
 at com.sun.jts.CosTransactions.TerminatorImpl.rollback(TerminatorImpl.java:266)
 at com.sun.jts.CosTransactions.CurrentImpl.rollback(CurrentImpl.java:728)
 at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:308)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1058)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
 at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
 at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
 at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
 at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
 at $Proxy22.processUser(Unknown Source)
 at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)

These exceptions were really frustrating as they appeared to occur only at transaction commit time that somehow the commit failed and the EJB container was trying to roll back and then encountered an unexpected CORBA or null pointer error.

That led us to think that our configuration for Spring and Hibernate to work with EJB 2.1 was not set up properly.

Well, all that was just red herring. With debugging turned on, it was clear to us that a minor and seemingly innocent change in the web tier resulted in invoking the EJB with illegal arguments and that the POJO implementation wrapped by the EJB threw an IllegalArgumentException, which was swallowed by Sun's EJB container and triggered the transaction to be rolled back.

Once we fixed the bug at the web tier, the issue went away immediately.

So, the problem is with Sun's EJB container: when it catches a runtime exception, it should have printed the stack trace of the root cause of the exception instead of swallowing it. It wasted us several hours to figure out what went wrong.

It also taught us a few lessons:

1. Debug early on may save you many hours of code reading and googling for a problem that was hidden/eclipse by a seemingly complex problem.
2. Write more test cases for web tier codes, preferably with EasyMock 2 or jMock 2. If we had written the test cases for the UI, this problem would probably occur in the first place.

So, if you see similar JTS5031 or JTS5068 errors on Sun Application Server 8.x, make sure you do some debugging to verify that it was not caused by a runtime exception thrown by your application code...