Showing posts with label s1as. Show all posts
Showing posts with label s1as. Show all posts

Saturday, March 8, 2008

More on How to make Acegi work on Sun Application Server 8.x?

Applying the solution mentioned in my previous post IMHO: How to make Acegi work on Sun Application Server 8.x? has been proved working quite well. Well, until I encountered the same ClassCastException again. This time it was because the access was denied. Looking into the stack trace, I found that it was because Acegi's AccessDeniedHandlerImpl forwards the SecurityContextHolderAwareRequestWrapper to show the error page.

Similar to the solution in the previous post, I creates a NullPrincipalAccessDeniedHandlerImpl, which extends Acegi's AccessDeniedHandlerImpl but wrap the HttpServletRequest with the NullPrincipalHttpServletRequestWrapper.

Following is the source code of NullPrincipalAccessDeniedHandlerImpl:

package au.net.ozgwei.util.spring.security;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;

import org.acegisecurity.AccessDeniedException;
import org.acegisecurity.ui.AccessDeniedHandlerImpl;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import au.net.ozgwei.util.httpservlet.NullPrincipalHttpServletRequestWrapper;

/**
* An Acegi AccessDeniedHandler implementation designed specific to get around
* the bug in Sun Application Server 8.x where a custom security framework's
* implementation of Principal is casted to Sun Application Server's own
* implementation.
*
* @author Alex
* @version 1.0
*/
public class NullPrincipalAccessDeniedHandlerImpl extends
AccessDeniedHandlerImpl {

@SuppressWarnings("unused")
private static final Log log =
LogFactory.getTrace(
NullPrincipalAccessDeniedHandlerImpl.class);

/**
* Default no-arg constructor.
*/
public NullPrincipalAccessDeniedHandlerImpl() {
super();
}

@Override
public void handle(ServletRequest aRequest, ServletResponse aResponse,
AccessDeniedException aAccessDeniedException) throws IOException,
ServletException {

super.handle(new NullPrincipalHttpServletRequestWrapper(
(HttpServletRequest)aRequest), aResponse, aAccessDeniedException);
}

}

Of course, the Spring application context must be changed to replace the original AccessDeniedHandler implementation with this in the definition of the exceptionTranslationFilter bean:
<bean id="exceptionTranslationFilter"
class="org.acegisecurity.ui.ExceptionTranslationFilter">
<property name="authenticationEntryPoint">
<ref local="authenticationProcessingFilterEntryPoint"/>
</property>

<property name="accessDeniedHandler">
<bean class="au.com.cardlink.common.util.spring.security.NullPrincipalAccessDeniedHandlerImpl">
<property name="errorPage" value="/faces/ForbiddenAccess.jsp"/>
</bean>
</property>

</bean>

Now it works even if user's access to a protected URL is denied by Acegi.

Monday, January 21, 2008

How to make Acegi work on Sun Application Server 8.x?

We are currently developing an application that employs Acegi and runs on Sun Application Server 8 (Sun AS 8.x).

Being a mature and widely adopted security solution, we did not have many problems until we encountered the following puzzling exception:
[#|2008-01-18T16:58:28.984+1100|SEVERE|sun-appserver-pe8.2|javax.enterprise.system.container.web|_ThreadID=22;|ApplicationDispatcher[/express_portal] Servlet.service() for servlet jsp threw exception
java.lang.ClassCastException: org.acegisecurity.providers.UsernamePasswordAuthenticationToken
at com.sun.web.server.J2EEInstanceListener.handleBeforeEvent(J2EEInstanceListener.java:130)
at com.sun.web.server.J2EEInstanceListener.instanceEvent(J2EEInstanceListener.java:68)
at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:300)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:482)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:417)
at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:80)
at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:95)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313)
at org.acegisecurity.ui.AccessDeniedHandlerImpl.handle(AccessDeniedHandlerImpl.java:65)
at org.acegisecurity.ui.ExceptionTranslationFilter.handleException(ExceptionTranslationFilter.java:166)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:106)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:161)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:263)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:132)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:933)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:189)
at com.sun.enterprise.web.connector.grizzly.ProcessorTask.doProcess(ProcessorTask.java:604)
at com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:475)
at com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:371)
at com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:264)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:281)
at com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:83)
|#]


We traced the application server and found that it was likely caused by the Sun AS 8.x trying to cast the Acegi-implemented Principal to an internal Sun AS implementation.

We did some googling and found that Andrey Grebnev blogged about this two years ago, and he suggested a workaround by overriding the getUserPrincipal() method (of SecurityContextHolderAwareRequestWrapper) by always returning a null.

Because SecurityContextHolderAwareRequestWrapper and its subclasses are used internally by Acegi, his workaround implied changing the source code of Acegi, which we are reluctant to do.

Realizing that the it was only unsafe for the HttpServletRequest to return the Acegi-implemented Principal when the servlet filter chain has been executed and the control is handed over to the AS and the running application, we came up with a NullPrincipalFilter that wraps the incoming HttpServletRequest with a NullPrincipalHttpServletRequestWrapper, which returns null for getUserPrincipal(), and hands the control over to the AS. This filter should always be placed at the end of the filter proxy chain in Acegi. And of course, the application must not use HttpServletRequest's getUserPrincipal() method to retrieve the user principal, which is very easy to do, as it can invoke SecurityContextHolder.getContext().getAuthentication() to achieve the same goal, without coupling to the Servlet API.

The following is the source code of NullPrincipalFilter:

package au.net.ozgwei.util.httpservlet;

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
* A filter designed specific to get around the bug in Sun Application Server
* 8.x where a custom security framework's implementation of Principal is
* casted to Sun Application Server's own implementation.
*
* @author Alex
* @version 1.0
*/
public class NullPrincipalFilter implements Filter {

@SuppressWarnings("unused")
private static final Log log = LogFactory.getLog(NullPrincipalFilter.class);

/* (non-Javadoc)
* @see javax.servlet.Filter#destroy()
*/
public void destroy() {
}

/* (non-Javadoc)
* @see javax.servlet.Filter#doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain)
*/
public void doFilter(ServletRequest aRequest, ServletResponse aResponse, FilterChain aFileterChain) throws IOException, ServletException {
aFileterChain.doFilter(
new NullPrincipalHttpServletRequestWrapper(
(HttpServletRequest) aRequest), aResponse);
}

/* (non-Javadoc)
* @see javax.servlet.Filter#init(javax.servlet.FilterConfig)
*/
public void init(FilterConfig aArg0) throws ServletException {
}

}


And the source code of NullPrincipalHttpServletRequestWrapper:

package au.net.ozgwei.util.httpservlet;

import java.security.Principal;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
* A HttpServletRequestWrapper that always return null for Principal.
*
* @author Alex
* @version 1.0
*/
public class NullPrincipalHttpServletRequestWrapper extends HttpServletRequestWrapper {

@SuppressWarnings("unused")
private static final Log log = LogFactory.getLog(
NullPrincipalHttpServletRequestWrapper.class);

public NullPrincipalHttpServletRequestWrapper(HttpServletRequest aReq) {
super(aReq);
}

@Override
public Principal getUserPrincipal() {
return null;
}

}


And a nullPrincipalFilter bean should be defined in the Spring application context and added to the end of the filterChainProxy, as following:
<bean class="org.acegisecurity.util.FilterChainProxy" id="filterChainProxy">
<property name="filterInvocationDefinitionSource">
<value>
CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON
PATTERN_TYPE_APACHE_ANT
/**=concurrentSessionFilter,httpSessionContextIntegrationFilter,logoutFilter,authenticationProcessingFilter,securityContextHolderAwareRequestFilter,anonymousProcessingFilter,exceptionTranslationFilter,filterInvocationInterceptor,nullPrincipalFilter
</value>
</property>
</bean>

<bean class="au.net.ozgwei.util.httpservlet.NullPrincipalFilter" id="nullPrincipalFilter">


Finally, this bug has been fixed in GlassFish eventually.

Thursday, October 25, 2007

Cryptic JTS5031 and JTS5068 errors on Sun Application Server 8.1 and 8.2

We encountered the following exceptions when testing our EJB 2.1 + Spring + Hibernate + Osworkflow (also using Hibernate) application.

When running on Sun Application Server 8.1 EE, the stack trace is listed below:
[#2007-10-25T10:50:34.347+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--Created new J2EETransaction, txId = 25#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;
In J2EETransactionManagerOpt.enlistResource, h=5 h.xares=com.sun.gjc.spi.XAResourceImpl@c21e52 h.alloc=com.sun.enterprise.resource.ConnectorAllocator@54d24d tx=J2EETransaction: txId=25 nonXAResource=null jtsTx=null localTxStatus=0 syncs=[]#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: begin#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Control object :com.sun.jts.CosTransactions.ControlImpl@162aeda corresponding to this transaction has been createdGTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistResource#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.enlistResource, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.404+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.405+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: delistResource#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13; ejbDestroyed: AccountProcessServiceBean; id: [B@7219a#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: rollback#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.409+1000SEVEREsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation.#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;context with empty container in ContainerSynchronization.afterCompletion#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.411+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5092:Exception occurred in postInvokeTx : [{0}]
javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation. vmcid: 0x0 minor code: 0 completed: No
at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:295)
at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1054)
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
at $Proxy22.processUser(Unknown Source)
at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)
...
[#2007-10-25T10:50:34.414+1000INFOsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5018: An exception was thrown during an ejb invocation on [UserProcessServiceBean]#]


When running on Sun Application Server 8.2 PE, the stack trace is listed below:
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO  org.hibernate.impl.SessionFactoryObjectFactory  - Not binding factory to JNDI, no JNDI name configured
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO org.hibernate.util.NamingHelper - JNDI InitialContext properties:{}
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO org.springframework.context.support.ClassPathXmlApplicationContext - Bean 'siteManagerSessionFactory' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO org.springframework.context.support.ClassPathXmlApplicationContext - Bean 'org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO org.springframework.beans.factory.support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1956ba5: defining beans [siteManager,siteRepository,siteAssembler,org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor,dataSource,transactionManager,siteManagerSessionFactory,eventLogService]; root of factory hierarchy
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA UserTransaction: com.sun.enterprise.distributedtx.UserTransactionImpl@ad9064
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA TransactionManager: com.sun.ejb.containers.PMTransactionManagerImpl@1eafdce
#]
[#2007-10-25T17:52:26.027+1000WARNINGsun-appserver-pe8.2javax.enterprise.system.core.transaction_ThreadID=15;JTS5068: Unexpected error occurred in rollback
java.lang.NullPointerException
at com.sun.gjc.spi.ManagedConnection.transactionCompleted(ManagedConnection.java:429)
at com.sun.gjc.spi.XAResourceImpl.rollback(XAResourceImpl.java:140)
at com.sun.jts.jta.TransactionState.rollback(TransactionState.java:168)
at com.sun.jts.jtsxa.OTSResourceImpl.rollback(OTSResourceImpl.java:271)
at com.sun.jts.CosTransactions.RegisteredResources.distributeRollback(RegisteredResources.java:971)
at com.sun.jts.CosTransactions.TopCoordinator.rollback(TopCoordinator.java:2240)
at com.sun.jts.CosTransactions.CoordinatorTerm.rollback(CoordinatorTerm.java:504)
at com.sun.jts.CosTransactions.TerminatorImpl.rollback(TerminatorImpl.java:266)
at com.sun.jts.CosTransactions.CurrentImpl.rollback(CurrentImpl.java:728)
at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:308)
at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1058)
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
at $Proxy22.processUser(Unknown Source)
at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)

These exceptions were really frustrating as they appeared to occur only at transaction commit time that somehow the commit failed and the EJB container was trying to roll back and then encountered an unexpected CORBA or null pointer error.

That led us to think that our configuration for Spring and Hibernate to work with EJB 2.1 was not set up properly.

Well, all that was just red herring. With debugging turned on, it was clear to us that a minor and seemingly innocent change in the web tier resulted in invoking the EJB with illegal arguments and that the POJO implementation wrapped by the EJB threw an IllegalArgumentException, which was swallowed by Sun's EJB container and triggered the transaction to be rolled back.

Once we fixed the bug at the web tier, the issue went away immediately.

So, the problem is with Sun's EJB container: when it catches a runtime exception, it should have printed the stack trace of the root cause of the exception instead of swallowing it. It wasted us several hours to figure out what went wrong.

It also taught us a few lessons:

  1. Debug early on may save you many hours of code reading and googling for a problem that was hidden/eclipse by a seemingly complex problem.
  2. Write more test cases for web tier codes, preferably with EasyMock 2 or jMock 2. If we had written the test cases for the UI, this problem would probably occur in the first place.

So, if you see similar JTS5031 or JTS5068 errors on Sun Application Server 8.x, make sure you do some debugging to verify that it was not caused by a runtime exception thrown by your application code...