Saturday, October 27, 2007

Integrating Spring, Hibernate and EJB 2.x

How to configure Hibernate Session Factory in a Spring application context used in an EJB 2.x stateless session bean?

Most of the examples found on the internet about Hibernate and Spring integration are targeted at web applications. How to configure Hibernate and Spring inside an EJB 2.x stateless session bean is rarely mentioned, and it is not as simple and straightforward as many assume.

The most important issue of the integration is around the management of JDBC connections:
In EJB 2.x, JDBC connections are managed by the data source registered with JNDI on the application server:

  • Applications are not expected to hold on JDBC connections after each use, that is, applications should aggressively release JDBC connections, preferrably after each statement.

  • Applications can acquire JDBC connections multiple times during the execution of one EJB invocation. The application server may return the same JDBC connection, or it may return different JDBC connections, but it assures that all JDBC connections returned are registered within the same Container Managed Transaction.


When database is accessed outside EJB 2.x, such as in a web application running in Tomcat, the application itself is responsible for transaction demarcation. When using a local database transaction, the application code must ensure that the same JDBC connection be used in all database access within the same transaction. Because database access occurs in multiple classes that collaborate to complete a transaction, it is very unwieldy for applications to pass around the JDBC connection in order to re-use the same JDBC connection.

Enter Spring and Hibernate.

Spring guarantees the same JDBC connection is reused, provided the same Spring-managed data source is re-used. The magic happens when a JDBC connection is retrieved for the first time from a Spring-managed data source, Spring binds the JDBC connection to the current execution thread. When another JDBC connection is requested within the same transaction from the Spring-managed data source, Spring returns the thread-bound JDBC connection. When the transaction ends, Spring invokes the commit() or rollback() method of the thread-bound JDBC connection and then unbinds it from the thread.

Hibernate 3.x employs similar technique. By default, a Hibernate session is bound to the thread and returned whenever SessionFactory.getCurrentSession() is invoked within the same transaction. The same JDBC connection is used for all JDBC operations initiated by the same Hibernate session. It is possible to specify other current session context class other than the default thread local context since Hibernate 3.1.

As can be seen from the above discussion, Spring and Hibernate by default re-use the same thread-bound JDBC connection. This setting does not go well with the aggressive connection release mode expected by EJB 2.x. Therefore, non-default settings must be configured for both Spring and Hibernate for Spring and Hibernate combination to be integrated within EJB 2.x.

Below is the recommended configuration. The reasons behind this configuration are explained following the configuration.

<jee:jndi-lookup id="dataSource"
jndi-name="jdbc/datasource" resource-ref="true"/>

<bean id="transactionManager"
class="org.springframework.transaction.jta.JtaTransactionManager"/>

<bean class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor"/>

<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<!-- default value is "false"
<property name="useTransactionAwareDataSource" value="false"/>
-->
<property name="exposeTransactionAwareSessionFactory" value="false"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.SomeDialect</prop>
<!--
WARNING! 'hibernate.connection.release_mode' must not be set to
'after_statement'. Otherwise, it will be overriden with 'after_transaction'
by Hibernate because Spring's LocalConnectionProvider does not support
aggressive connection release.
-->
<prop key="hibernate.connection.release_mode">auto</prop>
<prop key="hibernate.current_session_context_class">jta</prop>
<prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.SunONETransactionManagerLookup</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.transaction.CMTTransactionFactory</prop>
<prop key="hibernate.transaction.flush_before_completion">true</prop>
<prop key="hibernate.transaction.auto_close_session">true</prop>
</props>
</property>
<property name="mappingResources">
<list>...</list>
</property>
</bean>


The reasons behind this configuration are explained below. It is recommended to have your IDE opened and source code of Spring and Hibernate 3.1 or above ready.

Spring uses LocalSessionFactoryBean, which is a subclass of AbstractSessionFactoryBean, to configure a Hibernate session factory.

There are lots of javadoc to read for each bean property that can be set for a LocalSessionFactoryBean. It's better off to read the buildSessionFactory() method to understand what each property value does in the build time.

Note that the above configuration has two parts for LocalSessionFactoryBean, the 'normal' Spring bean properties, such as 'dataSource', and the 'hibernateProperties' that takes a java.util.Properties value. The 'normal' properties are used to configure the default settings, which can be overriden by the 'hibernateProperties'.

As a side note, we don't set the 'jtaTransactionManager' property. Note that the class for this property is javax.transaction.TransactionManager, not the JtaTransactionManager that implements Spring's PlatformTransactionManager. In order to set this property, we need to know the JNDI name that the target application server binds its JTA transaction manager. The benefit of setting this property would be that we don't need to configure Hibernate's 'hibernate.transaction.manager_lookup_class' and 'hibernate.transaction.factory_class' properties. We concluded that this benefit could not justify explicitly specifying the application server specific JNDI name for transaction manager. We would rather set those two properties in Hibernate configuration.

If 'jtaTransactionManager' property is not set, Spring automatically set the 'hibernate.connection.release_mode' property to 'on_close'. Because we are running Hibernate inside an EJB 2.x, we must set this property to other value in the overriding 'hibernateProperties'.

The 'exposeTransactionAwareSessionFactory' property has a default value of true. If set to true, Spring will set the 'hibernate.current_session_context_class' property to Spring's own thread-bound implementation, which must be overriden by a value of 'jta' in the 'hibernateProperties'.

The 'useTransactionAwareDataSource' property must be left to the default 'false' value. Otherwise, Spring will wrap the data source with a TransactionAwareDataSourceProxy, which will effectively re-use the same JDBC transaction within the same transaction, even if Hibernate aggressively releases connections.

It must be pointed out that when 'useTransactionAwareDataSource' is set to false, Spring will supply LocalDataSourceConnectionProvider as the implementation of Hibernate's ConnectionProvider. LocalDataSourceConnectionProvider informs Hibernate that it does not support aggressive release of connection. However, in Hibernate's SettingsFactory, if the ConnectionProvider does not support aggressive release of connections and connection release mode is set to 'after_statement', the connection release mode will be automatically rectified to 'after_transaction', which effectively re-uses the same JDBC transaction for the whole transaction. A warning message "Overriding release mode as connection provider does not support 'after_statement'" is logged. Therefore, the connection release mode in the 'hibernateProperties' must be set to 'auto' instead of 'after_statement'. When the transaction factory is set to 'CMTTransactionFactory', the default connection release mode is 'after_statement', which is precisely what we want.

The other property settings are self-explanatory:

  • 'hibernate.current_session_context_class' should be set to 'jta'.

  • 'hibernate.transaction.manager_lookup_class' should be set the a class mapped to the target application server.

  • 'hibernate.transaction.factory_class' should be set to 'org.hibernate.transaction.CMTTransactionFactory'.


I hope this post will help anyone who has encountered mysterious connection problems when integrating Spring, Hibernate and EJB 2.x

Thursday, October 25, 2007

Cryptic JTS5031 and JTS5068 errors on Sun Application Server 8.1 and 8.2

We encountered the following exceptions when testing our EJB 2.1 + Spring + Hibernate + Osworkflow (also using Hibernate) application.

When running on Sun Application Server 8.1 EE, the stack trace is listed below:
[#2007-10-25T10:50:34.347+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--Created new J2EETransaction, txId = 25#]
[#2007-10-25T10:50:34.400+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistComponentResources#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;
In J2EETransactionManagerOpt.enlistResource, h=5 h.xares=com.sun.gjc.spi.XAResourceImpl@c21e52 h.alloc=com.sun.enterprise.resource.ConnectorAllocator@54d24d tx=J2EETransaction: txId=25 nonXAResource=null jtsTx=null localTxStatus=0 syncs=[]#]
[#2007-10-25T10:50:34.401+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: begin#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Control object :com.sun.jts.CosTransactions.ControlImpl@162aeda corresponding to this transaction has been createdGTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: enlistResource#]
[#2007-10-25T10:50:34.402+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.enlistResource, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.404+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.405+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;--In J2EETransaction.registerSynchronization, jtsTx=com.sun.jts.jta.TransactionImpl@ffe966e9 nonXAResource=null#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: delistResource#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13; ejbDestroyed: AccountProcessServiceBean; id: [B@7219a#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.resource.jta_ThreadID=13;TM: rollback#]
[#2007-10-25T10:50:34.406+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.409+1000SEVEREsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation.#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;context with empty container in ContainerSynchronization.afterCompletion#]
[#2007-10-25T10:50:34.410+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.core.transaction_ThreadID=13;Within TopCoordinator.rollback() :GTID is : 19000000BBF79CD4616476627661707030312C5033373030#]
[#2007-10-25T10:50:34.411+1000FINEsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5092:Exception occurred in postInvokeTx : [{0}]
javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL: vmcid: 0x0 minor code: 0 completed: Maybe] on Resource [rollback] operation. vmcid: 0x0 minor code: 0 completed: No
at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:295)
at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1054)
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
at $Proxy22.processUser(Unknown Source)
at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)
...
[#2007-10-25T10:50:34.414+1000INFOsun-appserver-ee8.1_02javax.enterprise.system.container.ejb_ThreadID=13;EJB5018: An exception was thrown during an ejb invocation on [UserProcessServiceBean]#]


When running on Sun Application Server 8.2 PE, the stack trace is listed below:
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO  org.hibernate.impl.SessionFactoryObjectFactory  - Not binding factory to JNDI, no JNDI name configured
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO org.hibernate.util.NamingHelper - JNDI InitialContext properties:{}
#]
[#2007-10-25T17:52:04.079+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454609 [httpWorkerThread-2189-4] INFO org.springframework.context.support.ClassPathXmlApplicationContext - Bean 'siteManagerSessionFactory' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO org.springframework.context.support.ClassPathXmlApplicationContext - Bean 'org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
#]
[#2007-10-25T17:52:04.110+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;454640 [httpWorkerThread-2189-4] INFO org.springframework.beans.factory.support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1956ba5: defining beans [siteManager,siteRepository,siteAssembler,org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor,dataSource,transactionManager,siteManagerSessionFactory,eventLogService]; root of factory hierarchy
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA UserTransaction: com.sun.enterprise.distributedtx.UserTransactionImpl@ad9064
#]
[#2007-10-25T17:52:05.207+1000INFOsun-appserver-pe8.2javax.enterprise.system.stream.out_ThreadID=14;455737 [httpWorkerThread-2189-4] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA TransactionManager: com.sun.ejb.containers.PMTransactionManagerImpl@1eafdce
#]
[#2007-10-25T17:52:26.027+1000WARNINGsun-appserver-pe8.2javax.enterprise.system.core.transaction_ThreadID=15;JTS5068: Unexpected error occurred in rollback
java.lang.NullPointerException
at com.sun.gjc.spi.ManagedConnection.transactionCompleted(ManagedConnection.java:429)
at com.sun.gjc.spi.XAResourceImpl.rollback(XAResourceImpl.java:140)
at com.sun.jts.jta.TransactionState.rollback(TransactionState.java:168)
at com.sun.jts.jtsxa.OTSResourceImpl.rollback(OTSResourceImpl.java:271)
at com.sun.jts.CosTransactions.RegisteredResources.distributeRollback(RegisteredResources.java:971)
at com.sun.jts.CosTransactions.TopCoordinator.rollback(TopCoordinator.java:2240)
at com.sun.jts.CosTransactions.CoordinatorTerm.rollback(CoordinatorTerm.java:504)
at com.sun.jts.CosTransactions.TerminatorImpl.rollback(TerminatorImpl.java:266)
at com.sun.jts.CosTransactions.CurrentImpl.rollback(CurrentImpl.java:728)
at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:308)
at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1058)
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:391)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:2711)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:2521)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:819)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:137)
at $Proxy22.processUser(Unknown Source)
at au.net.ozgwei.services.userprocess.UserProcessServiceDelegate.processUser(UserProcessServiceDelegate.java:96)

These exceptions were really frustrating as they appeared to occur only at transaction commit time that somehow the commit failed and the EJB container was trying to roll back and then encountered an unexpected CORBA or null pointer error.

That led us to think that our configuration for Spring and Hibernate to work with EJB 2.1 was not set up properly.

Well, all that was just red herring. With debugging turned on, it was clear to us that a minor and seemingly innocent change in the web tier resulted in invoking the EJB with illegal arguments and that the POJO implementation wrapped by the EJB threw an IllegalArgumentException, which was swallowed by Sun's EJB container and triggered the transaction to be rolled back.

Once we fixed the bug at the web tier, the issue went away immediately.

So, the problem is with Sun's EJB container: when it catches a runtime exception, it should have printed the stack trace of the root cause of the exception instead of swallowing it. It wasted us several hours to figure out what went wrong.

It also taught us a few lessons:

  1. Debug early on may save you many hours of code reading and googling for a problem that was hidden/eclipse by a seemingly complex problem.
  2. Write more test cases for web tier codes, preferably with EasyMock 2 or jMock 2. If we had written the test cases for the UI, this problem would probably occur in the first place.

So, if you see similar JTS5031 or JTS5068 errors on Sun Application Server 8.x, make sure you do some debugging to verify that it was not caused by a runtime exception thrown by your application code...