Ajug hibernate-dos-donts

Page | 1© 2011 Altisource Portfolio Solutions S.A. All rights reserved. Proprietary and Confidential.© 2011 Altisource Portfolio Solutions. All rights reserved. Altisource™, Altisource Portfolio Solutions™, the Altisource Logo, the "REAL" family of trademarks and service marks, and all other marks identified herein are trademarks or service marks of Altisource

Portfolio Solutions S.A. or its subsidiaries and may be registered with the United States Patent and Trademark Office and in other countries. Proprietary and Confidential.

When Hibernate Attacks!Hibernate Best Practice and Pitfalls

Page | 2© 2011 Altisource Portfolio Solutions S.A. All rights reserved. Proprietary and Confidential.

– Me– Hibernate in two slides– Hibernate

The Good, The Bad Sanity Check My Hibernate Divorce

Agenda


– Roy Russo– Former JBoss Portal Co-Founder– LoopFuse Co-Founder– Senior Software Architect @ AltiSource

We’re Hiring… and we don’t suck.

Me


– Addresses the ‘Object-Relational Impedance Mismatch’

– Persistence classes using idiomatic Java– Transparent

No interfaces or build dependencies– Querying Facilities:

HQL, Criteria API, JPAQL, SQL– Database-Agnostic– Performance:

Caching , Lazy initialization, fetching strategies, versioning, etc…

– Automatic DDL generation

Hibernate in Two Slides

public class User implements Serializable{ private long userid = -1; private String userName; private String firstName; private String lastName; private String password; private Map props; private Set roles;…

http://java.sun.com/j2se/1.5.0/docs/api/java/io/Serializable.html

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html




http://java.sun.com/j2se/1.5.0/docs/api/java/util/Map.html

http://java.sun.com/j2se/1.5.0/docs/api/java/util/Set.html


Hibernate in 2 Slides

public Task getUserByID(long ID, Session session) throws DataSourceException { Transaction tx = session.beginTransaction(); User user = null; try { Query query = session.createQuery("from User where userID=? "); query.setParameter(0, ID); user = (User) query.uniqueResult(); tx.commit(); } catch (RuntimeException e) { … } return user; }

public class User implements Serializable{ private long userid; private String userName; private String firstName; private String lastName; private String password; private String email; private Map props; private Set roles;…

<hibernate-mapping> <class name="com.loopfuse.domain.auth.User" table="LF_USERS"> <id name="userid" column="USER_ID" type="java.lang.Long"> <generator class="native"/> </id> <property name="userName" column="USERNAME" type="java.lang.String" unique="true"/>… <set name="roles" table="LF_ROLE_MEMBERS" lazy="false" inverse="false" cascade="none" sort="unsorted"> <key column="USER_ID"/> <many-to-many class="com.loopfuse.domain.auth.Role" column="ROLE_ID" outer-join="true"/> </set> </class>


http://java.sun.com/j2se/1.5.0/docs/api/javax/management/Query.html

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/RuntimeException.html

http://java.sun.com/j2se/1.5.0/docs/api/java/io/Serializable.html






http://java.sun.com/j2se/1.5.0/docs/api/java/util/Map.html

http://java.sun.com/j2se/1.5.0/docs/api/java/util/Set.html


First… The Basics

“Some developers come to using a tool like Hibernate because they are uncomfortable with SQL and with relational databases. We would say that this is exactly the wrong reason to use Hibernate. You should be very comfortable with SQL and JDBC before you start using Hibernate - Hibernate builds on JDBC, it does not replace it. “ Gavin King, Hibernate Founder

– It’s JDBC– It’s a Database

– (Please, work with a DBA)


– Lazy developers love Hibernate: Near-flat learning curve Little knowledge of JDBC or DB schema

design Little knowledge of Hibernate mapping

– Lazy loading, cache, session mgmt Little knowledge on how to tune Hibernate or

its JDBC parameters Ignore Hibernate SQL Query log

Why we love Hibernate


The Good and the Bad

– Hibernate can rock: Rapid development Cache Pooling Support / Docs

– Hibernate can suck: Performance problems Session Management Development Time


– Who owns the DB schema?– Marketing called and needs 500 reports written.

Who delivers?– Sales needs integration with Crystal Reports– Dual metadata: Changes to either schema requires

changes in the other. (IDEs won’t refactor/migrate/adapt DB schema)

Problem: Schema-Ownership

Development 101:1. Developers begin project with a clean slate.2. Developers define schema in the DB AND *.hbm.xml3. Developers ship product4. DBAs held accountable for performance5. DBAs call for refactoring / denormalization6. Now what?


– Work with a DBA Be prepared: Likely minimize reliance on Hibernate DBA provides Query plan.

– Consider Hibernate for simple CRUD– Set up a mirror DB specifically for reports

Possible silo

Solution: Schema-Ownership, cont.


Problem: n+1 SELECT

Iterator items = session.createCriteria(Item.class).add( Expression.eq("item.seller", user) ).list().iterator();

List maxAmounts = new ArrayList();while (items.hasNext()) {Item item = (Item) items.next();BigDecimal maxAmount = new BigDecimal("0");for ( Iterator b = item.getBids().iterator(); b.hasNext(); ) {Bid bid = (Bid) b.next();if ( bid.getAmount().compareTo(maxAmount) == 1 )maxAmount = bid.getAmount();}maxAmounts.add( new MaxAmount( item.getId(), maxAmount ) );}

Retrieve all Items for User:

So you set lazy=“true”…

Hibernate issues 1 SELECT per Bid…

Find maximum Bid for Items:

The n+1 problem is difficult to detect, as it is usually hidden inside application logic.

http://java.sun.com/j2se/1.5.0/docs/api/java/util/Iterator.html

http://java.sun.com/j2se/1.5.0/docs/api/java/beans/Expression.html

http://www.google.com/search?sitesearch=java.sun.com&q=allinurl:j2se/1+5+0/docs/api+List

http://java.sun.com/j2se/1.5.0/docs/api/java/util/ArrayList.html

http://java.sun.com/j2se/1.5.0/docs/api/java/math/BigDecimal.html




Solution: n+1 SELECT

<set name="bids" lazy="true" inverse="true" batch size="10"‐ >

String query = "select MaxAmount( item.id, max(bid.amount) )" + " from Item item join fetch item.bids bid" + " where item.seller = :user group by item.id";

List maxAmounts = session.createQuery(query).setEntity("user", user).list();

Enable Batch Fetching:

HQL Aggregation:

– Hibernate pre-fetches the next 10 collections– Problem is now reduced to n/10+1 SELECTs– A little better, but now other transactions will fetch collections

unnecessarily.

– Possible solution, unless we want to do complex processing of Bids.

<set name="bids" inverse="true" outer join="true"‐ >

Enable Eager Fetching:

– Note: HQL ignores outer-join, but you can use the Criteria API– You should be fired.




Solution: n+1 SELECT

List results = session.createCriteria(Item.class).add( Expression.eq("item.seller", user) ).setFetchMode("bids", FetchMode.EAGER).list();

Iterator items = new HashSet(results).iterator();List maxAmounts = new ArrayList();for ( ; items.hasNext(); ) {Item item = (Item) items.next();BigDecimal maxAmount = new BigDecimal("0");for ( Iterator b = item.getBids().iterator(); b.hasNext(); ) {Bid bid = (Bid) b.next();if ( bid.getAmount().compareTo(maxAmount) == 1 )maxAmount = bid.getAmount();}maxAmounts.add( new MaxAmount( item.getId(), maxAmount ) );}

Runtime Declaration of Fetch Strategy:

– No guarantee of distinct Items returned.


http://java.sun.com/j2se/1.5.0/docs/api/java/beans/Expression.html


http://java.sun.com/j2se/1.5.0/docs/api/java/util/HashSet.html


http://java.sun.com/j2se/1.5.0/docs/api/java/util/ArrayList.html





– Cause: Thrown when an unitialized collection (or proxy)

is accessed outside of the scope of the Session.– Detached collections can’t be initialized

Lazy initialization turned on by default v3.– Recommended best practice to leave it turned on.

Problem: LazyInitializationException


– If Session is open, use Hibernate.initialize(item), forcing initialization, as long as the Session is open.

– Keep the Session open: Open Session in View pattern: Use a servlet filter to close the Session

at the end of a request.– You must alter application architecture– You must address exception handling

Call Hibernate.initialize(item) in the business layer before returning to the web tier, or resort to eager fetching.

May not be a feasible strategy in n-tier environments

Solution: LazyInitializationException


– Hibernate addresses performance: Cache Amount of data loaded

– Finding the Cause: Monitor the DB

– In MySQL:• show processlist;• log_slow_queries = /var/log/mysql/mysql-slow.log &• long_query_time = 1• log-queries-not-using-indexes

Monitor Hibernate logging/queries:– hibernate.show_sql=true / log4j.logger.org.hibernate.type=debug– The output will scare you, but you can use it to execute a test query…

• EXPLAIN [SQL] Use a profiler

Problem: “Hibernate is Slow”


– Are tables properly indexed? Call the DBA.– Slow INSERT?

Consider “batch” INSERT– Explicit flushing:

• Dirty checking of objects can take a lot of time– Slow SELECT?

SELECT on PK. Avoids DB call – uses cache Are you loading everything in memory and looping through it? Don’t.

– 2nd Level cache on read-only entities– Use SQL ;-)– If appropriate, consider batch processing…

Addressing: “Hibernate is Slow”


– Above code will lead to OOM Hibernate caches all newly created Customer instances

Problem: OOM on Batch Processing

Session session = sessionFactory.openSession();Transaction tx = session.beginTransaction();for ( int i=0; i<100000; i++ ) { Customer customer = new Customer(.....); session.save(customer);}tx.commit();session.close();


– Flush and Clear the Session regularly– Use the StatelessSession Interface

Bypass cache No dirty-checking Ignores collections on entities Bypass Hibernate interceptors and event model

Solution: OOM on Batch Processing

Session session = sessionFactory.openSession();Transaction tx = session.beginTransaction(); for ( int i=0; i<100000; i++ ) { Customer customer = new Customer(.....); session.save(customer); if ( i % 20 == 0 ) { //20, same as the JDBC batch size //flush a batch of inserts and release memory: session.flush(); session.clear(); }} tx.commit();session.close();

StatelessSession session = sessionFactory.openStatelessSession();Transaction tx = session.beginTransaction(); ScrollableResults customers = session.getNamedQuery("GetCustomers") .scroll(ScrollMode.FORWARD_ONLY);while ( customers.next() ) { Customer customer = (Customer) customers.get(0); customer.updateStuff(...); session.update(customer);} tx.commit();session.close();


Sanity Check

– Addresses the ‘Object-Relational Impedance Mismatch’ Good marketing. ORM is the impedance

mismatch. It creates the inefficient connection. DBs provide permanent storage. Programming

languages process data in step-wide fashion. Morphing both is a mismatch. Promoting their strengths is the ideal scenario.

– Transparent You’re in control of the level of depth

Hibernate permeates your codebase. Expect some magic.

– Querying Facilities (HQL, Criteria API, JPAQL, SQL) In the end, they output SQL. Ugly SQL.


Sanity Check

– Database-Agnostic Who cares?

– “Performance”: Lazy initialization, fetching strategies, versioning, etc… Faster than what? If performance is a concern, raw JDBC/SQL

is your answer.– Caching:

Databases already offer this. Roll your own cache Beware of cached data modified by other

applications.– Automatic DDL generation

Never use this!


– Scenario: Analytics company processing several million

hits, email clicks, email opens, and form posts per day.

Over 800 customer in multi-tenant-style schema

75+ tables per schema 2 MySQL servers in Master-Slave configuration Application servers from 8GB to 16GB physical

memory.– Hibernate Use:

100% Originally JSF+Hibernate = Rapid Development

My Hibernate Divorce


– Problem #1: Random OOMs

– Diagnosis: Boot up Jprofiler. Nothing there. Analyze heap dump file:

– Hibernate holding 5MB per SessionFactory in memory, ~4GB per server Reporting and other intensive operations cause OOM By design, lazy-loading was disabled.



– Solution: Add more RAM. Could not scale horizontally. Load balance customers across servers (ala SFDC) Rethink lazy-loading strategy:

– Massive re-architecture– Suboptimal in certain parts of the system.

• Recording hits/clicks/opens• Reporting

Rip out Hibernate for memory intensive operations.– Table per Class – this was going to take a while.– Perform processing in DB, not code! That’s what they’re designed to do!– Some stored proc. – Get a DBA.



– Problem #2: Slow boot time: ~20 minutes. With hibernate update 2+ hours. (and system crashes) Marketing people aren’t patient.

– Diagnosis: Hibernate SessionFactory creation across 800 schema can be slow. ;-) hbm2ddl.auto=update is evil

– Solution: hbm2ddl.auto=none

– Handle schema updates manually (yay, PHP!)– Migration and adapting data a manual process.

Load SessionFactories on-demand.– When user logs in, load SessionFactory

Reap stale SessionFactories



If you really must use Hibernate:– Never let Hibernate design your schema– Your schema will outlive your application– Learn SQL, Stored Proc, and database design principles.– Proper indexing is paramount! Get a DBA.– Consider a mix: 50% Hibernate (75% or 90%)

Final Thoughts

Anti-Pattern:1. Some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results, and2. A refactored solution exists that is clearly documented, proven in actual practice and repeatable.


“... ‘a lot of open source projects just outgrow Hibernate. Hibernate was great to get started and that although it has a richer set of capabilities than iBatis, sometimes it just gets in the way’.”- John Newton, CTO, Alfresco

And Today…


Questions?

Technology

Ajug hibernate-dos-donts