We released an EJB benchmark in early 2002 based on some work we did early on with EJBs. With the new site design, the EJB benchmark would have been lost, and that's why it ended up on my blog for archiving.

The Original EJB Benchmark

As developers of EJB applications, we are often faced with the question of what architecture or design to use. There is no shortage of proposed and touted architectures and designs out there. We can use fine grained entity beans, coarse grained entity beans, entity beans with a session bean front end, and even stateless session beans to access our data. Lacking any hard data on which we can base our decision, we are often forced to rely on claimed best practices and pure marketing hype.

This benchmark seeks to change that. The goal of this benchmark is to provide some objective basis for the hard design decisions every EJB development team has to make. This is not a benchmark of application servers. We do not test the performance of one application server versus any other app. server. The benchmark does test the performance of five EJB design idioms all running on the same application server and using the same database. The benchmark seeks to answer questions such as:

• how should I design my EJBs to maximize performance when accessing a single instance of data at a time?
• how should I design my EJBs to maximize performance when accessing a collection of data instances at a time?
• how should I design my EJBs to maximize the performance under a load?

In our desire to make the benchmark as pragmatic and as representative of real world results as possible, we chose to use a portion of an EJB based application as the benchmark. The portion of an application that we chose deals with inventory items and allows the user to access any individual item or lists of items. The concept of an item used in the benchmark application consists of the following attributes: item number, name, description, price, weight, weight unit of measure, manufacturer name.

In our desire to make the benchmark as pragmatic and as representative of real world results as possible, we chose to use a portion of an EJB based application as the benchmark. The portion of an application that we chose deals with inventory items and allows the user to access any individual item or lists of items. The concept of an item used in the benchmark application consists of the following attributes: item number, name, description, price, weight, weight unit of measure, manufacturer name.

What are design idioms?

Idioms are very similar to patterns. The difference between patterns and idioms is that patterns apply to general programming concepts where as idioms are specific to a programming language.

Idioms used in the benchmark

The design idioms covered by the benchmark are:

1. Fine Grained Entity Bean (FGE)
   often used by beginners to Enterprise Java Beans
2. Coarse Grained Entity Bean (CGE)
   the best performing Entity Bean idiom typically used in practice
3. Coarse Grained Container Managed Entity Bean (CGCME)
   touted as faster then Bean Managed Persistent Entity Beans
4. Optimized Entity Bean (OE)
   the best performing contender that still uses Entity Beans
5. Session over Entity Bean (SE)
   typically advanced as a "best practice" in trade media
6. Coarse Grained Session Beans (CGS)
   the proposed best performer

The entity bean based idioms covered by this benchmark used bean managed persistence rather than contain managed persistence. Further, all idioms used as much of the same SQL to converse with the database as possible (entity beans make some queries that are different from those made by session beans).

Fine Grained Entity Bean Idiom

The Fine Grained Entity Bean idiom is perhaps the most simple one for beginners to grasp. That's because this idiom presents a pattern of usage that is most similar to the one found in standard object oriented applications. When developing an application using the Fine Grained Entity Bean idiom the developer obtains reference to the entity bean and makes method calls to the individual accessor and mutator methods. This idiom is fine grained because every data attributes is accessed individually via an accessor method.

Coarse Grained Entity Bean Idiom

The Coarse Grained Entity Bean idiom makes a simple but profound optimization to the Fine Grained Entity Bean idiom. Rather than accessing each attribute individually, the Coarse Grained Entity Bean idiom allows the client application to access all the attributes of an instance at once. This is accomplished with the use of a state object. The entity bean has a single accessor method called getState() that returns an instance of the state object populated with the values of the instance attributes. This idiom allows the client application to incur the overhead of making a call to an EJB method only once and yet obtain the values for all attributes of the instance.

Coarse Grained Container Managed Entity Bean Idiom

The Coarse Grained Container Managed Entity Bean idiom is identical to the Coarse Grained Entity Bean except that the container manages the persistance. This allows the container to make optimizations. This also frees developers from having to write persistence code that is often not portable across data sources.

Optimized Entity Bean Idiom

The Optimized Entity Bean idiom is almost the same as the Coarse Grained Entity Bean Idiom with one exception - the inherent persistence mechanism present in entity beans is optimized. This optimization is made in the ejbStore() method. Normally the ejbStore() method executes a database update query every time it is invoked. That means that normally an update query is executed every time the client application calls the getState() method (if we wrap the getState() method with a transaction as we most certainly should). So the database is being updated even though the state has not changed. The optimization in this idiom involves using an is dirty flag. If the flag is true then the ejbStore() method executes the update query. If the flag is false then the ejbStore() method does nothing. Mutator methods would set the dirty flag to true but accessor methods such as getState() method would leave the flag set to false.

Session over Entity Bean Idiom

The Session over Entity Bean Idiom is perhaps the most widely touted idiom. It involves a session bean that communicates with the client application in a coarse grained manner and communicates with the database via a fine grained entity bean. When the client wants a reference to a data instance it calls a getState<>() method on the session bean. The session bean then creates a state object and populates it with attribute values that it obtains by calling the fine grained accessor methods on an entity bean. Because the transaction wraps the coarse grained getState<>() method on the session bean, the normally high overhead of calling the fine grained accessor methods on the entity bean is not incurred here.

Coarse Grained Session Bean Idiom

The Coarse Grained Session Bean idiom involves a stateless session bean that provides persistence services for a domain level concept such as items. That stateless session bean allows the client application to restore a particular instance of the domain level concept or to create a new instance of that concept or to update the representation of the concept instance in the persistent data store. The client application communicates with the stateless session bean in a coarse grained manner using state objects.

Test Results

Test 1 - Restoring a Single Instance

This test is designed to measure relative performance of each design idiom when accessing a single instance of data. A real world example that displays the same performance characteristics as this test is an order entry scenario where a customer requests to purchase a specific item. When entering the order into the order entry application the salesperson enters the specified item number and the application returns detailed information about the item. This test is designed to measure the amount of time it takes the application to return the detailed information about the item.

Test 2 - Restoring a Small Collection

This test is designed to measure the relative performance of each design idiom when accessing a small collection of data instances. A real world example that displays the same performance characteristics as this test is an order entry scenario where a customer requests to purchase an item such as a quarter inch bolt when there are 10 different quarter inch bolts available. When entering this order into the order entry application the sales person selects to view all the available quarter inch bolts and the application responds with a list of available bolts. This test is designed to measure the amount of time it takes the application to respond with the short list of bolts available. This test measures the performance of each idiom when accessing a collection whose size is between 10 and 100 elements.

As you can see from the graph below the coarse grain Session Bean idiom restored a list of 100 elements in 20 milliseconds. The next best performing idiom, the optimized entity idiom took 550 milliseconds to accomplish the same task. The Coarse Grained Entity idiom and Session over Entity idiom displayed very similar performance characteristics (something we will see throughout all the tests) and restored a list of 100 elements in over 800 milliseconds. The Fine Grained Entity Bean idiom was by far the worst performer requiring almost 3000 milliseconds to accomplish the same task .

By visually inspecting the chart of results we noticed that the performance graph of each idiom seemed linear. So we performed a regression analysis of the test results and found that the linear model was quite good, being able to explain upwards of 82% of the variations in the test data. We then divided the X coefficient of each idiom's model by the X coefficient of the best performing idiom - the Coarse Grain Session Bean idiom. This gave us a measure of how much better performing the best performing idiom was versus each other idiom tested. Based on the regression analysis, it seems that the Coarse Grained Session Bean idiom is about 20 times better performing than the Optimized Entity bean idiom when restoring short collection, and about 120 times better performing than the Fine Grained Entity Bean idiom. The results of the regression analysis as well as the performance factor are summarized in the table below.

Test 3 - Restoring a Large Collection

This test is exactly the same as the previous test except that the size of the collection is allowed to vary up to 1000 elements. The reason for splitting apart test 2 and 3 is that the performance characteristics encountered with small collections could the different than those encountered with large collections. In most applications we'll know at design time whether a particular collection is going to be small or large in the target deployment environment. Thus knowing the performance characteristics of each design idiom when faced with a small collection and with a large collection allows us to make the correct choice of idiom at design time.

The graph below summarizes the results of Test 3. Again the Coarse Grained Session Bean idiom prove to be the best performing one being able to restore a collection of 1000 elements in 451 milliseconds. The next best performing optimized entity bean idiom took 2264 milliseconds to accomplish the same task. The Coarse Grained Entity and Session over Entity Bean idioms clocked in with very similar performances at approximately 5958 milliseconds. And once again the Fine Grained Entity Bean idiom was by far the worst performing one having taken 26,458 milliseconds to restore a collection of 1000 elements.

We performed a regression analysis of the data obtained from test three as well. The results of that analysis indicate that this is the Coarse Grained Session Bean idiom is about 7.5 times faster than the optimized entity bean idiom and about 87 times faster than the Fine Grained Entity Bean idiom. The regression analysis results are summarized in the table below.

Test 4 - Load Test

The load test is designed to measure the relative performance of each idiom when accessing a collection of 50 data instances under a load of between 10 and 300 concurrent requests. The idea of behind this test is very simple - to see how these idioms scale.

The graph below summarizes the results of test 4. As you can see, the Coarse Grained Session Bean idiom scaled much better than the Optimized Entity Bean idiom. We only included the two top performing idioms in this test. At a load of 300 concurrent requests the Coarse Grained Session Bean idiom required less than 10,000 milliseconds on average to service each request whereas the optimized entity bean idiom required almost 70,000 milliseconds to accomplish the same task.

We performed a regression analysis of the data obtained from this test. The regression analysis yielded an X coefficient of 24.28 for the Coarse Grained Session Bean idiom. Meaning that if we increase the number of concurrent clients from 300 to 301 we could expect the average response time to each request to increase by 24.28 milliseconds. Further the regression analysis revealed that we can expect the Coarse Grained Session Bean idiom scale about 8.6 times better than the next best performing idiom the Optimized Entity Bean idiom.

Add a comment

Posted by Maciej Zawadzki on October 9, 2006 3:37:00 PM EDT #