Embedded Comparison Onyx Database Embedded Performance Benchmarks

Onyx Development Tools has created a series of performance benchmark tests using embedded Java databases including H2, Derby, Onyx Database, and HSQL. The purpose of these tests is to compare the performance of various JPA ORMs vs. the Onyx Database platform.

Rich Feature Set Realistic Throughput Simulation for Embedded Databases

Other than the Onyx Database environment, all embedded databases use the JPA library and their respective drivers. Onyx is a NoSQL database and an ORM (Object Relational Map) therefore it does not require a JPA implementation. This test only submits embedded Java databases. Further NoSQL benchmark comparisons and Graph Database performance reviews will be posted on a later date.

The data model has been set up to represent a production-like application. In this case, the data model is used to track NFL Player statistics. It has been pulled from one of our examples. The data model contains various relationships, indexes, and entities with attributes.

If you have a comment regarding our findings, or suggestions for benchmark test methodology, we would like to hear from you! Contact Onyx Development Tools, LLC.

Contact Us

Benchmark Results Compare Embedded Database Performance

The benchmarks were designed to test the speed and scalability of embedded databases. In each scenario, the database was pushed beyond a basic use load and used to determine how each individual database performs when it is pushed beyond a simple storage and throughput. In each test, Onyx Database performed better than each of the ORM database implementations. Based on our results, listed below are the databases ordered from best to worst.

Single Threaded Insertion Multi Threaded Insertion Delete Transaction Update Transaction Random Transactions *
Onyx Database 7,981 operations/second Finished in 2.506 seconds 8,357 operations/second Finished in 5.983 seconds 33,501 operations/second Finished in 0.597 seconds 13,514 operations/second Completed in 1.480 seconds 9,080 operations/second Finished in 3.304 seconds
H2 82 operations/second Finished in 245.011 seconds 130 operations/second Finished in 385.436 seconds 614 operations/second Finished in 32.582 seconds 444 operations/second Finished in 45.085 seconds 721 operations/second Finished in 38.117 seconds
Derby 70 operations/second Finished in 284.827 seconds 150 operations/second Finished in 333.791 seconds 652 operations/second Finished in 30.665 seconds 423 operations/second Finished in 47.242 seconds 721 operations/second Finished in 41.605 seconds
HSQL 29 operations/second Finished in 695.781 seconds 22 operations/second Finished in 2,232.081 seconds 105 operations/second Finished in 190.529 seconds 135 operations/second Finished in 147.971 seconds 132 operations/second Finished in 227.699 seconds
* The random transactions consist of insertion, update, delete, and queries. The breakdown of transactions are below:
  • 22% Database Insertions
  • 28% Querying various entities where 1/5 of the queries are indexed.
  • 6% Delete Transactions
  • 17% Delete Transactions
  • 17% Find Transactions
  • 6% Delete Transactions
  • 11% Update Transactions

Setup and Environment Test Methodology and Configuration

The tests were designed to measure insertion, read, delete, update, and multi-thread concurrency throughput. In addition to basic transactions, included is a test designed to simulate a realistic load by selecting a random distribution of events. The test was run on a 2015 MacBook Pro with El Capitan. The MacBook Pro consisted of 16 GB 1600 MHz DDR3, 2.5 GHz Intel Core i7, and SSD.

Each database benchmark test contains a reasonable ramp up followed by a fixed amount of iterations of given task(s). In most cases, the test was configured to have a maximum of 8 concurrent threads. In addition to that the JVM arguments included an -Xmx2048 argument. The performance tests were run 3 times and the average value was recorded for each scenario.

The Performance Benchmarks included are:
  • InsertionSingleThreadBenchmark Designed to evaluate the maximum insertion throughput of a single dedicated thread.
  • InsertionBenchmarkTest Used to show maximum throughput of multiple concurrent threads.
  • DeleteBenchmarkTest Illustrates the speed in which entities can be deleted.
  • UpdateBenchmarkTest Shows the performance of updating ORM objects concurrently.
  • RandomTransactionBenchmarkTest Randomly creates a workload of weighted transactions that is used to simulate a realistic production environment with high throughput.

The Performance Benchmark source code can be found: Onyx Embedded Database Performance on Github.