Brian Bulkowski, Aerospike Founder and CTO Benchmark

As we announced in our Database Manifesto, Aerospike is committed to fair and honest benchmarks. To this end, Aerospike will publish its own benchmarks and attempt to duplicate results from benchmarks from other vendors and independent sources. In that way, Aerospike hopes to provide transparency and to separate science from marketing.

This review focuses on a Redis Labs press release from June 2015 announcing the results of a new, independent NoSQL performance benchmark performed by Avalon Consulting. The study claimed to measure the speed and scale of different NoSQL database solutions in a “real-world application scenario.” Not surprisingly, the press release gave Redis Labs top scores over alternatives such as Couchbase, DataStax Enterprise In-Memory, and Aerospike. The test scenario consisted of an application featuring a “real-time voting process during massive live events” – such as televised game shows – with a high volume of ‘write’ operations. Scale is mentioned repeatedly as a key objective of the test.

At Aerospike, we couldn’t agree more that scale is a crucial part of NoSQL benchmarks. So, we decided to run the benchmark ourselves to try to understand how Redis could claim – contrary to all our own measurements – to perform so well. We spent several weeks analyzing the test framework, application design, and study results. We were able to confirm that the published results are technically accurate within the narrow confines of the custom-designed application. However, the test scenario is not particularly useful, because the test failed to simulate a real-life “Big Data” NoSQL implementation. This benchmark scenario only works for small deployments. Indeed, this benchmark tests a small and exceptionally narrow workload.

We observed that:

  • The application doesn’t scale. For a test that purports to demonstrate scalability, the application design contains a flaw. There are two “hot keys” – the two counters in the voting application which count the global number of votes. These two keys provided ⅔ (⅓ per key) of the database load. This is a non-scalable application and data design. With this application design, you can’t add cores or machines and have the app run faster. It’s as if a performance governor was deliberately embedded inside the application.
  • The environment is tested with 1 GB of data and is NOT configured to persist data. In Aerospike’s experience, the point of using a database is to allow restarts and recovery from challenging situations – especially in cloud deployments that Avalon Consulting states is the point of the test. Without persistence – not even a master-slave configuration for improved availability and thus, high availability – this test is not indicative of a standard deployment environment. Such a small data set, like 1GB, would be better done in-application using a local heap.
  • The benchmark is done on a single 2-core server. This configuration was presumably chosen because, unique among NoSQL technology, Redis is a single-threaded server. When we added more cores and re-ran the test, Redis – as expected – went no faster. Even a 4-core or 8-core server can’t be fully utilized by this test configuration. All the other NoSQL technologies tested were able to scale, albeit to different extents.

Our conclusion is that this application – with its two hot keys – can only be run for small, non-scalable deployments. This benchmark fails to demonstrate enterprise-scale, “Big Data”, or a typical NoSQL implementation. It is a non-scalable benchmark which appears contrived to claim scalable performance when none was demonstrated.

We hope this insight into the application and data design of the benchmark provides additional perspective to help the reader make informed technology decisions about NoSQL databases and choose the best for their particular use case.

About Author

mm

Brian Bulkowski, Aerospike Founder and CTO

All posts by this author
Brian is a Founder and the CTO of Aerospike. With almost 30 years in Silicon Valley, his motivation for starting Aerospike was the confluence of what he saw as the rapidly advancing flash storage technology with lower costs that weren’t being fully leveraged by database systems as well as the scaling limitations of sharded MySQL systems and the need for a new distributed database. He was able to see these needs as both a Lead Engineer at Novell and Chief Architect at Cable Solutions at Liberate - where he built a high-performance, embedded networking stack and high scale broadcast server infrastructure.