Aerospike’s dynamic cluster management and unique in-memory technology enable our database to add capacity while in production and reliably handle millions of transactions per second (TPS), efficiently scaling to meet some of the largest data volume needs.
It isn’t always easy for a project team to determine data size and velocity requirements at the beginning of a project. Both variables often depend on the success of the project and on which data the data scientists find to be actionable and/or profitable. Data sizes and speeds could eventually scale up or down, and object sizes will likely vary.
The ability to move from a dataset of only a few gigabytes to one of hundreds of terabytes, and from just a few servers to hundreds of servers, is one of the most desired and critical attributes of a modern database. For this reason, clustered databases are now the primary technology chosen for new big data projects. While over-engineering is antithetical to lean ideals, having a scalable database prevents the need to replace the data layer as project requirements grow. If the data layer can’t scale, a project may fail.
Aerospike provides horizontal scalability via dynamic cluster management. The cluster management system automatically moves data as the cluster size changes and provides re-sharding on the fly, if required. Network communication is reduced by the use of optimized, compressed management protocols. Cluster sizes of hundreds of nodes, each with potentially hundreds of terabytes of flash that provide millions of transaction per second, become practical.
The ability to use all of the hardware capabilities of large, multi-core servers with fast storage, fast networks, and fast memory provides crucial scale-up capabilities. Thanks to the fact that the Aerospike in-memory database optimizes hardware efficiency, customers are often able to reduce production hardware footprints by up to an order of magnitude—which provides scale at an affordable price.
When it comes time to change hardware generations to achieve higher scale, Aerospike’s operational capabilities enable in-flight database change-over. This allows ingest of new hardware types, permitting a business to continue to take on hardware cost reduction and not be overtaken by competition using up-to-date hardware. A new cluster can be provisioned, writes are replicated through XDR, a backup and restore is used to move unchanging data, and the new system can be completely configured and ready to use without any service downtime.
These capabilities have allowed Aerospike deployments to achieve extraordinary multi-year uptime in production situations. Being able to scale horizontally as well as vertically opens up the potential to explore new and expanded business opportunities.
eBay Advertising: Scaling Operations with Ease
Aerospike supports eBay in growing the company’s massive retail shopping platform to support two petabytes of behavioral data on 100 million active users, conducting 1.8 billion searches per month.
“There are two problems that Aerospike helped us solve: speed and scale. […] There are a lot of technologies that can do clusters. But how many of them actually scale horizontally? […] If you think, ‘Oh this is great, it’s horizontally scaling’, there comes a point where it doesn’t work anymore. So far, we have been actually very good with [Aerospike]. […] Talking about operations ease […] yes, systems will go down. When you’re talking about big numbers, you want something that actually fails over automatically. And if you want to add new servers, you don’t want to bring down everything. As eBay, if you bring down everything before we have to upgrade… that’s just not good enough for us.”
– Prakash Chandra, Head of Engineering, eBay Advertising (11:40-11:48 and 12:47-13:33 in the video).
Snappy Growth: How Snapdeal Scaled by 500%
How does a young company navigate the challenges of a competitive market to achieve spectacular growth? In this webinar, Snapdeal VP and Head of Engineering Amitabh Misra joins Aerospike founders Brian Bulkowski and Srini Srinivasan to discuss his vision for Snapdeal, the company’s architecture infrastructure, share his insights on the choices he’s made along the way, and how his company is planning for future growth.
“In two years, we have scaled more than 200 times—the number of products listed, the number of sellers we have, the amount of business they do, the number of servers, storage, and the technology team—everything has grown. […] With Aerospike, we can push through huge price changes while maintaining the same response time experience on the buyer’s side—even with millions of buyers. That has been the biggest advantage.”
– Amitabh Misra, Vice President of Engineering, Snapdeal