Aerospike Technology

Digitally transform and modernize both legacy real-time operational and transactional databases

Are you hitting limits and capabilities with your database technologies? Learn how the Aerospike technology provides mainframe levels of reliability for distributed, cost-efficient systems of record and edge-based systems. Aerospike combines performance with predictability and strong consistency with the ability to readily scale up and out.

Have questions before you get started?
Contact Us

Why Aerospike Technology at Scale

 Cache-plus DBMainframeAerospike
Scale upAverageExcellentExcellent
Scale outAveragePoorExcellent
ConsistencyPoorExcellentExcellent 1
PerformanceExcellentExcellentExcellent
Predictable PerformancePoorExcellentExcellent
ReliabilityPoorExcellentExcellent
Low TCOPoorPoorExcellent
Ease of DeploymentPoorAverageExcellent
 

Poor Poor        Average Average        Excellent Excellent

Aerospike Database Technology

The Aerospike Data Distribution results in unique, data partitioning that has uniform distribution of keys in the digest space, avoiding the creation of hotspots during data access, which helps achieve high levels of scale and fault tolerance. Aerospike colocates indexes and data to avoid any cross-node traffic when running read operations or queries.

The benefits of uniform data distribution across nodes result in:

  1. Application workload is uniformly distributed across the cluster,
  2. Performance of database operations is predictable,
  3. Scaling the cluster up and down is easy, and
  4. Live cluster reconfiguration and subsequent data rebalancing is simple, non-disruptive and efficient.

Partition assignment is deterministic which allows:

  • Each node can compute the same partition map
  • Uniform distribution of master partitions and replicas
  • Minimizes partition movement

Easy-to-program data structures such as lists and maps improve performance and allow indexed lookups on field values.

Data Structure & Document indexing

The Real-Time Transaction Engine is responsible for reading and writing data upon request while providing consistency and isolation. This involves synchronous and asynchronous replication, requests to alternate nodes if a node becomes unavailable as well as conflict/duplicate resolution after a node rejoins a cluster (after being unavailable).

Multi-Core System

Improves latency by reducing shared data across NUMA regions by grouping multiple threads per CPU socket (instead of per core).

Context Switch (Offloading)

Avoids costs of typical context switches, some operations are run in the network listener thread itself without relinquishing the CPU (and the system creates as many network listener threads as there are cores).

Memory Allocation 

Alleviates the application or a runtime environment while minimizing the need for memory defragmentation. System resources are leveraged by keeping the index packed into RAM. Fragmentation is minimized by grouping data objects by namespace into the same slab or arena, so the long-term object creation, access, modification and deletion patterns are optimized.

Data Structure Design

Safe and concurrent read, write, and delete access to the index tree, without holding multiple locks is due to each tree element having both a reference count and its own lock. This allows for access to nested data structures like index trees and does not involve acquiring multiple locks at each level, thereby reducing contention across partitions.

Scheduling and Prioritization

In addition to basic Key-Value Store operations, Aerospike supports batch queries, scans, and secondary index queries. Prioritization via throughput and fairness is achieved via job partitioning based on type, CPU-effort required and controlling the load being generated.

Cross Datacenter Replication (XDR) supports different replication topologies, including active-active, active-passive, chain, star, and multi-hop configurations.

Load sharing

If a node fails, all the other nodes detect this failure and takeover the pending work on behalf of the failed node. This scheme scales horizontally as one can just add more nodes to handle increasing replication load.

Data shipping

Namespace and set-level granularity for data shipping is supported. Optimizations are in place to reduce the amount of data shipped, especially in the case of hot keys. Compression is also supported (beneficial if records are larger than 1KB; especially so for strings and blobs).

Remote Cluster Management

Keeps track of remote cluster state changes, connects to all the nodes of the remote cluster, maintains connection pools, and performs all the roles just like a regular client. A proportional shipping/receiving model allows clusters of disparate sizes to send/receive data. New changes continue to be shipped as long as one node is viable, and adjusts easily to node additions and subtractions.

Pipelining

For cross data-center shipping, Aerospike uses an asynchronous pipelined scheme; each node in the source cluster communicates with all the nodes in the destination cluster. At any given point in time, there can be multiple records on the connection waiting to be written at the destination. This pipelined model is the main way we are able to deliver high throughput on high-latency connections over WAN.

+ Data Distributor

The Aerospike Data Distribution results in unique, data partitioning that has uniform distribution of keys in the digest space, avoiding the creation of hotspots during data access, which helps achieve high levels of scale and fault tolerance. Aerospike colocates indexes and data to avoid any cross-node traffic when running read operations or queries.

The benefits of uniform data distribution across nodes result in:

  1. Application workload is uniformly distributed across the cluster,
  2. Performance of database operations is predictable,
  3. Scaling the cluster up and down is easy, and
  4. Live cluster reconfiguration and subsequent data rebalancing is simple, non-disruptive and efficient.

Partition assignment is deterministic which allows:

  • Each node can compute the same partition map
  • Uniform distribution of master partitions and replicas
  • Minimizes partition movement

Easy-to-program data structures such as lists and maps improve performance and allow indexed lookups on field values.

Data Structure & Document indexing

+ Real-time Transaction Engine

The Real-Time Transaction Engine is responsible for reading and writing data upon request while providing consistency and isolation. This involves synchronous and asynchronous replication, requests to alternate nodes if a node becomes unavailable as well as conflict/duplicate resolution after a node rejoins a cluster (after being unavailable).

Multi-Core System

Improves latency by reducing shared data across NUMA regions by grouping multiple threads per CPU socket (instead of per core).

Context Switch (Offloading)

Avoids costs of typical context switches, some operations are run in the network listener thread itself without relinquishing the CPU (and the system creates as many network listener threads as there are cores).

Memory Allocation 

Alleviates the application or a runtime environment while minimizing the need for memory defragmentation. System resources are leveraged by keeping the index packed into RAM. Fragmentation is minimized by grouping data objects by namespace into the same slab or arena, so the long-term object creation, access, modification and deletion patterns are optimized.

Data Structure Design

Safe and concurrent read, write, and delete access to the index tree, without holding multiple locks is due to each tree element having both a reference count and its own lock. This allows for access to nested data structures like index trees and does not involve acquiring multiple locks at each level, thereby reducing contention across partitions.

Scheduling and Prioritization

In addition to basic Key-Value Store operations, Aerospike supports batch queries, scans, and secondary index queries. Prioritization via throughput and fairness is achieved via job partitioning based on type, CPU-effort required and controlling the load being generated.

+ Cross-datacenter Replication (XDR)

Cross Datacenter Replication (XDR) supports different replication topologies, including active-active, active-passive, chain, star, and multi-hop configurations.

Load sharing

If a node fails, all the other nodes detect this failure and takeover the pending work on behalf of the failed node. This scheme scales horizontally as one can just add more nodes to handle increasing replication load.

Data shipping

Namespace and set-level granularity for data shipping is supported. Optimizations are in place to reduce the amount of data shipped, especially in the case of hot keys. Compression is also supported (beneficial if records are larger than 1KB; especially so for strings and blobs).

Remote Cluster Management

Keeps track of remote cluster state changes, connects to all the nodes of the remote cluster, maintains connection pools, and performs all the roles just like a regular client. A proportional shipping/receiving model allows clusters of disparate sizes to send/receive data. New changes continue to be shipped as long as one node is viable, and adjusts easily to node additions and subtractions.

Pipelining

For cross data-center shipping, Aerospike uses an asynchronous pipelined scheme; each node in the source cluster communicates with all the nodes in the destination cluster. At any given point in time, there can be multiple records on the connection waiting to be written at the destination. This pipelined model is the main way we are able to deliver high throughput on high-latency connections over WAN.

It is not just the throughput and latency characteristic, but also the ability to store and process large swaths of data that defines the ability of a DBMS to scale up. Aerospike has been designed from the ground up to leverage SSD technology. This allows Aerospike to manage dozens of terabytes of data on a single machine with sub-millisecond record access times. Aerospike supports three kinds of storage structures: Hybrid-Memory, All-Flash, and In-Memory:

Hybrid-Memory storage subsystem

Aerospike implements a hybrid model wherein the index is purely in memory (not persisted), and data is only on a persistent storage (SSD) and is read directly from the disk. Disk I/O is not required to access the index, which makes performance predictable. Aerospike improves SSD device durability by avoiding in-place updates in favor of copy-on-write using large block writes. Data can be committed to disk before a confirmation is received from the application. Access to SSDs is done in a massively parallel manner with an even amount of data stored on every node and flash device, further avoiding hot spots avoiding reconfigurations as workloads change.

All-Flash storage subsystem

This feature extends Hybrid Memory to provide Aerospike reliability, consistency, and ease of use to areas where traditional relational systems may seem the first choice. By adding to our deployment options, Aerospike provides tiers of deployment strategies suitable for large-scale systems of record where HBase or HDFS may have been the first choice previously. All Flash feature extends Aerospike’s offerings in Hybrid Memory to support a broader set of cases. These uses surround having a very large number ( 100’s of billions ) of small ( < 1000 byte ) objects , which is common in architectures which store individual behaviors as separate database elements, or the need to segregate data elements for GDPR-style data layouts.

In-memory storage subsystem

In Aerospike’s in-memory storage architecture, all of the indexes and the entire data set are always present in DRAM and the data alone is backed up on persistent storage (can be rotational disk, i.e., HDD). Therefore, all application reads are from DRAM. The writes, however, are first applied to an in-memory buffer that will be subsequently flushed to persistent storage (HDD or SSD). Additionally, in strong consistency mode, Aerospike allows applications to commit every write directly to persistent storage.

The data rebalancing mechanism ensures that the transaction volume is distributed evenly across all nodes and is robust in the event of node failure happening during rebalancing itself. The system is designed to be continuously available, so data rebalancing doesn’t impact cluster behavior.

Benefits

  • Optimizes transactional simplicity in a scalable shared-nothing environment.
  • Clusters will self-heal even at demanding times by not requiring operator intervention.
  • Minimize service loss due to unforeseen failures via capacity planning and system monitoring.
  • Users can be insulated from database failures via hardware capacity configuring and provisioning and replication/synchronization policies.

Migrations

The process of moving records from one node to another node is termed a migration. After every cluster view change, the objective of data migration is to have the latest version of each record available at the current master and replica nodes for each of the data partitions.

Migration Optimization

Aerospike optimizes migrations by reducing the effort and time they take due to:

  • Nodes negotiating only the delta in records (especially useful for rolling upgrades).
  • Avoiding migration if content is known to be a subset of the same partition.
  • Duplicate resolution during migrations to ensure that partial writes to a record are completed.

The Self-Healing Smart Cluster Manager is responsible for adding and removing nodes seamlessly to the cluster.

Heartbeat Subsystem

Stores and exchanges information (status of neighboring nodes in adjacency lists) between nodes. (Heartbeats are used to locate new nodes.)

Clustering Subsystem

Maintains node membership information (node succession list) corresponding to the current active cluster for consensus.

Exchange Subsystem

Exchanges the partition state and triggers the rebalancing algorithm.

+ Storage Engine

It is not just the throughput and latency characteristic, but also the ability to store and process large swaths of data that defines the ability of a DBMS to scale up. Aerospike has been designed from the ground up to leverage SSD technology. This allows Aerospike to manage dozens of terabytes of data on a single machine with sub-millisecond record access times. Aerospike supports three kinds of storage structures: Hybrid-Memory, All-Flash, and In-Memory:

Hybrid-Memory storage subsystem

Aerospike implements a hybrid model wherein the index is purely in memory (not persisted), and data is only on a persistent storage (SSD) and is read directly from the disk. Disk I/O is not required to access the index, which makes performance predictable. Aerospike improves SSD device durability by avoiding in-place updates in favor of copy-on-write using large block writes. Data can be committed to disk before a confirmation is received from the application. Access to SSDs is done in a massively parallel manner with an even amount of data stored on every node and flash device, further avoiding hot spots avoiding reconfigurations as workloads change.

All-Flash storage subsystem

This feature extends Hybrid Memory to provide Aerospike reliability, consistency, and ease of use to areas where traditional relational systems may seem the first choice. By adding to our deployment options, Aerospike provides tiers of deployment strategies suitable for large-scale systems of record where HBase or HDFS may have been the first choice previously. All Flash feature extends Aerospike’s offerings in Hybrid Memory to support a broader set of cases. These uses surround having a very large number ( 100’s of billions ) of small ( < 1000 byte ) objects , which is common in architectures which store individual behaviors as separate database elements, or the need to segregate data elements for GDPR-style data layouts.

In-memory storage subsystem

In Aerospike’s in-memory storage architecture, all of the indexes and the entire data set are always present in DRAM and the data alone is backed up on persistent storage (can be rotational disk, i.e., HDD). Therefore, all application reads are from DRAM. The writes, however, are first applied to an in-memory buffer that will be subsequently flushed to persistent storage (HDD or SSD). Additionally, in strong consistency mode, Aerospike allows applications to commit every write directly to persistent storage.

+ Dynamic Data Rebalancer

The data rebalancing mechanism ensures that the transaction volume is distributed evenly across all nodes and is robust in the event of node failure happening during rebalancing itself. The system is designed to be continuously available, so data rebalancing doesn’t impact cluster behavior.

Benefits

  • Optimizes transactional simplicity in a scalable shared-nothing environment.
  • Clusters will self-heal even at demanding times by not requiring operator intervention.
  • Minimize service loss due to unforeseen failures via capacity planning and system monitoring.
  • Users can be insulated from database failures via hardware capacity configuring and provisioning and replication/synchronization policies.

Migrations

The process of moving records from one node to another node is termed a migration. After every cluster view change, the objective of data migration is to have the latest version of each record available at the current master and replica nodes for each of the data partitions.

Migration Optimization

Aerospike optimizes migrations by reducing the effort and time they take due to:

  • Nodes negotiating only the delta in records (especially useful for rolling upgrades).
  • Avoiding migration if content is known to be a subset of the same partition.
  • Duplicate resolution during migrations to ensure that partial writes to a record are completed.
+ Self-Healing Smart Cluster

The Self-Healing Smart Cluster Manager is responsible for adding and removing nodes seamlessly to the cluster.

Heartbeat Subsystem

Stores and exchanges information (status of neighboring nodes in adjacency lists) between nodes. (Heartbeats are used to locate new nodes.)

Clustering Subsystem

Maintains node membership information (node succession list) corresponding to the current active cluster for consensus.

Exchange Subsystem

Exchanges the partition state and triggers the rebalancing algorithm.

Aerospike Database Technology

Data Distributor

The Aerospike Data Distribution results in unique, data partitioning that has uniform distribution of keys in the digest space, avoiding the creation of hotspots during data access, which helps achieve high levels of scale and fault tolerance. Aerospike colocates indexes and data to avoid any cross-node traffic when running read operations or queries.

The benefits of uniform data distribution across nodes result in:

  1. Application workload is uniformly distributed across the cluster,
  2. Performance of database operations is predictable,
  3. Scaling the cluster up and down is easy, and
  4. Live cluster reconfiguration and subsequent data rebalancing is simple, non-disruptive and efficient.

Partition assignment is deterministic which allows:

  • Each node can compute the same partition map
  • Uniform distribution of master partitions and replicas
  • Minimizes partition movement

Easy-to-program data structures such as lists and maps improve performance and allow indexed lookups on field values.

Data Structure & Document indexing

Real-time Transaction Engine

The Real-Time Transaction Engine is responsible for reading and writing data upon request while providing consistency and isolation. This involves synchronous and asynchronous replication, requests to alternate nodes if a node becomes unavailable as well as conflict/duplicate resolution after a node rejoins a cluster (after being unavailable).

Multi-Core System

Improves latency by reducing shared data across NUMA regions by grouping multiple threads per CPU socket (instead of per core).

Context Switch (Offloading)

Avoids costs of typical context switches, some operations are run in the network listener thread itself without relinquishing the CPU (and the system creates as many network listener threads as there are cores).

Memory Allocation 

Alleviates the application or a runtime environment while minimizing the need for memory defragmentation. System resources are leveraged by keeping the index packed into RAM. Fragmentation is minimized by grouping data objects by namespace into the same slab or arena, so the long-term object creation, access, modification and deletion patterns are optimized.

Data Structure Design

Safe and concurrent read, write, and delete access to the index tree, without holding multiple locks is due to each tree element having both a reference count and its own lock. This allows for access to nested data structures like index trees and does not involve acquiring multiple locks at each level, thereby reducing contention across partitions.

Scheduling and Prioritization

In addition to basic Key-Value Store operations, Aerospike supports batch queries, scans, and secondary index queries. Prioritization via throughput and fairness is achieved via job partitioning based on type, CPU-effort required and controlling the load being generated.

Cross-datacenter Replication (XDR)

Cross Datacenter Replication (XDR) supports different replication topologies, including active-active, active-passive, chain, star, and multi-hop configurations.

Load sharing

If a node fails, all the other nodes detect this failure and takeover the pending work on behalf of the failed node. This scheme scales horizontally as one can just add more nodes to handle increasing replication load.

Data shipping

Namespace and set-level granularity for data shipping is supported. Optimizations are in place to reduce the amount of data shipped, especially in the case of hot keys. Compression is also supported (beneficial if records are larger than 1KB; especially so for strings and blobs).

Remote Cluster Management

Keeps track of remote cluster state changes, connects to all the nodes of the remote cluster, maintains connection pools, and performs all the roles just like a regular client. A proportional shipping/receiving model allows clusters of disparate sizes to send/receive data. New changes continue to be shipped as long as one node is viable, and adjusts easily to node additions and subtractions.

Pipelining

For cross data-center shipping, Aerospike uses an asynchronous pipelined scheme; each node in the source cluster communicates with all the nodes in the destination cluster. At any given point in time, there can be multiple records on the connection waiting to be written at the destination. This pipelined model is the main way we are able to deliver high throughput on high-latency connections over WAN.

Storage Engine

It is not just the throughput and latency characteristic, but also the ability to store and process large swaths of data that defines the ability of a DBMS to scale up. Aerospike has been designed from the ground up to leverage SSD technology. This allows Aerospike to manage dozens of terabytes of data on a single machine with sub-millisecond record access times. Aerospike supports three kinds of storage structures: Hybrid-Memory, All-Flash, and In-Memory:

Hybrid-Memory storage subsystem

Aerospike implements a hybrid model wherein the index is purely in memory (not persisted), and data is only on a persistent storage (SSD) and is read directly from the disk. Disk I/O is not required to access the index, which makes performance predictable. Aerospike improves SSD device durability by avoiding in-place updates in favor of copy-on-write using large block writes. Data can be committed to disk before a confirmation is received from the application. Access to SSDs is done in a massively parallel manner with an even amount of data stored on every node and flash device, further avoiding hot spots avoiding reconfigurations as workloads change.

All-Flash storage subsystem

This feature extends Hybrid Memory to provide Aerospike reliability, consistency, and ease of use to areas where traditional relational systems may seem the first choice. By adding to our deployment options, Aerospike provides tiers of deployment strategies suitable for large-scale systems of record where HBase or HDFS may have been the first choice previously. All Flash feature extends Aerospike’s offerings in Hybrid Memory to support a broader set of cases. These uses surround having a very large number ( 100’s of billions ) of small ( < 1000 byte ) objects , which is common in architectures which store individual behaviors as separate database elements, or the need to segregate data elements for GDPR-style data layouts.

In-memory storage subsystem

In Aerospike’s in-memory storage architecture, all of the indexes and the entire data set are always present in DRAM and the data alone is backed up on persistent storage (can be rotational disk, i.e., HDD). Therefore, all application reads are from DRAM. The writes, however, are first applied to an in-memory buffer that will be subsequently flushed to persistent storage (HDD or SSD). Additionally, in strong consistency mode, Aerospike allows applications to commit every write directly to persistent storage.

Dynamic Data Rebalancer

The data rebalancing mechanism ensures that the transaction volume is distributed evenly across all nodes and is robust in the event of node failure happening during rebalancing itself. The system is designed to be continuously available, so data rebalancing doesn’t impact cluster behavior.

Benefits

  • Optimizes transactional simplicity in a scalable shared-nothing environment.
  • Clusters will self-heal even at demanding times by not requiring operator intervention.
  • Minimize service loss due to unforeseen failures via capacity planning and system monitoring.
  • Users can be insulated from database failures via hardware capacity configuring and provisioning and replication/synchronization policies.

Migrations

The process of moving records from one node to another node is termed a migration. After every cluster view change, the objective of data migration is to have the latest version of each record available at the current master and replica nodes for each of the data partitions.

Migration Optimization

Aerospike optimizes migrations by reducing the effort and time they take due to:

  • Nodes negotiating only the delta in records (especially useful for rolling upgrades).
  • Avoiding migration if content is known to be a subset of the same partition.
  • Duplicate resolution during migrations to ensure that partial writes to a record are completed.

Self-Healing Smart Cluster

The Self-Healing Smart Cluster Manager is responsible for adding and removing nodes seamlessly to the cluster.

Heartbeat Subsystem

Stores and exchanges information (status of neighboring nodes in adjacency lists) between nodes. (Heartbeats are used to locate new nodes.)

Clustering Subsystem

Maintains node membership information (node succession list) corresponding to the current active cluster for consensus.

Exchange Subsystem

Exchanges the partition state and triggers the rebalancing algorithm.

Get started with Aerospike

Have questions before you get started?