Recommended Minimum Requirements

OS

For AWS, we recommend using the latest version of Amazon Linux for performance reasons. We have observed Amazon Linux to have the best performance on AWS EC2 instances. The rest of supported OS's also work in AWS, but their performance might lag as compared to AWS Linux.

Virtualization Type

We recommend using Hardware Virtual Machine (HVM) based AMIs for usage with Aerospike. In our benchmarks, we have seen an approximate 3x performance gain without any other tuning when using HVM instances instead of PV instances.

Cloud Considerations

1. Lack of Multicast Support: Cloud providers (e.g. Amazon, Google Compute Engine) do not support multicast networking. For these providers, we offer Mesh heartbeats, which uses point-to-point TCP connections for heartbeats.

2. Network Variability: Often, the network latency on cloud platforms is not consistent over time. This can cause problems with heartbeat packet delivery times. For these providers, we recommend setting the heartbeat interval to 150 and the heartbeat timeout to 20.

Network Setup

As a prerequisite of using HVM and Enhanced Networking, we recommend you use a VPC based setup. You cannot use HVM AMIs and Enhanced networking in Classic EC2 mode.

We also recommend the use of placement groups for Aerospike server instances.

We suggest you use an AWS private IP for setup of mesh heartbeat rather than the public IP. Private IPs are allocated to each EC2 instance by default. You may also need to add a public IP to your instance if you need direct access to the instance. There are two ways to add a public IP to your instance:

• During instance launch: You can explicitly set the option to enable a public IP for your instance while launching the instance. You can also modify the behavior of your VPC subnet to enable it by default.
• Attach an Elastic IP after launching instance: An Elastic IP address is a static, public IP address designed for dynamic cloud computing. You can associate an Elastic IP address with any instance or network interface for your VPC.

Network Interface

Each network interface on an amazon Linux HVM instance can handle about 250K packets per second. If you need higher performance per instance, you need to do one of the following:

• Add More NIC/ENI Elastic Network Interfaces (ENI) provide a way of adding multiple (virtual) NICs to an instance. A single NIC peaks at around 250k TPS, bottlenecking on cores processing interrupts. Adding more interfaces help to process more packets per second on the same instance. Using ENIs with private IPs is free of cost in AWS.

• Receive Packet Steering

  RPS is only available in kernel version 2.6.35 and above.
  Another-simpler-approach is to distribute IRQ over multiple cores using RPS

  echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

  With Aerospike, this eliminates the need to use multiple NICs/ENIs, making management easier, and result in similar TPS. A single NIC with RPS enabled can achieve up to 800K TPS with interrupts spread over 4 cores.

Security Group

Aerospike needs TCP ports 3000-3003 for intra-cluster communication. These ports need not be open to the rest of Internet.

Additionally, if your are running XDR, port 3004 will be required. If using XDR, port 3000 (or the info port for remote data centers' aerospike) of destination datacenter should be reachable from the source datacenter.

AMC requires port 8081 for access from the Internet.

Additionally, you will need a port for SSH access to your instances (default TCP port 22)

Cloud Formation

We have published a cloud formation script which should be helpful in quickly setting up a cluster with recommended setup.

This is available on Github

Redundancy Using Availability Zone

To add further redundancy to Aerospike in AWS using Availability Zone (AZ), you can set up two cluster across two different availability zones such that there is one set of data in each AZ.

If you wish to configure an Aerospike cluster across two zones or regions, we suggest you use an Application-level Queue like Kafka or RabbitMQ.

Aerospike Enterprise Edition has Cross Data Center Replication (XDR) feature, which handles clusters located in multiple data centers. Please contact us for further details.

Pre-warming EBS and Ephemeral disks

Similar to the initialization of Flash on bare metal machines, we also recommend that AWS disks initialized (or pre-warmed) using the dd tool.

sudo dd if=/dev/zero of=/dev/<deviceID> bs=1M&

Shadow device configuration

As noted above, some EC2 instance types have direct-attached SSDs called Instance Store Volumes. These can be significantly faster than EBS volumes (as EBS volumes are network attached). But, it is recommended by AWS to not rely on instance store volumes for valuable, long-term data.

To take advantage of the fast direct-attached instance store SSDs, Aerospike provides the concept of shadow device where all writes are also propagated to these shadow devices. This is configured by specifying an additional device name in the storage engine block of the namespace configuration.

storage-engine device {
        device /dev/sdb /dev/sdf
        write-block-size 1024K
    }

In this case, /dev/sdb is the Instance store volume where all reads and writes will be directed to. The other shadow device /dev/sdf is the EBS volume where only the writes are propagated. In this way, we can achieve the high speeds of direct-attached SSDs while not compromising on the durability guarantees of EBS volumes. Note that the write throughput will still be limited by the EBS volume and hence this strategy will give good results when the percentage of writes is low.

Additional Information

• Tuning for 1 Million TPS on an AWS instance
• EBS Pre-warming
• EC2 Disk Performance