Qualifying Cloud Instances
The Aerospike Cloud Qualifying Process evaluates cloud providers' instance and device performance by simulating a steady read and write workload, and measuring response time. These are tests that should be performed to get an idea of the level of performance you can expect from cloud instances.
If your instance is not one that has been tested by Aerospike, you can test it yourself by downloading the open sourced Aerospike Cloud Qualification package to test your instances, or by utilizing YCSB directly.
Qualified Instance Types
Cloud instance types will give you a certain amount of resources (CPU, Memory, Disk and Networking), but it can be hard to translate that into expected performance. The Cloud Qualification Process will take these instances and qualify it capable of a certain operational level, as an overall metric. It does not focus on individual resource components or optimizations.
This page will hopefully provide a starting point into what you can expect from each instance and how the instance class/family is more suitable toward different object sizes and use cases.
The Cloud Qualification Process is done using Aerospike's fork of the Yahoo! Cloud Serving Benchmark.
The testing methodology is as follows:
| Workload | Access Pattern | Record Count* | Record Size* | Cluster Size | YCSB Duration |
|---|---|---|---|---|---|
| 50/50 R/U | Uniform | 60% of memory | 50% of disk | 3 | 3 * Record Count |
* Based on a 2 node cluster
The testing client is a single instance of of a top-end system focused on compute power and network.
The entire test is repeated at least 3 times for verification
This workload is chosen as a worst-case scenario for Aerospike, as a uniform access pattern will result in all blocks entering defragmentation at nearly the same time.
Aerospike is configured to utilize local SSD storage (when available), but is otherwise left as default, as are the instances that Aerospike is deployed on. You may see improved performance from additional tuning. You may also choose to alter these numbers and percentages in your own run of YCSB.
An instance is considered passing at an operational level (eg: 30,000 ops/s) if the average read latency does not exceed 1 millisecond from the client. 95 and 99 percentiles, as well as object size and count, are provided as reference.
Larger operational levels are better.
Your results may vary from these published numbers due to differences in server, topology, and even variation between systems of the same type.
The table below shows the results from the Aerospike Cloud Qualification Process on some popular instance types.
Qualified Instances
AWS
| Instance Type | Client Transactions/s+ | 95th Percentile (ms) | 99th Percentile (ms) | Object Size (bytes) | Object Count | Disk Size | Cost++ |
|---|---|---|---|---|---|---|---|
| i3.xlarge | 68,000 | 3.3 | 5.4 | 1424 | 240,000,000 | 950GB | $0.312/hr |
| c3.4xlarge+++ | 66,000 | 4.1 | 11 | 554 | 240,000,000 | 2x160GB | $0.84/hr |
| i2.xlarge | 36,000 | 3.5 | 5.8 | 1424 | 240,000,000 | 800GB | $0.424/hr |
++ Price in us-west-2, at time of writing.
+++ All available ephemeral drives were utilized concurrently.
Some ephemeral drives require initialization before reaching maximum performance. You can read more on Amazon's Documentation.
Initialization is not done for these tests. The loading phase, as well as having multiple verification runs, acts as an effective initialization process.
Your numbers may not necessarily match these exactly, due to variance within AWS's environment.
Azure
| Instance Type | Client Transactions/s+ | 95th Percentile (ms) | 99th Percentile (ms) | Object Size (bytes) | Object Count | Disk Size | Cost++ |
|---|---|---|---|---|---|---|---|
| Standard_Gs1 | 31,000 | 3.5 | 9.1 | 13+++ | 240,000,000 | 56GB | $0.61/hr |
| Standard_L4s | 41,000 | 1.2 | 5.8 | 1500 | 240,000,000 | 678GB | $0.344/hr |
++ Price in West US, at time of writing.
+++ Low object size is due to fitting 240m objects into a mere 56GB disk. Larger object sizes can be used with the following optimizations:
- shorten set name from 9 characters
- Enable single-bin for the namespace. This will make 28 additional bytes available.
- 3 additional bytes are available for Integer or Float types.
Your numbers may not necessarily match these exactly, due to variance within Azure's environment.
GCP
| Instance Type | Client Transactions/s+ | 95th Percentile (ms) | 99th Percentile (ms) | Object Size (bytes) | Object Count | Disk Size | Cost++ |
|---|---|---|---|---|---|---|---|
| n1_standard_8 | 82,000 | 3.2 | 5.9 | 768 | 240,000,000 | 375GB | $0.492/hr+++ |
++ Price in us-central1-a, at time of writing.
+++ Cost includes $0.112/hr for a single local ssd.
Your numbers may not necessarily match these exactly, due to variance within Google's environment.
IBM Cloud
| Instance Type | Client Transactions/s+ | 95th Percentile (ms) | 99th Percentile (ms) | Object Size (bytes) | Object Count | Disk Size | Cost++ |
|---|---|---|---|---|---|---|---|
| Xeon E3 1270v3 1Gig Networking VM to Baremetal | 75,000 | 2.7 | 4.3 | 500 | 240,000,000 | 1.2TB | $415/mo ($0.576/hr) |
| Xeon E3 1270v3 1Gig Networking Baremetal to Baremetal | 130,000 | 3.4 | 5.5 | 500 | 240,000,000 | 1.2TB | $415/mo ($0.576/hr) |
| Xeon E3 1270v3 1Gig Networking Baremetal to Baremetal | 30,000 | 3.0 | 4.6 | 3000 | 240,000,000 | 1.2TB | $415/mo ($0.576/hr) |
| 2x Xeon E5 2620v4 10Gig Networking Baremetal | 222,000 | 1.7 | 3.2 | 500 | 240,000,000 | 1.2TB | $1109/mo ($1.54/hr) |
| 2x Xeon E5 2620v4 10Gig Networking Baremetal +++ | 100,000 | 2.6 | 3.4 | 3000 | 240,000,000 | 1.2TB | $1109/mo ($1.54/hr) |
++ Price in SJC01/SJC04, at time of writing. System, RAM, Networking and Disk are all separate billable items, shown as a combined sum. Servers were on monthly provisioning, with hourly costs extrapolated using 30-day months.
+++ SSD Device overloaded and testing was terminated early. Testing proceeded at a faster pace than the device was capable of while still able to maintain latency profile. This caused defragmentation to fall behind and exhausted free blocks, therefore Aerospike Server entered into
stop_writes. Read more about stop-writes and how it relates toYour numbers may not necessarily match these exactly, due to variance within IBM's environment.